Hi, I'm Timo

A few probability facts kept as a reference: distribution notation, joint and conditional, the law of total expectation, why a squared-error fit returns a conditional mean, and the reparametrization trick.

Constrained optimization from the ground up: Lagrange multipliers for equality constraints, the KKT conditions for inequalities, and the Lagrangian dual. Why the dual is the clean way to handle constraints, and where it shows up in RL.

Information, entropy, cross-entropy, and KL divergence built up one step at a time, with interactive figures. Then total variation distance, and the Fisher information matrix derived as the curvature of KL.

Differential dynamic programming for trajectory optimization. Derivation, regularization, line search, and constraints via augmented Lagrangian.

Imitation learning on Push-T. Behavioral cloning, distribution shift, action chunking, and flow matching policies, with full math derivations.

REINFORCE derived from first principles, then variance reduction through reward-to-go, baselines, discounting, and advantage normalization, with experiments on HalfCheetah.

From REINFORCE to actor-critic: bootstrapping and what makes a critic, the TD error, and generalized advantage estimation (GAE).

Crossing from on-policy to off-policy: why a replay buffer breaks actor-critic, how learning a Q-function fixes it, the reparametrization trick, and the greedy limit that drops the actor and points to Q-learning.

From the argmax over Q to Q-learning: policy and value iteration, fitted value iteration, fitted Q-iteration, online Q-learning and exploration, and why the naive replay-buffer version will not train yet.

Rainbow extensions stacked on plain DQN. Dueling networks (the value/advantage split) is covered, with a derivation and a working critic; prioritized replay, distributional RL (C51), and noisy nets are next.

Coming soon. Off-policy actor-critic for continuous actions: replacing the argmax with a learned deterministic actor (DDPG), the fixes that make it stable (TD3), and the maximum-entropy version (SAC).

A physics based rocket battle game, with a write-up of the dynamics behind it: rigid-body motion, thrust and torque, semi-implicit integration, and impulse-based collisions. Playable in the browser.

A list of subjects I would like to explore further, kept as a reminder for myself. A small description of each, with a page for them hopefully someday.