Exploration list
This is just a list of subjects I would like to explore further. I use it as a reminder for myself. I am putting a small description of each and hope to eventually build a page for them.
RL and planning
- MPPI (path-integral MPC). Sampling-based MPC: sample many control sequences, roll each through a model, and take a softmax-weighted average by cost. It is derivative-free, so it removes a lot of headaches.
- PETS. Learn an ensemble of probabilistic dynamics models, then plan through them with sampling-based MPC, propagating uncertainty by particle rollouts. Known to be sample efficient.
- Dreamer. Learn a latent world model from pixels and train an actor-critic entirely "in imagination" by backpropagating through the learned dynamics. Seems to be popular.
- RRT* hand implementation. Idk why but I've always wanted to implement it on a complex dynamical system for fun. Probably obsolete technology at this point.
- Offline RL and stitching. Learn a value or policy from a fixed dataset. You can only back up a cost that something in the data actually saw. Makes coverage and safety concerns interesting (covered by CS285, so I should get to it soon).
Cost and safety
- Safety filters and control barrier functions. Let a learned policy propose actions, then project them onto a safe set with a CBF or a shield. Seems useful for real deployment with safety guarantees.
- Inverse RL. Learn the cost or reward from demonstrations instead of hand-designing it. I played with inverse optimal control before and it was fun.