setup RL experiments

[Armandpl]

• Improve the simulation and sim2real #52

• evaluate recurrent RL algos #53

• write script to bench the max control freq when using PPO, SAC etc #55

• try out dreamer #62

• sanity check by training with sac or tqc #64

• check markov assumption is not violated by:

  • the timelimit wrapper https://arxiv.org/abs/1712.00378 -> use TimeFeature wrapper
  • limits on angles while the agent can only see cos/sin of angles -> add actual angles to the obs, scale them based on safety limits. add option to pass None for angle/speed limits individually, so if there is no limits, no need to add the obs, and if there is one it shouldn't be too big so rescaling makes sense

• eval tqc vs. sac

  • what do about hyperparameters though

• based on output from write script to bench the max control freq when using PPO, SAC etc #55 try higher control freq, see what is changes, is it better at rejecting perturbations? Is it making it easier/faster to learn?

• do a sweep on state limits, see if they have an effect on the time to convergence #58

• try fine-tuning a policy trained in sim #61

• eval oob penalty

• tune hyper-parameters:

  • gamma, ent_coef. aim to reduce global step to speed up training
  • sde sample freq
  • higher UTD
  • batch size

• investigate gSDE issue where, in sim, and on the real robot, the agent takes actions that are too small to move the robot

• train one model from scratch on the robot

• eval vec normalize wrapper

• dreamer v3, mu dreamer, decision transformer? are those worth exploring? I feel like this might be too simple of a problem to do anything interesting with these approaches? although since the problem is easy maybe it's a good way to learn the techniques? but they might just show the same result than normal RL?

• eval if random init state is useful

• eval encoder resolution

  • sweep over encoder resolutions and see how low we can go and when we start seeing diminishing returns