sim2sim for stompymicro

[henri123lemoine]

Summary:

• Added base Env and base MujocoEnv environments
• Added ~working (meh) sim2sim, robot sometimes stands for quite a while. More tests/param opt is needed
• Added Bayesian parameter optimizer for sim2sim. The idea, in short, is to initialize parameters you're unsure about, and have the optimizer run a bunch of sims for ~randomly selected params, try find a pattern through the noise, test it, and iterate (until you have params that do well according to your reward function). Should be portable over to other envs with minor modifs.
• Restructured sim2sim-related files
• Added drawing of vectors for direction of command & empirical direction of robot to compare for play.py
• Improved(?) arg-parsing
• Added command manager
• Added policies of some successful experiments
• Added auto-config-saving at start of training runs
• Added arms to stompymicro robot

Notes: will need to clean up tomorrow, revert some changes, uncommit some crimes

[henri123lemoine]

Still a few confusions:

• Does this look right? @budzianowski I chose values from previous sim2sim attempts that I was told work, but I have not independently confirmed that.

  <default>
    <joint limited="true" damping="0.53" armature="0.008793405204572328" frictionloss="0.001"/>
    <geom condim="4" contype="1" conaffinity="15" friction="0.9 0.2 0.2" solref="0.001 2"/>
    <motor ctrllimited="true"/>
    <equality solref="0.001 2"/>
    <default class="visualgeom">
      <geom material="visualgeom" condim="1" contype="0" conaffinity="0"/>
    </default>
  </default>

Also curious what are all the parameters worth modifying, if the list is reasonably short?

• For a bit I was investigating kds and kps parameters, what impact they have on the standing behavior, etc. But I just found out that the mujoco policy that uses ONNXModel("kinfer_test.onnx") generated from examples/experiments/standing/robustv1/policy_1.pt does not produce the same behavior as the policy produced through make_alg_runner. I still don't know why. I'm guessing I'm not giving in the inputs properly?