Adding multi-discrete pretraining

[MadcowD]

Adds multi-discrete action space support to pretraining.

[araffin]

Hello, thanks for the PR, please do not forget to read the contributing guide first ;)

[MadcowD]

Okay, sorry! I'll go check it out. Sorry in the middle of finishing a late NeurIPS review :S

[MadcowD]

I think I've followed the steps (ran the tests, etc) I can't seem to find a change-log. Is there anything else I am missing?

[araffin]

The changelog is in the doc and you should add a test that check that new feature.

[araffin]

And next time, please open an issue first, so we can discuss the feature.

[MadcowD]

Okay got it :) Sorry again to bother!

[MadcowD]

I can think of any multi-discret environments packaged with OpenAI gym. I'm currently using minerl's (https://github.com/minerllabs/minerl) environments to test, but obviously that wouldn't be a great dependency to add to the test suite unless we went through and added baselines for all of those environments (A fun task I would certainly take on, later). There is currently not a pretrain test in the test suite (it's just a part of some of the existing tests).

We could potentially add the minerl expert data to the test suite and go from there. Otherwise I'm not sure how I can provide a good test case for this PR.

[araffin]

I think a variation of the identity env (in the common folder) should make the trick.

[DanielPBak]

This PR should also include a change to generate_expert_traj(), right?

[araffin]

This PR should also include a change to generate_expert_traj(), right?

I would say yes.