Enable Python 3.8 testing

[carmocca]

No description provided.

[carmocca]

I couldnt get this to work

[Andrei-Aksionov]

I couldnt get this to work

Because it's not possible.
As I stated in #632 Python 3.8 isn't supported by lm-eval.
It's possible to install packages by providing the latest supported by 3.8 packages in requirements-all.txt:

-r requirements.txt
bitsandbytes==0.41.0    # quantization
scipy~=1.10.1           # required by bitsandbytes
sentencepiece           # pythia, falcon, redpajama
tokenizers~=0.15.0      # llama-based models
datasets                # quantize/gptq.py
zstandard               # scripts/prepare_redpajama.py, scripts/prepare_starcoder.py
pandas~=2.0.3           # scripts/prepare_csv.py, scripts/prepare_starcoder.py
transformers~=4.35.2
pyarrow                 # scripts/prepare_starcoder.py
# eval
git+https://github.com/EleutherAI/lm-evaluation-harness.git@master
# scripts/prepare_slimpajama.py, scripts/prepare_starcoder.py, pretrain/tinyllama.py
lightning[data] @ git+https://github.com/Lightning-AI/lightning@532c723c8584903dc719458d0ad52861d51bc395

... but it doesn't matter as in the end you get the final error:

ERROR: Package 'lm-eval' requires a different Python: 3.8.18 not in '>=3.9'

[Andrei-Aksionov]

Don't know what is wrong with me, but when someone says that it's not possible, it triggers me, and I want to prove that it's indeed possible. In this case, I triggered myself 🙈.

So, yes, it's possible. But you have to provide --ignore-requires-python to force to install lm-eval and that argument requires fixing even more packages.
But at least it works: Andrei-Aksionov#1
Too hacky for my liking though.

[carmocca]

Thank you @Andrei-Aksionov! I didn't know that lm eval is >=3.9. Pip is weird, it should mention this before going down the rabbit hole of trying different dependency combinations

[Andrei-Aksionov]

Glad that I was helpful.

Your solution is much nicer, btw 👍

[Andrei-Aksionov]

A new twist: now lm-eval supports python 3.8 😆.