Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pyproject.toml #1074

Merged
merged 8 commits into from
Mar 11, 2024
Merged

pyproject.toml #1074

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
83ff557
convert to pyproject.toml
rasbt Mar 8, 2024
a228473
update
rasbt Mar 8, 2024
c3fcabd
update
rasbt Mar 8, 2024
01db2ec
Merge branch 'wip' into toml
rasbt Mar 8, 2024
05b99e9
Add test dependencies
rasbt Mar 8, 2024
4bc698c
simplify and update tests
rasbt Mar 11, 2024
a7ab349
Merge branch 'wip' into toml
rasbt Mar 11, 2024
25d4654
add back comments
rasbt Mar 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 70 additions & 0 deletions pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
[project]
name = "litgpt"
version = "0.1.0"
description = "Open source large language model implementation"
authors = [
{ name = "Lightning AI", email = "[email protected]" },
]

readme = "README.md"

[project.license]
file = "LICENSE"

[project.urls]
homepage = "https://github.com/lightning-AI/litgpt"

[project.optional-dependencies]
test = [
"pytest",
"pytest-rerunfailures",
"pytest-timeout",
"transformers>=4.38.0",
"einops",
"protobuf",
]
all = [
"jsonargparse[signatures]",
"bitsandbytes==0.41.0",
"scipy",
"sentencepiece",
"tokenizers",
"datasets",
"requests",
"litdata",
"zstandard",
"pandas",
"pyarrow",
"tensorboard",
"torchmetrics",
"lm_eval @ git+https://github.com/EleutherAI/lm-evaluation-harness.git@115206dc89dad67b8b",
]

[build-system]
requires = [
"setuptools",
"wheel",
]
build-backend = "setuptools.build_meta"

dynamic = [
"dependencies",
]

[tool.setuptools.dynamic.dependencies]
file = "requirements.txt"

[tool.setuptools.packages.find]
include = [
"litgpt",
"litgpt.*",
]
exclude = []

[tool.setuptools.package-data]
litgpt = [
"LICENSE",
"README.md",
"requirements.txt",
"requirements-all.txt",
]
20 changes: 20 additions & 0 deletions pyproject_toml_generate.py
View file
Open in desktop
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think having this file is very confusing and complex. If we want a single place where dependencies are updated then we should get rid of the requirements files.

https://stackoverflow.com/a/73600610 says that we can use files for optional dependencies but then we would need to add a requirements-test.txt file too because test dependencies are currently only defined in pyproject.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That file also says

Note that the referenced file will use a requirements.txt-like syntax; each line must conform to PEP 508, so flags like -r, -c, and -e are not supported inside this requirements.txt.

So would need to drop the -r we have, making it so users always need to do

pip install -r requirements.txt -r requirements-all.txt.

In my opinion, this is impractical and would stick to a single file that contains all the package metadata

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm yeah, 3 requirements files + .toml is probably too much. Should I combine everything into the .toml directly? I think if we add a clear installation instruction

# basic installation
pip install -e .

# install with all dependencies
pip install -e ".[all]"

# install test dependencies
pip install -e ".[test]"

then I guess it should be fine not having requirements.txt files. I think .toml files are very readable. And ours is pretty small too.

Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
import tomli, tomli_w
from pathlib import Path

root = Path(__file__).parent
requirements_path = Path(root / 'requirements-all.txt')
pyproject_path = Path(root / 'pyproject.toml')

requirements_all = (requirements_path).read_text().split("\n")
requirements_all = [
r.split("#")[0].strip()
for r in requirements_all if r and not r.strip().startswith("-r")
]

with pyproject_path.open('rb') as f:
pyproject_data = tomli.load(f)

pyproject_data.setdefault('project', {}).setdefault('optional-dependencies', {})['all'] = requirements_all

with pyproject_path.open('wb') as f:
tomli_w.dump(pyproject_data, f)
27 changes: 0 additions & 27 deletions setup.py
View file
Open in desktop

This file was deleted.

Loading