Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend arena to support for training dataset or model storage management #28

Open
wsxiaozhang opened this issue Aug 22, 2018 · 0 comments

Comments

@wsxiaozhang
Copy link
Collaborator

It should be helpful for data scientists to use command like "arena create data imagenet-full" to create, index and manage different training datasets for different training jobs.
Then when use arena submit training job, the "imagenet-full" can be passed in as parameter "data" directly.
The data actually can be a pointer to specific PVC or hdfs path, etc. It's easy to use 'data' to manage and record which training datasets to load, while needn't care about which storage backend is used for persistent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants