I didn't expect this

[Teddy563]

1.5k forks in under 12h.. I'm very curious who will get it running first on a normal CPU and under 64gb VRAM

[khanhnd0408]

Did you successfully load the model and run it with only 64GB of VRAM?

[khanhnd0408]

Quick check on the GCP instance with 4 A100 80GB; it cost around $20 per hour; this is not for me LOL.

[supersal3478]

daym... ok am a bit of a no code typa guy, just wondering if you wanna run the model and work with it off line, you gotta pay $20 per hour? for the GPUs? does not make any sense... could someone break it down for me please?

[khanhnd0408]

I mean, if I want to run this model and this model may require more than 300GB of VRAM, then I should look for a cloud VM, e.g., GCP Engine, AWS EC2, etc. And I just did a quick check on GCP Cloud: to rent 4 GPU Nvidia A100 80GB (to meet the requirement), it costs around $20 per hour. But also, all the VMs will include CPU, RAM, network, and maybe storage, not just GPUs.

[Zoobdude]

I suggest y'all go check out vast.ai or tensordock.com for far cheaper GPU VMs

[Teddy563]

Yeah.. it would be great to run on something smaller scale like you can with llama or mistral. I hope for it to be able to be ran with less vram but I think this would only be possible with very small fine tuned models

[goeielewe]

Neither did I, how cool of them!

[robvdl]

It's likely people are interested in just reading the source, or making a small PR for fame and glory like a typo fix.

[robvdl]

I am reluctant to send any PRs at least for now as the project is new so will likely get bombarded with useless PRs for a while.

I'm wondering why it's just a bunch of .py files in a directory rather than a package, move the requirements.txt into the pyproject.toml and expand it. Use a src-based layout. But that is just me. Not everything needs to be a package, but it's nice.

[elysiumplain]

I'll wait for the news to pass before I cycle back around - I expect, as intended, there to be a media cycle.

[wangzehai1]

一般人还真用不上这玩意儿