Litgpt serve API

[rasbt]

The LitGPT serve API. E.g., use as

litgpt serve --checkpoint_dir checkpoints/microsoft/phi-2

and then send requests via

In [1]:  import requests, json

In [2]:  response = requests.post(
   ...:      "http://127.0.0.1:8000/predict",
   ...:          json={"prompt": "Fix typos in the following sentence: Exampel input"}
   ...:          )

[lantiga]

Looks great, added a few comments. We also need a test for this, feel free to inherit test code from litserve since it's using clients etc

@aniketmaurya will be able to advise here

[williamFalcon]

this is sick. super excited about this landing. nice job.

[rasbt]

It would be nice of you could review @lantiga and @aniketmaurya . Btw I am currently having problems with getting quantize (e.g., --quantize bnb.nf4) etc to work. Somehow it's a None type even if I pass it via the CLI. It's been a long day and maybe I'm overlooking something here ...

[carmocca]

@rasbt Did you make this a required dependency (instead of an "all") purposefully?

[rasbt]

When I remember correctly, I had problems with tugging the import so that it doesn't always gets called. I think that was due to how we use inheritance in

litgpt/litgpt/deploy/serve.py

Line 18 in 57f0a61

class SimpleLitAPI(LitAPI):

But I suppose one can define the class in a function to avoid the import at the top. But that would be ugly. What do you think?

[carmocca]

You can follow this pattern:

litgpt/litgpt/data/prepare_starcoder.py

Lines 8 to 20 in 43c4432

	from lightning_utilities.core.imports import RequirementCache
	from litgpt import Tokenizer
	from litgpt.utils import CLI
	_LITDATA_AVAILABLE = RequirementCache("litdata")
	if _LITDATA_AVAILABLE:
	from litdata.processing.data_processor import DataChunkRecipe
	else:
	DataChunkRecipe = object
	class StarcoderDataRecipe(DataChunkRecipe):

And then inside the __init__(), before calling super().__init__() check that _LITSERVE_AVAILABLE is True (this part is missing from the code I linked)

[carmocca]

I think you should specify this as a test dependency too