Add support to parse VertexAI Model garden Llama2 model output

[izo0x90]

• Description: The output of Llama2 models in GCP VertexAI Model garden is not compatible with the generic VertexAiModelGarden handler class. This PR adds a class to handle/ properly parse the output returned by those models. The handler also adds support for storing/ passing parameters to the model that is consistent with the core VertexAI LLM models.

• Dependencies: None

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Jan 4, 2024 3:20pm

[izo0x90]

@baskaryan

[baskaryan]

thanks for contribution @izo0x90! the main vertexai integrations have moved to their own package in libs/partners/google-vertexai. any interest in porting this change over there?

cc @efriis @lkuligin

[izo0x90]

thanks for contribution @izo0x90! the main vertexai integrations have moved to their own package in libs/partners/google-vertexai. any interest in porting this change over there?

cc @efriis @lkuligin

@baskaryan Yes absolutely if this has not been accomplished there already I would to love to make that change. Thanks.

[lkuligin]

@izo0x90 could you share the exact error you're getting, please?

there's an integration tests for Llama and it seems to be working fine:

langchain/libs/partners/google-vertexai/tests/integration_tests/test_llms.py

Line 110 in 2145636

[("FALCON_ENDPOINT_ID", "generated_text"), ("LLAMA_ENDPOINT_ID", None)],

[izo0x90]

@izo0x90 could you share the exact error you're getting, please?

there's an integration tests for Llama and it seems to be working fine:

langchain/libs/partners/google-vertexai/tests/integration_tests/test_llms.py

Line 110 in 2145636

[("FALCON_ENDPOINT_ID", "generated_text"), ("LLAMA_ENDPOINT_ID", None)],

Sorry for the tardy response, I finally had the time to create an update PR with the new location that houses this functionality. langchain-ai/langchain-google#41

As to why the test passes successfully, the core of the issue is that the generic class interprets each character of the Llama2 response as a "prediction" as such it created numerous Generation objects for each char of the response text. The test only checks if the "correct" objects are generated and the type of the response is string, which a singe letter of the response text satisfies, and the test technically passes, of course this is not what we actually need as an end user.