From 482df66db3510b3078766cfc0386dcbaaf531b06 Mon Sep 17 00:00:00 2001 From: LangChain4j Date: Fri, 22 Nov 2024 14:54:02 +0100 Subject: [PATCH] docu: WIP: structured outputs --- .../language-models/google-ai-gemini.md | 4 +- .../integrations/language-models/index.md | 42 +++++++++---------- .../integrations/language-models/open-ai.md | 12 +++--- docs/docs/tutorials/structured-outputs.md | 28 ++++++------- 4 files changed, 45 insertions(+), 41 deletions(-) diff --git a/docs/docs/integrations/language-models/google-ai-gemini.md b/docs/docs/integrations/language-models/google-ai-gemini.md index 46a0b3694f7..ef65fac855c 100644 --- a/docs/docs/integrations/language-models/google-ai-gemini.md +++ b/docs/docs/integrations/language-models/google-ai-gemini.md @@ -168,7 +168,9 @@ System.out.println("Gemini> " + tokyoWeather); // with a temperature of 32 degrees. ``` -## Structured output +## Structured Outputs + +See more info on Structured Outputs [here](/tutorials/structured-outputs). ### JSON mode diff --git a/docs/docs/integrations/language-models/index.md b/docs/docs/integrations/language-models/index.md index 55bc86e2f10..58c4f79c9ff 100644 --- a/docs/docs/integrations/language-models/index.md +++ b/docs/docs/integrations/language-models/index.md @@ -4,31 +4,31 @@ hide_title: false sidebar_position: 0 --- -| Provider | [Streaming](/tutorials/response-streaming) | [Tools](/tutorials/tools) (sync/streaming) | [Structured Outputs](/tutorials/ai-services#structured-outputs) | [JSON mode](/tutorials/ai-services#json-mode) | Supported Modalities (Input) | [Observability](/tutorials/observability) | Local | Native | Comments | -|----------------------------------------------------------------------------------|--------------------------------------------|--------------------------------------------------------------------|--------------------------------------------------------------------|-----------------------------------------------|--------------------------------|-------------------------------------------|---------------------------------------------------|--------|-----------------------------| -| [Amazon Bedrock](/integrations/language-models/amazon-bedrock) | ✅ | ✅/⏳[#2114](https://github.com/langchain4j/langchain4j/issues/2114) | | | text | ✅ | | | | -| [Anthropic](/integrations/language-models/anthropic) | ✅ | ✅/✅ | | | text, image | | | ✅ | | -| [Azure OpenAI](/integrations/language-models/azure-open-ai) | ✅ | ✅/✅ | 🔜 [#1982](https://github.com/langchain4j/langchain4j/pull/1982) | ✅ | text, image | ✅ | | | | -| [ChatGLM](/integrations/language-models/chatglm) | | | | | text | | | | | -| [DashScope](/integrations/language-models/dashscope) | ✅ | ✅/✅ | | | text, image, audio | ✅ | | | | -| [GitHub Models](/integrations/language-models/github-models) | ✅ | ✅/✅ | 🔜 [#1911](https://github.com/langchain4j/langchain4j/issues/1911) | ✅ | text | ✅ | | | | -| [Google AI Gemini](/integrations/language-models/google-ai-gemini) | ✅ | ✅/✅ | ✅ | ✅ | text, image, audio, video, PDF | ✅ | | | | -| [Google Vertex AI Gemini](/integrations/language-models/google-vertex-ai-gemini) | ✅ | ✅/✅ | ⏳ [#1717](https://github.com/langchain4j/langchain4j/issues/1717) | ✅ | text, image, audio, video, PDF | ✅ | | | | -| [Google Vertex AI PaLM 2](/integrations/language-models/google-palm) | | | | | text | | | ✅ | | -| [Hugging Face](/integrations/language-models/hugging-face) | | | | | text | | | | | -| [Jlama](/integrations/language-models/jlama) | ✅ | ✅/⏳[#2115](https://github.com/langchain4j/langchain4j/issues/2115) | | | text | | ✅ | ✅ | | -| [LocalAI](/integrations/language-models/local-ai) | ✅ | ✅/✅ | | | text | | ✅ | | | -| [Mistral AI](/integrations/language-models/mistral-ai) | ✅ | ✅/✅ | | ✅ | text | | | | | -| [Ollama](/integrations/language-models/ollama) | ✅ | ✅/❌ [Issue](https://github.com/ollama/ollama/issues/5796) | | ✅ | text, image | ✅ | ✅ | | | -| [OpenAI](/integrations/language-models/open-ai) | ✅ | ✅/✅ | ✅ | ✅ | text, image | ✅ | Compatible with: Ollama, LM Studio, GPT4All, etc. | ✅ | Compatible with: Groq, etc. | -| [Qianfan](/integrations/language-models/qianfan) | ✅ | ✅/✅ | | | text | | | | | -| [Cloudflare Workers AI](/integrations/language-models/workers-ai) | | | | | text | | | | | -| [Zhipu AI](/integrations/language-models/zhipu-ai) | ✅ | ✅/✅ | | | text, image | ✅ | | | | +| Provider | [Streaming](/tutorials/response-streaming) | [Tools](/tutorials/tools) (sync/streaming) | [Structured Outputs](/tutorials/structured-outputs#structured-outputs-1) | [JSON Mode](/tutorials/ai-services#json-mode) | Supported Modalities (Input) | [Observability](/tutorials/observability) | Local Deployment | Supports Native Image | Comments | +|----------------------------------------------------------------------------------|--------------------------------------------|---------------------------------------------------------------------|--------------------------------------------------------------------------|-----------------------------------------------|--------------------------------|-------------------------------------------|---------------------------------------------------|-----------------------|-----------------------------| +| [Amazon Bedrock](/integrations/language-models/amazon-bedrock) | ✅ | ✅/🆘[#2114](https://github.com/langchain4j/langchain4j/issues/2114) | | | text | ✅ | | | | +| [Anthropic](/integrations/language-models/anthropic) | ✅ | ✅/✅ | | | text, image | | | ✅ | | +| [Azure OpenAI](/integrations/language-models/azure-open-ai) | ✅ | ✅/✅ | 🔜 [#1982](https://github.com/langchain4j/langchain4j/pull/1982) | ✅ | text, image | ✅ | | | | +| [ChatGLM](/integrations/language-models/chatglm) | | | | | text | | | | | +| [DashScope](/integrations/language-models/dashscope) | ✅ | ✅/✅ | | | text, image, audio | ✅ | | | | +| [GitHub Models](/integrations/language-models/github-models) | ✅ | ✅/✅ | 🔜 [#1911](https://github.com/langchain4j/langchain4j/issues/1911) | ✅ | text | ✅ | | | | +| [Google AI Gemini](/integrations/language-models/google-ai-gemini) | ✅ | ✅/✅ | ✅ | ✅ | text, image, audio, video, PDF | ✅ | | | | +| [Google Vertex AI Gemini](/integrations/language-models/google-vertex-ai-gemini) | ✅ | ✅/✅ | 🆘 [#1717](https://github.com/langchain4j/langchain4j/issues/1717) | ✅ | text, image, audio, video, PDF | ✅ | | | | +| [Google Vertex AI PaLM 2](/integrations/language-models/google-palm) | | | | | text | | | ✅ | | +| [Hugging Face](/integrations/language-models/hugging-face) | | | | | text | | | | | +| [Jlama](/integrations/language-models/jlama) | ✅ | ✅/🆘[#2115](https://github.com/langchain4j/langchain4j/issues/2115) | | | text | | ✅ | ✅ | | +| [LocalAI](/integrations/language-models/local-ai) | ✅ | ✅/✅ | | | text | | ✅ | | | +| [Mistral AI](/integrations/language-models/mistral-ai) | ✅ | ✅/✅ | | ✅ | text | | | | | +| [Ollama](/integrations/language-models/ollama) | ✅ | ✅/❌ [#5796](https://github.com/ollama/ollama/issues/5796) | | ✅ | text, image | ✅ | ✅ | | | +| [OpenAI](/integrations/language-models/open-ai) | ✅ | ✅/✅ | ✅ | ✅ | text, image | ✅ | Compatible with: Ollama, LM Studio, GPT4All, etc. | ✅ | Compatible with: Groq, etc. | +| [Qianfan](/integrations/language-models/qianfan) | ✅ | ✅/✅ | | | text | | | | | +| [Cloudflare Workers AI](/integrations/language-models/workers-ai) | | | | | text | | | | | +| [Zhipu AI](/integrations/language-models/zhipu-ai) | ✅ | ✅/✅ | | | text, image | ✅ | | | | Legend: - ✅ means "supported" -- ⏳ means "not supported yet; please help us implement it" +- 🆘 means "not supported yet; please help us implement it" - 🔜 means "it is being implemented; please wait" - ❌ means "not supported by the LLM provider" - no mark means "not sure, need to double-check" diff --git a/docs/docs/integrations/language-models/open-ai.md b/docs/docs/integrations/language-models/open-ai.md index 03da422d777..dfa5d72e4b1 100644 --- a/docs/docs/integrations/language-models/open-ai.md +++ b/docs/docs/integrations/language-models/open-ai.md @@ -120,9 +120,11 @@ class ChatLanguageModelController { ## Structured Outputs The [Structured Outputs](https://openai.com/index/introducing-structured-outputs-in-the-api/) feature is supported -for both [tools](/tutorials/tools) and [JSON mode](/tutorials/ai-services#json-mode). +for both [tools](/tutorials/tools) and [response format](/tutorials/ai-services#json-mode). -### Structured Outputs for tools +See more info on Structured Outputs [here](/tutorials/structured-outputs). + +### Structured Outputs for Tools To enable Structured Outputs feature for tools, set `.strictTools(true)` when building the model: ```java OpenAiChatModel.builder() @@ -133,9 +135,9 @@ OpenAiChatModel.builder() Please note that this will automatically make all tool parameters mandatory (`required` in json schema) and set `additionalProperties=false` for each `object` in json schema. This is due to the current OpenAI limitations. -### Structured Outputs for JSON mode -To enable Structured Outputs feature for JSON mode, set `.responseFormat("json_schema")` and `.strictJsonSchema(true)` -when building the model: +### Structured Outputs for Response Format +To enable Structured Outputs feature for response format, +set `.responseFormat("json_schema")` and `.strictJsonSchema(true)` when building the model: ```java OpenAiChatModel.builder() ... diff --git a/docs/docs/tutorials/structured-outputs.md b/docs/docs/tutorials/structured-outputs.md index 7a3180aa035..73419362f15 100644 --- a/docs/docs/tutorials/structured-outputs.md +++ b/docs/docs/tutorials/structured-outputs.md @@ -21,9 +21,9 @@ Currently unmarried, he enjoys the freedom to focus on his personal goals and in Currently, depending on the LLM and the LLM provider, there are four ways how this can be achieved (from most to least reliable): -- [Structured Outputs](/tutorials/structured-outputs#structured-outputs) +- [Structured Outputs](/tutorials/structured-outputs#structured-outputs-1) - [Tools (Function Calling)](/tutorials/structured-outputs#tools-function-calling) -- [Prompting + JSON Mode](/tutorials/structured-outputs#prompting-json-mode) +- [Prompting + JSON Mode](/tutorials/structured-outputs#prompting--json-mode) - [Prompting](/tutorials/structured-outputs#prompting) @@ -96,8 +96,8 @@ Person person = new ObjectMapper().readValue(output, Person.class); System.out.println(person); // Person[name=John, age=42, height=1.75, married=false] ``` Notes: -- [1] - This is required to activate the Structured Outputs feature for OpenAI, see more details [here](/integrations/language-models/open-ai#structured-outputs-for-json-mode). -- [2] - This is required to activate the Structured Outputs feature for [Google AI Gemini](/integrations/language-models/google-ai-gemini). +- [1] - This is required to enable the Structured Outputs feature for OpenAI, see more details [here](/integrations/language-models/open-ai#structured-outputs-for-response-format). +- [2] - This is required to enable the Structured Outputs feature for [Google AI Gemini](/integrations/language-models/google-ai-gemini). - [3] - Response format type can be either `TEXT` (default) or `JSON`. - [4] - OpenAI requires specifying the name for the schema. - [5] - In most cases, the root element must be of `JsonObjectSchema` type, @@ -113,16 +113,15 @@ LangChain4j offers `ResponseFormat` and `JsonSchema` types. The structure of the schema is defined using `JsonSchemaElement` interface, with the following subtypes: -- `JsonStringSchema` - to support `String`, `char`/`Character`, etc. -- `JsonIntegerSchema` - to support `int`/`Integer`, `long`/`Long`, `BigInteger`, etc. -- `JsonNumberSchema` - to support `float`/`Float`, `double`/`Double`, `BigDecimal`, etc. +- `JsonStringSchema` - to support `String`, `char`/`Character` types. +- `JsonIntegerSchema` - to support `int`/`Integer`, `long`/`Long`, `BigInteger` types. +- `JsonNumberSchema` - to support `float`/`Float`, `double`/`Double`, `BigDecimal` types. - `JsonBooleanSchema` - to support `boolean`/`Boolean` types. -- `JsonEnumSchema` - to support `enum`s. -- `JsonArraySchema` - to support arrays and other collection types. +- `JsonEnumSchema` - to support `enum` types. +- `JsonArraySchema` - to support arrays and collection (e.g., `List`, `Set`) types. - `JsonObjectSchema` - to support object types. - `JsonReferenceSchema` - to support recursion (e.g., `Person` has a `Set children` field). -See more information in the Javadoc of these types. ### High Level Structured Outputs API @@ -167,10 +166,10 @@ Notes: as these beans are created automatically. More info on this: [for Quarkus](https://docs.quarkiverse.io/quarkus-langchain4j/dev/ai-services.html), [for Spring Boot](https://docs.langchain4j.dev/tutorials/spring-boot-integration#spring-boot-starter-for-declarative-ai-services). -- [2] - This is required to activate the Structured Outputs feature for OpenAI, see more details [here](/integrations/language-models/open-ai#structured-outputs-for-json-mode). -- [3] - This is required to activate the Structured Outputs feature for [Google AI Gemini](/integrations/language-models/google-ai-gemini). +- [2] - This is required to enable the Structured Outputs feature for OpenAI, see more details [here](/integrations/language-models/open-ai#structured-outputs-for-response-format). +- [3] - This is required to enable the Structured Outputs feature for [Google AI Gemini](/integrations/language-models/google-ai-gemini). -When AI Service returns a POJO **and** used `ChatLanguageModel` supports/enables Structured Outputs feature, +When AI Service returns a POJO **and** used `ChatLanguageModel` supports Structured Outputs **and** Structured Outputs are enabled, `JsonSchema`/`ResponseFormat` will be generated automatically from the specified return type. :::note Make sure to explicitly enable Structured Outputs feature when configuring `ChatLanguageModel`, @@ -200,7 +199,8 @@ We are [working](https://github.com/langchain4j/langchain4j/pull/1938) on suppor - Nested POJOs - `List`, `Set` and `T[]`, where `T` is a scalar, an enum or a POJO - All fields and sub-fields in the generated `JsonSchema` are marked as `required`, there is currently no way to make them optional. -- Classes and fields can be annotated with `@Description` to guide the LLM, foe example: +- Classes and fields can be (optionally) annotated with `@Description` to provide more information to the LLM. +For example: ```java @Description("a person") record Person(@Description("person's name") String name,