diff --git a/docs/core_docs/docs/integrations/text_embedding/togetherai.ipynb b/docs/core_docs/docs/integrations/text_embedding/togetherai.ipynb new file mode 100644 index 000000000000..ce2a931a3f37 --- /dev/null +++ b/docs/core_docs/docs/integrations/text_embedding/togetherai.ipynb @@ -0,0 +1,302 @@ +{ + "cells": [ + { + "cell_type": "raw", + "id": "afaf8039", + "metadata": { + "vscode": { + "languageId": "raw" + } + }, + "source": [ + "---\n", + "sidebar_label: TogetherAI\n", + "---" + ] + }, + { + "cell_type": "markdown", + "id": "9a3d6f34", + "metadata": {}, + "source": [ + "# TogetherAIEmbeddings\n", + "\n", + "This will help you get started with TogetherAIEmbeddings [embedding models](/docs/concepts#embedding-models) using LangChain. For detailed documentation on `TogetherAIEmbeddings` features and configuration options, please refer to the [API reference](https://api.js.langchain.com/classes/langchain_community_embeddings_togetherai.TogetherAIEmbeddings.html).\n", + "\n", + "## Overview\n", + "### Integration details\n", + "\n", + "| Class | Package | Local | [Py support](https://python.langchain.com/docs/integrations/text_embedding/together/) | Package downloads | Package latest |\n", + "| :--- | :--- | :---: | :---: | :---: | :---: |\n", + "| [TogetherAIEmbeddings](https://api.js.langchain.com/classes/langchain_community_embeddings_togetherai.TogetherAIEmbeddings.html) | [@langchain/community](https://api.js.langchain.com/modules/langchain_community_embeddings_togetherai.html) | ❌ | ✅ | ![NPM - Downloads](https://img.shields.io/npm/dm/@langchain/community?style=flat-square&label=%20&) | ![NPM - Version](https://img.shields.io/npm/v/@langchain/community?style=flat-square&label=%20&) |\n", + "\n", + "## Setup\n", + "\n", + "To access TogetherAI embedding models you'll need to create a TogetherAI account, get an API key, and install the `@langchain/community` integration package.\n", + "\n", + "### Credentials\n", + "\n", + "You can sign up for a Together account and create an API key [here](https://api.together.xyz/). Once you've done this set the `TOGETHER_AI_API_KEY` environment variable:\n", + "\n", + "```bash\n", + "export TOGETHER_AI_API_KEY=\"your-api-key\"\n", + "```\n", + "\n", + "If you want to get automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:\n", + "\n", + "```bash\n", + "# export LANGCHAIN_TRACING_V2=\"true\"\n", + "# export LANGCHAIN_API_KEY=\"your-api-key\"\n", + "```\n", + "\n", + "### Installation\n", + "\n", + "The LangChain TogetherAIEmbeddings integration lives in the `@langchain/community` package:\n", + "\n", + "```{=mdx}\n", + "import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n", + "import Npm2Yarn from \"@theme/Npm2Yarn\";\n", + "\n", + "\n", + "\n", + "\n", + " @langchain/community\n", + "\n", + "```" + ] + }, + { + "cell_type": "markdown", + "id": "45dd1724", + "metadata": {}, + "source": [ + "## Instantiation\n", + "\n", + "Now we can instantiate our model object and generate chat completions:" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "9ea7a09b", + "metadata": {}, + "outputs": [], + "source": [ + "import { TogetherAIEmbeddings } from \"@langchain/community/embeddings/togetherai\";\n", + "\n", + "const embeddings = new TogetherAIEmbeddings({\n", + " model: \"togethercomputer/m2-bert-80M-8k-retrieval\", // Default value\n", + "});" + ] + }, + { + "cell_type": "markdown", + "id": "77d271b6", + "metadata": {}, + "source": [ + "## Indexing and Retrieval\n", + "\n", + "Embedding models are often used in retrieval-augmented generation (RAG) flows, both as part of indexing data as well as later retrieving it. For more detailed instructions, please see our RAG tutorials under the [working with external knowledge tutorials](/docs/tutorials/#working-with-external-knowledge).\n", + "\n", + "Below, see how to index and retrieve data using the `embeddings` object we initialized above. In this example, we will index and retrieve a sample document using the demo [`MemoryVectorStore`](/docs/integrations/vectorstores/memory)." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "d817716b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "LangChain is the framework for building context-aware reasoning applications\n" + ] + } + ], + "source": [ + "// Create a vector store with a sample text\n", + "import { MemoryVectorStore } from \"langchain/vectorstores/memory\";\n", + "\n", + "const text = \"LangChain is the framework for building context-aware reasoning applications\";\n", + "\n", + "const vectorstore = await MemoryVectorStore.fromDocuments(\n", + " [{ pageContent: text, metadata: {} }],\n", + " embeddings,\n", + ");\n", + "\n", + "// Use the vector store as a retriever that returns a single document\n", + "const retriever = vectorstore.asRetriever(1);\n", + "\n", + "// Retrieve the most similar text\n", + "const retrievedDocuments = await retriever.invoke(\"What is LangChain?\");\n", + "\n", + "retrievedDocuments[0].pageContent;" + ] + }, + { + "cell_type": "markdown", + "id": "e02b9855", + "metadata": {}, + "source": [ + "## Direct Usage\n", + "\n", + "Under the hood, the vectorstore and retriever implementations are calling `embeddings.embedDocument(...)` and `embeddings.embedQuery(...)` to create embeddings for the text(s) used in `fromDocuments` and the retriever's `invoke` operations, respectively.\n", + "\n", + "You can directly call these methods to get embeddings for your own use cases.\n", + "\n", + "### Embed single texts\n", + "\n", + "You can embed queries for search with `embedQuery`. This generates a vector representation specific to the query:" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "0d2befcd", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[\n", + " 0.3812227, -0.052848946, -0.10564975, 0.03480297, 0.2878488,\n", + " 0.0084609175, 0.11605915, 0.05303011, 0.14711718, -0.14407106,\n", + " -0.29865336, -0.15807179, -0.068397366, -0.2708063, 0.056596708,\n", + " -0.07656515, 0.052995138, -0.11275427, 0.028096694, 0.123501234,\n", + " -0.039519835, 0.12148692, -0.12820457, 0.15691335, 0.033519063,\n", + " -0.27026987, -0.08460162, -0.23792154, -0.234982, -0.05786798,\n", + " 0.016467346, -0.17168592, -0.060787182, 0.038752213, -0.08169927,\n", + " 0.09327062, 0.29490772, 0.0167866, -0.32224452, -0.2037822,\n", + " -0.10284172, -0.124050565, 0.25344968, -0.06275548, -0.14180769,\n", + " 0.0046709594, 0.073105976, 0.12004031, 0.19224276, -0.022589967,\n", + " 0.102790825, 0.1138286, -0.057701062, -0.050010648, -0.1632584,\n", + " -0.18942119, -0.12018798, 0.15288158, 0.07941474, 0.10440051,\n", + " -0.13257962, -0.19282033, 0.044656333, 0.13560675, -0.068929024,\n", + " 0.028590716, 0.055663664, 0.04652713, 0.014936657, 0.120679885,\n", + " 0.053866718, -0.16296014, 0.119450666, -0.29559663, 0.008097747,\n", + " 0.07380408, -0.09010084, -0.0687739, -0.08575685, -0.07202606,\n", + " 0.18868081, -0.08392917, 0.014016109, 0.15435852, -0.030115498,\n", + " -0.16927013, 0.02836557, -0.050763763, 0.0840437, -0.22718845,\n", + " 0.111397505, 0.033395614, -0.123287566, -0.2111604, -0.1580479,\n", + " 0.05520573, -0.1422921, 0.08828953, 0.051058788, -0.13312188\n", + "]\n" + ] + } + ], + "source": [ + "const singleVector = await embeddings.embedQuery(text);\n", + "\n", + "console.log(singleVector.slice(0, 100));" + ] + }, + { + "cell_type": "markdown", + "id": "1b5a7d03", + "metadata": {}, + "source": [ + "### Embed multiple texts\n", + "\n", + "You can embed multiple texts for indexing with `embedDocuments`. The internals used for this method may (but do not have to) differ from embedding queries:" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "2f4d6e97", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[\n", + " 0.3812227, -0.052848946, -0.10564975, 0.03480297, 0.2878488,\n", + " 0.0084609175, 0.11605915, 0.05303011, 0.14711718, -0.14407106,\n", + " -0.29865336, -0.15807179, -0.068397366, -0.2708063, 0.056596708,\n", + " -0.07656515, 0.052995138, -0.11275427, 0.028096694, 0.123501234,\n", + " -0.039519835, 0.12148692, -0.12820457, 0.15691335, 0.033519063,\n", + " -0.27026987, -0.08460162, -0.23792154, -0.234982, -0.05786798,\n", + " 0.016467346, -0.17168592, -0.060787182, 0.038752213, -0.08169927,\n", + " 0.09327062, 0.29490772, 0.0167866, -0.32224452, -0.2037822,\n", + " -0.10284172, -0.124050565, 0.25344968, -0.06275548, -0.14180769,\n", + " 0.0046709594, 0.073105976, 0.12004031, 0.19224276, -0.022589967,\n", + " 0.102790825, 0.1138286, -0.057701062, -0.050010648, -0.1632584,\n", + " -0.18942119, -0.12018798, 0.15288158, 0.07941474, 0.10440051,\n", + " -0.13257962, -0.19282033, 0.044656333, 0.13560675, -0.068929024,\n", + " 0.028590716, 0.055663664, 0.04652713, 0.014936657, 0.120679885,\n", + " 0.053866718, -0.16296014, 0.119450666, -0.29559663, 0.008097747,\n", + " 0.07380408, -0.09010084, -0.0687739, -0.08575685, -0.07202606,\n", + " 0.18868081, -0.08392917, 0.014016109, 0.15435852, -0.030115498,\n", + " -0.16927013, 0.02836557, -0.050763763, 0.0840437, -0.22718845,\n", + " 0.111397505, 0.033395614, -0.123287566, -0.2111604, -0.1580479,\n", + " 0.05520573, -0.1422921, 0.08828953, 0.051058788, -0.13312188\n", + "]\n", + "[\n", + " 0.066308185, -0.032866564, 0.115751594, 0.19082588, 0.14017,\n", + " -0.26976448, -0.056340694, -0.26923394, 0.2548541, -0.27271318,\n", + " -0.2244126, 0.07949589, -0.27710953, -0.17993368, 0.09681616,\n", + " -0.08692256, 0.22127126, -0.14512022, -0.18016525, 0.14892976,\n", + " -0.0526347, -0.008140617, -0.2916987, 0.23706906, -0.38488507,\n", + " -0.35881752, 0.09276949, -0.07051063, -0.07778231, 0.12552947,\n", + " 0.06256748, -0.25832427, 0.025054429, -0.1451448, -0.2662871,\n", + " 0.13676351, -0.07413256, 0.14966589, -0.39968985, 0.15542287,\n", + " -0.13107607, 0.02761394, 0.108077586, -0.12076956, 0.128296,\n", + " -0.05625126, 0.15723586, -0.056932643, 0.23720805, 0.23993455,\n", + " -0.035553705, -0.053907514, -0.11852807, 0.07005695, -0.06317475,\n", + " 0.070009425, 0.284697, 0.2212059, 0.018890115, 0.16924675,\n", + " 0.21651487, 0.07259682, 0.1328156, 0.3261852, 0.1914124,\n", + " -0.10120423, 0.03450111, -0.22588971, -0.04458192, 0.24116798,\n", + " -0.021830376, -0.30731413, 0.08586451, -0.058835756, 0.0010347435,\n", + " 0.0031927782, -0.09403646, -0.22608931, 0.15865424, 0.15738021,\n", + " 0.23582733, 0.1714161, 0.1585189, -0.18085755, 0.019376995,\n", + " -0.026587496, -0.017079154, -0.04588549, -0.047336094, -0.082413346,\n", + " -0.1114185, -0.05403556, 0.12438637, -0.20476522, 0.073182,\n", + " -0.12210378, -0.010543863, -0.09767598, 0.1057683, -0.050204434\n", + "]\n" + ] + } + ], + "source": [ + "const text2 = \"LangGraph is a library for building stateful, multi-actor applications with LLMs\";\n", + "\n", + "const vectors = await embeddings.embedDocuments([text, text2]);\n", + "\n", + "console.log(vectors[0].slice(0, 100));\n", + "console.log(vectors[1].slice(0, 100));" + ] + }, + { + "cell_type": "markdown", + "id": "8938e581", + "metadata": {}, + "source": [ + "## API reference\n", + "\n", + "For detailed documentation of all TogetherAIEmbeddings features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_community_embeddings_togetherai.TogetherAIEmbeddings.html" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "TypeScript", + "language": "typescript", + "name": "tslab" + }, + "language_info": { + "codemirror_mode": { + "mode": "typescript", + "name": "javascript", + "typescript": true + }, + "file_extension": ".ts", + "mimetype": "text/typescript", + "name": "typescript", + "version": "3.7.2" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/core_docs/docs/integrations/text_embedding/togetherai.mdx b/docs/core_docs/docs/integrations/text_embedding/togetherai.mdx deleted file mode 100644 index a7578aa07616..000000000000 --- a/docs/core_docs/docs/integrations/text_embedding/togetherai.mdx +++ /dev/null @@ -1,28 +0,0 @@ ---- -sidebar_label: Together AI ---- - -# Together AI - -The `TogetherAIEmbeddings` class uses the Together AI API to generate embeddings for a given text. - -## Setup - -In order to use the Together API you'll need an API key. You can sign up for a Together account and create an API key [here](https://api.together.xyz/). - -You'll first need to install the [`@langchain/community`](https://www.npmjs.com/package/@langchain/community) package: - -import IntegrationInstallTooltip from "@mdx_components/integration_install_tooltip.mdx"; - - - -```bash npm2yarn -npm install @langchain/community -``` - -## Usage - -import CodeBlock from "@theme/CodeBlock"; -import TogetherExample from "@examples/embeddings/togetherai.ts"; - -{TogetherExample} diff --git a/libs/langchain-scripts/src/cli/README.md b/libs/langchain-scripts/src/cli/README.md new file mode 100644 index 000000000000..6dfe68d7601e --- /dev/null +++ b/libs/langchain-scripts/src/cli/README.md @@ -0,0 +1,39 @@ +# Integration doc CLI + +Generate integration documentation using the CLI. + +## Supported integration types + +- Chat models +- LLMs +- Text embeddings +- Retrievers +- Document loaders + +## Usage + +1. Build the CLI: + +```bash +yarn build --filter=@langchain/scripts +``` + +2. Run the CLI: + +```bash +yarn create:integration:doc --classname --type +``` + +The `--classname` field should be passed the full class name of the integration, e.g `ChatOpenAI` or `RecursiveUrlLoader`. + +The `--type` field should be passed the type of the integration. It must be one of the following: + +- `chat` +- `llm` +- `embeddings` +- `retriever` +- `doc_loader` + +After invoking the script, you'll be prompted to fill out more integration-specific information. + +Finally, the script will log the path of the newly created integration documentation. You should open this notebook, run all the cells, handle and remove any TODOs, and verify all links work as expected. diff --git a/libs/langchain-scripts/src/cli/constants.ts b/libs/langchain-scripts/src/cli/constants.ts new file mode 100644 index 000000000000..936fd4164422 --- /dev/null +++ b/libs/langchain-scripts/src/cli/constants.ts @@ -0,0 +1,13 @@ +export const SIDEBAR_LABEL_PLACEHOLDER = "__sidebar_label__"; +export const MODULE_NAME_PLACEHOLDER = "__module_name__"; +export const PACKAGE_NAME_PLACEHOLDER = "__package_name__"; +export const FULL_IMPORT_PATH_PLACEHOLDER = "__full_import_path__"; +export const ENV_VAR_NAME_PLACEHOLDER = "__env_var_name__"; + +export const API_REF_MODULE_PLACEHOLDER = "__api_ref_module__"; +export const API_REF_PACKAGE_PLACEHOLDER = "__api_ref_package__"; +export const PYTHON_DOC_URL_PLACEHOLDER = "__python_doc_url__"; + +export const SERIALIZABLE_PLACEHOLDER = "__serializable__"; +export const LOCAL_PLACEHOLDER = "__local__"; +export const PY_SUPPORT_PLACEHOLDER = "__py_support__"; diff --git a/libs/langchain-scripts/src/cli/docs/chat.ts b/libs/langchain-scripts/src/cli/docs/chat.ts index 196caabddc19..9a18f6d3b321 100644 --- a/libs/langchain-scripts/src/cli/docs/chat.ts +++ b/libs/langchain-scripts/src/cli/docs/chat.ts @@ -6,16 +6,20 @@ import { greenText, redBackground, } from "../utils/get-input.js"; - -const PACKAGE_NAME_PLACEHOLDER = "__package_name__"; -const PACKAGE_NAME_SHORT_SNAKE_CASE_PLACEHOLDER = - "__package_name_short_snake_case__"; -const PACKAGE_NAME_SNAKE_CASE_PLACEHOLDER = "__package_name_snake_case__"; -const PACKAGE_NAME_PRETTY_PLACEHOLDER = "__package_name_pretty__"; -const PACKAGE_IMPORT_PATH_PLACEHOLDER = "__import_path__"; -const MODULE_NAME_PLACEHOLDER = "__ModuleName__"; -// This should not be prefixed with `Chat` as it's used for API keys. -const MODULE_NAME_ALL_CAPS_PLACEHOLDER = "__MODULE_NAME_ALL_CAPS__"; +import { fetchURLStatus } from "../utils/fetch-url-status.js"; +import { + SIDEBAR_LABEL_PLACEHOLDER, + MODULE_NAME_PLACEHOLDER, + PACKAGE_NAME_PLACEHOLDER, + FULL_IMPORT_PATH_PLACEHOLDER, + ENV_VAR_NAME_PLACEHOLDER, + API_REF_MODULE_PLACEHOLDER, + API_REF_PACKAGE_PLACEHOLDER, + PYTHON_DOC_URL_PLACEHOLDER, + LOCAL_PLACEHOLDER, + SERIALIZABLE_PLACEHOLDER, + PY_SUPPORT_PLACEHOLDER, +} from "../constants.js"; const TOOL_CALLING_PLACEHOLDER = "__tool_calling__"; const JSON_MODE_PLACEHOLDER = "__json_mode__"; @@ -26,30 +30,11 @@ const TOKEN_LEVEL_STREAMING_PLACEHOLDER = "__token_level_streaming__"; const TOKEN_USAGE_PLACEHOLDER = "__token_usage__"; const LOGPROBS_PLACEHOLDER = "__logprobs__"; -const SERIALIZABLE_PLACEHOLDER = "__serializable__"; -const LOCAL_PLACEHOLDER = "__local__"; -const PY_SUPPORT_PLACEHOLDER = "__py_support__"; - -const API_REF_BASE_PACKAGE_URL = `https://api.js.langchain.com/modules/langchain_${PACKAGE_NAME_PLACEHOLDER}.html`; -const API_REF_BASE_MODULE_URL = `https://api.js.langchain.com/classes/langchain_${PACKAGE_NAME_PLACEHOLDER}.${MODULE_NAME_PLACEHOLDER}.html`; - const TEMPLATE_PATH = path.resolve("./src/cli/docs/templates/chat.ipynb"); const INTEGRATIONS_DOCS_PATH = path.resolve( "../../docs/core_docs/docs/integrations/chat" ); -const fetchAPIRefUrl = async (url: string): Promise => { - try { - const res = await fetch(url); - if (res.status !== 200) { - throw new Error(`API Reference URL ${url} not found.`); - } - return true; - } catch (_) { - return false; - } -}; - type ExtraFields = { /** * If tool calling is true, structured output will also be true. @@ -65,9 +50,14 @@ type ExtraFields = { local: boolean; serializable: boolean; pySupport: boolean; + envVarName: string; + fullImportPath: string; + packageName: string; }; -async function promptExtraFields(): Promise { +async function promptExtraFields(fields: { + envVarGuess: string; +}): Promise { const hasToolCalling = await getUserInput( "Does this integration support tool calling? (y/n) ", undefined, @@ -124,6 +114,46 @@ async function promptExtraFields(): Promise { true ); + const importPath = await getUserInput( + "What is the full import path of the integration? (e.g @langchain/community/chat_models/togetherai) ", + undefined, + true + ); + + let packageName = ""; + if (importPath.startsWith("langchain/")) { + packageName = "langchain"; + } else { + packageName = importPath.split("/").slice(0, 2).join("/"); + } + + const verifyPackageName = await getUserInput( + `Is ${packageName} the correct package name? (y/n) `, + undefined, + true + ); + if (verifyPackageName.toLowerCase() === "n") { + packageName = await getUserInput( + "Please enter the full package name (e.g @langchain/community) ", + undefined, + true + ); + } + + const isEnvGuessCorrect = await getUserInput( + `Is the environment variable for the API key named ${fields.envVarGuess}? (y/n) `, + undefined, + true + ); + let envVarName = fields.envVarGuess; + if (isEnvGuessCorrect.toLowerCase() === "n") { + envVarName = await getUserInput( + "Please enter the correct environment variable name ", + undefined, + true + ); + } + return { toolCalling: hasToolCalling.toLowerCase() === "y", jsonMode: hasJsonMode.toLowerCase() === "y", @@ -136,86 +166,50 @@ async function promptExtraFields(): Promise { local: hasLocal.toLowerCase() === "y", serializable: hasSerializable.toLowerCase() === "y", pySupport: hasPySupport.toLowerCase() === "y", + envVarName, + fullImportPath: importPath, + packageName, }; } export async function fillChatIntegrationDocTemplate(fields: { - packageName: string; - moduleName: string; - isCommunity: boolean; + className: string; }) { - // Ask the user if they'd like to fill in extra fields, if so, prompt them. - let extraFields: ExtraFields | undefined; - const shouldPromptExtraFields = await getUserInput( - "Would you like to fill out optional fields? (y/n) ", - "white_background" - ); - if (shouldPromptExtraFields.toLowerCase() === "y") { - extraFields = await promptExtraFields(); - } - - let formattedApiRefPackageUrl = ""; - let formattedApiRefModuleUrl = ""; - if (fields.isCommunity) { - formattedApiRefPackageUrl = API_REF_BASE_PACKAGE_URL.replace( - PACKAGE_NAME_PLACEHOLDER, - `community_chat_models_${fields.packageName}` - ); - formattedApiRefModuleUrl = API_REF_BASE_MODULE_URL.replace( - PACKAGE_NAME_PLACEHOLDER, - `community_chat_models_${fields.packageName}` - ).replace(MODULE_NAME_PLACEHOLDER, fields.moduleName); - } else { - formattedApiRefPackageUrl = API_REF_BASE_PACKAGE_URL.replace( - PACKAGE_NAME_PLACEHOLDER, - fields.packageName - ); - formattedApiRefModuleUrl = API_REF_BASE_MODULE_URL.replace( - PACKAGE_NAME_PLACEHOLDER, - fields.packageName - ).replace(MODULE_NAME_PLACEHOLDER, fields.moduleName); - } - - const success = await Promise.all([ - fetchAPIRefUrl(formattedApiRefPackageUrl), - fetchAPIRefUrl(formattedApiRefModuleUrl), + const sidebarLabel = fields.className.replace("Chat", ""); + const pyDocUrl = `https://python.langchain.com/v0.2/docs/integrations/chat/${sidebarLabel.toLowerCase()}/`; + let envVarName = `${sidebarLabel.toUpperCase()}_API_KEY`; + const extraFields = await promptExtraFields({ + envVarGuess: envVarName, + }); + envVarName = extraFields.envVarName; + + const apiRefModuleUrl = `https://api.js.langchain.com/classes/${extraFields.fullImportPath + .replace("@", "") + .replaceAll("/", "_") + .replaceAll("-", "_")}.${fields.className}.html`; + const apiRefPackageUrl = apiRefModuleUrl + .replace("/classes/", "/modules/") + .replace(`.${fields.className}.html`, ".html"); + + const apiRefUrlSuccesses = await Promise.all([ + fetchURLStatus(apiRefModuleUrl), + fetchURLStatus(apiRefPackageUrl), ]); - if (success.some((s) => s === false)) { - // Don't error out because this might be used before the package is released. - console.error("Invalid package or module name. API reference not found."); - } - - const packageNameShortSnakeCase = fields.packageName.replaceAll("-", "_"); - let fullPackageNameSnakeCase = ""; - let packageNamePretty = ""; - let fullPackageImportPath = ""; - - if (fields.isCommunity) { - fullPackageNameSnakeCase = `langchain_community_chat_models_${packageNameShortSnakeCase}`; - fullPackageImportPath = `@langchain/community/chat_models/${fields.packageName}`; - packageNamePretty = "@langchain/community"; - } else { - fullPackageNameSnakeCase = `langchain_${packageNameShortSnakeCase}`; - packageNamePretty = `@langchain/${fields.packageName}`; - fullPackageImportPath = packageNamePretty; - } - - let moduleNameAllCaps = fields.moduleName.toUpperCase(); - if (moduleNameAllCaps.startsWith("CHAT")) { - moduleNameAllCaps = moduleNameAllCaps.replace("CHAT", ""); + if (apiRefUrlSuccesses.find((s) => !s)) { + console.warn( + "API ref URLs invalid. Please manually ensure they are correct." + ); } const docTemplate = (await fs.promises.readFile(TEMPLATE_PATH, "utf-8")) - .replaceAll(PACKAGE_NAME_PLACEHOLDER, fields.packageName) - .replaceAll(PACKAGE_NAME_SNAKE_CASE_PLACEHOLDER, fullPackageNameSnakeCase) - .replaceAll( - PACKAGE_NAME_SHORT_SNAKE_CASE_PLACEHOLDER, - packageNameShortSnakeCase - ) - .replaceAll(PACKAGE_NAME_PRETTY_PLACEHOLDER, packageNamePretty) - .replaceAll(PACKAGE_IMPORT_PATH_PLACEHOLDER, fullPackageImportPath) - .replaceAll(MODULE_NAME_PLACEHOLDER, fields.moduleName) - .replaceAll(MODULE_NAME_ALL_CAPS_PLACEHOLDER, moduleNameAllCaps) + .replaceAll(SIDEBAR_LABEL_PLACEHOLDER, sidebarLabel) + .replaceAll(MODULE_NAME_PLACEHOLDER, fields.className) + .replaceAll(PACKAGE_NAME_PLACEHOLDER, extraFields.packageName) + .replaceAll(FULL_IMPORT_PATH_PLACEHOLDER, extraFields.fullImportPath) + .replaceAll(ENV_VAR_NAME_PLACEHOLDER, extraFields.envVarName) + .replaceAll(API_REF_MODULE_PLACEHOLDER, apiRefModuleUrl) + .replaceAll(API_REF_PACKAGE_PLACEHOLDER, apiRefPackageUrl) + .replaceAll(PYTHON_DOC_URL_PLACEHOLDER, pyDocUrl) .replaceAll( TOOL_CALLING_PLACEHOLDER, extraFields?.toolCalling ? "✅" : "❌" @@ -237,10 +231,8 @@ export async function fillChatIntegrationDocTemplate(fields: { ) .replace(PY_SUPPORT_PLACEHOLDER, extraFields?.pySupport ? "✅" : "❌"); - const docPath = path.join( - INTEGRATIONS_DOCS_PATH, - `${packageNameShortSnakeCase}.ipynb` - ); + const docFileName = extraFields.fullImportPath.split("/").pop(); + const docPath = path.join(INTEGRATIONS_DOCS_PATH, `${docFileName}.ipynb`); await fs.promises.writeFile(docPath, docTemplate); const prettyDocPath = docPath.split("docs/core_docs/")[1]; diff --git a/libs/langchain-scripts/src/cli/docs/document_loaders.ts b/libs/langchain-scripts/src/cli/docs/document_loaders.ts index 9fe5118af0b4..bf3b058831f3 100644 --- a/libs/langchain-scripts/src/cli/docs/document_loaders.ts +++ b/libs/langchain-scripts/src/cli/docs/document_loaders.ts @@ -1,33 +1,30 @@ import * as path from "node:path"; import * as fs from "node:fs"; -import _ from "lodash"; import { boldText, getUserInput, greenText, redBackground, } from "../utils/get-input.js"; - -const NODE_OR_WEB_PLACEHOLDER = "__fs_or_web__"; -const NODE_OR_WEB_IMPORT_PATH_PLACEHOLDER = "__fs_or_web_import_path__"; -const FILE_NAME_PLACEHOLDER = "__file_name__"; -const MODULE_NAME_PLACEHOLDER = "__ModuleName__"; - -const API_REF_BASE_PACKAGE_URL = `https://api.js.langchain.com/modules/langchain_community_document_loaders_${NODE_OR_WEB_PLACEHOLDER}_${FILE_NAME_PLACEHOLDER}.html`; -const API_REF_BASE_MODULE_URL = `https://v02.api.js.langchain.com/classes/langchain_community_document_loaders_${NODE_OR_WEB_PLACEHOLDER}_${FILE_NAME_PLACEHOLDER}.${MODULE_NAME_PLACEHOLDER}.html`; - -const SERIALIZABLE_PLACEHOLDER = "__serializable__"; -const LOCAL_PLACEHOLDER = "__local__"; -const PY_SUPPORT_PLACEHOLDER = "__py_support__"; +import { fetchURLStatus } from "../utils/fetch-url-status.js"; +import { + SIDEBAR_LABEL_PLACEHOLDER, + MODULE_NAME_PLACEHOLDER, + PACKAGE_NAME_PLACEHOLDER, + FULL_IMPORT_PATH_PLACEHOLDER, + ENV_VAR_NAME_PLACEHOLDER, + PYTHON_DOC_URL_PLACEHOLDER, + API_REF_MODULE_PLACEHOLDER, + API_REF_PACKAGE_PLACEHOLDER, + LOCAL_PLACEHOLDER, + SERIALIZABLE_PLACEHOLDER, + PY_SUPPORT_PLACEHOLDER, +} from "../constants.js"; const NODE_SUPPORT_PLACEHOLDER = "__fs_support__"; - const NODE_ONLY_SIDEBAR_BADGE_PLACEHOLDER = "__node_only_sidebar__"; const NODE_ONLY_TOOL_TIP_PLACEHOLDER = "__node_only_tooltip__"; -// This should not be suffixed with `Loader` as it's used for API keys. -const MODULE_NAME_ALL_CAPS_PLACEHOLDER = "__MODULE_NAME_ALL_CAPS__"; - const TEMPLATE_PATH = path.resolve( "./src/cli/docs/templates/document_loaders.ipynb" ); @@ -36,30 +33,23 @@ const INTEGRATIONS_DOCS_PATH = path.resolve( ); const NODE_ONLY_TOOLTIP = - "```{=mdx}\n\n:::tip Compatibility\n\nOnly available on Node.js.\n\n:::\n\n```\n"; + "```{=mdx}\\n\\n:::tip Compatibility\\n\\nOnly available on Node.js.\\n\\n:::\\n\\n```\\n"; const NODE_ONLY_SIDEBAR_BADGE = `sidebar_class_name: node-only`; -const fetchAPIRefUrl = async (url: string): Promise => { - try { - const res = await fetch(url); - if (res.status !== 200) { - throw new Error(`API Reference URL ${url} not found.`); - } - return true; - } catch (_) { - return false; - } -}; - type ExtraFields = { webLoader: boolean; nodeOnly: boolean; serializable: boolean; pySupport: boolean; local: boolean; + envVarName: string; + fullImportPath: string; + packageName: string; }; -async function promptExtraFields(): Promise { +async function promptExtraFields(fields: { + envVarGuess: string; +}): Promise { const isWebLoader = await getUserInput( "Is this integration a web loader? (y/n) ", undefined, @@ -85,6 +75,45 @@ async function promptExtraFields(): Promise { undefined, true ); + const importPath = await getUserInput( + "What is the full import path of the integration? (e.g @langchain/community/llms/togetherai) ", + undefined, + true + ); + + let packageName = ""; + if (importPath.startsWith("langchain/")) { + packageName = "langchain"; + } else { + packageName = importPath.split("/").slice(0, 2).join("/"); + } + + const verifyPackageName = await getUserInput( + `Is ${packageName} the correct package name? (y/n) `, + undefined, + true + ); + if (verifyPackageName.toLowerCase() === "n") { + packageName = await getUserInput( + "Please enter the full package name (e.g @langchain/community) ", + undefined, + true + ); + } + + const isEnvGuessCorrect = await getUserInput( + `Is the environment variable for the API key named ${fields.envVarGuess}? (y/n) `, + undefined, + true + ); + let envVarName = fields.envVarGuess; + if (isEnvGuessCorrect.toLowerCase() === "n") { + envVarName = await getUserInput( + "Please enter the correct environment variable name ", + undefined, + true + ); + } return { webLoader: isWebLoader.toLowerCase() === "y", @@ -92,63 +121,53 @@ async function promptExtraFields(): Promise { serializable: isSerializable.toLowerCase() === "y", pySupport: hasPySupport.toLowerCase() === "y", local: hasLocalSupport.toLowerCase() === "y", + envVarName, + fullImportPath: importPath, + packageName, }; } export async function fillDocLoaderIntegrationDocTemplate(fields: { - packageName: string; - moduleName: string; - webSupport?: boolean; - nodeSupport?: boolean; + className: string; }) { - // Ask the user if they'd like to fill in extra fields, if so, prompt them. - let extraFields: ExtraFields | undefined; - const shouldPromptExtraFields = await getUserInput( - "Would you like to fill out optional fields? (y/n) ", - "white_background" - ); - if (shouldPromptExtraFields.toLowerCase() === "y") { - extraFields = await promptExtraFields(); - } - - const formattedPackageApiRefUrl = API_REF_BASE_PACKAGE_URL.replace( - NODE_OR_WEB_PLACEHOLDER, - extraFields?.webLoader ? "web" : "fs" - ).replace(FILE_NAME_PLACEHOLDER, fields.packageName); - - const formattedApiRefModuleUrl = API_REF_BASE_MODULE_URL.replace( - NODE_OR_WEB_PLACEHOLDER, - extraFields?.webLoader ? "web" : "fs" - ) - .replace(FILE_NAME_PLACEHOLDER, fields.packageName) - .replace(MODULE_NAME_PLACEHOLDER, fields.moduleName); - - const success = await Promise.all([ - fetchAPIRefUrl(formattedApiRefModuleUrl), - fetchAPIRefUrl(formattedPackageApiRefUrl), + const sidebarLabel = fields.className.replace("Loader", ""); + const pyDocUrl = `https://python.langchain.com/v0.2/docs/integrations/document_loaders/${sidebarLabel.toLowerCase()}/`; + let envVarName = `${sidebarLabel.toUpperCase()}_API_KEY`; + const extraFields = await promptExtraFields({ + envVarGuess: envVarName, + }); + envVarName = extraFields.envVarName; + const importPathEnding = extraFields.fullImportPath.split("/").pop() ?? ""; + const apiRefModuleUrl = `https://api.js.langchain.com/classes/${extraFields.fullImportPath + .replace("@", "") + .replaceAll("/", "_") + .replaceAll("-", "_")}_${importPathEnding}.${fields.className}.html`; + const apiRefPackageUrl = apiRefModuleUrl + .replace("/classes/", "/modules/") + .replace(`.${fields.className}.html`, ".html"); + + const apiRefUrlSuccesses = await Promise.all([ + fetchURLStatus(apiRefModuleUrl), + fetchURLStatus(apiRefPackageUrl), ]); - if (success.find((s) => s === false)) { - // Don't error out because this might be used before the package is released. - console.error("Invalid package or module name. API reference not found."); - } - - let moduleNameAllCaps = _.snakeCase(fields.moduleName).toUpperCase(); - if (moduleNameAllCaps.endsWith("_LOADER")) { - moduleNameAllCaps = moduleNameAllCaps.replace("_LOADER", ""); + if (apiRefUrlSuccesses.find((s) => !s)) { + console.warn( + "API ref URLs invalid. Please manually ensure they are correct." + ); } const docTemplate = (await fs.promises.readFile(TEMPLATE_PATH, "utf-8")) - .replaceAll(NODE_OR_WEB_PLACEHOLDER, extraFields?.webLoader ? "web" : "fs") - .replaceAll(MODULE_NAME_PLACEHOLDER, fields.moduleName) - .replaceAll(MODULE_NAME_ALL_CAPS_PLACEHOLDER, moduleNameAllCaps) - .replaceAll( - NODE_OR_WEB_IMPORT_PATH_PLACEHOLDER, - extraFields?.webLoader ? "web" : "fs" - ) - .replaceAll(FILE_NAME_PLACEHOLDER, fields.packageName) + .replaceAll(SIDEBAR_LABEL_PLACEHOLDER, sidebarLabel) + .replaceAll(MODULE_NAME_PLACEHOLDER, fields.className) + .replaceAll(PACKAGE_NAME_PLACEHOLDER, extraFields.packageName) + .replaceAll(FULL_IMPORT_PATH_PLACEHOLDER, extraFields.fullImportPath) + .replaceAll(ENV_VAR_NAME_PLACEHOLDER, envVarName) + .replaceAll(PYTHON_DOC_URL_PLACEHOLDER, pyDocUrl) + .replaceAll(API_REF_MODULE_PLACEHOLDER, apiRefModuleUrl) + .replaceAll(API_REF_PACKAGE_PLACEHOLDER, apiRefPackageUrl) .replaceAll( NODE_ONLY_SIDEBAR_BADGE_PLACEHOLDER, - extraFields?.nodeOnly ? NODE_ONLY_SIDEBAR_BADGE : "" + extraFields.nodeOnly ? NODE_ONLY_SIDEBAR_BADGE : "" ) .replaceAll( NODE_ONLY_TOOL_TIP_PLACEHOLDER, @@ -168,7 +187,7 @@ export async function fillDocLoaderIntegrationDocTemplate(fields: { const docPath = path.join( INTEGRATIONS_DOCS_PATH, extraFields?.webLoader ? "web_loaders" : "file_loaders", - `${fields.packageName}.ipynb` + `${importPathEnding}.ipynb` ); await fs.promises.writeFile(docPath, docTemplate); const prettyDocPath = docPath.split("docs/core_docs/")[1]; diff --git a/libs/langchain-scripts/src/cli/docs/embeddings.ts b/libs/langchain-scripts/src/cli/docs/embeddings.ts index 03e1cc96f99e..f87ce6d8ecd2 100644 --- a/libs/langchain-scripts/src/cli/docs/embeddings.ts +++ b/libs/langchain-scripts/src/cli/docs/embeddings.ts @@ -6,17 +6,19 @@ import { greenText, redBackground, } from "../utils/get-input.js"; - -const PACKAGE_NAME_PLACEHOLDER = "__package_name__"; -const MODULE_NAME_PLACEHOLDER = "__ModuleName__"; -const SIDEBAR_LABEL_PLACEHOLDER = "__sidebar_label__"; -const FULL_IMPORT_PATH_PLACEHOLDER = "__full_import_path__"; -const LOCAL_PLACEHOLDER = "__local__"; -const PY_SUPPORT_PLACEHOLDER = "__py_support__"; -const ENV_VAR_NAME_PLACEHOLDER = "__env_var_name__"; -const API_REF_MODULE_PLACEHOLDER = "__api_ref_module__"; -const API_REF_PACKAGE_PLACEHOLDER = "__api_ref_package__"; -const PYTHON_DOC_URL_PLACEHOLDER = "__python_doc_url__"; +import { fetchURLStatus } from "../utils/fetch-url-status.js"; +import { + PACKAGE_NAME_PLACEHOLDER, + MODULE_NAME_PLACEHOLDER, + SIDEBAR_LABEL_PLACEHOLDER, + FULL_IMPORT_PATH_PLACEHOLDER, + LOCAL_PLACEHOLDER, + PY_SUPPORT_PLACEHOLDER, + ENV_VAR_NAME_PLACEHOLDER, + API_REF_MODULE_PLACEHOLDER, + API_REF_PACKAGE_PLACEHOLDER, + PYTHON_DOC_URL_PLACEHOLDER, +} from "../constants.js"; const TEMPLATE_PATH = path.resolve( "./src/cli/docs/templates/text_embedding.ipynb" @@ -25,18 +27,6 @@ const INTEGRATIONS_DOCS_PATH = path.resolve( "../../docs/core_docs/docs/integrations/text_embedding" ); -const fetchAPIRefUrl = async (url: string): Promise => { - try { - const res = await fetch(url); - if (res.status !== 200) { - throw new Error(`API Reference URL ${url} not found.`); - } - return true; - } catch (_) { - return false; - } -}; - type ExtraFields = { local: boolean; pySupport: boolean; @@ -47,10 +37,9 @@ type ExtraFields = { async function promptExtraFields(fields: { envVarGuess: string; - packageNameGuess: string; - isCommunity: boolean; }): Promise { - const { envVarGuess, packageNameGuess, isCommunity } = fields; + const { envVarGuess } = fields; + const canRunLocally = await getUserInput( "Does this embeddings model support local usage? (y/n) ", undefined, @@ -62,73 +51,63 @@ async function promptExtraFields(fields: { true ); - let packageName = packageNameGuess; - if (!isCommunity) { - // If it's not community, get the package name. + const importPath = await getUserInput( + "What is the full import path of the integration? (e.g @langchain/community/embeddings/togetherai) ", + undefined, + true + ); - const isOtherPackageName = await getUserInput( - `Is this integration part of the ${packageNameGuess} package? (y/n) ` - ); - if (isOtherPackageName.toLowerCase() === "n") { - packageName = await getUserInput( - "What is the name of the package this integration is located in? (e.g @langchain/openai) ", - undefined, - true - ); - if ( - !packageName.startsWith("@langchain/") && - !packageName.startsWith("langchain/") - ) { - packageName = await getUserInput( - "Packages must start with either '@langchain/' or 'langchain/'. Please enter a valid package name: ", - undefined, - true - ); - } - } + let packageName = ""; + if (importPath.startsWith("langchain/")) { + packageName = "langchain"; + } else { + packageName = importPath.split("/").slice(0, 2).join("/"); } - // If it's community or langchain, ask for the full import path - let fullImportPath: string | undefined; - if ( - packageName.startsWith("@langchain/community") || - packageName.startsWith("langchain/") - ) { - fullImportPath = await getUserInput( - "What is the full import path of the package? (e.g '@langchain/community/embeddings/togetherai') ", + const verifyPackageName = await getUserInput( + `Is ${packageName} the correct package name? (y/n) `, + undefined, + true + ); + if (verifyPackageName.toLowerCase() === "n") { + packageName = await getUserInput( + "Please enter the full package name (e.g @langchain/community) ", undefined, true ); } - const envVarName = await getUserInput( - `Is the environment variable for the API key named ${envVarGuess}? If it is, reply with 'y', else reply with the correct name: `, + const isEnvGuessCorrect = await getUserInput( + `Is the environment variable for the API key named ${envVarGuess}? (y/n) `, undefined, true ); + let envVarName = envVarGuess; + if (isEnvGuessCorrect.toLowerCase() === "n") { + envVarName = await getUserInput( + "Please enter the correct environment variable name ", + undefined, + true + ); + } return { local: canRunLocally.toLowerCase() === "y", pySupport: hasPySupport.toLowerCase() === "y", packageName, - fullImportPath, - envVarName: - envVarName.toLowerCase() === "y" ? envVarGuess : envVarName.toUpperCase(), + fullImportPath: importPath, + envVarName, }; } export async function fillEmbeddingsIntegrationDocTemplate(fields: { - packageName: string; - moduleName: string; - isCommunity: boolean; + className: string; }) { - const sidebarLabel = fields.moduleName.replace("Embeddings", ""); + const sidebarLabel = fields.className.replace("Embeddings", ""); const pyDocUrl = `https://python.langchain.com/docs/integrations/text_embedding/${sidebarLabel.toLowerCase()}/`; let envVarName = `${sidebarLabel.toUpperCase()}_API_KEY`; const extraFields = await promptExtraFields({ - packageNameGuess: `@langchain/${fields.packageName}`, envVarGuess: envVarName, - isCommunity: fields.isCommunity, }); envVarName = extraFields.envVarName; const { pySupport } = extraFields; @@ -139,14 +118,14 @@ export async function fillEmbeddingsIntegrationDocTemplate(fields: { const apiRefModuleUrl = `https://api.js.langchain.com/classes/${fullImportPath .replace("@", "") .replaceAll("/", "_") - .replaceAll("-", "_")}.${fields.moduleName}.html`; + .replaceAll("-", "_")}.${fields.className}.html`; const apiRefPackageUrl = apiRefModuleUrl .replace("/classes/", "/modules/") - .replace(`.${fields.moduleName}.html`, ".html"); + .replace(`.${fields.className}.html`, ".html"); const apiRefUrlSuccesses = await Promise.all([ - fetchAPIRefUrl(apiRefModuleUrl), - fetchAPIRefUrl(apiRefPackageUrl), + fetchURLStatus(apiRefModuleUrl), + fetchURLStatus(apiRefPackageUrl), ]); if (apiRefUrlSuccesses.find((s) => !s)) { console.warn( @@ -156,7 +135,7 @@ export async function fillEmbeddingsIntegrationDocTemplate(fields: { const docTemplate = (await fs.promises.readFile(TEMPLATE_PATH, "utf-8")) .replaceAll(PACKAGE_NAME_PLACEHOLDER, packageName) - .replaceAll(MODULE_NAME_PLACEHOLDER, fields.moduleName) + .replaceAll(MODULE_NAME_PLACEHOLDER, fields.className) .replaceAll(SIDEBAR_LABEL_PLACEHOLDER, sidebarLabel) .replaceAll(FULL_IMPORT_PATH_PLACEHOLDER, fullImportPath) .replaceAll(LOCAL_PLACEHOLDER, localSupport ? "✅" : "❌") diff --git a/libs/langchain-scripts/src/cli/docs/index.ts b/libs/langchain-scripts/src/cli/docs/index.ts index 9d7f5aa35a12..a4ab18784d36 100644 --- a/libs/langchain-scripts/src/cli/docs/index.ts +++ b/libs/langchain-scripts/src/cli/docs/index.ts @@ -9,68 +9,51 @@ import { fillRetrieverIntegrationDocTemplate } from "./retrievers.js"; import { fillEmbeddingsIntegrationDocTemplate } from "./embeddings.js"; type CLIInput = { - package: string; - module: string; type: string; community: boolean; + classname: string; }; async function main() { const program = new Command(); program .description("Create a new integration doc.") - .option("--package ", "Package name, eg openai.") - .option("--module ", "Module name, e.g ChatOpenAI") - .option("--type ", "Type of integration, e.g. 'chat'") .option( - "--community", - "If the integration is a community integration. Will effect the fields populated in the template." - ); + "--classname ", + "Class name of the integration. e.g ChatOpenAI" + ) + .option("--type ", "Type of integration, e.g. 'chat'"); program.parse(); const options = program.opts(); - const { module: moduleName, type, community: isCommunity } = options; - let { package: packageName } = options; - - if (packageName.startsWith("@langchain/")) { - packageName = packageName.replace("@langchain/", ""); - } + const { classname: className, type } = options; switch (type) { case "chat": await fillChatIntegrationDocTemplate({ - packageName, - moduleName, - isCommunity, - }); - break; - case "doc_loader": - await fillDocLoaderIntegrationDocTemplate({ - packageName, - moduleName, + className, }); break; case "llm": await fillLLMIntegrationDocTemplate({ - packageName, - moduleName, - isCommunity, + className, + }); + break; + case "embeddings": + await fillEmbeddingsIntegrationDocTemplate({ + className, }); break; case "retriever": await fillRetrieverIntegrationDocTemplate({ - packageName, - moduleName, - isCommunity, + className, }); break; - case "embeddings": - await fillEmbeddingsIntegrationDocTemplate({ - packageName, - moduleName, - isCommunity, + case "doc_loader": + await fillDocLoaderIntegrationDocTemplate({ + className, }); break; default: diff --git a/libs/langchain-scripts/src/cli/docs/llms.ts b/libs/langchain-scripts/src/cli/docs/llms.ts index bd538bb55ac1..214f5cc38711 100644 --- a/libs/langchain-scripts/src/cli/docs/llms.ts +++ b/libs/langchain-scripts/src/cli/docs/llms.ts @@ -6,48 +6,38 @@ import { greenText, redBackground, } from "../utils/get-input.js"; - -const PACKAGE_NAME_PLACEHOLDER = "__package_name__"; -const PACKAGE_NAME_SHORT_SNAKE_CASE_PLACEHOLDER = - "__package_name_short_snake_case__"; -const PACKAGE_NAME_SNAKE_CASE_PLACEHOLDER = "__package_name_snake_case__"; -const PACKAGE_NAME_PRETTY_PLACEHOLDER = "__package_name_pretty__"; -const PACKAGE_IMPORT_PATH_PLACEHOLDER = "__import_path__"; -const MODULE_NAME_PLACEHOLDER = "__ModuleName__"; -// This should not be prefixed with `Chat` as it's used for API keys. -const MODULE_NAME_ALL_CAPS_PLACEHOLDER = "__MODULE_NAME_ALL_CAPS__"; - -const SERIALIZABLE_PLACEHOLDER = "__serializable__"; -const LOCAL_PLACEHOLDER = "__local__"; -const PY_SUPPORT_PLACEHOLDER = "__py_support__"; - -const API_REF_BASE_PACKAGE_URL = `https://api.js.langchain.com/modules/langchain_${PACKAGE_NAME_PLACEHOLDER}.html`; -const API_REF_BASE_MODULE_URL = `https://api.js.langchain.com/classes/langchain_${PACKAGE_NAME_PLACEHOLDER}.${MODULE_NAME_PLACEHOLDER}.html`; +import { fetchURLStatus } from "../utils/fetch-url-status.js"; +import { + API_REF_MODULE_PLACEHOLDER, + API_REF_PACKAGE_PLACEHOLDER, + ENV_VAR_NAME_PLACEHOLDER, + FULL_IMPORT_PATH_PLACEHOLDER, + LOCAL_PLACEHOLDER, + MODULE_NAME_PLACEHOLDER, + PACKAGE_NAME_PLACEHOLDER, + PY_SUPPORT_PLACEHOLDER, + PYTHON_DOC_URL_PLACEHOLDER, + SERIALIZABLE_PLACEHOLDER, + SIDEBAR_LABEL_PLACEHOLDER, +} from "../constants.js"; const TEMPLATE_PATH = path.resolve("./src/cli/docs/templates/llms.ipynb"); const INTEGRATIONS_DOCS_PATH = path.resolve( "../../docs/core_docs/docs/integrations/llms" ); -const fetchAPIRefUrl = async (url: string): Promise => { - try { - const res = await fetch(url); - if (res.status !== 200) { - throw new Error(`API Reference URL ${url} not found.`); - } - return true; - } catch (_) { - return false; - } -}; - type ExtraFields = { local: boolean; serializable: boolean; pySupport: boolean; + packageName: string; + fullImportPath: string; + envVarName: string; }; -async function promptExtraFields(): Promise { +async function promptExtraFields(fields: { + envVarGuess: string; +}): Promise { const hasLocal = await getUserInput( "Does this integration support local usage? (y/n) ", undefined, @@ -64,92 +54,94 @@ async function promptExtraFields(): Promise { true ); + const importPath = await getUserInput( + "What is the full import path of the integration? (e.g @langchain/community/llms/togetherai) ", + undefined, + true + ); + + let packageName = ""; + if (importPath.startsWith("langchain/")) { + packageName = "langchain"; + } else { + packageName = importPath.split("/").slice(0, 2).join("/"); + } + + const verifyPackageName = await getUserInput( + `Is ${packageName} the correct package name? (y/n) `, + undefined, + true + ); + if (verifyPackageName.toLowerCase() === "n") { + packageName = await getUserInput( + "Please enter the full package name (e.g @langchain/community) ", + undefined, + true + ); + } + + const isEnvGuessCorrect = await getUserInput( + `Is the environment variable for the API key named ${fields.envVarGuess}? (y/n) `, + undefined, + true + ); + let envVarName = fields.envVarGuess; + if (isEnvGuessCorrect.toLowerCase() === "n") { + envVarName = await getUserInput( + "Please enter the correct environment variable name ", + undefined, + true + ); + } + return { local: hasLocal.toLowerCase() === "y", serializable: hasSerializable.toLowerCase() === "y", pySupport: hasPySupport.toLowerCase() === "y", + packageName, + fullImportPath: importPath, + envVarName, }; } export async function fillLLMIntegrationDocTemplate(fields: { - packageName: string; - moduleName: string; - isCommunity: boolean; + className: string; }) { - // Ask the user if they'd like to fill in extra fields, if so, prompt them. - let extraFields: ExtraFields | undefined; - const shouldPromptExtraFields = await getUserInput( - "Would you like to fill out optional fields? (y/n) ", - "white_background" - ); - if (shouldPromptExtraFields.toLowerCase() === "y") { - extraFields = await promptExtraFields(); - } - - let formattedApiRefPackageUrl = ""; - let formattedApiRefModuleUrl = ""; - if (fields.isCommunity) { - formattedApiRefPackageUrl = API_REF_BASE_PACKAGE_URL.replace( - PACKAGE_NAME_PLACEHOLDER, - `community_llms_${fields.packageName}` - ); - formattedApiRefModuleUrl = API_REF_BASE_MODULE_URL.replace( - PACKAGE_NAME_PLACEHOLDER, - `community_llms_${fields.packageName}` - ).replace(MODULE_NAME_PLACEHOLDER, fields.moduleName); - } else { - formattedApiRefPackageUrl = API_REF_BASE_PACKAGE_URL.replace( - PACKAGE_NAME_PLACEHOLDER, - fields.packageName - ); - formattedApiRefModuleUrl = API_REF_BASE_MODULE_URL.replace( - PACKAGE_NAME_PLACEHOLDER, - fields.packageName - ).replace(MODULE_NAME_PLACEHOLDER, fields.moduleName); - } - - const success = await Promise.all([ - fetchAPIRefUrl(formattedApiRefPackageUrl), - fetchAPIRefUrl(formattedApiRefModuleUrl), + const sidebarLabel = fields.className.replace("LLM", "").replace("Llm", ""); + const pyDocUrl = `https://python.langchain.com/v0.2/docs/integrations/llms/${sidebarLabel.toLowerCase()}/`; + let envVarName = `${sidebarLabel.toUpperCase()}_API_KEY`; + const extraFields = await promptExtraFields({ + envVarGuess: envVarName, + }); + envVarName = extraFields.envVarName; + + const apiRefModuleUrl = `https://api.js.langchain.com/classes/${extraFields.fullImportPath + .replace("@", "") + .replaceAll("/", "_") + .replaceAll("-", "_")}.${fields.className}.html`; + const apiRefPackageUrl = apiRefModuleUrl + .replace("/classes/", "/modules/") + .replace(`.${fields.className}.html`, ".html"); + + const apiRefUrlSuccesses = await Promise.all([ + fetchURLStatus(apiRefModuleUrl), + fetchURLStatus(apiRefPackageUrl), ]); - if (success.some((s) => s === false)) { - // Don't error out because this might be used before the package is released. - console.error("Invalid package or module name. API reference not found."); - } - - const packageNameShortSnakeCase = fields.packageName.replaceAll("-", "_"); - let fullPackageNameSnakeCase = ""; - let packageNamePretty = ""; - let fullPackageImportPath = ""; - - if (fields.isCommunity) { - fullPackageNameSnakeCase = `langchain_community_llms_${packageNameShortSnakeCase}`; - fullPackageImportPath = `@langchain/community/llms/${fields.packageName}`; - packageNamePretty = "@langchain/community"; - } else { - fullPackageNameSnakeCase = `langchain_${packageNameShortSnakeCase}`; - packageNamePretty = `@langchain/${fields.packageName}`; - fullPackageImportPath = packageNamePretty; - } - - let moduleNameAllCaps = fields.moduleName.toUpperCase(); - if (moduleNameAllCaps.endsWith("_LLM")) { - moduleNameAllCaps = moduleNameAllCaps.replace("_LLM", ""); - } else if (moduleNameAllCaps.endsWith("LLM")) { - moduleNameAllCaps = moduleNameAllCaps.replace("LLM", ""); + if (apiRefUrlSuccesses.find((s) => !s)) { + console.warn( + "API ref URLs invalid. Please manually ensure they are correct." + ); } const docTemplate = (await fs.promises.readFile(TEMPLATE_PATH, "utf-8")) - .replaceAll(PACKAGE_NAME_PLACEHOLDER, fields.packageName) - .replaceAll(PACKAGE_NAME_SNAKE_CASE_PLACEHOLDER, fullPackageNameSnakeCase) - .replaceAll( - PACKAGE_NAME_SHORT_SNAKE_CASE_PLACEHOLDER, - packageNameShortSnakeCase - ) - .replaceAll(PACKAGE_NAME_PRETTY_PLACEHOLDER, packageNamePretty) - .replaceAll(PACKAGE_IMPORT_PATH_PLACEHOLDER, fullPackageImportPath) - .replaceAll(MODULE_NAME_PLACEHOLDER, fields.moduleName) - .replaceAll(MODULE_NAME_ALL_CAPS_PLACEHOLDER, moduleNameAllCaps) + .replaceAll(SIDEBAR_LABEL_PLACEHOLDER, sidebarLabel) + .replaceAll(MODULE_NAME_PLACEHOLDER, fields.className) + .replaceAll(PACKAGE_NAME_PLACEHOLDER, extraFields.packageName) + .replaceAll(FULL_IMPORT_PATH_PLACEHOLDER, extraFields.fullImportPath) + .replaceAll(ENV_VAR_NAME_PLACEHOLDER, extraFields.envVarName) + .replaceAll(PYTHON_DOC_URL_PLACEHOLDER, pyDocUrl) + .replaceAll(API_REF_MODULE_PLACEHOLDER, apiRefModuleUrl) + .replaceAll(API_REF_PACKAGE_PLACEHOLDER, apiRefPackageUrl) .replace(LOCAL_PLACEHOLDER, extraFields?.local ? "✅" : "❌") .replace( SERIALIZABLE_PLACEHOLDER, @@ -157,6 +149,9 @@ export async function fillLLMIntegrationDocTemplate(fields: { ) .replace(PY_SUPPORT_PLACEHOLDER, extraFields?.pySupport ? "✅" : "❌"); + const packageNameShortSnakeCase = fields.className + .replace(/-/g, "_") + .toLowerCase(); const docPath = path.join( INTEGRATIONS_DOCS_PATH, `${packageNameShortSnakeCase}.ipynb` diff --git a/libs/langchain-scripts/src/cli/docs/retrievers.ts b/libs/langchain-scripts/src/cli/docs/retrievers.ts index fc68cb96d5e3..f94613027901 100644 --- a/libs/langchain-scripts/src/cli/docs/retrievers.ts +++ b/libs/langchain-scripts/src/cli/docs/retrievers.ts @@ -6,28 +6,19 @@ import { greenText, redBackground, } from "../utils/get-input.js"; +import { fetchURLStatus } from "../utils/fetch-url-status.js"; +import { + PACKAGE_NAME_PLACEHOLDER, + MODULE_NAME_PLACEHOLDER, + SIDEBAR_LABEL_PLACEHOLDER, + FULL_IMPORT_PATH_PLACEHOLDER, + PY_SUPPORT_PLACEHOLDER, + API_REF_MODULE_PLACEHOLDER, + PYTHON_DOC_URL_PLACEHOLDER, +} from "../constants.js"; -const PACKAGE_NAME_PLACEHOLDER = "__package_name__"; -const MODULE_NAME_PLACEHOLDER = "__ModuleName__"; -const SIDEBAR_LABEL_PLACEHOLDER = "__sidebar_label__"; -const FULL_IMPORT_PATH_PLACEHOLDER = "__full_import_path__"; const HAS_CLOUD_OFFERING_PLACEHOLDER = "__has_cloud_offering__"; const CAN_SELF_HOST_PLACEHOLDER = "__can_self_host__"; -const PY_SUPPORT_PLACEHOLDER = "__py_support__"; -const API_REF_MODULE_PLACEHOLDER = "__api_ref_module__"; -const PYTHON_DOC_URL_PLACEHOLDER = "__python_doc_url__"; - -const fetchAPIRefUrl = async (url: string): Promise => { - try { - const res = await fetch(url); - if (res.status !== 200) { - throw new Error(`API Reference URL ${url} not found.`); - } - return true; - } catch (_) { - return false; - } -}; const TEMPLATE_PATH = path.resolve("./src/cli/docs/templates/retrievers.ipynb"); const INTEGRATIONS_DOCS_PATH = path.resolve( @@ -42,12 +33,7 @@ type ExtraFields = { pySupport: boolean; }; -async function promptExtraFields(fields: { - packageNameGuess: string; - isCommunity: boolean; -}): Promise { - const { packageNameGuess, isCommunity } = fields; - +async function promptExtraFields(): Promise { const hasCloudOffering = await getUserInput( "Does this retriever support self hosting? (y/n) ", undefined, @@ -64,40 +50,27 @@ async function promptExtraFields(fields: { true ); - let packageName = packageNameGuess; - if (!isCommunity) { - // If it's not community, get the package name. + const importPath = await getUserInput( + "What is the full import path of the integration? (e.g @langchain/community/retrievers/my_retriever) ", + undefined, + true + ); - const isOtherPackageName = await getUserInput( - `Is this integration part of the ${packageNameGuess} package? (y/n) ` - ); - if (isOtherPackageName.toLowerCase() === "n") { - packageName = await getUserInput( - "What is the name of the package this integration is located in? (e.g @langchain/openai) ", - undefined, - true - ); - if ( - !packageName.startsWith("@langchain/") && - !packageName.startsWith("langchain/") - ) { - packageName = await getUserInput( - "Packages must start with either '@langchain/' or 'langchain/'. Please enter a valid package name: ", - undefined, - true - ); - } - } + let packageName = ""; + if (importPath.startsWith("langchain/")) { + packageName = "langchain"; + } else { + packageName = importPath.split("/").slice(0, 2).join("/"); } - // If it's community or langchain, ask for the full import path - let fullImportPath: string | undefined; - if ( - packageName.startsWith("@langchain/community") || - packageName.startsWith("langchain/") - ) { - fullImportPath = await getUserInput( - "What is the full import path of the module? (e.g '@langchain/community/retrievers/tavily_search_api') ", + const verifyPackageName = await getUserInput( + `Is ${packageName} the correct package name? (y/n) `, + undefined, + true + ); + if (verifyPackageName.toLowerCase() === "n") { + packageName = await getUserInput( + "Please enter the full package name (e.g @langchain/community) ", undefined, true ); @@ -105,7 +78,7 @@ async function promptExtraFields(fields: { return { packageName, - fullImportPath, + fullImportPath: importPath, canSelfHost: canSelfHost.toLowerCase() === "y", hasCloudOffering: hasCloudOffering.toLowerCase() === "y", pySupport: hasPySupport.toLowerCase() === "y", @@ -113,16 +86,11 @@ async function promptExtraFields(fields: { } export async function fillRetrieverIntegrationDocTemplate(fields: { - packageName: string; - moduleName: string; - isCommunity: boolean; + className: string; }) { - const sidebarLabel = fields.moduleName.replace("Retriever", ""); + const sidebarLabel = fields.className.replace("Retriever", ""); const pyDocUrl = `https://python.langchain.com/v0.2/docs/integrations/retrievers/${sidebarLabel.toLowerCase()}/`; - const extraFields = await promptExtraFields({ - packageNameGuess: `@langchain/${fields.packageName}`, - isCommunity: fields.isCommunity, - }); + const extraFields = await promptExtraFields(); const { pySupport } = extraFields; const { canSelfHost } = extraFields; const { hasCloudOffering } = extraFields; @@ -132,14 +100,14 @@ export async function fillRetrieverIntegrationDocTemplate(fields: { const apiRefModuleUrl = `https://api.js.langchain.com/classes/${fullImportPath .replace("@", "") .replaceAll("/", "_") - .replaceAll("-", "_")}.${fields.moduleName}.html`; + .replaceAll("-", "_")}.${fields.className}.html`; const apiRefPackageUrl = apiRefModuleUrl .replace("/classes/", "/modules/") - .replace(`.${fields.moduleName}.html`, ".html"); + .replace(`.${fields.className}.html`, ".html"); const apiRefUrlSuccesses = await Promise.all([ - fetchAPIRefUrl(apiRefModuleUrl), - fetchAPIRefUrl(apiRefPackageUrl), + fetchURLStatus(apiRefModuleUrl), + fetchURLStatus(apiRefPackageUrl), ]); if (apiRefUrlSuccesses.find((s) => !s)) { console.warn( @@ -149,7 +117,7 @@ export async function fillRetrieverIntegrationDocTemplate(fields: { const docTemplate = (await fs.promises.readFile(TEMPLATE_PATH, "utf-8")) .replaceAll(PACKAGE_NAME_PLACEHOLDER, packageName) - .replaceAll(MODULE_NAME_PLACEHOLDER, fields.moduleName) + .replaceAll(MODULE_NAME_PLACEHOLDER, fields.className) .replaceAll(SIDEBAR_LABEL_PLACEHOLDER, sidebarLabel) .replaceAll(FULL_IMPORT_PATH_PLACEHOLDER, fullImportPath) .replace(HAS_CLOUD_OFFERING_PLACEHOLDER, hasCloudOffering ? "✅" : "❌") @@ -158,7 +126,9 @@ export async function fillRetrieverIntegrationDocTemplate(fields: { .replaceAll(API_REF_MODULE_PLACEHOLDER, apiRefModuleUrl) .replaceAll(PYTHON_DOC_URL_PLACEHOLDER, pyDocUrl); - const packageNameShortSnakeCase = fields.packageName.replace(/-/g, "_"); + const packageNameShortSnakeCase = fields.className + .replace(/-/g, "_") + .toLowerCase(); const docPath = path.join( INTEGRATIONS_DOCS_PATH, `${packageNameShortSnakeCase}.ipynb` diff --git a/libs/langchain-scripts/src/cli/docs/templates/chat.ipynb b/libs/langchain-scripts/src/cli/docs/templates/chat.ipynb index 1f6508d8a861..5a296128d703 100644 --- a/libs/langchain-scripts/src/cli/docs/templates/chat.ipynb +++ b/libs/langchain-scripts/src/cli/docs/templates/chat.ipynb @@ -3,10 +3,14 @@ { "cell_type": "raw", "id": "afaf8039", - "metadata": {}, + "metadata": { + "vscode": { + "languageId": "raw" + } + }, "source": [ "---\n", - "sidebar_label: __ModuleName__\n", + "sidebar_label: __sidebar_label__\n", "---" ] }, @@ -15,18 +19,18 @@ "id": "e49f1e0d", "metadata": {}, "source": [ - "# __ModuleName__\n", + "# __module_name__\n", "\n", - "This will help you getting started with `__ModuleName__` [chat models](/docs/concepts/#chat-models). For detailed documentation of all `__ModuleName__` features and configurations head to the [API reference](https://api.js.langchain.com/classes/__package_name_snake_case__.__ModuleName__.html).\n", + "This will help you getting started with [`__module_name__`](/docs/concepts/#chat-models). For detailed documentation of all `__module_name__` features and configurations head to the [API reference](__api_ref_module__).\n", "\n", "## Overview\n", "### Integration details\n", "\n", "- TODO: Make sure Python integration doc link is correct, if applicable.\n", "\n", - "| Class | Package | Local | Serializable | [PY support](https://python.langchain.com/docs/integrations/chat/__package_name_short_snake_case__) | Package downloads | Package latest |\n", + "| Class | Package | Local | Serializable | [PY support](__python_doc_url__) | Package downloads | Package latest |\n", "| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n", - "| [__ModuleName__](https://api.js.langchain.com/classes/__package_name_snake_case__.__ModuleName__.html) | [__package_name_pretty__](https://api.js.langchain.com/modules/__package_name_snake_case__.html) | __local__ | __serializable__ | __py_support__ | ![NPM - Downloads](https://img.shields.io/npm/dm/__package_name_pretty__?style=flat-square&label=%20&) | ![NPM - Version](https://img.shields.io/npm/v/__package_name_pretty__?style=flat-square&label=%20&) |\n", + "| [__module_name__](__api_ref_module__) | [__package_name__](__api_ref_package__) | __local__ | __serializable__ | __py_support__ | ![NPM - Downloads](https://img.shields.io/npm/dm/__package_name__?style=flat-square&label=%20&) | ![NPM - Version](https://img.shields.io/npm/v/__package_name__?style=flat-square&label=%20&) |\n", "\n", "### Model features\n", "| [Tool calling](/docs/how_to/tool_calling) | [Structured output](/docs/how_to/structured_output/) | JSON mode | [Image input](/docs/how_to/multimodal_inputs/) | Audio input | Video input | [Token-level streaming](/docs/how_to/chat_streaming/) | [Token usage](/docs/how_to/chat_token_usage_tracking/) | [Logprobs](/docs/how_to/logprobs/) |\n", @@ -37,16 +41,16 @@ "\n", "- TODO: Update with relevant info.\n", "\n", - "To access `__ModuleName__` models you'll need to create a/an `__ModuleName__` account, get an API key, and install the `__package_name_pretty__` integration package.\n", + "To access __sidebar_label__ models you'll need to create a/an __sidebar_label__ account, get an API key, and install the `__package_name__` integration package.\n", "\n", "### Credentials\n", "\n", "- TODO: Update with relevant info.\n", "\n", - "Head to (TODO: link) to sign up to `__ModuleName__` and generate an API key. Once you've done this set the `__MODULE_NAME_ALL_CAPS___API_KEY` environment variable:\n", + "Head to (TODO: link) to sign up to __sidebar_label__ and generate an API key. Once you've done this set the `__env_var_name__` environment variable:\n", "\n", "```bash\n", - "export __MODULE_NAME_ALL_CAPS___API_KEY=\"your-api-key\"\n", + "export __env_var_name__=\"your-api-key\"\n", "```\n", "\n", "If you want to get automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:\n", @@ -58,7 +62,7 @@ "\n", "### Installation\n", "\n", - "The LangChain __ModuleName__ integration lives in the `__package_name_pretty__` package:\n", + "The LangChain __module_name__ integration lives in the `__package_name__` package:\n", "\n", "```{=mdx}\n", "import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n", @@ -67,7 +71,7 @@ "\n", "\n", "\n", - " __package_name_pretty__\n", + " __package_name__\n", "\n", "\n", "```" @@ -96,15 +100,15 @@ }, "outputs": [], "source": [ - "import { __ModuleName__ } from \"__import_path__\"\n", - "\n", - "const llm = new __ModuleName__({\n", - " model: \"model-name\",\n", - " temperature: 0,\n", - " maxTokens: undefined,\n", - " timeout: undefined,\n", - " maxRetries: 2,\n", - " // other params...\n", + "import { __module_name__ } from \"__full_import_path__\"\n", + "\n", + "const llm = new __module_name__({\n", + " model: \"model-name\",\n", + " temperature: 0,\n", + " maxTokens: undefined,\n", + " timeout: undefined,\n", + " maxRetries: 2,\n", + " // other params...\n", "})" ] }, @@ -131,11 +135,11 @@ "outputs": [], "source": [ "const aiMsg = await llm.invoke([\n", - " [\n", - " \"system\",\n", - " \"You are a helpful assistant that translates English to French. Translate the user sentence.\",\n", - " ],\n", - " [\"human\", \"I love programming.\"],\n", + " [\n", + " \"system\",\n", + " \"You are a helpful assistant that translates English to French. Translate the user sentence.\",\n", + " ],\n", + " [\"human\", \"I love programming.\"],\n", "])\n", "aiMsg" ] @@ -180,22 +184,22 @@ "import { ChatPromptTemplate } from \"@langchain/core/prompts\"\n", "\n", "const prompt = ChatPromptTemplate.fromMessages(\n", + " [\n", " [\n", - " [\n", - " \"system\",\n", - " \"You are a helpful assistant that translates {input_language} to {output_language}.\",\n", - " ],\n", - " [\"human\", \"{input}\"],\n", - " ]\n", + " \"system\",\n", + " \"You are a helpful assistant that translates {input_language} to {output_language}.\",\n", + " ],\n", + " [\"human\", \"{input}\"],\n", + " ]\n", ")\n", "\n", "const chain = prompt.pipe(llm);\n", "await chain.invoke(\n", - " {\n", - " input_language: \"English\",\n", - " output_language: \"German\",\n", - " input: \"I love programming.\",\n", - " }\n", + " {\n", + " input_language: \"English\",\n", + " output_language: \"German\",\n", + " input: \"I love programming.\",\n", + " }\n", ")" ] }, @@ -216,7 +220,7 @@ "source": [ "## API reference\n", "\n", - "For detailed documentation of all __ModuleName__ features and configurations head to the API reference: https://api.js.langchain.com/classes/__package_name_snake_case__.__ModuleName__.html" + "For detailed documentation of all __module_name__ features and configurations head to the API reference: __api_ref_module__" ] } ], diff --git a/libs/langchain-scripts/src/cli/docs/templates/document_loaders.ipynb b/libs/langchain-scripts/src/cli/docs/templates/document_loaders.ipynb index 09781453dea8..de5c4704ad5f 100644 --- a/libs/langchain-scripts/src/cli/docs/templates/document_loaders.ipynb +++ b/libs/langchain-scripts/src/cli/docs/templates/document_loaders.ipynb @@ -2,10 +2,14 @@ "cells": [ { "cell_type": "raw", - "metadata": {}, + "metadata": { + "vscode": { + "languageId": "raw" + } + }, "source": [ "---\n", - "sidebar_label: __ModuleName__\n", + "sidebar_label: __sidebar_label__\n", "__node_only_sidebar__\n", "---" ] @@ -14,13 +18,13 @@ "cell_type": "markdown", "metadata": {}, "source": [ - "# __ModuleName__\n", + "# __module_name__\n", "\n", "- TODO: Make sure API reference link is correct.\n", "\n", "__node_only_tooltip__\n", "\n", - "This notebook provides a quick overview for getting started with `__ModuleName__` [document loaders](/docs/concepts/#document-loaders). For detailed documentation of all `__ModuleName__` features and configurations head to the [API reference](https://api.js.langchain.com/classes/langchain_community_document_loaders___fs_or_web_____file_name__.__ModuleName__.html).\n", + "This notebook provides a quick overview for getting started with `__module_name__` [document loaders](/docs/concepts/#document-loaders). For detailed documentation of all `__module_name__` features and configurations head to the [API reference](__api_ref_module__).\n", "\n", "- TODO: Add any other relevant links, like information about underlying API, etc.\n", "\n", @@ -31,29 +35,29 @@ "- TODO: Remove JS support link if not relevant, otherwise ensure link is correct.\n", "- TODO: Make sure API reference links are correct.\n", "\n", - "| Class | Package | Compatibility | Local | [PY support](https://python.langchain.com/docs/integrations/document_loaders/__file_name__)| \n", + "| Class | Package | Compatibility | Local | [PY support](__python_doc_url__)| \n", "| :--- | :--- | :---: | :---: | :---: |\n", - "| [__ModuleName__](https://api.js.langchain.com/classes/langchain_community_document_loaders___fs_or_web_____file_name__.__ModuleName__.html) | [@langchain/community](https://api.js.langchain.com/modules/langchain_community_document_loaders___fs_or_web_____file_name__.html) | __fs_support__ | __local__ | __py_support__ |\n", + "| [__module_name__](__api_ref_module__) | [__package_name__](__api_ref_package__) | __fs_support__ | __local__ | __py_support__ |\n", "\n", "## Setup\n", "\n", "- TODO: Update with relevant info.\n", "\n", - "To access `__ModuleName__` document loader you'll need to install the `@langchain/community` integration package, and create a **__ModuleName__** account and get an API key.\n", + "To access the __sidebar_label__ document loader you'll need to install the `__package_name__` integration package, and create a **__sidebar_label__** account and get an API key.\n", "\n", "### Credentials\n", "\n", "- TODO: Update with relevant info.\n", "\n", - "Head to (TODO: link) to sign up to __ModuleName__ and generate an API key. Once you've done this set the `__MODULE_NAME_ALL_CAPS___API_KEY` environment variable:\n", + "Head to (TODO: link) to sign up to __sidebar_label__ and generate an API key. Once you've done this set the `__env_var_name__` environment variable:\n", "\n", "```bash\n", - "export __MODULE_NAME_ALL_CAPS___API_KEY=\"your-api-key\"\n", + "export __env_var_name__=\"your-api-key\"\n", "```\n", "\n", "### Installation\n", "\n", - "The LangChain __ModuleName__ integration lives in the `@langchain/community` package:\n", + "The LangChain __module_name__ integration lives in the `__package_name__` package:\n", "\n", "```{=mdx}\n", "import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n", @@ -62,7 +66,7 @@ "\n", "\n", "\n", - " @langchain/community\n", + " __package_name__\n", "\n", "\n", "```" @@ -89,9 +93,9 @@ }, "outputs": [], "source": [ - "import { __ModuleName__ } from \"@langchain/community/document_loaders/__fs_or_web_import_path__/__file_name__\"\n", + "import { __module_name__ } from \"__full_import_path__\"\n", "\n", - "const loader = new __ModuleName__({\n", + "const loader = new __module_name__({\n", " // required params = ...\n", " // optional params = ...\n", "})" @@ -148,7 +152,7 @@ "source": [ "## API reference\n", "\n", - "For detailed documentation of all __ModuleName__ features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_community_document_loaders___fs_or_web_____file_name__.__ModuleName__.html" + "For detailed documentation of all __module_name__ features and configurations head to the API reference: __api_ref_module__" ] }, { diff --git a/libs/langchain-scripts/src/cli/docs/templates/llms.ipynb b/libs/langchain-scripts/src/cli/docs/templates/llms.ipynb index eac6a24b611d..0fe09cf75048 100644 --- a/libs/langchain-scripts/src/cli/docs/templates/llms.ipynb +++ b/libs/langchain-scripts/src/cli/docs/templates/llms.ipynb @@ -3,10 +3,14 @@ { "cell_type": "raw", "id": "67db2992", - "metadata": {}, + "metadata": { + "vscode": { + "languageId": "raw" + } + }, "source": [ "---\n", - "sidebar_label: __ModuleName__\n", + "sidebar_label: __sidebar_label__\n", "---" ] }, @@ -15,11 +19,11 @@ "id": "9597802c", "metadata": {}, "source": [ - "# __ModuleName__\n", + "# __module_name__\n", "\n", "- [ ] TODO: Make sure API reference link is correct\n", "\n", - "This will help you get started with __ModuleName__ completion models (LLMs) using LangChain. For detailed documentation on `__ModuleName__` features and configuration options, please refer to the [API reference](https://api.js.langchain.com/classes/__package_name_snake_case__.__ModuleName__.html).\n", + "This will help you get started with `__module_name__` completion models (LLMs) using LangChain. For detailed documentation on `__module_name__` features and configuration options, please refer to the [API reference](__api_ref_module__).\n", "\n", "## Overview\n", "### Integration details\n", @@ -28,24 +32,24 @@ "- TODO: Remove JS support link if not relevant, otherwise ensure link is correct.\n", "- TODO: Make sure API reference links are correct.\n", "\n", - "| Class | Package | Local | Serializable | [PY support](https://python.langchain.com/docs/integrations/llms/__package_name_short_snake_case__) | Package downloads | Package latest |\n", + "| Class | Package | Local | Serializable | [PY support](__python_doc_url__) | Package downloads | Package latest |\n", "| :--- | :--- | :---: | :---: | :---: | :---: | :---: |\n", - "| [__ModuleName__](https://api.js.langchain.com/classes/__package_name_snake_case__.__ModuleName__.html) | [__package_name_pretty__](https://api.js.langchain.com/modules/__package_name_snake_case__.html) | __local__ | __serializable__ | __py_support__ | ![NPM - Downloads](https://img.shields.io/npm/dm/__package_name_pretty__?style=flat-square&label=%20&) | ![NPM - Version](https://img.shields.io/npm/v/__package_name_pretty__?style=flat-square&label=%20&) |\n", + "| [__module_name__](__api_ref_module__) | [__package_name__](__api_ref_package__) | __local__ | __serializable__ | __py_support__ | ![NPM - Downloads](https://img.shields.io/npm/dm/__package_name__?style=flat-square&label=%20&) | ![NPM - Version](https://img.shields.io/npm/v/__package_name__?style=flat-square&label=%20&) |\n", "\n", "## Setup\n", "\n", "- [ ] TODO: Update with relevant info.\n", "\n", - "To access __ModuleName__ models you'll need to create a/an __ModuleName__ account, get an API key, and install the `__package_name_pretty__` integration package.\n", + "To access __sidebar_label__ models you'll need to create a/an __sidebar_label__ account, get an API key, and install the `__package_name__` integration package.\n", "\n", "### Credentials\n", "\n", "- TODO: Update with relevant info.\n", "\n", - "Head to (TODO: link) to sign up to __ModuleName__ and generate an API key. Once you've done this set the `__MODULE_NAME_ALL_CAPS___API_KEY` environment variable:\n", + "Head to (TODO: link) to sign up to __sidebar_label__ and generate an API key. Once you've done this set the `__env_var_name__` environment variable:\n", "\n", "```bash\n", - "export __MODULE_NAME_ALL_CAPS___API_KEY=\"your-api-key\"\n", + "export __env_var_name__=\"your-api-key\"\n", "```\n", "\n", "If you want to get automated tracing of your model calls you can also set your [LangSmith](https://docs.smith.langchain.com/) API key by uncommenting below:\n", @@ -57,7 +61,7 @@ "\n", "### Installation\n", "\n", - "The LangChain __ModuleName__ integration lives in the `__package_name_pretty__` package:\n", + "The LangChain __module_name__ integration lives in the `__package_name__` package:\n", "\n", "```{=mdx}\n", "import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n", @@ -66,7 +70,7 @@ "\n", "\n", "\n", - " __package_name_pretty__\n", + " __package_name__\n", "\n", "\n", "```" @@ -95,9 +99,9 @@ }, "outputs": [], "source": [ - "import { __ModuleName__ } from \"__import_path__\"\n", + "import { __module_name__ } from \"__full_import_path__\"\n", "\n", - "const llm = new __ModuleName__({\n", + "const llm = new __module_name__({\n", " model: \"model-name\",\n", " temperature: 0,\n", " maxTokens: undefined,\n", @@ -129,7 +133,7 @@ }, "outputs": [], "source": [ - "const inputText = \"__ModuleName__ is an AI company that \"\n", + "const inputText = \"__module_name__ is an AI company that \"\n", "\n", "const completion = await llm.invoke(inputText)\n", "completion" @@ -188,7 +192,7 @@ "source": [ "## API reference\n", "\n", - "For detailed documentation of all __ModuleName__ features and configurations head to the API reference: https://api.js.langchain.com/classes/__package_name_snake_case__.__ModuleName__.html" + "For detailed documentation of all `__module_name__` features and configurations head to the API reference: __api_ref_module__" ] } ], diff --git a/libs/langchain-scripts/src/cli/docs/templates/retrievers.ipynb b/libs/langchain-scripts/src/cli/docs/templates/retrievers.ipynb index 236fec3abfc4..3ea8a62d5d21 100644 --- a/libs/langchain-scripts/src/cli/docs/templates/retrievers.ipynb +++ b/libs/langchain-scripts/src/cli/docs/templates/retrievers.ipynb @@ -19,13 +19,13 @@ "id": "e49f1e0d", "metadata": {}, "source": [ - "# __ModuleName__\n", + "# __module_name__\n", "\n", "## Overview\n", "\n", "- TODO: Make sure API reference link is correct.\n", "\n", - "This will help you getting started with the [__ModuleName__](/docs/concepts/#retrievers). For detailed documentation of all __ModuleName__ features and configurations head to the [API reference](__api_ref_module__).\n", + "This will help you getting started with the [__module_name__](/docs/concepts/#retrievers). For detailed documentation of all __module_name__ features and configurations head to the [API reference](__api_ref_module__).\n", "\n", "### Integration details\n", "\n", @@ -35,13 +35,13 @@ "\n", "| Retriever | Self-host | Cloud offering | Package | [Py support](__python_doc_url__) |\n", "| :--- | :--- | :---: | :---: | :---: |\n", - "[__ModuleName__](__api_ref_module__) | __can_self_host__ | __has_cloud_offering__ | __package_name__ | __py_support__ |\n", + "[__module_name__](__api_ref_module__) | __can_self_host__ | __has_cloud_offering__ | __package_name__ | __py_support__ |\n", "\n", "2: External index (e.g., constructed from Internet data or similar):\n", "\n", "| Retriever | Source | Package |\n", "| :--- | :--- | :---: |\n", - "[__ModuleName__](__api_ref_module__) | Source description | __package_name__ |\n", + "[__module_name__](__api_ref_module__) | Source description | __package_name__ |\n", "\n", "## Setup\n", "\n", @@ -89,9 +89,9 @@ "metadata": {}, "outputs": [], "source": [ - "import { __ModuleName__ } from \"__full_import_path__\";\n", + "import { __module_name__ } from \"__full_import_path__\";\n", "\n", - "const retriever = new __ModuleName__(\n", + "const retriever = new __module_name__(\n", " // ...\n", ");" ] @@ -123,7 +123,7 @@ "source": [ "## Use within a chain\n", "\n", - "Like other retrievers, __ModuleName__ can be incorporated into LLM applications via [chains](/docs/how_to/sequence/).\n", + "Like other retrievers, __module_name__ can be incorporated into LLM applications via [chains](/docs/how_to/sequence/).\n", "\n", "We will need a LLM or chat model:\n", "\n", @@ -214,7 +214,7 @@ "source": [ "## API reference\n", "\n", - "For detailed documentation of all __ModuleName__ features and configurations head to the [API reference](__api_ref_module__)." + "For detailed documentation of all __module_name__ features and configurations head to the [API reference](__api_ref_module__)." ] } ], diff --git a/libs/langchain-scripts/src/cli/docs/templates/text_embedding.ipynb b/libs/langchain-scripts/src/cli/docs/templates/text_embedding.ipynb index 48d85f8a3afc..40fb944453bf 100644 --- a/libs/langchain-scripts/src/cli/docs/templates/text_embedding.ipynb +++ b/libs/langchain-scripts/src/cli/docs/templates/text_embedding.ipynb @@ -19,11 +19,11 @@ "id": "9a3d6f34", "metadata": {}, "source": [ - "# __ModuleName__\n", + "# __module_name__\n", "\n", "- [ ] TODO: Make sure API reference link is correct\n", "\n", - "This will help you get started with __ModuleName__ [embedding models](/docs/concepts#embedding-models) using LangChain. For detailed documentation on `__ModuleName__` features and configuration options, please refer to the [API reference](__api_ref_module__).\n", + "This will help you get started with __module_name__ [embedding models](/docs/concepts#embedding-models) using LangChain. For detailed documentation on `__module_name__` features and configuration options, please refer to the [API reference](__api_ref_module__).\n", "\n", "## Overview\n", "### Integration details\n", @@ -34,13 +34,13 @@ "\n", "| Class | Package | Local | [Py support](__python_doc_url__) | Package downloads | Package latest |\n", "| :--- | :--- | :---: | :---: | :---: | :---: |\n", - "| [__ModuleName__](__api_ref_module__) | [__package_name__](__api_ref_package__) | __local__ | __py_support__ | ![NPM - Downloads](https://img.shields.io/npm/dm/__package_name__?style=flat-square&label=%20&) | ![NPM - Version](https://img.shields.io/npm/v/__package_name__?style=flat-square&label=%20&) |\n", + "| [__module_name__](__api_ref_module__) | [__package_name__](__api_ref_package__) | __local__ | __py_support__ | ![NPM - Downloads](https://img.shields.io/npm/dm/__package_name__?style=flat-square&label=%20&) | ![NPM - Version](https://img.shields.io/npm/v/__package_name__?style=flat-square&label=%20&) |\n", "\n", "## Setup\n", "\n", "- [ ] TODO: Update with relevant info.\n", "\n", - "To access __sidebar_label__ embedding models you'll need to create a/an __ModuleName__ account, get an API key, and install the `__package_name__` integration package.\n", + "To access __sidebar_label__ embedding models you'll need to create a/an __sidebar_label__ account, get an API key, and install the `__package_name__` integration package.\n", "\n", "### Credentials\n", "\n", @@ -61,7 +61,7 @@ "\n", "### Installation\n", "\n", - "The LangChain __ModuleName__ integration lives in the `__package_name__` package:\n", + "The LangChain __module_name__ integration lives in the `__package_name__` package:\n", "\n", "```{=mdx}\n", "import IntegrationInstallTooltip from \"@mdx_components/integration_install_tooltip.mdx\";\n", @@ -94,9 +94,9 @@ "metadata": {}, "outputs": [], "source": [ - "import { __ModuleName__ } from \"__full_import_path__\";\n", + "import { __module_name__ } from \"__full_import_path__\";\n", "\n", - "const embeddings = new __ModuleName__({\n", + "const embeddings = new __module_name__({\n", " model: \"model-name\",\n", " // ...\n", "});" @@ -135,9 +135,9 @@ "const retriever = vectorstore.asRetriever(1);\n", "\n", "// Retrieve the most similar text\n", - "const retrievedDocument = await retriever.invoke(\"What is LangChain?\");\n", + "const retrievedDocuments = await retriever.invoke(\"What is LangChain?\");\n", "\n", - "retrievedDocument.pageContent;" + "retrievedDocuments[0].pageContent;" ] }, { @@ -200,7 +200,7 @@ "source": [ "## API reference\n", "\n", - "For detailed documentation of all __ModuleName__ features and configurations head to the API reference: __api_ref_module__" + "For detailed documentation of all __module_name__ features and configurations head to the API reference: __api_ref_module__" ] } ], diff --git a/libs/langchain-scripts/src/cli/utils/fetch-url-status.ts b/libs/langchain-scripts/src/cli/utils/fetch-url-status.ts new file mode 100644 index 000000000000..afadde4337fe --- /dev/null +++ b/libs/langchain-scripts/src/cli/utils/fetch-url-status.ts @@ -0,0 +1,11 @@ +export const fetchURLStatus = async (url: string): Promise => { + try { + const res = await fetch(url); + if (res.status !== 200) { + throw new Error(`API Reference URL ${url} not found.`); + } + return true; + } catch (_) { + return false; + } +};