diff --git a/libs/checkpoint-validation/README.md b/libs/checkpoint-validation/README.md index 0dc841faa..d0e9a8b1b 100644 --- a/libs/checkpoint-validation/README.md +++ b/libs/checkpoint-validation/README.md @@ -1,38 +1,38 @@ # @langchain/langgraph-checkpoint-validation -The checkpoint saver validation tool is used to validate that custom checkpoint saver implementations conform to LangGraph's requirements. LangGraph uses [checkpoint savers](https://langchain-ai.github.io/langgraphjs/concepts/persistence/#checkpointer-libraries) for persisting workflow state, providing the ability to "rewind" your workflow to some earlier point in time, and continue execution from there. +The checkpointer validation tool is used to validate that custom checkpointer implementations conform to LangGraph's requirements. LangGraph uses [checkpointers](https://langchain-ai.github.io/langgraphjs/concepts/persistence/#checkpointer-libraries) for persisting workflow state, providing the ability to "rewind" your workflow to some earlier point in time, and continue execution from there. The overall process for using this tool is as follows: -1. Write your custom checkpoint saver implementation. -2. Add a file to your project that defines a [`CheckpointSaverTestInitializer`](./src/types.ts) as its default export. -3. Run the checkpoint saver validation tool to test your checkpoint saver and determine whether it meets LangGraph's requirements. -4. Iterate on your custom checkpoint saver as required, until tests pass. +1. Write your custom checkpointer implementation. +2. Add a file to your project that defines a [`CheckpointerTestInitializer`](./src/types.ts) as its default export. +3. Run the checkpointer validation tool to test your checkpointer and determine whether it meets LangGraph's requirements. +4. Iterate on your custom checkpointer as required, until tests pass. The tool can be executed from the terminal as a CLI, or you can use it as a library to integrate it into your test suite. -## Writing a CheckpointSaverTestInitializer +## Writing a CheckpointerTestInitializer -The `CheckpointSaverTestInitializer` interface ([example](./src/tests/postgresInitializer.ts)) is used by the test harness to create instances of your custom checkpoint saver, and any infrastructure that it requires for testing purposes. +The `CheckpointerTestInitializer` interface ([example](./src/tests/postgres_initializer.ts)) is used by the test harness to create instances of your custom checkpointer, and any infrastructure that it requires for testing purposes. -If you intend to execute the tool via the CLI, your `CheckpointSaverTestInitializer` **must** be the default export of the module in which it is defined. +If you intend to execute the tool via the CLI, your `CheckpointerTestInitializer` **must** be the default export of the module in which it is defined. -**Synchronous vs Asynchronous initializer functions**: You may return promises from any functions defined in your `CheckpointSaverTestInitializer` according to your needs and the test harness will behave accordingly. +**Synchronous vs Asynchronous initializer functions**: You may return promises from any functions defined in your `CheckpointerTestInitializer` according to your needs and the test harness will behave accordingly. -**IMPORTANT**: You must take care to write your `CheckpointSaverTestInitializer` such that instances of your custom checkpoint saver are isolated from one another with respect to persisted state, or else some tests (particularly the ones that exercise the `list` method) will fail. That is, state written by one instance of your checkpoint saver MUST NOT be readable by another instance of your checkpoint saver. That said, there will only ever be one instance of your checkpoint saver live at any given time, so **you may use shared storage, provided it is cleared when your checkpoint saver is created or destroyed.** The structure of the `CheckpointSaverTestInitializer` interface should make this relatively easy to achieve, per the sections below. +**IMPORTANT**: You must take care to write your `CheckpointerTestInitializer` such that instances of your custom checkpointer are isolated from one another with respect to persisted state, or else some tests (particularly the ones that exercise the `list` method) will fail. That is, state written by one instance of your checkpointer MUST NOT be readable by another instance of your checkpointer. That said, there will only ever be one instance of your checkpointer live at any given time, so **you may use shared storage, provided it is cleared when your checkpointer is created or destroyed.** The structure of the `CheckpointerTestInitializer` interface should make this relatively easy to achieve, per the sections below. -### (Required) `saverName`: Define a name for your checkpoint saver +### (Required) `checkpointerName`: Define a name for your checkpointer -`CheckpointSaverTestInitializer` requires you to define a `saverName` field (of type `string`) for use in the test output. +`CheckpointerTestInitializer` requires you to define a `checkpointerName` field (of type `string`) for use in the test output. ### `beforeAll`: Set up required infrastructure -If your checkpoint saver requires some external infrastructure to be provisioned, you may wish to provision this via the **optional** `beforeAll` function. This function executes exactly once, at the very start of the testing lifecycle. If defined, it is the first function that will be called from your `CheckpointSaverTestInitializer`. +If your checkpointer requires some external infrastructure to be provisioned, you may wish to provision this via the **optional** `beforeAll` function. This function executes exactly once, at the very start of the testing lifecycle. If defined, it is the first function that will be called from your `CheckpointerTestInitializer`. -**Timeout duration**: If your `beforeAll` function may take longer than 10 seconds to execute, you can assign a custom timeout duration (as milliseconds) to the optional `beforeAllTimeout` field of your `CheckpointSaverTestInitializer`. +**Timeout duration**: If your `beforeAll` function may take longer than 10 seconds to execute, you can assign a custom timeout duration (as milliseconds) to the optional `beforeAllTimeout` field of your `CheckpointerTestInitializer`. -**State isolation note**: Depending on the cost/performance/requirements of your checkpoint saver infrastructure, it **may** make more sense for you to provision it during the `createSaver` step, so you can provide each checkpoint saver instance with its own isolated storage backend. However as mentioned above, you may also provision a single shared storage backend, provided you clear any stored data during the `createSaver` or `destroySaver` step. +**State isolation note**: Depending on the cost/performance/requirements of your checkpointer infrastructure, it **may** make more sense for you to provision it during the `createCheckpointer` step, so you can provide each checkpointer instance with its own isolated storage backend. However as mentioned above, you may also provision a single shared storage backend, provided you clear any stored data during the `createCheckpointer` or `destroyCheckpointer` step. ### `afterAll`: Tear down required infrastructure @@ -40,31 +40,25 @@ If you set up infrastructure during the `beforeAll` step, you may need to tear i **IMPORTANT**: If you kill the test runner early this function may not be called. To avoid manual clean-up, give preference to test infrastructure management tools like [TestContainers](https://testcontainers.com/guides/getting-started-with-testcontainers-for-nodejs/), as these tools are designed to detect when this happens and clean up after themselves once the controlling process dies. -### (Required) `createSaver`: Construct your checkpoint saver +### (Required) `createCheckpointer`: Construct your checkpointer -`CheckpointSaverTestInitializer` requires you to define a `createSaver(config: RunnableConfig)` function that returns an instance of your custom checkpoint saver. The `config` argument is provided to this function in case it is necessary for the construction of your custom checkpoint saver. +`CheckpointerTestInitializer` requires you to define a `createCheckpointer()` function that returns an instance of your custom checkpointer. -**State isolation note:** If you're provisioning storage during this step, make sure that it is "fresh" storage for each instance of your checkpoint saver. Otherwise if you are using a shared storage setup, be sure to clear it either in this function, or in the `destroySaver` function (described in the section below). +**State isolation note:** If you're provisioning storage during this step, make sure that it is "fresh" storage for each instance of your checkpointer. Otherwise if you are using a shared storage setup, be sure to clear it either in this function, or in the `destroyCheckpointer` function (described in the section below). -### `destroySaver`: Destroy your checkpoint saver +### `destroyCheckpointer`: Destroy your checkpointer -If your custom checkpoint saver requires an explicit teardown step (for example, to clean up database connections), you can define this in the **optional** `destroySaver(saver: CheckpointSaverT, config: RunnableConfig)` function. +If your custom checkpointer requires an explicit teardown step (for example, to clean up database connections), you can define this in the **optional** `destroyCheckpointer(checkpointer: CheckpointerT)` function. -**State isolation note:** If you are using a shared storage setup, be sure to clear it either in this function, or in the `createSaver` function (described in the section above). - -### `configure`: Customize the `RunnableConfig` object that is passed during testing - -If you need to customize the config argument that is passed to your custom checkpoint saver during testing, you can implement the **optional** `configure(config: RunnableConfig)` function. This function may inspect the default configuration (passed as the `config` argument) and must return an instance of `RunnableConfig`. The `RunnableConfig` returned by this function will be merged with the default config and passed to your checkpoint saver during testing. - -Some custom checkpoint savers may require additional custom configuration data to be present in the `configurable` object of the `RunnableConfig` in order to work correctly. For example, custom checkpoint savers that work as part of a multi-tenant application may require authentication details to be passed along in order to enforce data isolation requirements. +**State isolation note:** If you are using a shared storage setup, be sure to clear it either in this function, or in the `createCheckpointer` function (described in the section above). ## CLI usage -You may use this tool's CLI either via `npx`, `yarn dlx`, or by installing globally and executing it via the `validate-saver` command. +You may use this tool's CLI either via `npx`, `yarn dlx`, or by installing globally and executing it via the `validate-checkpointer` command. -The only required argument to the tool is the import path for your `CheckpointSaverTestInitializer`. Relative paths must begin with a leading `./` (or `.\`, for Windows), otherwise the path will be interpreted as a module name rather than a relative path. +The only required argument to the tool is the import path for your `CheckpointerTestInitializer`. Relative paths must begin with a leading `./` (or `.\`, for Windows), otherwise the path will be interpreted as a module name rather than a relative path. -You may optionally pass one or more test filters as positional arguments after the import path argument (separated by spaces). Valid values are `getTuple`, `list`, `put`, and `putWrites`. If present, only the test suites specified in the filter list will be executed. This is useful for working through smaller sets of test failures as you're validating your checkpoint saver. +You may optionally pass one or more test filters as positional arguments after the import path argument (separated by spaces). Valid values are `getTuple`, `list`, `put`, and `putWrites`. If present, only the test suites specified in the filter list will be executed. This is useful for working through smaller sets of test failures as you're validating your checkpointer. TypeScript imports **are** supported, so you may pass a path directly to your TypeScript source file. @@ -73,14 +67,13 @@ TypeScript imports **are** supported, so you may pass a path directly to your Ty NPX: ```bash -cd MySaverProject -npx @langchain/langgraph-checkpoint-validation ./src/mySaverInitializer.ts +npx @langchain/langgraph-checkpoint-validation ./src/my_initializer.ts ``` Yarn: ```bash -yarn dlx @langchain/langgraph-checkpoint-validation ./src/mySaverInitializer.ts +yarn dlx @langchain/langgraph-checkpoint-validation ./src/my_initializer.ts ``` ### Global install @@ -89,7 +82,7 @@ NPM: ```bash npm install -g @langchain/langgraph-checkpoint-validation -validate-saver ./src/mySaverInitializer.ts +validate-checkpointer ./src/my_initializer.ts ``` ## Usage in existing Jest test suite @@ -99,5 +92,5 @@ If you wish to integrate this tooling into your existing Jest test suite, you im ```ts import { validate } from "@langchain/langgraph-validation"; -validate(MyCheckpointSaverInitializer); +validate(MyCheckpointerInitializer); ``` diff --git a/libs/checkpoint-validation/bin/jest.config.cjs b/libs/checkpoint-validation/bin/jest.config.cjs index a99445d3d..98e95f92b 100644 --- a/libs/checkpoint-validation/bin/jest.config.cjs +++ b/libs/checkpoint-validation/bin/jest.config.cjs @@ -2,6 +2,7 @@ // For the Jest config for the tests in this project, see the `jest.config.cjs` in the root of the package workspace. const path = require("path"); +/** @type {import('ts-jest').JestConfigWithTsJest} */ const config = { preset: "ts-jest/presets/default-esm", rootDir: path.resolve(__dirname, "..", "dist"), diff --git a/libs/checkpoint-validation/package.json b/libs/checkpoint-validation/package.json index d87e01bda..2d20e22ed 100644 --- a/libs/checkpoint-validation/package.json +++ b/libs/checkpoint-validation/package.json @@ -95,7 +95,7 @@ "./package.json": "./package.json" }, "bin": { - "validate-saver": "./bin/cli.js" + "validate-checkpointer": "./bin/cli.js" }, "files": [ "dist/", diff --git a/libs/checkpoint-validation/src/cli.ts b/libs/checkpoint-validation/src/cli.ts index b65f17600..299d669b4 100644 --- a/libs/checkpoint-validation/src/cli.ts +++ b/libs/checkpoint-validation/src/cli.ts @@ -4,12 +4,12 @@ import { runCLI } from "@jest/core"; import yargs, { ArgumentsCamelCase } from "yargs"; import { BaseCheckpointSaver } from "@langchain/langgraph-checkpoint"; import { - CheckpointSaverTestInitializer, - checkpointSaverTestInitializerSchema, + CheckpointerTestInitializer, + checkpointerTestInitializerSchema, GlobalThis, TestTypeFilter, } from "./types.js"; -import { dynamicImport, resolveImportPath } from "./importUtils.js"; +import { dynamicImport, resolveImportPath } from "./import_utils.js"; // make it so we can import/require .ts files import "@swc-node/register/esm-register"; @@ -21,14 +21,14 @@ export async function main() { await builder .command( "* [filters..]", - "Validate a checkpoint saver", + "Validate a checkpointer", { builder: (args) => { return args .positional("initializerImportPath", { type: "string", describe: - "The import path of the CheckpointSaverTestInitializer for the checkpoint saver (passed to 'import()'). " + + "The import path of the CheckpointSaverTestInitializer for the checkpointer (passed to 'import()'). " + "Must be the default export.", demandOption: true, }) @@ -70,9 +70,9 @@ export async function main() { process.exit(1); } - let initializer: CheckpointSaverTestInitializer; + let initializer: CheckpointerTestInitializer; try { - initializer = checkpointSaverTestInitializerSchema.parse( + initializer = checkpointerTestInitializerSchema.parse( (initializerExport as { default?: unknown }).default ?? initializerExport ); diff --git a/libs/checkpoint-validation/src/importUtils.ts b/libs/checkpoint-validation/src/import_utils.ts similarity index 100% rename from libs/checkpoint-validation/src/importUtils.ts rename to libs/checkpoint-validation/src/import_utils.ts diff --git a/libs/checkpoint-validation/src/index.ts b/libs/checkpoint-validation/src/index.ts index 0b5729efe..538cf8a97 100644 --- a/libs/checkpoint-validation/src/index.ts +++ b/libs/checkpoint-validation/src/index.ts @@ -1,8 +1,8 @@ import type { BaseCheckpointSaver } from "@langchain/langgraph-checkpoint"; import { specTest } from "./spec/index.js"; -import type { CheckpointSaverTestInitializer } from "./types.js"; +import type { CheckpointerTestInitializer } from "./types.js"; -export { CheckpointSaverTestInitializer } from "./types.js"; +export { CheckpointerTestInitializer as CheckpointSaverTestInitializer } from "./types.js"; export { getTupleTests, listTests, @@ -12,7 +12,7 @@ export { } from "./spec/index.js"; export function validate( - initializer: CheckpointSaverTestInitializer + initializer: CheckpointerTestInitializer ) { specTest(initializer); } diff --git a/libs/checkpoint-validation/src/runner.ts b/libs/checkpoint-validation/src/runner.ts index ad5a022ce..908e18e78 100644 --- a/libs/checkpoint-validation/src/runner.ts +++ b/libs/checkpoint-validation/src/runner.ts @@ -1,6 +1,6 @@ -// This file is used by the CLI to dynamically execute tests against the user-provided checkpoint saver. It's written -// as a Jest test file because unfortunately there's no good way to just pass Jest a test definition function and tell -// it to run it. +// This file is used by the CLI to dynamically execute tests against the user-provided checkpointer. It's written as a +// Jest test file because unfortunately there's no good way to just pass Jest a test definition function and tell it to +// run it. import { specTest } from "./spec/index.js"; import type { GlobalThis } from "./types.js"; diff --git a/libs/checkpoint-validation/src/spec/getTuple.ts b/libs/checkpoint-validation/src/spec/get_tuple.ts similarity index 77% rename from libs/checkpoint-validation/src/spec/getTuple.ts rename to libs/checkpoint-validation/src/spec/get_tuple.ts index a31c16470..be06f54bb 100644 --- a/libs/checkpoint-validation/src/spec/getTuple.ts +++ b/libs/checkpoint-validation/src/spec/get_tuple.ts @@ -5,30 +5,23 @@ import { uuid6, type BaseCheckpointSaver, } from "@langchain/langgraph-checkpoint"; -import { mergeConfigs, RunnableConfig } from "@langchain/core/runnables"; -import { CheckpointSaverTestInitializer } from "../types.js"; -import { parentAndChildCheckpointTuplesWithWrites } from "./data.js"; -import { putTuples } from "./util.js"; +import { CheckpointerTestInitializer } from "../types.js"; +import { + parentAndChildCheckpointTuplesWithWrites, + putTuples, +} from "../test_utils.js"; export function getTupleTests( - initializer: CheckpointSaverTestInitializer + initializer: CheckpointerTestInitializer ) { - describe(`${initializer.saverName}#getTuple`, () => { - let saver: T; - let initializerConfig: RunnableConfig; + describe(`${initializer.checkpointerName}#getTuple`, () => { + let checkpointer: T; beforeAll(async () => { - const baseConfig = { - configurable: {}, - }; - initializerConfig = mergeConfigs( - baseConfig, - await initializer.configure?.(baseConfig) - ); - saver = await initializer.createSaver(initializerConfig); + checkpointer = await initializer.createCheckpointer(); }); afterAll(async () => { - await initializer.destroySaver?.(saver, initializerConfig); + await initializer.destroyCheckpointer?.(checkpointer); }); describe.each(["root", "child"])("namespace: %s", (namespace) => { @@ -50,10 +43,6 @@ export function getTupleTests( parentCheckpointId = uuid6(-3); childCheckpointId = uuid6(-3); - const config = mergeConfigs(initializerConfig, { - configurable: { thread_id, checkpoint_ns }, - }); - const writesToParent = [ { taskId: "pending_sends_task", @@ -70,7 +59,7 @@ export function getTupleTests( ({ parent: generatedParentTuple, child: generatedChildTuple } = parentAndChildCheckpointTuplesWithWrites({ - config, + thread_id, parentCheckpointId, childCheckpointId, checkpoint_ns, @@ -81,31 +70,25 @@ export function getTupleTests( writesToChild, })); - const storedTuples = putTuples( - saver, - [ - { - tuple: generatedParentTuple, - writes: writesToParent, - newVersions: { animals: 1 }, - }, - { - tuple: generatedChildTuple, - writes: writesToChild, - newVersions: { animals: 2 }, - }, - ], - config - ); + const storedTuples = putTuples(checkpointer, [ + { + tuple: generatedParentTuple, + writes: writesToParent, + newVersions: { animals: 1 }, + }, + { + tuple: generatedChildTuple, + writes: writesToChild, + newVersions: { animals: 2 }, + }, + ]); parentTuple = (await storedTuples.next()).value; childTuple = (await storedTuples.next()).value; - latestTuple = await saver.getTuple( - mergeConfigs(config, { - configurable: { checkpoint_ns, checkpoint_id: undefined }, - }) - ); + latestTuple = await checkpointer.getTuple({ + configurable: { thread_id, checkpoint_ns }, + }); }); describe("success cases", () => { @@ -236,33 +219,29 @@ export function getTupleTests( describe("failure cases", () => { it("should return undefined if the checkpoint_id is not found", async () => { - const configWithInvalidCheckpointId = mergeConfigs( - initializerConfig, - { - configurable: { - thread_id: uuid6(-3), - checkpoint_ns, - checkpoint_id: uuid6(-3), - }, - } - ); - const checkpointTuple = await saver.getTuple( + const configWithInvalidCheckpointId = { + configurable: { + thread_id: uuid6(-3), + checkpoint_ns, + checkpoint_id: uuid6(-3), + }, + }; + const checkpointTuple = await checkpointer.getTuple( configWithInvalidCheckpointId ); expect(checkpointTuple).toBeUndefined(); }); it("should return undefined if the thread_id is undefined", async () => { - const missingThreadIdConfig: RunnableConfig = { - ...initializerConfig, - configurable: Object.fromEntries( - Object.entries(initializerConfig.configurable ?? {}).filter( - ([key]) => key !== "thread_id" - ) - ), + const missingThreadIdConfig = { + configurable: { + checkpoint_ns, + }, }; - expect(await saver.getTuple(missingThreadIdConfig)).toBeUndefined(); + expect( + await checkpointer.getTuple(missingThreadIdConfig) + ).toBeUndefined(); }); }); }); diff --git a/libs/checkpoint-validation/src/spec/index.ts b/libs/checkpoint-validation/src/spec/index.ts index ccce443d3..646408f3d 100644 --- a/libs/checkpoint-validation/src/spec/index.ts +++ b/libs/checkpoint-validation/src/spec/index.ts @@ -1,9 +1,9 @@ import { type BaseCheckpointSaver } from "@langchain/langgraph-checkpoint"; -import { CheckpointSaverTestInitializer, TestTypeFilter } from "../types.js"; +import { CheckpointerTestInitializer, TestTypeFilter } from "../types.js"; import { putTests } from "./put.js"; -import { putWritesTests } from "./putWrites.js"; -import { getTupleTests } from "./getTuple.js"; +import { putWritesTests } from "./put_writes.js"; +import { getTupleTests } from "./get_tuple.js"; import { listTests } from "./list.js"; const testTypeMap = { @@ -14,12 +14,15 @@ const testTypeMap = { }; /** - * Kicks off a test suite to validate that the provided checkpoint saver conforms to the specification for classes that extend @see BaseCheckpointSaver. - * @param initializer A @see CheckpointSaverTestInitializer, providing methods for setup and cleanup of the test, and for creation of the saver instance being tested. + * Kicks off a test suite to validate that the provided checkpointer conforms to the specification for classes that + * extend @see BaseCheckpointSaver. + * + * @param initializer A @see CheckpointerTestInitializer, providing methods for setup and cleanup of the test, + * and for creation of the checkpointer instance being tested. * @param filters If specified, only the test suites in this list will be executed. */ export function specTest( - initializer: CheckpointSaverTestInitializer, + initializer: CheckpointerTestInitializer, filters?: TestTypeFilter[] ) { beforeAll(async () => { @@ -30,7 +33,7 @@ export function specTest( await initializer.afterAll?.(); }); - describe(initializer.saverName, () => { + describe(initializer.checkpointerName, () => { if (!filters || filters.length === 0) { putTests(initializer); putWritesTests(initializer); diff --git a/libs/checkpoint-validation/src/spec/list.ts b/libs/checkpoint-validation/src/spec/list.ts index a38992d6f..d9f630971 100644 --- a/libs/checkpoint-validation/src/spec/list.ts +++ b/libs/checkpoint-validation/src/spec/list.ts @@ -4,10 +4,14 @@ import { uuid6, type BaseCheckpointSaver, } from "@langchain/langgraph-checkpoint"; -import { mergeConfigs, RunnableConfig } from "@langchain/core/runnables"; -import { CheckpointSaverTestInitializer } from "../types.js"; -import { generateTuplePairs } from "./data.js"; -import { putTuples, toArray, toMap } from "./util.js"; +import { RunnableConfig } from "@langchain/core/runnables"; +import { CheckpointerTestInitializer } from "../types.js"; +import { + generateTuplePairs, + putTuples, + toArray, + toMap, +} from "../test_utils.js"; interface ListTestCase { description: string; @@ -20,15 +24,15 @@ interface ListTestCase { } /** - * Exercises the `list` method of the CheckpointSaver. + * Exercises the `list` method of the checkpointer. * - * IMPORTANT NOTE: This test relies on the `getTuple` method of the saver functioning properly. If you have failures in - * `getTuple`, you should fix them before addressing the failures in this test. + * IMPORTANT NOTE: This test relies on the `getTuple` method of the checkpointer functioning properly. If you have + * failures in `getTuple`, you should fix them before addressing the failures in this test. * - * @param initializer the initializer for the CheckpointSaver + * @param initializer the initializer for the checkpointer */ export function listTests( - initializer: CheckpointSaverTestInitializer + initializer: CheckpointerTestInitializer ) { const invalidThreadId = uuid6(-3); @@ -38,7 +42,7 @@ export function listTests( tuple: CheckpointTuple; writes: { writes: PendingWrite[]; taskId: string }[]; newVersions: Record; - }[] = Array.from(generateTuplePairs({ configurable: {} }, 2, namespaces)); + }[] = Array.from(generateTuplePairs(2, namespaces)); const argumentRanges = setupArgumentRanges( generatedTuples.map(({ tuple }) => tuple), @@ -52,34 +56,21 @@ export function listTests( ) ); - describe(`${initializer.saverName}#list`, () => { - let saver: T; - let initializerConfig: RunnableConfig; - + describe(`${initializer.checkpointerName}#list`, () => { + let checkpointer: T; const storedTuples: Map = new Map(); beforeAll(async () => { - const baseConfig = { - configurable: {}, - }; - initializerConfig = mergeConfigs( - baseConfig, - await initializer.configure?.(baseConfig) - ); - saver = await initializer.createSaver(initializerConfig); + checkpointer = await initializer.createCheckpointer(); // put all the tuples - for await (const tuple of putTuples( - saver, - generatedTuples, - initializerConfig - )) { + for await (const tuple of putTuples(checkpointer, generatedTuples)) { storedTuples.set(tuple.checkpoint.id, tuple); } }); afterAll(async () => { - await initializer.destroySaver?.(saver, initializerConfig); + await initializer.destroyCheckpointer?.(checkpointer); }); // can't reference argumentCombinations directly here because it isn't built at the time this is evaluated. @@ -95,7 +86,7 @@ export function listTests( expectedCheckpointIds, }: ListTestCase) => { const actualTuplesArray = await toArray( - saver.list( + checkpointer.list( { configurable: { thread_id, checkpoint_ns } }, { limit, before, filter } ) @@ -128,14 +119,14 @@ export function listTests( // TODO: MongoDBSaver and SQLiteSaver don't return pendingWrites on list, so we need to special case them // see: https://github.com/langchain-ai/langgraphjs/issues/589 // see: https://github.com/langchain-ai/langgraphjs/issues/590 - const saverIncludesPendingWritesOnList = - initializer.saverName !== + const checkpointerIncludesPendingWritesOnList = + initializer.checkpointerName !== "@langchain/langgraph-checkpoint-mongodb" && - initializer.saverName !== + initializer.checkpointerName !== "@langchain/langgraph-checkpoint-sqlite"; const expectedTuple = expectedTuplesMap.get(key); - if (!saverIncludesPendingWritesOnList) { + if (!checkpointerIncludesPendingWritesOnList) { delete expectedTuple?.pendingWrites; } @@ -180,7 +171,8 @@ export function listTests( filter: // TODO: MongoDBSaver support for filter is broken and can't be fixed without a breaking change // see: https://github.com/langchain-ai/langgraphjs/issues/581 - initializer.saverName === "@langchain/langgraph-checkpoint-mongodb" + initializer.checkpointerName === + "@langchain/langgraph-checkpoint-mongodb" ? [undefined] : [undefined, {}, { source: "input" }, { source: "loop" }], }; diff --git a/libs/checkpoint-validation/src/spec/put.ts b/libs/checkpoint-validation/src/spec/put.ts index d2c16bfde..5ecc343dc 100644 --- a/libs/checkpoint-validation/src/spec/put.ts +++ b/libs/checkpoint-validation/src/spec/put.ts @@ -5,45 +5,34 @@ import { uuid6, type BaseCheckpointSaver, } from "@langchain/langgraph-checkpoint"; -import { mergeConfigs, RunnableConfig } from "@langchain/core/runnables"; -import { CheckpointSaverTestInitializer } from "../types.js"; -import { initialCheckpointTuple } from "./data.js"; -import { putTuples } from "./util.js"; -import { it_skipForSomeModules } from "../testUtils.js"; +import { RunnableConfig } from "@langchain/core/runnables"; +import { CheckpointerTestInitializer } from "../types.js"; +import { + initialCheckpointTuple, + it_skipForSomeModules, + putTuples, +} from "../test_utils.js"; export function putTests( - initializer: CheckpointSaverTestInitializer + initializer: CheckpointerTestInitializer ) { - describe(`${initializer.saverName}#put`, () => { - let saver: T; - let initializerConfig: RunnableConfig; + describe(`${initializer.checkpointerName}#put`, () => { + let checkpointer: T; let thread_id: string; let checkpoint_id1: string; beforeEach(async () => { thread_id = uuid6(-3); checkpoint_id1 = uuid6(-3); - - const baseConfig = { - configurable: { - thread_id, - }, - }; - - initializerConfig = mergeConfigs( - baseConfig, - await initializer.configure?.(baseConfig) - ); - saver = await initializer.createSaver(initializerConfig); + checkpointer = await initializer.createCheckpointer(); }); afterEach(async () => { - await initializer.destroySaver?.(saver, initializerConfig); + await initializer.destroyCheckpointer?.(checkpointer); }); describe.each(["root", "child"])("namespace: %s", (namespace) => { const checkpoint_ns = namespace === "root" ? "" : namespace; - let configArgument: RunnableConfig; let checkpointStoredWithoutIdInConfig: Checkpoint; let metadataStoredWithoutIdInConfig: CheckpointMetadata | undefined; @@ -56,51 +45,49 @@ export function putTests( checkpoint: checkpointStoredWithoutIdInConfig, metadata: metadataStoredWithoutIdInConfig, } = initialCheckpointTuple({ - config: initializerConfig, + thread_id, checkpoint_id: checkpoint_id1, checkpoint_ns, })); - configArgument = mergeConfigs(initializerConfig, { - configurable: { checkpoint_ns }, - }); - // validate assumptions - the test checkpoints must not already exist - const existingCheckpoint1 = await saver.get( - mergeConfigs(configArgument, { - configurable: { - checkpoint_id: checkpoint_id1, - }, - }) - ); + const existingCheckpoint1 = await checkpointer.get({ + configurable: { + thread_id, + checkpoint_ns, + checkpoint_id: checkpoint_id1, + }, + }); - const existingCheckpoint2 = await saver.get( - mergeConfigs(configArgument, { - configurable: { - checkpoint_id: checkpoint_id1, - }, - }) - ); + const existingCheckpoint2 = await checkpointer.get({ + configurable: { + thread_id, + checkpoint_ns, + checkpoint_id: checkpoint_id1, + }, + }); expect(existingCheckpoint1).toBeUndefined(); expect(existingCheckpoint2).toBeUndefined(); // set up // call put without the `checkpoint_id` in the config - basicPutReturnedConfig = await saver.put( - mergeConfigs(configArgument, { + basicPutReturnedConfig = await checkpointer.put( + { configurable: { + thread_id, + checkpoint_ns, // adding this to ensure that additional fields are not stored in the checkpoint tuple canary: "tweet", }, - }), + }, checkpointStoredWithoutIdInConfig, metadataStoredWithoutIdInConfig!, {} ); - basicPutRoundTripTuple = await saver.getTuple( - mergeConfigs(configArgument, basicPutReturnedConfig) + basicPutRoundTripTuple = await checkpointer.getTuple( + basicPutReturnedConfig ); }); @@ -157,14 +144,11 @@ export function putTests( describe("failure cases", () => { it("should fail if config.configurable is missing", async () => { - const missingConfigurableConfig: RunnableConfig = { - ...configArgument, - configurable: undefined, - }; + const missingConfigurableConfig: RunnableConfig = {}; await expect( async () => - await saver.put( + await checkpointer.put( missingConfigurableConfig, checkpointStoredWithoutIdInConfig, metadataStoredWithoutIdInConfig!, @@ -175,17 +159,14 @@ export function putTests( it("should fail if the thread_id is missing", async () => { const missingThreadIdConfig: RunnableConfig = { - ...configArgument, - configurable: Object.fromEntries( - Object.entries(configArgument.configurable ?? {}).filter( - ([key]) => key !== "thread_id" - ) - ), + configurable: { + checkpoint_ns, + }, }; await expect( async () => - await saver.put( + await checkpointer.put( missingThreadIdConfig, checkpointStoredWithoutIdInConfig, metadataStoredWithoutIdInConfig!, @@ -196,7 +177,7 @@ export function putTests( }); }); - it_skipForSomeModules(initializer.saverName, { + it_skipForSomeModules(initializer.checkpointerName, { // TODO: MemorySaver throws instead of defaulting to empty namespace // see: https://github.com/langchain-ai/langgraphjs/issues/591 MemorySaver: "TODO: throws instead of defaulting to empty namespace", @@ -208,21 +189,16 @@ export function putTests( "should default to empty namespace if the checkpoint namespace is missing from config.configurable", async () => { const missingNamespaceConfig: RunnableConfig = { - ...initializerConfig, - configurable: Object.fromEntries( - Object.entries(initializerConfig.configurable ?? {}).filter( - ([key]) => key !== "checkpoint_ns" - ) - ), + configurable: { thread_id }, }; const { checkpoint, metadata } = initialCheckpointTuple({ - config: missingNamespaceConfig, + thread_id, checkpoint_id: checkpoint_id1, checkpoint_ns: "", }); - const returnedConfig = await saver.put( + const returnedConfig = await checkpointer.put( missingNamespaceConfig, checkpoint, metadata!, @@ -236,8 +212,8 @@ export function putTests( } ); - it_skipForSomeModules(initializer.saverName, { - // TODO: all of the savers below store full channel_values on every put, rather than storing deltas + it_skipForSomeModules(initializer.checkpointerName, { + // TODO: all of the checkpointers below store full channel_values on every put, rather than storing deltas // see: https://github.com/langchain-ai/langgraphjs/issues/593 // see: https://github.com/langchain-ai/langgraphjs/issues/594 // see: https://github.com/langchain-ai/langgraphjs/issues/595 @@ -256,7 +232,7 @@ export function putTests( const generatedPuts = newVersions.map((newVersions) => ({ tuple: initialCheckpointTuple({ - config: initializerConfig, + thread_id, checkpoint_id: uuid6(-3), checkpoint_ns: "", channel_values: { @@ -269,11 +245,7 @@ export function putTests( })); const storedTuples: CheckpointTuple[] = []; - for await (const tuple of putTuples( - saver, - generatedPuts, - initializerConfig - )) { + for await (const tuple of putTuples(checkpointer, generatedPuts)) { storedTuples.push(tuple); } diff --git a/libs/checkpoint-validation/src/spec/putWrites.ts b/libs/checkpoint-validation/src/spec/put_writes.ts similarity index 54% rename from libs/checkpoint-validation/src/spec/putWrites.ts rename to libs/checkpoint-validation/src/spec/put_writes.ts index 43f179d26..f6ed22e9d 100644 --- a/libs/checkpoint-validation/src/spec/putWrites.ts +++ b/libs/checkpoint-validation/src/spec/put_writes.ts @@ -5,16 +5,15 @@ import { uuid6, type BaseCheckpointSaver, } from "@langchain/langgraph-checkpoint"; -import { mergeConfigs, RunnableConfig } from "@langchain/core/runnables"; -import { CheckpointSaverTestInitializer } from "../types.js"; -import { initialCheckpointTuple } from "./data.js"; +import { RunnableConfig } from "@langchain/core/runnables"; +import { CheckpointerTestInitializer } from "../types.js"; +import { initialCheckpointTuple } from "../test_utils.js"; export function putWritesTests( - initializer: CheckpointSaverTestInitializer + initializer: CheckpointerTestInitializer ) { - describe(`${initializer.saverName}#putWrites`, () => { - let saver: T; - let initializerConfig: RunnableConfig; + describe(`${initializer.checkpointerName}#putWrites`, () => { + let checkpointer: T; let thread_id: string; let checkpoint_id: string; @@ -22,25 +21,15 @@ export function putWritesTests( thread_id = uuid6(-3); checkpoint_id = uuid6(-3); - const baseConfig = { - configurable: { - thread_id, - }, - }; - initializerConfig = mergeConfigs( - baseConfig, - await initializer.configure?.(baseConfig) - ); - saver = await initializer.createSaver(initializerConfig); + checkpointer = await initializer.createCheckpointer(); }); afterEach(async () => { - await initializer.destroySaver?.(saver, initializerConfig); + await initializer.destroyCheckpointer?.(checkpointer); }); describe.each(["root", "child"])("namespace: %s", (namespace) => { const checkpoint_ns = namespace === "root" ? "" : namespace; - let configArgument: RunnableConfig; let checkpoint: Checkpoint; let metadata: CheckpointMetadata | undefined; @@ -50,52 +39,52 @@ export function putWritesTests( beforeEach(async () => { ({ checkpoint, metadata } = initialCheckpointTuple({ - config: initializerConfig, - checkpoint_id, + thread_id, checkpoint_ns, + checkpoint_id, })); - configArgument = mergeConfigs(initializerConfig, { - configurable: { checkpoint_ns }, + // ensure the test checkpoint does not already exist + const existingCheckpoint = await checkpointer.get({ + configurable: { + thread_id, + checkpoint_ns, + checkpoint_id, + }, }); + expect(existingCheckpoint).toBeUndefined(); // our test checkpoint should not exist yet - // ensure the test checkpoint does not already exist - const existingCheckpoint = await saver.get( - mergeConfigs(configArgument, { + returnedConfig = await checkpointer.put( + { configurable: { - checkpoint_id, + thread_id, + checkpoint_ns, }, - }) - ); - expect(existingCheckpoint).toBeUndefined(); // our test checkpoint should not exist yet - - returnedConfig = await saver.put( - configArgument, + }, checkpoint, metadata!, {} /* not sure what to do about newVersions, as it's unused */ ); - await saver.putWrites( - mergeConfigs(configArgument, returnedConfig), + await checkpointer.putWrites( + returnedConfig, [["animals", "dog"]], "pet_task" ); - savedCheckpointTuple = await saver.getTuple( - mergeConfigs(configArgument, returnedConfig) - ); + savedCheckpointTuple = await checkpointer.getTuple(returnedConfig); // fail here if `put` or `getTuple` is broken so we don't get a bunch of noise from the actual test cases below expect(savedCheckpointTuple).not.toBeUndefined(); expect(savedCheckpointTuple?.checkpoint).toEqual(checkpoint); expect(savedCheckpointTuple?.metadata).toEqual(metadata); - expect(savedCheckpointTuple?.config).toEqual( - expect.objectContaining( - // allow the saver to add additional fields to the config - mergeConfigs(configArgument, { configurable: { checkpoint_id } }) - ) - ); + expect(savedCheckpointTuple?.config).toEqual({ + configurable: { + thread_id, + checkpoint_ns, + checkpoint_id, + }, + }); }); it("should store writes to the checkpoint", async () => { @@ -107,18 +96,16 @@ export function putWritesTests( describe("failure cases", () => { it("should fail if the thread_id is missing", async () => { - const missingThreadIdConfig: RunnableConfig = { - ...configArgument, - configurable: Object.fromEntries( - Object.entries(configArgument.configurable ?? {}).filter( - ([key]) => key !== "thread_id" - ) - ), + const missingThreadIdConfig = { + configurable: { + checkpoint_ns, + checkpoint_id, + }, }; await expect( async () => - await saver.putWrites( + await checkpointer.putWrites( missingThreadIdConfig, [["animals", "dog"]], "pet_task" @@ -128,17 +115,15 @@ export function putWritesTests( it("should fail if the checkpoint_id is missing", async () => { const missingCheckpointIdConfig: RunnableConfig = { - ...configArgument, - configurable: Object.fromEntries( - Object.entries(configArgument.configurable ?? {}).filter( - ([key]) => key !== "checkpoint_id" - ) - ), + configurable: { + thread_id, + checkpoint_ns, + }, }; await expect( async () => - await saver.putWrites( + await checkpointer.putWrites( missingCheckpointIdConfig, [["animals", "dog"]], "pet_task" diff --git a/libs/checkpoint-validation/src/spec/util.ts b/libs/checkpoint-validation/src/spec/util.ts deleted file mode 100644 index 36f0dab7f..000000000 --- a/libs/checkpoint-validation/src/spec/util.ts +++ /dev/null @@ -1,76 +0,0 @@ -import { mergeConfigs, RunnableConfig } from "@langchain/core/runnables"; -import { - BaseCheckpointSaver, - CheckpointTuple, - PendingWrite, -} from "@langchain/langgraph-checkpoint"; - -export async function* putTuples( - saver: BaseCheckpointSaver, - generatedTuples: { - tuple: CheckpointTuple; - writes: { writes: PendingWrite[]; taskId: string }[]; - newVersions: Record; - }[], - initializerConfig: RunnableConfig -): AsyncGenerator { - for (const generated of generatedTuples) { - const { thread_id, checkpoint_ns } = generated.tuple.config - .configurable as { thread_id: string; checkpoint_ns: string }; - - const checkpoint_id = generated.tuple.parentConfig?.configurable - ?.checkpoint_id as string | undefined; - - const config = mergeConfigs(initializerConfig, { - configurable: { - thread_id, - checkpoint_ns, - checkpoint_id, - }, - }); - - const existingTuple = await saver.getTuple( - mergeConfigs(initializerConfig, generated.tuple.config) - ); - - expect(existingTuple).toBeUndefined(); - - const newConfig = await saver.put( - config, - generated.tuple.checkpoint, - generated.tuple.metadata!, - generated.newVersions - ); - - for (const write of generated.writes) { - await saver.putWrites(newConfig, write.writes, write.taskId); - } - - const expectedTuple = await saver.getTuple(newConfig); - - expect(expectedTuple).not.toBeUndefined(); - - if (expectedTuple) { - yield expectedTuple; - } - } -} - -export async function toArray( - generator: AsyncGenerator -): Promise { - const result = []; - for await (const item of generator) { - result.push(item); - } - return result; -} - -export function toMap(tuples: CheckpointTuple[]): Map { - const result = new Map(); - for (const item of tuples) { - const key = item.checkpoint.id; - result.set(key, item); - } - return result; -} diff --git a/libs/checkpoint-validation/src/testUtils.ts b/libs/checkpoint-validation/src/testUtils.ts deleted file mode 100644 index af2d661b1..000000000 --- a/libs/checkpoint-validation/src/testUtils.ts +++ /dev/null @@ -1,55 +0,0 @@ -// to make the type signature of the skipOnModules function a bit more readable -export type SaverName = string; -export type WhySkipped = string; - -/** - * Conditionally skips a test for a specific checkpoint saver implementation. When the test is skipped, - * the reason for skipping is provided. - * - * @param saverName - The name of the current module being tested (as passed via the `name` argument in the top-level suite entrypoint). - * @param skippedSavers - A list of modules for which the test should be skipped. - * @returns A function that can be used in place of the Jest @see it function and conditionally skips the test for the provided module. - */ -export function it_skipForSomeModules( - saverName: string, - skippedSavers: Record -): typeof it | typeof it.skip { - const skipReason = skippedSavers[saverName]; - - if (skipReason) { - const skip = ( - name: string, - test: jest.ProvidesCallback | undefined, - timeout?: number - ) => { - it.skip(`[because ${skipReason}] ${name}`, test, timeout); - }; - skip.prototype = it.skip.prototype; - return skip as typeof it.skip; - } - - return it; -} - -export function it_skipIfNot( - saverName: string, - ...savers: SaverName[] -): typeof it | typeof it.skip { - if (!savers.includes(saverName)) { - const skip = ( - name: string, - test: jest.ProvidesCallback | undefined, - timeout?: number - ) => { - it.skip( - `[only passes for "${savers.join('", "')}"] ${name}`, - test, - timeout - ); - }; - skip.prototype = it.skip.prototype; - return skip as typeof it.skip; - } - - return it; -} diff --git a/libs/checkpoint-validation/src/spec/data.ts b/libs/checkpoint-validation/src/test_utils.ts similarity index 59% rename from libs/checkpoint-validation/src/spec/data.ts rename to libs/checkpoint-validation/src/test_utils.ts index cbc928196..60b6896df 100644 --- a/libs/checkpoint-validation/src/spec/data.ts +++ b/libs/checkpoint-validation/src/test_utils.ts @@ -1,5 +1,5 @@ -import { mergeConfigs, RunnableConfig } from "@langchain/core/runnables"; import { + BaseCheckpointSaver, ChannelVersions, CheckpointPendingWrite, PendingWrite, @@ -9,24 +9,74 @@ import { type CheckpointTuple, } from "@langchain/langgraph-checkpoint"; +// to make the type signature of the skipOnModules function a bit more readable +export type CheckpointerName = string; +export type WhySkipped = string; + +/** + * Conditionally skips a test for a specific checkpointer implementation. When the test is skipped, the reason for + * skipping is provided. + * + * @param checkpointerName - The name of the current module being tested (as passed via the `name` argument in the top-level suite entrypoint). + * @param skippedCheckpointers - A list of modules for which the test should be skipped. + * @returns A function that can be used in place of the Jest @see it function and conditionally skips the test for the provided module. + */ +export function it_skipForSomeModules( + checkpointerName: string, + skippedCheckpointers: Record +): typeof it | typeof it.skip { + const skipReason = skippedCheckpointers[checkpointerName]; + + if (skipReason) { + const skip = ( + name: string, + test: jest.ProvidesCallback | undefined, + timeout?: number + ) => { + it.skip(`[because ${skipReason}] ${name}`, test, timeout); + }; + skip.prototype = it.skip.prototype; + return skip as typeof it.skip; + } + + return it; +} + +export function it_skipIfNot( + checkpointerName: string, + ...checkpointers: CheckpointerName[] +): typeof it | typeof it.skip { + if (!checkpointers.includes(checkpointerName)) { + const skip = ( + name: string, + test: jest.ProvidesCallback | undefined, + timeout?: number + ) => { + it.skip( + `[only passes for "${checkpointers.join('", "')}"] ${name}`, + test, + timeout + ); + }; + skip.prototype = it.skip.prototype; + return skip as typeof it.skip; + } + + return it; +} export interface InitialCheckpointTupleConfig { - config: RunnableConfig; + thread_id: string; checkpoint_id: string; - checkpoint_ns?: string; + checkpoint_ns: string; channel_values?: Record; channel_versions?: ChannelVersions; } export function initialCheckpointTuple({ - config, + thread_id, checkpoint_id, checkpoint_ns, channel_values = {}, }: InitialCheckpointTupleConfig): CheckpointTuple { - if (checkpoint_ns === undefined) { - // eslint-disable-next-line no-param-reassign - checkpoint_ns = config.configurable?.checkpoint_ns; - } - if (checkpoint_ns === undefined) { throw new Error("checkpoint_ns is required"); } @@ -35,13 +85,16 @@ export function initialCheckpointTuple({ Object.keys(channel_values).map((key) => [key, 1]) ); + const config = { + configurable: { + thread_id, + checkpoint_id, + checkpoint_ns, + }, + }; + return { - config: mergeConfigs(config, { - configurable: { - checkpoint_id, - checkpoint_ns, - }, - }), + config, checkpoint: { v: 1, ts: new Date().toISOString(), @@ -49,7 +102,7 @@ export function initialCheckpointTuple({ channel_values, channel_versions, versions_seen: { - // I think this is meant to be opaque to checkpoint savers, so I'm just stuffing the data in here to make sure it's stored and retrieved + // this is meant to be opaque to checkpointers, so we just stuff dummy data in here to make sure it's stored and retrieved "": { someChannel: 1, }, @@ -67,17 +120,17 @@ export function initialCheckpointTuple({ } export interface ParentAndChildCheckpointTuplesWithWritesConfig { - config: RunnableConfig; + thread_id: string; parentCheckpointId: string; childCheckpointId: string; - checkpoint_ns?: string; + checkpoint_ns: string; initialChannelValues?: Record; writesToParent?: { taskId: string; writes: PendingWrite[] }[]; writesToChild?: { taskId: string; writes: PendingWrite[] }[]; } export function parentAndChildCheckpointTuplesWithWrites({ - config, + thread_id, parentCheckpointId, childCheckpointId, checkpoint_ns, @@ -88,11 +141,6 @@ export function parentAndChildCheckpointTuplesWithWrites({ parent: CheckpointTuple; child: CheckpointTuple; } { - if (checkpoint_ns === undefined) { - // eslint-disable-next-line no-param-reassign - checkpoint_ns = config.configurable?.checkpoint_ns; - } - if (checkpoint_ns === undefined) { throw new Error("checkpoint_ns is required"); } @@ -159,7 +207,7 @@ export function parentAndChildCheckpointTuplesWithWrites({ channel_values: initialChannelValues, channel_versions: parentChannelVersions, versions_seen: { - // I think this is meant to be opaque to checkpoint savers, so I'm just stuffing the data in here to make sure it's stored and retrieved + // this is meant to be opaque to checkpointers, so we just stuff dummy data in here to make sure it's stored and retrieved "": { someChannel: 1, }, @@ -172,12 +220,13 @@ export function parentAndChildCheckpointTuplesWithWrites({ writes: null, parents: {}, }, - config: mergeConfigs(config, { + config: { configurable: { + thread_id, checkpoint_ns, checkpoint_id: parentCheckpointId, }, - }), + }, parentConfig: undefined, pendingWrites: parentPendingWrites, }, @@ -189,7 +238,7 @@ export function parentAndChildCheckpointTuplesWithWrites({ channel_values: childChannelValues, channel_versions: childChannelVersions, versions_seen: { - // I think this is meant to be opaque to checkpoint savers, so I'm just stuffing the data in here to make sure it's stored and retrieved + // this is meant to be opaque to checkpointers, so we just stuff dummy data in here to make sure it's stored and retrieved "": { someChannel: 1, }, @@ -200,34 +249,32 @@ export function parentAndChildCheckpointTuplesWithWrites({ source: "loop", step: 0, writes: { - // I think this is meant to be opaque to checkpoint savers, so I'm just stuffing the data in here to make sure it's stored and retrieved someNode: parentPendingWrites, }, parents: { - // I think this is meant to be opaque to checkpoint savers, so I'm just stuffing the data in here to make sure it's stored and retrieved - // I think this is roughly what it'd look like if it were generated by the pregel loop, though - checkpoint_ns: parentCheckpointId, + [checkpoint_ns]: parentCheckpointId, }, }, - config: mergeConfigs(config, { + config: { configurable: { + thread_id, checkpoint_ns, checkpoint_id: childCheckpointId, }, - }), - parentConfig: mergeConfigs(config, { + }, + parentConfig: { configurable: { + thread_id, checkpoint_ns, checkpoint_id: parentCheckpointId, }, - }), + }, pendingWrites: childPendingWrites, }, }; } export function* generateTuplePairs( - config: RunnableConfig, countPerNamespace: number, namespaces: string[] ): Generator<{ @@ -258,12 +305,8 @@ export function* generateTuplePairs( }; const { parent, child } = parentAndChildCheckpointTuplesWithWrites({ - config: mergeConfigs(config, { - configurable: { - thread_id, - checkpoint_ns, - }, - }), + thread_id, + checkpoint_ns, parentCheckpointId, childCheckpointId, initialChannelValues, @@ -288,3 +331,70 @@ export function* generateTuplePairs( } } } + +export async function* putTuples( + checkpointer: BaseCheckpointSaver, + generatedTuples: { + tuple: CheckpointTuple; + writes: { writes: PendingWrite[]; taskId: string }[]; + newVersions: Record; + }[] +): AsyncGenerator { + for (const generated of generatedTuples) { + const { thread_id, checkpoint_ns } = generated.tuple.config + .configurable as { thread_id: string; checkpoint_ns: string }; + + const checkpoint_id = generated.tuple.parentConfig?.configurable + ?.checkpoint_id as string | undefined; + + const config = { + configurable: { + thread_id, + checkpoint_ns, + checkpoint_id, + }, + }; + + const existingTuple = await checkpointer.getTuple(generated.tuple.config); + + expect(existingTuple).toBeUndefined(); + + const newConfig = await checkpointer.put( + config, + generated.tuple.checkpoint, + generated.tuple.metadata!, + generated.newVersions + ); + + for (const write of generated.writes) { + await checkpointer.putWrites(newConfig, write.writes, write.taskId); + } + + const expectedTuple = await checkpointer.getTuple(newConfig); + + expect(expectedTuple).not.toBeUndefined(); + + if (expectedTuple) { + yield expectedTuple; + } + } +} + +export async function toArray( + generator: AsyncGenerator +): Promise { + const result = []; + for await (const item of generator) { + result.push(item); + } + return result; +} + +export function toMap(tuples: CheckpointTuple[]): Map { + const result = new Map(); + for (const item of tuples) { + const key = item.checkpoint.id; + result.set(key, item); + } + return result; +} diff --git a/libs/checkpoint-validation/src/tests/memory.spec.ts b/libs/checkpoint-validation/src/tests/memory.spec.ts index 90ea4dca4..cbb6775dc 100644 --- a/libs/checkpoint-validation/src/tests/memory.spec.ts +++ b/libs/checkpoint-validation/src/tests/memory.spec.ts @@ -1,4 +1,4 @@ import { specTest } from "../spec/index.js"; -import { initializer } from "./memoryInitializer.js"; +import { initializer } from "./memory_initializer.js"; specTest(initializer); diff --git a/libs/checkpoint-validation/src/tests/memoryInitializer.ts b/libs/checkpoint-validation/src/tests/memoryInitializer.ts deleted file mode 100644 index c634fdfe2..000000000 --- a/libs/checkpoint-validation/src/tests/memoryInitializer.ts +++ /dev/null @@ -1,8 +0,0 @@ -import { MemorySaver } from "@langchain/langgraph-checkpoint"; - -export const initializer = { - saverName: "MemorySaver", - createSaver: () => new MemorySaver(), -}; - -export default initializer; diff --git a/libs/checkpoint-validation/src/tests/memory_initializer.ts b/libs/checkpoint-validation/src/tests/memory_initializer.ts new file mode 100644 index 000000000..8a8f7dc29 --- /dev/null +++ b/libs/checkpoint-validation/src/tests/memory_initializer.ts @@ -0,0 +1,9 @@ +import { MemorySaver } from "@langchain/langgraph-checkpoint"; +import { CheckpointerTestInitializer } from "../types.js"; + +export const initializer: CheckpointerTestInitializer = { + checkpointerName: "MemorySaver", + createCheckpointer: () => new MemorySaver(), +}; + +export default initializer; diff --git a/libs/checkpoint-validation/src/tests/mongodb.spec.ts b/libs/checkpoint-validation/src/tests/mongodb.spec.ts index 2135c4ffd..8a235cfc5 100644 --- a/libs/checkpoint-validation/src/tests/mongodb.spec.ts +++ b/libs/checkpoint-validation/src/tests/mongodb.spec.ts @@ -1,9 +1,9 @@ import { specTest } from "../spec/index.js"; -import { initializer } from "./mongoInitializer.js"; -import { isCI, osHasSupportedContainerRuntime } from "./utils.js"; +import { initializer } from "./mongodb_initializer.js"; +import { isSkippedCIEnvironment } from "./utils.js"; -if (osHasSupportedContainerRuntime() || !isCI()) { - specTest(initializer); +if (isSkippedCIEnvironment()) { + it.skip(`${initializer.checkpointerName} skipped in CI because no container runtime is available`, () => {}); } else { - it.skip(`${initializer.saverName} skipped in CI because no container runtime is available`, () => {}); + specTest(initializer); } diff --git a/libs/checkpoint-validation/src/tests/mongoInitializer.ts b/libs/checkpoint-validation/src/tests/mongodb_initializer.ts similarity index 84% rename from libs/checkpoint-validation/src/tests/mongoInitializer.ts rename to libs/checkpoint-validation/src/tests/mongodb_initializer.ts index 40f144805..0a6002785 100644 --- a/libs/checkpoint-validation/src/tests/mongoInitializer.ts +++ b/libs/checkpoint-validation/src/tests/mongodb_initializer.ts @@ -8,7 +8,7 @@ import { // eslint-disable-next-line import/no-extraneous-dependencies import { MongoClient } from "mongodb"; -import type { CheckpointSaverTestInitializer } from "../types.js"; +import type { CheckpointerTestInitializer } from "../types.js"; const dbName = "test_db"; @@ -17,8 +17,8 @@ const container = new MongoDBContainer("mongo:6.0.1"); let startedContainer: StartedMongoDBContainer; let client: MongoClient | undefined; -export const initializer: CheckpointSaverTestInitializer = { - saverName: "@langchain/langgraph-checkpoint-mongodb", +export const initializer: CheckpointerTestInitializer = { + checkpointerName: "@langchain/langgraph-checkpoint-mongodb", async beforeAll() { startedContainer = await container.start(); @@ -30,7 +30,7 @@ export const initializer: CheckpointSaverTestInitializer = { beforeAllTimeout: 300_000, // five minutes, to pull docker container - async createSaver() { + async createCheckpointer() { // ensure fresh database for each test const db = await client!.db(dbName); await db.dropDatabase(); diff --git a/libs/checkpoint-validation/src/tests/postgres.spec.ts b/libs/checkpoint-validation/src/tests/postgres.spec.ts index ca357de55..f2cff9d1e 100644 --- a/libs/checkpoint-validation/src/tests/postgres.spec.ts +++ b/libs/checkpoint-validation/src/tests/postgres.spec.ts @@ -1,9 +1,9 @@ import { specTest } from "../spec/index.js"; -import { initializer } from "./postgresInitializer.js"; -import { isCI, osHasSupportedContainerRuntime } from "./utils.js"; +import { initializer } from "./postgres_initializer.js"; +import { isSkippedCIEnvironment } from "./utils.js"; -if (osHasSupportedContainerRuntime() || !isCI()) { - specTest(initializer); +if (isSkippedCIEnvironment()) { + it.skip(`${initializer.checkpointerName} skipped in CI because no container runtime is available`, () => {}); } else { - it.skip(`${initializer.saverName} skipped in CI because no container runtime is available`, () => {}); + specTest(initializer); } diff --git a/libs/checkpoint-validation/src/tests/postgresInitializer.ts b/libs/checkpoint-validation/src/tests/postgres_initializer.ts similarity index 75% rename from libs/checkpoint-validation/src/tests/postgresInitializer.ts rename to libs/checkpoint-validation/src/tests/postgres_initializer.ts index a0e128484..7ab466b4e 100644 --- a/libs/checkpoint-validation/src/tests/postgresInitializer.ts +++ b/libs/checkpoint-validation/src/tests/postgres_initializer.ts @@ -10,7 +10,7 @@ import { // eslint-disable-next-line import/no-extraneous-dependencies import pg from "pg"; -import type { CheckpointSaverTestInitializer } from "../types.js"; +import type { CheckpointerTestInitializer } from "../types.js"; const dbName = "test_db"; @@ -22,8 +22,8 @@ const container = new PostgreSqlContainer("postgres:16.2") let startedContainer: StartedPostgreSqlContainer; let client: pg.Pool | undefined; -export const initializer: CheckpointSaverTestInitializer = { - saverName: "@langchain/langgraph-checkpoint-postgres", +export const initializer: CheckpointerTestInitializer = { + checkpointerName: "@langchain/langgraph-checkpoint-postgres", async beforeAll() { startedContainer = await container.start(); @@ -35,7 +35,7 @@ export const initializer: CheckpointSaverTestInitializer = { await startedContainer.stop(); }, - async createSaver() { + async createCheckpointer() { client = new pg.Pool({ connectionString: startedContainer.getConnectionUri(), }); @@ -49,13 +49,13 @@ export const initializer: CheckpointSaverTestInitializer = { url.username = startedContainer.getUsername(); url.password = startedContainer.getPassword(); - const saver = PostgresSaver.fromConnString(url.toString()); - await saver.setup(); - return saver; + const checkpointer = PostgresSaver.fromConnString(url.toString()); + await checkpointer.setup(); + return checkpointer; }, - async destroySaver(saver) { - await saver.end(); + async destroyCheckpointer(checkpointer: PostgresSaver) { + await checkpointer.end(); await client?.query(`DROP DATABASE ${dbName}`); await client?.end(); }, diff --git a/libs/checkpoint-validation/src/tests/sqlite.spec.ts b/libs/checkpoint-validation/src/tests/sqlite.spec.ts index e3c250228..96f6687bd 100644 --- a/libs/checkpoint-validation/src/tests/sqlite.spec.ts +++ b/libs/checkpoint-validation/src/tests/sqlite.spec.ts @@ -1,5 +1,5 @@ // eslint-disable-next-line import/no-extraneous-dependencies import { specTest } from "../spec/index.js"; -import { initializer } from "./sqliteInitializer.js"; +import { initializer } from "./sqlite_initializer.js"; specTest(initializer); diff --git a/libs/checkpoint-validation/src/tests/sqliteInitializer.ts b/libs/checkpoint-validation/src/tests/sqliteInitializer.ts deleted file mode 100644 index 8fe6b7519..000000000 --- a/libs/checkpoint-validation/src/tests/sqliteInitializer.ts +++ /dev/null @@ -1,17 +0,0 @@ -// eslint-disable-next-line import/no-extraneous-dependencies -import { SqliteSaver } from "@langchain/langgraph-checkpoint-sqlite"; -import { CheckpointSaverTestInitializer } from "../types.js"; - -export const initializer: CheckpointSaverTestInitializer = { - saverName: "@langchain/langgraph-checkpoint-sqlite", - - async createSaver() { - return SqliteSaver.fromConnString(":memory:"); - }, - - destroySaver(saver) { - saver.db.close(); - }, -}; - -export default initializer; diff --git a/libs/checkpoint-validation/src/tests/sqlite_initializer.ts b/libs/checkpoint-validation/src/tests/sqlite_initializer.ts new file mode 100644 index 000000000..b526c2422 --- /dev/null +++ b/libs/checkpoint-validation/src/tests/sqlite_initializer.ts @@ -0,0 +1,17 @@ +// eslint-disable-next-line import/no-extraneous-dependencies +import { SqliteSaver } from "@langchain/langgraph-checkpoint-sqlite"; +import { CheckpointerTestInitializer } from "../types.js"; + +export const initializer: CheckpointerTestInitializer = { + checkpointerName: "@langchain/langgraph-checkpoint-sqlite", + + async createCheckpointer() { + return SqliteSaver.fromConnString(":memory:"); + }, + + async destroyCheckpointer(checkpointer: SqliteSaver) { + await checkpointer.db.close(); + }, +}; + +export default initializer; diff --git a/libs/checkpoint-validation/src/tests/utils.ts b/libs/checkpoint-validation/src/tests/utils.ts index 32c8871dc..d4139513a 100644 --- a/libs/checkpoint-validation/src/tests/utils.ts +++ b/libs/checkpoint-validation/src/tests/utils.ts @@ -8,7 +8,7 @@ function isWindows() { return platform() === "win32"; } -export function isCI() { +function isCI() { // eslint-disable-next-line no-process-env return (process.env.CI ?? "").toLowerCase() === "true"; } @@ -28,6 +28,6 @@ export function isCI() { * * */ -export function osHasSupportedContainerRuntime() { - return !isWindows() && !isMSeriesMac(); +export function isSkippedCIEnvironment() { + return isCI() && (isWindows() || isMSeriesMac()); } diff --git a/libs/checkpoint-validation/src/types.ts b/libs/checkpoint-validation/src/types.ts index 548457c07..b875b34b9 100644 --- a/libs/checkpoint-validation/src/types.ts +++ b/libs/checkpoint-validation/src/types.ts @@ -1,17 +1,16 @@ -import { RunnableConfig } from "@langchain/core/runnables"; import type { BaseCheckpointSaver } from "@langchain/langgraph-checkpoint"; import { z } from "zod"; -export interface CheckpointSaverTestInitializer< - CheckpointSaverT extends BaseCheckpointSaver +export interface CheckpointerTestInitializer< + CheckpointerT extends BaseCheckpointSaver > { /** - * The name of the checkpoint saver being tested. This will be used to identify the saver in test output. + * The name of the checkpointer being tested. This will be used to identify the checkpointer in test output. */ - saverName: string; + checkpointerName: string; /** - * Called once before any tests are run. Use this to perform any setup that your checkpoint saver may require, like + * Called once before any tests are run. Use this to perform any setup that your checkpoint checkpointer may require, like * starting docker containers, etc. */ beforeAll?(): void | Promise; @@ -25,49 +24,29 @@ export interface CheckpointSaverTestInitializer< beforeAllTimeout?: number; /** - * Called once after all tests are run. Use this to perform any infrastructure cleanup that your checkpoint saver may + * Called once after all tests are run. Use this to perform any infrastructure cleanup that your checkpointer may * require, like tearing down docker containers, etc. */ afterAll?(): void | Promise; /** - * Called before each set of validations is run, prior to calling @see createSaver. Use this to modify the @see - * RunnableConfig that will be used during the test, used to include any additional configuration that your - * checkpoint saver may require. + * Called before each set of validations is run. The checkpointer returned will be used during test execution. * - * @param config The @see RunnableConfig that will be used during the test. - * @returns an instance of @see RunnableConfig (or a promise that resolves to one) to be merged with the original - * config for use during the test execution. + * @returns A new checkpointer, or promise that resolves to a new checkpointer. */ - configure?(config: RunnableConfig): RunnableConfig | Promise; + createCheckpointer(): CheckpointerT | Promise; /** - * Called before each set of validations is run, after @see configure has been called. The checkpoint saver returned - * will be used during test execution. + * Called after each set of validations is run. Use this to clean up any resources that your checkpointer may + * have been using. This should include cleaning up any state that the checkpointer wrote during the tests that just ran. * - * @param config The @see RunnableConfig that will be used during the test. Can be used for constructing the saver, - * if required. - * @returns A new saver, or promise that resolves to a new saver. + * @param checkpointer The @see BaseCheckpointSaver that was used during the test. */ - createSaver( - config: RunnableConfig - ): CheckpointSaverT | Promise; - - /** - * Called after each set of validations is run. Use this to clean up any resources that your checkpoint saver may - * have been using. This should include cleaning up any state that the saver wrote during the tests that just ran. - * - * @param saver The @see BaseCheckpointSaver that was used during the test. - * @param config The @see RunnableConfig that was used during the test. - */ - destroySaver?( - saver: CheckpointSaverT, - config: RunnableConfig - ): void | Promise; + destroyCheckpointer?(checkpointer: CheckpointerT): void | Promise; } -export const checkpointSaverTestInitializerSchema = z.object({ - saverName: z.string(), +export const checkpointerTestInitializerSchema = z.object({ + checkpointerName: z.string(), beforeAll: z .function() .returns(z.void().or(z.promise(z.void()))) @@ -77,24 +56,16 @@ export const checkpointSaverTestInitializerSchema = z.object({ .function() .returns(z.void().or(z.promise(z.void()))) .optional(), - configure: z - .function() - .args(z.custom()) - .returns( - z.custom().or(z.promise(z.custom())) - ) - .optional(), - createSaver: z + createCheckpointer: z .function() - .args(z.custom()) .returns( z .custom() .or(z.promise(z.custom())) ), - destroySaver: z + destroyCheckpointer: z .function() - .args(z.custom(), z.custom()) + .args(z.custom()) .returns(z.void().or(z.promise(z.void()))) .optional(), }); @@ -102,6 +73,6 @@ export const checkpointSaverTestInitializerSchema = z.object({ export type TestTypeFilter = "getTuple" | "list" | "put" | "putWrites"; export type GlobalThis = typeof globalThis & { - __langgraph_checkpoint_validation_initializer?: CheckpointSaverTestInitializer; + __langgraph_checkpoint_validation_initializer?: CheckpointerTestInitializer; __langgraph_checkpoint_validation_filters?: TestTypeFilter[]; };