RFC: Schema-driven testing

[alessbell]

The status quo

In her GraphQL Conf 2023 talk "Sophisticated Schema Mocking", Stephanie Saunders from Coinbase outlined some challenges she and her team have encountered testing front-end applications that consume GraphQL data. (I recommend watching the whole talk!)

A common theme is the brittleness of tests that make use of static response mocks for individual operations. If you've written tests with Apollo Client's MockedProvider, you may have seen warnings logged when none of the remaining mocks can be matched, or when fields are missing when writing a result to the cache: maybe the query now includes some new fields, or variables have changed.

Note: a new MockedResponse.variableMatcher function was recently released in v3.9, in addition to the ability to reuse mocks, making it possible to write more flexible tests with MockedProvider, but the challenges posed by per-operation response mocks still stand.

MockedProvider is a useful tool for unit tests—and it's not going anywhere! But, in addition to some limitations inherent to static response mocking, it also doesn't allow developers to test the link chain, code in your application's critical path. Our team has been thinking about complementary testing utilities, and today I'm sharing this RFC for an API I'm calling proxiedSchema.

proxiedSchema

1. Summary

proxiedSchema will allow developers to take a schema-driven approach to testing by generating responses using mock resolvers with default values for scalar values. This tool can be used for integration tests that allow you to test your link chain, and can also be paired with tools like MSW.

An initial version should support the following features:

• schema mocking via custom resolvers
• default scalar values
• an API that allows resolvers to be overridden in individual tests and subsequently reset
• TypeScript support for strongly-typed resolvers

2. Detailed Design

The tool I’m proposing would require a static schema and a set of default mock resolvers, and accept optional scalar mocks. Here’s an example of how we’d configure it before writing our tests:

import { proxiedSchema, createMockSchema } from '@apollo/client/testing';
import schema from '../schema.graphql';

const schemaWithMocks = createMockSchema(schema, {
  ID: () => "1",
  Int: () => 42,
  Float: () => 22.1,
  String: () => "String",
  Date: () => new Date("January 1, 2024 01:00:00").toJSON().split("T")[0],
});

export const schema = proxiedSchema(schemaWithMocks, {
  Query: {
    dog: () => ({
      name: "Buck"
    }),
  },
});

Your testing setup can then either make use of a terminating SchemaLink directly configured via a custom render function, which would take our proxied schema:

import { render } from "@testing-library/react";
import { ApolloClient, ApolloProvider, InMemoryCache } from "@apollo/client";

const customRender = (ui: React.ReactElement, options = {}) => {
  const wrapper = ({ children }: { children: React.ReactElement }) => {
    return (
      <ApolloProvider
        client={
          new ApolloClient({
            cache: new InMemoryCache(),
            link: new SchemaLink({ // could be appended to existing links
              schema: options.schema
            })
          })
        }
      >
        {children}
      </ApolloProvider>
    );
  };

  return render(ui, { wrapper, ...options });
};

Using a custom fetch implementation

Since we'd like to enable testing of your entire link chain with minimal environment-specific alterations, we plan to provide a custom fetch implementation that would effectively turn your terminating HttpLink into a SchemaLink. This will be provided via context in your tests.

Usage with e.g. Jest and Testing Library

In our tests, we can now render a component that issues any query and make assertions against the output based on either data supplied by our mock resolvers or our scalar defaults if no resolver is found.

Consider the following component:

import { gql, useQuery, TypedDocumentNode } from "@apollo/client";

interface DogQueryData {
  dog: {
    id: string;
    name: string;
    breed: string;
  };
}

const GET_DOG_QUERY: TypedDocumentNode<DogQueryData> = gql`
  query GetDog($name: String) {
    dog(name: $name) {
      id
      name
      breed
    }
  }
`;

export function Dog({ name }: { name: string }) {
  const { loading, error, data, refetch } = useQuery(GET_DOG_QUERY, {
    variables: { name },
    errorPolicy: "all",
  });

  if (loading) return <p>Loading...</p>;

  return (
    <>
      {data ? (
        <p>
          {data?.dog.name} is a {data?.dog.breed}
        </p>
      ) : null}
      {error ? <p>{error.message}</p> : null}
      <button onClick={() => refetch()}>Refetch</button>
    </>
  );
}

Here's how we might test out Dog component using proxiedSchema:

import userEvent from "@testing-library/user-event";
import { ApolloError } from "@apollo/client";
import { proxiedSchema, createMockSchema } from '@apollo/client/testing';

import { render, screen } from "./test-utils";
import { Dog } from "./dog";

describe("schema proxy", () => {
  // can either be created once and imported in each test file,
  // or created in individual tests
  const schemaWithMocks = createMockSchema(schema, {
    ID: () => "1",
    Int: () => 42,
    Float: () => 22.1,
    String: () => "String",
    Date: () => new Date("January 1, 2024 01:00:00").toJSON().split("T")[0],
  });
  
  // similarly, you can create a base proxied schema that is imported
  // in individual test files and "forked" via `forkWithResolvers`,
  // as illustrated below by "forking" the proxied schema in each test
  // before calling `addResolvers`.
  // `addResolvers` alters a proxied schema going forward, and should
  // therefore almost always be called only after creating a fork via
  // `forkWithResolvers`
  const schema = proxiedSchema(schemaWithMocks, {
    Query: {
      dog: (_, { name }) => ({
        name
      }),
    },
  });

  it("renders with data from original resolvers", async () => {
    render(<Dog name="Buck" />, { schema });

    // default string scalar is rendered for `breed`
    expect(await screen.findByText(/Buck is a String/i)).toBeInTheDocument();
  });
  
  it("allows us to 'fork' resolvers without polluting original schema", async () => {
    const forkedSchema = schema.forkWithResolvers({
      Query: {
        dog: (_, { name }) => ({
          name,
          breed: new ApolloError({ errorMessage: "No breed found" }),
        }),
      },
    });
  
    const { rerender } = render(<Dog name="Buck" />, { schema: forkedSchema });
  
    // loading state
    expect(await screen.findByText(/loading/i)).toBeInTheDocument();
  
    // error state is present
    expect(await screen.findByText(/no breed found/i)).toBeInTheDocument();
  
    // in the component we've set errorPolicy: all, so we're rendering partial
    // data without the breed
    expect(await screen.findByText(/buck is a/i)).toBeInTheDocument();
    expect(screen.queryByText(/bulldog/i)).not.toBeInTheDocument();
  
    // we can specify a different breed in advance of a refetch
    // note that we have to specify both name and breed, since we're not deep merging
    // resolvers
    forkedSchema.addResolvers({
      Query: {
        dog: (_, { name }) => ({
          name,
          breed: "pug",
        }),
      },
    });

    rerender(<Dog name="Rover" />, { schema: forkedSchema });
  
    userEvent.click(screen.getByText(/refetch/i));
  
    // NB: we don't get a loading state again by default,
    // needs notifyOnNetworkStatusChange: true
    expect(await screen.findByText(/loading/i)).toBeInTheDocument();
  
    // now we have the breed and no error message
    expect(await screen.findByText(/rover is a pug/i)).toBeInTheDocument();
    expect(screen.queryByText(/no breed found/i)).not.toBeInTheDocument();
  });
  
  it("original resolvers are preserved", async () => {
    render(<Main />, { schema });

    expect(await screen.findByText(/Buck is a String/i)).toBeInTheDocument();
  });
});

Note: there may be a blocker to usage with Vitest, due to graphql's dual package hazard problem. More testing is needed.

Usage with MSW

proxiedSchema should be compatible with MSW's graphql.operation API for "resolving any outgoing GraphQL operations against a mock GraphQL schema." I've been focusing on API design but will share an example as soon as I can spin one up.

3. Drawbacks

Thinking in your GraphQL schema

While I’d argue schema-driven testing has advantages over static response mocking, there will be a learning curve for some users. Some developers may not be familiar with the schema of the API their app is consuming, and writing mock resolvers (as opposed to "thinking in response JSON") might feel strange at first. OTOH, this is a great opportunity to provide high quality documentation that teaches/reinforces core GraphQL concepts.

4. Alternatives

Instead of using an object proxy, we could use the “provider”-style approach, similar to the design of MockedProvider.

I'd argue that exposing a way of interfacing with the schema in a declarative way via object proxy has a DX benefit over the provider approach: mocking is decoupled from component rendering, so mock resolvers can be updated throughout a single test. It also importantly decouples this testing approach from a React-specific API, allowing developers outside of the React community to write tests in this style.

5. Unresolved Questions

• Merging resolvers: I initially thought we probably shouldn’t deep merge resolvers, i.e. if your root resolvers declare a Query.dog, and your test defines a Query.dog, we don’t merge fields. I'd love some feedback here once folks start trying this out.
• Subscriptions/@defer: What will support for features that use multipart over HTTP look like? With MSW?

[qswitcher]

I love the idea, but one thing to consider is how this will scale for companies with huge company-wide schemas. For example, Indeed's schema at present contains over 7000 types and 24000 fields. If a developer were to download that and feed that into createMockSchema I would expect it to be quite slow, especially if this were executed in a test setup function. In order for this to be performant I see two possible approaches.

1. The createMockSchema implementation would need implement it's own internal caching mechanism so that the full schema file does not need to be parsed each time.
2. Implement some sort of schema pruning utility to limit the schema to just the types the unit and integration tests care about. This pruned schema could be built once and reused for all subsequent tests.

Without an elegant way to handle huge schemas I fear that developers will resort to hand rolling their own pruned versions of schemas for the purpose of testing, which will create the new problem of keeping those pruned schemas in sync with the real schema.

[alessbell]

@qswitcher that's a great call out! I need to do some testing with much larger schemas - I suspect caching will allow us to achieve the performance we need, but a schema pruning utility is an interesting idea, too. Totally agree we need to avoid a scenario where developers are having to maintain their own condensed version of the schema - thanks for the feedback!

[stubailo]

This is awesome, really excited to have first-party tools for this! I published a post about a similar approach back when we were introducing GraphQL at Stripe: https://www.freecodecamp.org/news/a-new-approach-to-mocking-graphql-data-1ef49de3d491/

[alessbell]

Hey everyone 👋

Last week we released v3.10 with new experimental testing APIs! Please see the documentation for more information.

There's also a new pinned issue #11817 where we'd love to hear your feedback. Looking forward to hearing about what's working well and what can be improved. Thanks!

[github-actions]

Do you have any feedback for the maintainers? Please tell us by taking a one-minute survey. Your responses will help us understand Apollo Client usage and allow us to serve you better.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
For general questions, we recommend using StackOverflow or our discord server.