Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update AI-powered search tutorial #3056

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Update AI-powered search tutorial #3056

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 39 additions & 22 deletions learn/ai_powered_search/getting_started_with_ai_search.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@ description: AI-powered search is an experimental technology that uses LLMs to r

# Getting started with AI-powered search <NoticeTag type="experimental" label="experimental" />

[AI-powered search](https://meilisearch.com/solutions/vector-search?utm_campaign=vector-search&utm_source=docs&utm_content=getting-started-with-ai-search), sometimes also called vector search and hybrid search, is an experimental technology that uses [large language models](https://en.wikipedia.org/wiki/Large_language_model) to retrieve search results based on the meaning and context of a query.
[AI-powered search](https://meilisearch.com/solutions/vector-search), sometimes also called vector search or hybrid search, is an experimental technology that uses [large language models (LLMs)](https://en.wikipedia.org/wiki/Large_language_model) to retrieve search results based on the meaning and context of a query.

This tutorial will walk you through configuring AI-powered search in your Meilisearch project. You will activate the vector store setting, generate document embeddings with OpenAI, and perform your first search.
This tutorial will walk you through configuring AI-powered search in your Meilisearch project. You will see how to activate this feature, generate document embeddings with OpenAI, and perform your first search.

## Requirements

Expand All @@ -17,25 +17,27 @@ This tutorial will walk you through configuring AI-powered search in your Meilis

## Create a new index

Create a `kitchenware` index and add [this kitchenware products dataset](/assets/datasets/kitchenware.json) to it. If necessary, consult the quick start for instructions on how to configure a basic Meilisearch installation.
First, create a new Meilisearch project. If you need a refresher or if this is your first time using Meilisearch, follow the [getting started](/learn/getting_started/cloud_quick_start).

Next, create a `kitchenware` index and add [this kitchenware products dataset](/assets/datasets/kitchenware.json) to it. It will take Meilisearch a few moments to process the data.

## Activate AI-powered search

First, activate the AI-powered search experimental feature. Exactly how to do that depends on whether you are using [Meilisearch Cloud](#meilisearch-cloud-projects) or [self-hosting Meilisearch](#self-hosted-instances).
AI-powered search is an experimental feature and is disabled by default. You must manually activate it either via the Meilisearch Cloud UI, or with the experimental features endpoint.

### Meilisearch Cloud projects
<Capsule intent="tip" title="Meilisearch Cloud AI-powered search waitlist">
To use AI-powered search with Meilisearch Cloud, you must first enter the waitlist. You will not be able to activate vector search until your sign-up has been approved.
</Capsule>

If using Meilisearch Cloud, navigate to your project overview and find "Experimental features". Then check the "AI-powered search" box.
### Meilisearch Cloud UI

![A section of the project overview interface titled "Experimental features". The image shows a few options, including "Vector store".](https://raw.githubusercontent.com/meilisearch/documentation/main/assets/images/vector-search/01-cloud-vector-store.png)
Navigate to your project overview and find "Experimental features". Then click on the "AI-powered search" box.

<Capsule intent="note" title="Meilisearch Cloud AI-powered search waitlist">
To ensure proper scaling of Meilisearch Cloud's latest AI-powered search offering, you must enter the waitlist before activating vector search. You will not be able to activate vector search in the Cloud interface or via the `/experimental-features` route until your sign-up has been approved.
</Capsule>
![A section of the project overview interface titled "Experimental features". The image shows a few options, including "Vector store".](https://raw.githubusercontent.com/meilisearch/documentation/main/assets/images/vector-search/01-cloud-vector-store.png)

### Self-hosted instances
### Experimental features endpoint

Use [the `/experimental-features` route](/reference/api/experimental_features?utm_campaign=vector-search&utm_source=docs&utm_medium=vector-search-guide) to activate vector search during runtime:
Use [the `/experimental-features` route](/reference/api/experimental_features) to activate vector search during runtime:

```sh
curl \
Expand All @@ -48,9 +50,11 @@ curl \

## Generate vector embeddings with OpenAI

Next, you must generate vector embeddings for all documents in your dataset. Embeddings are mathematical representations of the meanings of words and sentences in your documents. Meilisearch relies on external providers to generate these embeddings. This tutorial uses an OpenAI embedder, but Meilisearch also supports embedders from HuggingFace, Ollama, and any embedder accessible via a RESTful API.
Next, you must generate vector embeddings for all documents in your dataset. Embeddings are mathematical representations of the meanings of words and sentences in your documents. Meilisearch relies on various external providers to generate these embeddings.

For this tutorial, you should use OpenAI. Log into OpenAI, or create an account if this is your first time using it. Generate a new API key using [OpenAI's web interface](https://platform.openai.com/api-keys).

Use the `embedders` index setting of the [update `/settings` endpoint](/reference/api/settings?utm_campaign=vector-search&utm_source=docs&utm_medium=vector-search-guide) to configure an [OpenAI](https://platform.openai.com/) embedder:
After copying the key value, open your terminal and send the following request to create a new embedder:

```sh
curl \
Expand All @@ -68,7 +72,13 @@ curl \
}'
```

Replace `OPEN_AI_API_KEY` with your [OpenAI API key](https://platform.openai.com/api-keys). You may use any key tier for this tutorial, but prefer [Tier 2 keys](https://platform.openai.com/docs/guides/rate-limits/usage-tiers?context=tier-two) for optimal performance in production environments.
Replace `localhost:7700` with the address of your Meilisearch project, and `OPEN_AI_API_KEY` with your [OpenAI API key](https://platform.openai.com/api-keys).

Meilisearch and OpenAI will start processing your documents and updating your index. This process may take a few moments, but once it's done you are ready to perform an AI-powered search.

<Capsule intent="tip">
You may use any key tier for this tutorial, but prefer [Tier 2 keys](https://platform.openai.com/docs/guides/rate-limits/usage-tiers?context=tier-two) for optimal performance in production environments.
</Capsule>

### `documentTemplate`

Expand All @@ -78,11 +88,13 @@ The resulting text is the prompt OpenAI uses to generate document embeddings.

For example, kitchenware documents have three fields: `id`, `name`, and `price`. If your `documentTemplate` is `"An object used in a kitchen named '{{doc.name}}'"`, the text Meilisearch will send to the embedder when indexing the first document is `"An object used in a kitchen named 'Wooden spoon'"`.

For the best results, always provide a `documentTemplate`. Keep your templates short and only include highly relevant information. This ensures optimal indexing performance and search result relevancy.
Keep your templates short and only include highly relevant information. This ensures optimal indexing performance and search result relevancy.

## Perform an AI-powered search

Perform AI-powered searches with `q` and `hybrid` to retrieve search results using the default embedder you configured in the previous step:
As you may have seen in the getting started, to perform basic text searches in Meilisearch you must send a request to the `/search` endpoint. This request usually specifies a `q` parameter, which contains the words you are looking for.

AI-powered searches are very similar to those basic text searches. You must query the `/search` endpoint with a request containing both `q` and `hybrid`. For this tutorial, `hybrid` is an object with a single `embedder` field:

```sh
curl \
Expand All @@ -91,16 +103,21 @@ curl \
--data-binary '{
"q": "kitchen utensils made of wood",
"hybrid": {
"embedder": "openai",
"semanticRatio": 0.7
"embedder": "openai"
}
}'
```

Meilisearch will return a mix of semantic and full-text matches, prioritizing results that match the query's meaning and context. If you want Meilisearch to return more results based on the meaning and context of a search, set `semanticRatio` to a value greater than `0.5`. Setting `semanticRatio` to a value lower than `0.5`, instead, will return more full-text matches.
Meilisearch will return an equal mix of semantic and full-text matches.

## Conclusion

You have seen how to set up and perform AI-powered searches with Meilisearch and OpenAI. For more in-depth information, consult the reference for embedders and the `hybrid` search parameter.
Congratulations. You have created an index, added a small dataset to it, and activated AI-powered search. You then used OpenAI to generate embeddings out of your documents, and performed your first AI-powered search.

## Next steps

Now you have a basic overview of the basic steps required for setting up and performing AI-powered searches, you might want to try and implement this feature in your own application.

For practical information on implementing AI-powered search with other services, consult our [guides section](/guides/ai/openai). There you will find specific instructions for embedders such as [LangChain](/guides/ai/langchain) and [Cloudflare](/guides/ai/cloudflare).

AI-powered search is an experimental Meilisearch feature and is undergoing active development—[join the discussion on GitHub](https://github.com/orgs/meilisearch/discussions/677).
For more in-depth information, consult the API reference for [embedder settings](/reference/api/settings#embedders-experimental) and [the `hybrid` search parameter](/reference/api/search#hybrid-search-experimental).