Skip to content

Commit

Permalink
New "Engine" section (#2829)
Browse files Browse the repository at this point in the history
  • Loading branch information
guimachiavelli authored May 30, 2024
1 parent fef5639 commit 539ea92
Show file tree
Hide file tree
Showing 18 changed files with 58 additions and 36 deletions.
16 changes: 16 additions & 0 deletions config/redirects.json
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,22 @@
"source": "/docs/learn/async/managing_tasks",
"destination": "/docs/learn/async/working_with_tasks"
},
{
"source": "/docs/learn/advanced/concat",
"destination": "/docs/learn/engine/concat"
},
{
"source": "/docs/learn/advanced/datatypes",
"destination": "/docs/learn/engine/datatypes"
},
{
"source": "/docs/learn/advanced/prefix",
"destination": "/docs/learn/engine/prefix"
},
{
"source": "/docs/learn/advanced/storage",
"destination": "/docs/learn/engine/storage"
},
{
"source": "/docs/learn/cloud/analytics",
"destination": "/docs/learn/analytics/analytics"
Expand Down
38 changes: 22 additions & 16 deletions config/sidebar-learn.json
Original file line number Diff line number Diff line change
Expand Up @@ -332,38 +332,44 @@
]
},
{
"title": "Inner workings",
"slug": "advanced",
"title": "Engine",
"slug": "engine",
"routes": [
{
"source": "learn/inner_workings/indexing.mdx",
"label": "Indexing and performance",
"slug": "indexing"
},
{
"source": "learn/inner_workings/concat.mdx",
"source": "learn/engine/concat.mdx",
"label": "Concatenate and split queries",
"slug": "concat"
},
{
"source": "learn/inner_workings/datatypes.mdx",
"source": "learn/engine/datatypes.mdx",
"label": "Data types",
"slug": "datatypes"
},
{
"source": "learn/inner_workings/known_limitations.mdx",
"label": "Known limitations",
"slug": "known_limitations"
},
{
"source": "learn/inner_workings/prefix.mdx",
"source": "learn/engine/prefix.mdx",
"label": "Prefix search",
"slug": "prefix"
},
{
"source": "learn/inner_workings/storage.mdx",
"source": "learn/engine/storage.mdx",
"label": "Storage",
"slug": "storage"
}
]
},
{
"title": "Inner workings",
"slug": "advanced",
"routes": [
{
"source": "learn/inner_workings/indexing.mdx",
"label": "Indexing and performance",
"slug": "indexing"
},
{
"source": "learn/inner_workings/known_limitations.mdx",
"label": "Known limitations",
"slug": "known_limitations"
},
{
"source": "learn/inner_workings/tokenization.mdx",
Expand Down
2 changes: 1 addition & 1 deletion learn/core_concepts/documents.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ That data is the attribute's **value**. In the case of strings, a value **[can c

Every field has a data type dictated by its value. Every value must be a valid [JSON data type](https://www.w3schools.com/js/js_json_datatypes.asp).

If a field contains an object, Meilisearch flattens it during indexing using dot notation and brings the object's keys and values to the root level of the document itself. This flattened object is only an intermediary representation—you will get the original structure upon search. You can read more about this in our [dedicated guide](/learn/advanced/datatypes#objects).
If a field contains an object, Meilisearch flattens it during indexing using dot notation and brings the object's keys and values to the root level of the document itself. This flattened object is only an intermediary representation—you will get the original structure upon search. You can read more about this in our [dedicated guide](/learn/engine/datatypes#objects).

With [ranking rules](/learn/core_concepts/relevancy#ranking-rules), you can decide which fields are more relevant than others. For example, you may decide recent movies should be more relevant than older ones. You can also designate certain fields as displayed or searchable.

Expand Down
2 changes: 1 addition & 1 deletion learn/deployment/aws.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ Choose the storage **device** and **size** for your Meilisearch instance.

![Page titled 'Step 4: Add Storage'. Text at bottom of screen: Free tier eligible users can get up to 30GB of EBS General Purpose (SSD) or Magnetic storage.](/assets/images/aws/05.storage.png)

The amount of storage space required can [vary drastically](/learn/advanced/storage#measured-disk-usage) depending on the data you plan to index. In this example, we will use 25 GiB, which is more than enough for most small datasets (< 1 million documents). We have the "Volume Type" set to "General Purpose SSD (gp2)".
The amount of storage space required can [vary drastically](/learn/engine/storage#measured-disk-usage) depending on the data you plan to index. In this example, we will use 25 GiB, which is more than enough for most small datasets (< 1 million documents). We have the "Volume Type" set to "General Purpose SSD (gp2)".

When you're ready, click on "Next: Add Tags" to continue.

Expand Down
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ Will be processed as if all elements were arranged at the same level:
"Bruce Willis. Vin Diesel. Kung Fu Panda."
```

Once the above array has been flattened, it will be parsed exactly as explained in the [string example](/learn/advanced/datatypes#examples).
Once the above array has been flattened, it will be parsed exactly as explained in the [string example](/learn/engine/datatypes#examples).

## Objects

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -40,5 +40,5 @@ response:
Meilisearch also handles typos while performing the prefix search. You can [read more about the typo rules on the dedicated page](/learn/configuration/typo_tolerance).

<Capsule intent="tip">
We also [apply splitting and concatenating on search queries](/learn/advanced/concat).
We also [apply splitting and concatenating on search queries](/learn/engine/concat).
</Capsule>
File renamed without changes.
2 changes: 1 addition & 1 deletion learn/fine_tuning_results/faceted_search.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -182,7 +182,7 @@ The above code sample sorts the `genres` facet by descending value count. If you

## Searching facet values

Once you have configured [filterableAttributes](/reference/api/settings#filterable-attributes), you can also search for facet values with the [facet search endpoint](/reference/api/facet_search). Facet search supports [prefix search](/learn/advanced/prefix) and [typo tolerance](/learn/configuration/typo_tolerance).
Once you have configured [filterableAttributes](/reference/api/settings#filterable-attributes), you can also search for facet values with the [facet search endpoint](/reference/api/facet_search). Facet search supports [prefix search](/learn/engine/prefix) and [typo tolerance](/learn/configuration/typo_tolerance).

<Capsule intent="note" title="Facet search on numbers">
Meilisearch does not support facet search on numbers. Convert numeric facets to strings to make them searchable.
Expand Down
2 changes: 1 addition & 1 deletion learn/fine_tuning_results/filtering.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -387,4 +387,4 @@ Use dot notation to filter results based on a document's nested fields. The foll

<CodeSamples id="filtering_guide_nested_1" />

[You can read more about nested fields in our guide on data types.](/learn/advanced/datatypes)
[You can read more about nested fields in our guide on data types.](/learn/engine/datatypes)
2 changes: 1 addition & 1 deletion learn/index_settings/synonyms.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Words set as synonyms won't always return the same results. With the default set

<CodeSamples id="synonyms_guide_1" />

With the new settings, searching for `great` returns 595 results and `fantastic` returns 423 results. This is due to various factors like [typos](/learn/configuration/typo_tolerance#minwordsizefortypos) and [splitting the query](/learn/advanced/concat#split-queries) to find relevant documents. The search for `great` will allow only one typo (for example, `create`) and take into account all variations of `great` (for instance, `greatest`) along with `fantastic`.
With the new settings, searching for `great` returns 595 results and `fantastic` returns 423 results. This is due to various factors like [typos](/learn/configuration/typo_tolerance#minwordsizefortypos) and [splitting the query](/learn/engine/concat#split-queries) to find relevant documents. The search for `great` will allow only one typo (for example, `create`) and take into account all variations of `great` (for instance, `greatest`) along with `fantastic`.

<Capsule intent="warning">
The number of search results may vary depending on changes to the `movies` dataset.
Expand Down
2 changes: 1 addition & 1 deletion learn/index_settings/typo_tolerance.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ We recommend keeping the value of `oneTypo` between `2` and `8` and the value of
Meilisearch considers a typo on a query's first character as two typos.

**Concatenation**
When considering possible candidates for typo tolerance, Meilisearch will concatenate multiple search terms separated by a [space separator](/learn/advanced/datatypes#string). This is treated as one typo. For example, a search for `any way` would match documents containing `anyway`.
When considering possible candidates for typo tolerance, Meilisearch will concatenate multiple search terms separated by a [space separator](/learn/engine/datatypes#string). This is treated as one typo. For example, a search for `any way` would match documents containing `anyway`.

For more about typo calculations, [see below](#understanding-typo-calculations).
</Capsule>
Expand Down
6 changes: 3 additions & 3 deletions learn/inner_workings/known_limitations.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: Meilisearch has a number of known limitations. These are hard limit

# Known limitations

Currently, Meilisearch has a number of known limitations. Some of these limitations are the result of intentional design trade-offs, while others can be attributed to [LMDB](/learn/advanced/storage), the key-value store that Meilisearch uses under the hood.
Currently, Meilisearch has a number of known limitations. Some of these limitations are the result of intentional design trade-offs, while others can be attributed to [LMDB](/learn/engine/storage), the key-value store that Meilisearch uses under the hood.

This guide covers hard limits that cannot be altered. Meilisearch also has some default limits that _can_ be changed, such as a [default payload limit of 100MB](/learn/configuration/instance_options#payload-limit-size) and a [default search limit of 20 hits](/reference/api/search#limit).

Expand Down Expand Up @@ -37,7 +37,7 @@ If your query is `Hello, World`:
- `World` takes the position `9` of the attribute

<Capsule intent="note">
`,` takes 8 positions as it is a hard separator. You can read more about word separators in our [article about data types](/learn/advanced/datatypes#string).
`,` takes 8 positions as it is a hard separator. You can read more about word separators in our [article about data types](/learn/engine/datatypes#string).
</Capsule>

If your query is `Hello - World`:
Expand All @@ -47,7 +47,7 @@ If your query is `Hello - World`:
- `World` takes the position `2` of the attribute

<Capsule intent="note">
`-` takes 1 position as it is a soft separator. You can read more about word separators in our [article about data types](/learn/advanced/datatypes#string).
`-` takes 1 position as it is a soft separator. You can read more about word separators in our [article about data types](/learn/engine/datatypes#string).
</Capsule>

## Maximum number of attributes per document
Expand Down
4 changes: 2 additions & 2 deletions learn/what_is_meilisearch/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ _Meilisearch helps you find where to watch a movie at [where2watch.meilisearch.c
## Features

- **Blazing fast**: Answers in less than 50 milliseconds
- **Search as you type**: Results are updated on each keystroke using [prefix-search](/learn/advanced/prefix#prefix-search)
- **Search as you type**: Results are updated on each keystroke using [prefix-search](/learn/engine/prefix#prefix-search)
- [Typo tolerance](/learn/configuration/typo_tolerance): Get relevant matches even when queries contain typos and misspellings
- [Comprehensive language support](/learn/what_is_meilisearch/language): Optimized support for **Chinese, Japanese, Hebrew, and languages using the Latin alphabet**
- **Returns the whole document**: The entire document is returned upon search
Expand All @@ -45,7 +45,7 @@ Meilisearch's ease-of-use goes hand-in-hand with ultra relevant search results.

Meilisearch should **not be your main data store**. It is a search engine, not a database. Meilisearch should contain only the data you want your users to search through. If you must add data that is irrelevant to search, be sure to [make those fields non-searchable](/learn/configuration/displayed_searchable_attributes#searchable-fields) to improve relevancy and response time.

Meilisearch provides an intuitive search-as-you-type experience with response times under 50 milliseconds, no matter whether you are developing a site or an app. This helps end-users find what they are looking for quickly and efficiently. To make that happen, we are fully committed to the philosophy of [prefix search](/learn/advanced/prefix).
Meilisearch provides an intuitive search-as-you-type experience with response times under 50 milliseconds, no matter whether you are developing a site or an app. This helps end-users find what they are looking for quickly and efficiently. To make that happen, we are fully committed to the philosophy of [prefix search](/learn/engine/prefix).

## Give it a try!

Expand Down
2 changes: 1 addition & 1 deletion reference/api/facet_search.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ description: The /facet-search route allows you to search for facet values.

# Facet search

The `/facet-search` route allows you to search for facet values. Facet search supports [prefix search](/learn/advanced/prefix) and [typo tolerance](/learn/configuration/typo_tolerance). The returned hits are sorted lexicographically in ascending order. You can configure how facets are sorted using the [`sortFacetValuesBy`](/reference/api/settings#faceting-object) property of the `faceting` index settings.
The `/facet-search` route allows you to search for facet values. Facet search supports [prefix search](/learn/engine/prefix) and [typo tolerance](/learn/configuration/typo_tolerance). The returned hits are sorted lexicographically in ascending order. You can configure how facets are sorted using the [`sortFacetValuesBy`](/reference/api/settings#faceting-object) property of the `faceting` index settings.

<Capsule intent="note" title="Facet search on numbers">
Meilisearch does not support facet search on numbers. Convert numeric facets to strings to make them searchable.
Expand Down
2 changes: 1 addition & 1 deletion reference/api/search.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@ Placeholder search is particularly useful when building a [faceted search interf

If you enclose search terms in double quotes (`"`), Meilisearch will only return documents containing those terms in the order they were given. This is called a **phrase search**.

Phrase searches are case-insensitive and ignore [soft separators such as `-`, `,`, and `:`](/learn/advanced/datatypes). Using a hard separator within a phrase search effectively splits it into multiple separate phrase searches: `"Octavia.Butler"` will return the same results as `"Octavia" "Butler"`.
Phrase searches are case-insensitive and ignore [soft separators such as `-`, `,`, and `:`](/learn/engine/datatypes). Using a hard separator within a phrase search effectively splits it into multiple separate phrase searches: `"Octavia.Butler"` will return the same results as `"Octavia" "Butler"`.

You can combine phrase search and normal queries in a single search request. In this case, Meilisearch will first fetch all documents with exact matches to the given phrase(s), and [then proceed with its default behavior](/learn/core_concepts/relevancy).

Expand Down
6 changes: 3 additions & 3 deletions reference/api/settings.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -1358,7 +1358,7 @@ Reset an index's search cutoff value to its default value, `null`. This translat

Configure strings as custom separator tokens indicating where a word ends and begins.

Tokens in the `separatorTokens` list are added on top of [Meilisearch's default list of separators](/learn/advanced/datatypes#string). To remove separators from the default list, use [the `nonSeparatorTokens` setting](#non-separator-tokens).
Tokens in the `separatorTokens` list are added on top of [Meilisearch's default list of separators](/learn/engine/datatypes#string). To remove separators from the default list, use [the `nonSeparatorTokens` setting](#non-separator-tokens).

### Get separator tokens

Expand Down Expand Up @@ -1452,7 +1452,7 @@ Use this `taskUid` to get more details on [the status of the task](/reference/ap

## Non-separator tokens

Remove tokens from Meilisearch's default [list of word separators](/learn/advanced/datatypes#string).
Remove tokens from Meilisearch's default [list of word separators](/learn/engine/datatypes#string).

### Get non-separator tokens

Expand Down Expand Up @@ -1494,7 +1494,7 @@ Update an index's list of non-separator tokens.
["@", "#"]
```

An array of strings, with each string indicating a token present in [list of word separators](/learn/advanced/datatypes#string).
An array of strings, with each string indicating a token present in [list of word separators](/learn/engine/datatypes#string).

#### Example

Expand Down
4 changes: 2 additions & 2 deletions resources/faq.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -113,12 +113,12 @@ Beware heavily multi-lingual datasets and datasets with many unique words, such

### Search speed

Because Meilisearch uses a [memory map](/learn/advanced/storage#lmdb), **search speed is based on the ratio between RAM and database size**. In other words:
Because Meilisearch uses a [memory map](/learn/engine/storage#lmdb), **search speed is based on the ratio between RAM and database size**. In other words:

- A big database + a small amount of RAM => slow search
- A small database + tons of RAM => lightning fast search

Meilisearch also uses disk space as [virtual memory](/learn/advanced/storage#memory-usage). This disk space does not correspond to database size; rather, it provides speed and flexibility to the engine by allowing it to go over the limits of physical RAM.
Meilisearch also uses disk space as [virtual memory](/learn/engine/storage#memory-usage). This disk space does not correspond to database size; rather, it provides speed and flexibility to the engine by allowing it to go over the limits of physical RAM.

At this time, the number of CPU cores has no direct impact on index or search speed. However, **the more cores you provide to the engine, the more search queries it will be able to process at the same time**.

Expand Down

0 comments on commit 539ea92

Please sign in to comment.