[Elasticsearch] fix: Filters not working with metadata that contain a space or capitalization

[julian-risch]

Related Issues

fixes #616

So far, metadata fields were analyzed during indexing, for example lowercased because the elasticsearch integration did not define a different mapping for them. When users tried to filter based on metadata fields by a string with uppercase letter or with whitespace, it is not automatically analyzed. Therefore the filter did not match.

Proposed Changes

• Add dynamic template that defines any additional index fields as mapping type keyword, which means they are not analyzed. This is the desired behavior. Same behavior as in other document stores such as OpenSearchDocumentStore.
• Explicitly defined the mapping type of the content field as text, which means it is analyzed

How did you test it?

I tested Elasticsearch and OpenSearch locally.

from haystack import Document
from haystack.components.retrievers import FilterRetriever
from haystack.document_stores.types import DuplicatePolicy
from haystack_integrations.document_stores.elasticsearch import ElasticsearchDocumentStore

docs = [
    Document(content="Python is a popular programming language",
            meta={"about": "Python", "language": "english"}),

    Document(content="python ist eine beliebte Programmiersprache",
             meta={"about": "Python", "language": "german"}),
]

document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200")
document_store.write_documents(docs, policy=DuplicatePolicy.OVERWRITE)
retriever = FilterRetriever(document_store)
result = retriever.run(filters={"field": "about", "operator": "==", "value": "Python"})
print(result["documents"])  # now returns documents but previously did not
result = retriever.run(filters={"field": "about", "operator": "==", "value": "python"})
print(result["documents"]). # now does not return documents but previously did

Notes for the reviewer

• I confirmed that the issue does not occur with the OpenSearch integration.
• I didn't add any integration tests. If we decide we need to add them, we should add similar new tests to OpenSearch too. I wouldn't add tests. They would test Elasticsearch more than the Haystack integration.

[masci]

Left a comment but I might be mistaken, feel free to merge

[masci]

Nit: not sure you need this line, the template will run on any string field anyways

[julian-risch]

Yes, that's correct. It's not needed and I removed it now. Thank you!