The filters are not working with metadata that contain a space.

[PAHXO]

Greetings.

Elasticsearch retrievers bm25, Embedding, and the filter retriever. Their filters don't select string metadata that a has space within them.

from haystack import Document
from haystack.components.retrievers import FilterRetriever
from haystack.document_stores.types import DuplicatePolicy
from haystack_integrations.document_stores.elasticsearch import ElasticsearchDocumentStore

docs = [
    Document(content="Python is a popular programming language",
            meta={"about": "Python language", "language": "english"}),

    Document(content="python ist eine beliebte Programmiersprache",
             meta={"about": "Python language", "language": "german"}),
]

document_store = ElasticsearchDocumentStore(hosts=something)
document_store.write_documents(docs, policy=DuplicatePolicy.OVERWRITE)
filters = {
    "operator": "AND",
    "conditions": [
        {"field": "meta.about", "operator": "in", "value": ["Python language"]},
        {"field": "meta.language", "operator": "in", "value": ["english"]}
    ]}
retriever = FilterRetriever(document_store)
result = retriever.run(filters={"field": "about", "operator": "==", "value": "Python language"})

"it does not work, because 'Python Language' has a space within it."

[julian-risch]

Hello @PAHXO thank you for reporting this issue. After having a first look, my understanding is that the issue is caused by using term instead of match on a text field under the hood in the implementation of the integration. The unexpected behavior is not limited to metadata with whitespaces. For example, I have two documents with "Python" as metadata but a filter "Python" will not retrieve them. Only "python". term queries are not analyzed by Elasticsearch, only match queries. That means if the meta field was analyzed, in this example here lowercased, then filters can unexpectedly not match.

haystack-core-integrations/integrations/elasticsearch/src/haystack_integrations/document_stores/elasticsearch/filters.py

Lines 62 to 63 in 9b922d6

	return {"match": {field: {"query": value, "minimum_should_match": "100%"}}}
	return {"term": {field: value}}

I'll continue looking into it.

from haystack import Document
from haystack.components.retrievers import FilterRetriever
from haystack.document_stores.types import DuplicatePolicy
from haystack_integrations.document_stores.elasticsearch import ElasticsearchDocumentStore

docs = [
    Document(content="Python is a popular programming language",
            meta={"about": "Python", "language": "english"}),

    Document(content="python ist eine beliebte Programmiersprache",
             meta={"about": "Python", "language": "german"}),
]

document_store = ElasticsearchDocumentStore(hosts="http://localhost:9200")
document_store.write_documents(docs, policy=DuplicatePolicy.OVERWRITE)
retriever = FilterRetriever(document_store)
result = retriever.run(filters={"field": "about", "operator": "==", "value": "Python"})
print(result["documents"])  # no document retrieved
result = retriever.run(filters={"field": "about", "operator": "==", "value": "python"})
print(result["documents"])  # both documents retrieved

For reference, here is the relevant documentation page from elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html#term-query-notes

[julian-risch]

One way to fix the behavior would be to adjust the mapping that you use for your elasticsearch index so that the metadata in the about field is not analyzed.

[julian-risch]

@PAHXO I opened a PR that fixes this issue: #639
Thanks again for bringing this issue to our attention. 👍

[julian-risch]

@PAHXO The bug is fixed now and there is a new release of elasticsearch-haystack on pypi: https://pypi.org/project/elasticsearch-haystack/
pip install --upgrade elasticsearch-haystack should do the trick.

[PAHXO]

I'll sure try it as soon as I can! Thanks, for taking the time to look into the issue @julian-risch 🫡