Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce the size of search results #1151

Open
hancush opened this issue Jul 24, 2024 · 6 comments
Open

Reduce the size of search results #1151

hancush opened this issue Jul 24, 2024 · 6 comments
Assignees

Comments

@hancush
Copy link
Collaborator

hancush commented Jul 24, 2024

One search result can be several MB in size. That can really add up! I suspect the biggest culprits are the full text fields. Figure out a way to index that information, but not return it in the results. Looks like ElasticSearch provides for this, if django-haystack does not: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-fields.html

@hancush
Copy link
Collaborator Author

hancush commented Jul 24, 2024

Related to #1145, #718

@hancush
Copy link
Collaborator Author

hancush commented Jul 24, 2024

Also related to efforts to reduce page load time.

@antidipyramid
Copy link
Collaborator

It doesn't look like there's a way to add a fields parameter to the Elastic query with Haystack. We can try extending Haystack's Elasticsearch7SearchBackend class.

@antidipyramid
Copy link
Collaborator

antidipyramid commented Jul 29, 2024

@hancush I tested #1152 on staging and found that it didn't lower the size of the index by very much. I've read that elastic uses a compression algorithm by default in indexes, so that may be why.

It seemed to reduce memory usage when searching, but not by much either.

@hancush
Copy link
Collaborator Author

hancush commented Jul 29, 2024

Hmm, bummer, @antidipyramid.

@antidipyramid
Copy link
Collaborator

antidipyramid commented Aug 2, 2024

@hancush I was able to get Elastic to exclude the text fields of bills from search results by customizing Haystack's Elastic backend. I deployed that branch and found that, again, it reduced memory usage somewhat but not enough to be considered a fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

No branches or pull requests

2 participants