QA2 - ES relevance improvements and query generation refactor #411
Conversation
Incorporate latest queries from Mike.
Also stubs in some keyword matching for standard_number searches
Also add relevance report to debug response
…' into qa-node20-es-tuning
- named queries - min_should_match - more targetted standard_numbers searches - include some contrib matchers in 'all' search
lib/elasticsearch/config.js
Outdated
| fields: null | ||
| }, | ||
| standard_number: { | ||
| fields: null |
There was a problem hiding this comment.
why are these fields null?
There was a problem hiding this comment.
Historically, all search-scopes were configured in this const - with an array of fields to use in the query_string/multi-match query. With ES search improvements, two of these search-scopes don't use multi-match at all, so there's no need for a central config. However, this const remains a central, authoritative config for what search-scopes are available. I think future refactoring may change this, but for now I've just removed the fields prop from these search-scopes and added a comment to clarify
swagger.v1.1.x.json
Outdated
There was a problem hiding this comment.
thanks for backfilling this documentation.
test/elastic-query-builder.test.js
Outdated
There was a problem hiding this comment.
These are great and actually go a long way in documenting how we're using ES. 🤘🏻
Updates: - Add search_scope=journal_title (to ease testing and because it really ought to be implemented in the discovery-api, not the front-end - Match on shelfmark lowercased - Allow smart-quotes in fully-quoted searches - Remove boost=1 on quoted creator/contrib phrase matching - Match on new idIsbn.clean and idIssn.clean (added 2024-10-17) - Updates qa index to 2024-10-16 (added shelfMark.keywordLowercased)
| /** | ||
| * Examine request for user-filters. When found, add them to query. | ||
| */ | ||
| applyFilters () { |
There was a problem hiding this comment.
Can this method be broken down any more?
There was a problem hiding this comment.
I agree it should be, but this specific code is unchanged from previous form (just moved). So I'd like to keep that as a project for a future refactor if that's alright.
This is a large PR. It includes two major efforts:
If you want to see the result of the first effort, this diff shows just the parts that are mainly about updating the client to work with the v8 index, adding an assessment script, and doing a rough implementation of the updated queries.
This PR builds on top of that first effort to also attempt to clarify search query generation. It adds simple tools for query construction so that the important decisions about matching are foregrounded, with the fiddly ES query generation happening in the background. The aim is to make it much clearer what matches are active for a given query and scope and make it clear how to change things.
Also: This includes a new
buildingLocationagg and support forfilters[buildingLocation]=...https://newyorkpubliclibrary.atlassian.net/browse/SCC-4312