search: demonstrate "keyword search" via zoekt

[keegancsmith]

We want to bring code search closer to the "keyword search" style that users are used to, where queries are broken up into individual terms that can match broadly across the file name, contents, and symbols.

This issue tracks us demonstrating the value of this approach. A successful outcome is buy in to further invest in making this a broad change across our product.

Ranking

The work done here benefits all customers even if we don't ship a change to our syntax. This is were the majority of our work has been concentrated so far.

Most of these tasks are not on the critical path for closing this issue, but we are capturing all the ideas so far.

• much tweaking of ranking in zoekt (just capturing all the work done before this issue)
• ranking integration test https://github.com/sourcegraph/sourcegraph/issues/57666
• ranking: introduce approximate file ranks https://github.com/sourcegraph/sourcegraph/issues/57950
• ranking: improve result diversity https://github.com/sourcegraph/sourcegraph/issues/58057
• Search: better search for docs files https://github.com/sourcegraph/sourcegraph/issues/58059
• ranking: boost atoms on same line https://github.com/sourcegraph/sourcegraph/issues/58195
• ranking: boost filename matches https://github.com/sourcegraph/sourcegraph/issues/58196
• ranking: boost document when all atoms have good quality matches https://github.com/sourcegraph/sourcegraph/issues/58197

Feedback

This is on the critical path to showing value and starting on the next phase of work.

• lookup smart search telemetry
• sqs style loom for "make search better"
• feature flag to make smart search use new keyword search https://github.com/sourcegraph/sourcegraph/issues/58125
• enable on s2

Symbols

A key part of what we like when using google code search is the symbols autocomplete UX.

• prototype keyword search for symbols in autocomplete https://github.com/sourcegraph/sourcegraph/issues/58198

RFC: Search Platform Q4 Plan - Google Docs

/cc @sourcegraph/search-platform/search-platform

[keegancsmith]

I've broken these up into seperate tracking issues. Additionally the single task for symbols will live by itself for now.

• https://github.com/sourcegraph/sourcegraph/issues/58234
• https://github.com/sourcegraph/sourcegraph/issues/58235