[FEATURE] Multi-model sparse search ensembling

[martin-gaievski]

Add support for ensemble-based neural sparse search that combines results from multiple sparse models to improve search quality and robustness.

Motivation

Research shows that ensemble of sparse retrievers provides:

• Better generalization across different query types
• Improved robustness to different document types
• Better effectiveness-efficiency trade-off

Key research:

• Ensemble of BERT-based Sparse Retrievers
• Sparse-Dense Hybrid Representations

Proposed Functionality

Ensemble Configuration

PUT _neural/sparse_model/ensemble
{
  "name": "sparse_ensemble",
  "models": [
    {
      "model_id": "splade_v2",
      "weight": 0.6
    },
    {
      "model_id": "unicoil",
      "weight": 0.4
    }
  ],
  "combination_method": "weighted_sum",  // or "max", "mean"
  "cache_policy": {
    "enabled": true,
    "ttl": "1h"
  }
}

Search API

GET my-index/_search
{
  "query": {
    "neural_sparse_ensemble": {
      "query_text": "search query",
      "ensemble_id": "sparse_ensemble",
      "k": 100
    }
  }
}

As shown in the configuration section, we can use caching to improve remote call latency.

[heemin32]

Does this imply that only the query embedding varies across models? Shouldn't the index embedding also differ for each model?

[martin-gaievski]

good question, yes we need to include the ingestion in the scope. Embeddings for multiple models can be generated during ingestion.