[FEATURE] Enhanced adaptive token pruning for neural sparse search

[martin-gaievski]

Enhance the basic token pruning mechanism with adaptive capabilities to optimize storage efficiency while preserving search quality.

The basic token pruning (covered in #946) uses fixed thresholds and limits. This enhancement proposes adaptive mechanisms that automatically adjust pruning parameters based on content characteristics and quality metrics.

Proposed Functionality

1. Dynamic Threshold Adjustment

PUT _neural/sparse_model/config
{
  "name": "adaptive_pruning_config",
  "pruning": {
    "mode": "adaptive",
    "quality_target": 0.95,
    "token_budget": {
      "min": 50,
      "max": 500
    },
    "weight_threshold": {
      "base": 0.001,
      "adaptive": true
    }
  }
}

2. Quality Preservation

• Monitor quality metrics during pruning
• Adjust parameters to maintain specified quality target
• Support different quality metrics (precision, recall, MRR)

3. Token Importance Analysis

• Analyze semantic importance of tokens
• Consider token relationships
• Preserve critical tokens for search quality

If implemented, solution promises following benefits:

• Improved storage efficiency
• Better search quality preservation
• Automatic adaptation to content
• Reduced manual configuration

As of now I do see following dependencies

• requires basic token pruning ([Enhancement] Implement pruning for neural sparse search #988)
• neural sparse search functionality
• metrics collection framework, need to implement stats collection for pruning metrics (new component), can leverage OpenSearch core stats functionality, will need to add pruning-specific metrics collection