Reindex after relation embedder to lambda implementation and infra complete

[kenoir]

Reindex to serve the dual purpose of populating aggregations for concept work and testing new relation embedder implementation.

Related:

• Update catalogue-api aggregations to use concept ID filters for subjects & contributors #5825
• Relation-embedder ECS to lambda infra catalogue-pipeline#2770
• Restructure Relation Embedder to allow different calling conventions catalogue-pipeline#2772

Checklist to completion:

• Merge @agnes's infra change: Relation embedder lambda infra build catalogue-pipeline#2771
• Run terraform apply on main in 2024-11-18
• Merge @paul Butcher's embedder change: Embedder reshuffle catalogue-pipeline#2777
• Merge @robert Kenny rules for requesting change: Update rules for requesting (03/11/24) catalogue-pipeline#2778
• Determine and set queue configuration for new relation embedder
• Allow reading of secrets to environment variable for containerised lambdas
• Clear denormalised and works/images indexed indices
  The ES delete by query API might be useful here.
• Run a partial reindex of a few thousand documents (that include images)
  • Check that image vector changes worked as expected
  • Check that nested aggregations changes worked as expected
  • Check that the lambda relation_embedder subsystems processed messages as expected
  • Run an analysis of records in new vs old denormalised indices for relation embedder
• Checkpoint - are we happy with everything so far?
• Run a full reindex
• Test and deploy catalogue API change: Aggregation updates catalogue-api#832