Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Indexer performance Improvement #3414

Open
t83714 opened this issue Oct 13, 2022 · 0 comments
Open

Indexer performance Improvement #3414

t83714 opened this issue Oct 13, 2022 · 0 comments

Comments

@t83714
Copy link
Contributor

t83714 commented Oct 13, 2022

Indexer Performance Improvement

At this moment, the indexer expects the registry API to search for all relevant events through aspect reference links and send over a list of updated datasets rather than raw events.

This might lay too much pressure on the registry API & database.

Instead, we should:

  • Indexer listens to the raw events instead, and the registry will only send over raw events
  • The indexer should combine the raw events (e.g. events for the same record in the same page only trigger one index action) and re-construct the required dataset data from the raw events.
  • Allow webhook to specify desired event page size in config
@t83714 t83714 added this to the v2.2.0 milestone Oct 13, 2022
@t83714 t83714 modified the milestones: v2.3.0, v2.2.2 Jan 31, 2023
@t83714 t83714 modified the milestones: v2.2.6, v2.2.5 May 22, 2023
@t83714 t83714 moved this to Todo in Magda dev plan Jul 13, 2023
@t83714 t83714 modified the milestones: v2.2.5, v2.3.0 Aug 2, 2023
@t83714 t83714 modified the milestones: v2.3.0, v2.3.1 Aug 29, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Todo
Development

No branches or pull requests

1 participant