Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Feeder logic #6

Open
Mikts opened this issue Oct 21, 2020 · 0 comments
Open

Add Feeder logic #6

Mikts opened this issue Oct 21, 2020 · 0 comments
Assignees
@kostaskol
Labels
enhancement New feature or request feeder

Comments

@Mikts
Copy link
Contributor

Mikts commented Oct 21, 2020

No description provided.

@Mikts Mikts changed the title Create feeder logic Add feeder logic Oct 21, 2020
@Mikts Mikts changed the title Add feeder logic Add Feeder logic Oct 21, 2020
@Mikts Mikts self-assigned this Oct 21, 2020
@kostaskol kostaskol mentioned this issue Dec 30, 2020
Open
@kostaskol kostaskol assigned kostaskol and unassigned Mikts Jan 7, 2021
@kostaskol kostaskol added enhancement New feature or request feeder labels Jan 7, 2021
kostaskol added a commit that referenced this issue Sep 20, 2023
@kostaskol
Add support of Redis configuration
3db4b90 
The relevant options have also been added to `config.tpl.yaml`

Ref #6
kostaskol added a commit that referenced this issue Sep 20, 2023
@kostaskol
Introduce Event struct
1933a99 
This struct contains methods for deserializing JSON strings into Events,
as well as inserting/retrieving Events from the DB

Ref #6
kostaskol added a commit that referenced this issue Sep 20, 2023
@kostaskol
Introduce Feeder
0c12ec8 
The function Feeder.start_feeders spawns N threads, each of which
immediately blocks on a Redis topic, waiting to pop the first available
event. When an event is popped, it is deserialized into an Event object
and written into a crossbeam channel, on the read-end of which exist
processor threads. The number of feeder threads to be spawned is
configurable through `workers:feeders` in `config.yaml`

Ref #6
kostaskol added a commit that referenced this issue Sep 20, 2023
@kostaskol
Add a trivial docker-compose
69c9995 
It does not create a container for the processor itself, simply the
required services (Postgres, Redis)

Ref #6
kostaskol added a commit that referenced this issue Sep 20, 2023
@kostaskol
Spawn feeder threads on process startup
0f3bd89 
Ref #6
kostaskol added a commit that referenced this issue Sep 23, 2023
@kostaskol
Add support of Redis configuration
edf5528 
The relevant options have also been added to `config.tpl.yaml`

Ref #6
kostaskol added a commit that referenced this issue Sep 23, 2023
@kostaskol
Introduce Event struct
3763c85 
This struct contains methods for deserializing JSON strings into Events,
as well as inserting/retrieving Events from the DB

Ref #6
kostaskol added a commit that referenced this issue Sep 23, 2023
@kostaskol
Introduce Feeder
4073683 
The function Feeder.start_feeders spawns N threads, each of which
immediately blocks on a Redis topic, waiting to pop the first available
event. When an event is popped, it is deserialized into an Event object
and written into a crossbeam channel, on the read-end of which exist
processor threads. The number of feeder threads to be spawned is
configurable through `workers:feeders` in `config.yaml`

Ref #6
kostaskol added a commit that referenced this issue Sep 23, 2023
@kostaskol
Add a trivial docker-compose
234027e 
It does not create a container for the processor itself, simply the
required services (Postgres, Redis)

Ref #6
kostaskol added a commit that referenced this issue Sep 23, 2023
@kostaskol
Spawn feeder threads on process startup
1698c8d 
Ref #6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kostaskol kostaskol

Labels
enhancement New feature or request feeder
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Mikts @kostaskol