Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] feat: copy log to central system #441

Open
wants to merge 4 commits into
base: transition-to-runkit
Choose a base branch
from
Open

[WIP] feat: copy log to central system #441

wants to merge 4 commits into from
+270 −14

Conversation

eduardacoppo
Copy link

No description provided.

@eduardacoppo eduardacoppo changed the base branch from master to transition-to-runkit December 4, 2024 12:23
@eduardacoppo eduardacoppo changed the title feat: copy log to central system [WIP] feat: copy log to central system Dec 4, 2024
@eduardacoppo eduardacoppo force-pushed the copy_log_to_central_system branch 2 times, most recently from 9d849f6 to d86cc46 Compare December 4, 2024 19:03
doudou
doudou reviewed Dec 5, 2024
View reviewed changes
lib/syskit/cli/log_runtime_archive.rb Outdated Show resolved Hide resolved
lib/syskit/cli/log_runtime_archive.rb Outdated Show resolved Hide resolved
lib/syskit/roby_app/plugin.rb Outdated Show resolved Hide resolved
@eduardacoppo eduardacoppo force-pushed the copy_log_to_central_system branch 3 times, most recently from b056ea9 to 34a737c Compare December 5, 2024 16:29
@eduardacoppo eduardacoppo force-pushed the copy_log_to_central_system branch 2 times, most recently from a23c520 to c101f33 Compare December 6, 2024 12:52
jhonasiv
jhonasiv reviewed Dec 6, 2024
View reviewed changes
lib/syskit/cli/log_runtime_archive.rb Outdated Show resolved Hide resolved
lib/syskit/cli/log_runtime_archive.rb Outdated Show resolved Hide resolved
@eduardacoppo eduardacoppo force-pushed the copy_log_to_central_system branch 2 times, most recently from be62e5a to 49bb197 Compare December 6, 2024 15:04
wvmcastro
wvmcastro reviewed Dec 6, 2024
View reviewed changes
Copy link

@wvmcastro wvmcastro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure you got the change between the current log transfer and archiving schema and the new one we are moving towards too. Just to be sure we are all at the same page

Current: There is a central node (LogTransferManager) that runs in the master machine and periodically tells the slave machines process servers to send their logs to a target dir. Then, another tool (LogRuntimeArchive) gets the data accumulated in this target folder and does the archiving (compress and move to another storage)

New: The LogTransferManager is taken out the dynamic, and now the slave machines will have a client that periodically sends its logs to a LogTransferServer at the master machine. Then, the master machine LogRuntimeArchive continues to do what it currently does

Given that, I think a great way to test your implementation is to setup a server in your machine and a use a client to get files from on folder to another

lib/syskit/cli/log_runtime_archive_main.rb Outdated Show resolved Hide resolved
@eduardacoppo
Copy link
Author

I am not sure you got the change between the current log transfer and archiving schema and the new one we are moving towards too. Just to be sure we are all at the same page

Current: There is a central node (LogTransferManager) that runs in the master machine and periodically tells the slave machines process servers to send their logs to a target dir. Then, another tool (LogRuntimeArchive) gets the data accumulated in this target folder and does the archiving (compress and move to another storage)

New: The LogTransferManager is taken out the dynamic, and now the slave machines will have a client that periodically sends its logs to a LogTransferServer at the master machine. Then, the master machine LogRuntimeArchive continues to do what it currently does

Given that, I think a great way to test your implementation is to setup a server in your machine and a use a client to get files from on folder to another

This is clear to me. The biggest difficulty lies in implementing it.

@eduardacoppo eduardacoppo force-pushed the copy_log_to_central_system branch from 49bb197 to 987a361 Compare December 6, 2024 17:57
@doudou
Copy link
Member

doudou commented Dec 6, 2024

I am not sure you got the change between the current log transfer and archiving schema and the new one we are moving towards too. Just to be sure we are all at the same page

Current: There is a central node (LogTransferManager) that runs in the master machine and periodically tells the slave machines process servers to send their logs to a target dir. Then, another tool (LogRuntimeArchive) gets the data accumulated in this target folder and does the archiving (compress and move to another storage)

New: The LogTransferManager is taken out the dynamic, and now the slave machines will have a client that periodically sends its logs to a LogTransferServer at the master machine. Then, the master machine LogRuntimeArchive continues to do what it currently does

Given that, I think a great way to test your implementation is to setup a server in your machine and a use a client to get files from on folder to another

There are unit tests for the current log transfer functionality. There should be unit tests for the new functionality as well.

@eduardacoppo eduardacoppo force-pushed the copy_log_to_central_system branch 4 times, most recently from cc2964c to d9bebde Compare December 9, 2024 17:47
@eduardacoppo eduardacoppo force-pushed the copy_log_to_central_system branch from d9bebde to 2d42928 Compare December 9, 2024 17:49
@eduardacoppo eduardacoppo force-pushed the copy_log_to_central_system branch from 2d42928 to daf2442 Compare December 9, 2024 20:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jhonasiv jhonasiv jhonasiv left review comments

@wvmcastro wvmcastro wvmcastro left review comments

@doudou doudou doudou left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@eduardacoppo @doudou @wvmcastro @jhonasiv