Write to standard out regularly instead of just the end of the process

[SebastianArcq]

When processing big tables there can be a long time when Sling sits on "INF streaming data". Only when everything is streamed then the output is written to the terminal. So in practice there can be minutes / hours where nothing seems to be happening, and at the very end the entire log (e.g., 7m57s 22,083,410 46359 r/s) is dumped.

For my own projects I have successfully followed the suggestion from this Stackoverflow article, i.e., to call sys.stdout.flush() regularly - maybe that's the solution here as well!

[flarco]

Ah I see, can you confirm that it works as expected using the Go binary directly (non-python wrapper)?
It should flush as you've described: https://github.com/slingdata-io/sling-cli/blob/main/core/sling/task_run_write.go#L90

[SebastianArcq]

Yes, if I use $ sling run directly the output works as expected:

With the Python wrapper it looks like this (with the whole log appearing at once at the end):

[flarco]

Ah i see, yeah It's the progress update that writes and overwrites the text (in-place) without creating a new line, I'm not too sure how to do this correctly in python since it's taking the output of the child process (go binary).
Do you mind opening a PR and testing locally? I'm pretty busy these days, so I may not get to this soon

[SebastianArcq]

Appreciate the invitation, but having never contributed to a public repo I don't feel experienced enough to work on your project.

However, this would be the correct way in Python to overwrite the output in-place:

import time

for n in range(1, 5):
    print(n, end='\r')
    time.sleep(1) 
    n += 1