SymlinkAndCleanup - make cleanup asynchronous and

[ctheune]

At least one customer has been bitten by the fact that cleanups may take a very long time.

The cleanup IMHO should spawn a systemd-run job for the rm -rf and limit IOPS to something like 100 iops/s to ensure the cleanup job doesn't overload the system.

[EdwardBock]

Any news here?

[ctheune]

@elikoga could we prioritize this?

[elikoga]

Sure. As far as I undestand, we want to make what happens in this loop:

batou_ext/src/batou_ext/file.py

Lines 99 to 108 in 1cb0884

	for el in self._list_removals():
	batou.output.annotate("Removing: {}".format(el))
	try:
	if os.path.isdir(el):
	shutil.rmtree(el)
	else:
	os.remove(el)
	except OSError as e:
	batou.output.error(f'Failed to remove "{el}": {e.strerror}')
	pass

            for el in self._list_removals():
                batou.output.annotate("Removing: {}".format(el))
                try:
                    if os.path.isdir(el):
                        shutil.rmtree(el)
                    else:
                        os.remove(el)
                except OSError as e:
                    batou.output.error(f'Failed to remove "{el}": {e.strerror}')
                    pass

Be defered to running in the background using systemd-run

[ctheune]

Yup.

[EdwardBock]

When can we use this?

[EdwardBock]

Any news here?

[EdwardBock]

I'm back. Any news?

[elikoga]

Yes, the use_systemd_run_async_cleanup option on SymlinkAndCleanup added in #159 has this behaviour

[EdwardBock]

We tested this. This was not working. @zagy knows more about the problems as he was talking to my colleague.