Hooks into the DNF HTTP Requests

[andrewegel]

Background:

My ultimate goal is to implement this yum plugin for DNF: https://github.com/seporaitis/yum-s3-iam for EL8.

TL;DR: This yum plugin "hooks" into yum's download process to add AWS Sigv4 Auth Headers:

https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html

So you can have your rhel/centos system download from a private S3 bucket as a yum repo. AWS IAM creds are either provided by the user as config values, or the plugin will query the AWS MDS API for role-based credentials (so secrets are not stored on the system).

Porting over to DNF isn't trivial, as not only has the plugin API changed, but a lot of tasks that handles repo management or download has been moved to libdnf (this project) or librepo.

My Issue:

I figured I would start at libdnf for this query before I moved it to librepo where I think the majority of my hooks would go, but this is just step one in my quest for advice. Is there any guidance libdnf contributors can provide for me to "hook" into the HTTP request code paths of the download (anything from whats deemed an "s3 repo") to add SigV4 headers to authenticate into AWS to download an RPM or repo metadata.

Preferably I can add that "hook" through a Python DNF plugin so I can implement this feature through a python plugin, and avoid touching c/c++ in these projects.

Thanks!

[jrohel]

The libdnf library has an API for setting HTTP headers.
I added public API to DNF rpm-software-management/dnf#1265 a long time ago (November 2018) as a solution for https://bugzilla.redhat.com/show_bug.cgi?id=1638225 .

HTTP headers are repository specific and can be set before download begins (eg in the config hook; changing HTTP headers during repository downloading is not supported, only before download begins). These headers are then sent in all HTTP headers for that repository. Is it enough?

Support for changing HTTP headers during repository downloads may require changes to DNF, libdnf, librepo, libcurl.

[andrewegel]

Thanks for getting back to me.

These headers are then sent in all HTTP headers for that repository. Is it enough?

I saw those methods, which is how I figured I could add them in. However, the Signature header [1] is going to change per request thats made, because the signature is calculated based on the HTTP request like method, path, body - the HTTP path being the changing piece of data for every HTTP request made. So unless theres a "hook" where I could get access to the "raw" request before its made to extract & calculate the signature from the method, path, body, etc, it doesn't sound like it will work.

Alternatively there is also a way to use query parameters in the URL path [2], but would (again) require some sort of hook with access to the raw request to append query parameters to the requested URL. Is there any hook I could put in that would intercept the HTTP request (repository download) before its made and write in some query parameters?

[1] https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-auth-using-authorization-header.html
[2] https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-query-string-auth.html

[miaoneng]

I am working on the same direction and facing the exact same issue. AWS Sigv4 Auth Headers change per request. In old yum, the way is to monkey-patch _getFile method to add computed headers there, but now actual downloading function has been moved to likely librepo (plz correct me if it is wrong).

A stupid duck-tape solution might be money-patch download_remote_payload function in DNF by reimplementing downloading function there, but it will ultimately break other plugins behavior in certain manner.

[andrewegel]

Thanks for the thoughts @miaoneng - I too had been thinking of your approach mentioned.

A stupid duck-tape solution might be money-patch download_remote_payload function in DNF by reimplementing downloading function there, but it will ultimately break other plugins behavior in certain manner.

Yeah, one could argue that a Minimum Viable Product initial release could call this out to have that kind of support added later. What plugins do you think would cause an issue?

[ozbenh]

Reviving this, I would like to be able to use librepo on sigv4 repositories, not just s3. Has anything been done since those old comments ? If not, should we propose a patch/PR to add a hook ?

[diegoestrada88]

@andrewegel did you ever got a solution working? I am facing the same issue. Or @ozbenh any chance you found something?

[kontura]

As far as upstream is concerned I don't think there were any developments.

This is a fairly complex RFE with low priority. As mentioned before it would require changes to librepo and libdnf (preferably libdnf5) at minimum.