reqwest basic auth fails but curl works

[nb-twy]

I use a DDNS service called ChangeIp. They have a very simple API for updating DNS records. I really like Rust and decided to update my ddns utility using Rust and reqwest. Unfortunately, I can't get basic auth to work with ChangeIp's API.

Here's a functional extract of the code, with sensitive data redacted.

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let set = 1;
    let host = "https://nic.changeip.com/nic/update";
    let username = "my_username";
    let password = "my_password";
    let new_ip = String::from("new_ip_address");

    let client = reqwest::Client::builder()
        .build()?;

    let res = client
        .get(host)
        .query(&[("set", set.to_string()), ("ip", new_ip)])
        .basic_auth(username, Some(password))
        .header("Accept", "*/*")
        .send()
        .await?;

    println!("\n<<< Response >>>");
    println!("Status: {}", res.status());
    println!("Content-length: {}", res.content_length().unwrap());
    for (header_name, header_value) in res.headers() {
        println!("{}: {:?}", header_name, header_value);
    }
    println!("\n{}", res.text().await?);

    Ok(())
}

I get a 401 response every time. Here's the output from a run.

<<< Response >>>
Status: 401 Unauthorized
Content-length: 11
server: "nginx/1.22.0"
date: "Thu, 27 Oct 2022 16:49:34 GMT"
content-type: "text/plain;charset=UTF-8"
content-length: "11"
connection: "keep-alive"
x-powered-by: "PHP/5.6.40"
www-authenticate: "basic realm=\"DDNS\""

401 badauth

Alternatively, the following curl command works just fine.

❯ curl -sSG "https://nic.changeip.com/nic/update" -u 'my_username:my_password'\
 --data-urlencode "set=1" --data-urlencode "ip=new_ip_address"
200 Successful Update

I wondered if somehow the basic auth header was not being constructed properly or if I had mangled the credentials somehow. So I pointed the request to a web server I control and inspected the request. I compared the Basic Auth header curl is sending, using the -v switch to see verbose output, with the header the web server received from the reqwest version, and they are identical.

If the request is constructed properly, then could it be TLS? I read some forum posts that said that there was something wonky with how TLS works with openssl and that they could get reqwest's Basic Auth to work if they used rustls-tls instead. Others said that it worked on Windows. I have tried using rustls-tls. I have tried building and executing from Ubuntu, Fedora, and Windows 11. I get the same response on every platform, whether I'm using native TLS or rustls-tls.

I just can't think of any other way to troubleshoot this problem, so I am hoping that someone here can offer a solution or, at least, a suggestion.

Thank you!

[seanmonstar]

Does it require a user-agent header? Some sites get grumpy if you don't send one...

[nb-twy]

That's a great question. I had considered that and tried to use the same user-agent string that curl sends with its request. Here's what the code looks like. Unfortunately, I wind up with the same 401 response.

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let set = 1;
    let host = "https://nic.changeip.com/nic/update";
    let username = "my_username";
    let password = "my_password";
    let new_ip = String::from("new_ip_address");

    let client = reqwest::Client::builder()
        .user_agent("curl/7.81.0")
        .build()?;

    let res = client
        .get(host)
        .query(&[("set", set.to_string()), ("ip", new_ip)])
        .basic_auth(username, Some(password))
        .header("Accept", "*/*")
        .send()
        .await?;

    println!("\n<<< Response >>>");
    println!("Status: {}", res.status());
    println!("Content-length: {}", res.content_length().unwrap());
    for (header_name, header_value) in res.headers() {
        println!("{}: {:?}", header_name, header_value);
    }
    println!("\n{}", res.text().await?);

    Ok(())
}

[nb-twy]

Okay! I have figured out what is wrong, but I don't know how to solve the problem quite yet. I just wanted to let you know, in case anyone is working on it right now. I'll post the details in a few hours.

Thanks for the help so far!

[nb-twy]

The problem is that ChangeIp's implementation of basic auth treats the Authorization header key name as case sensitive.

Here is the curl command with the header name uppercase: Authorization: Basic blahblahblah. Notice that the request is successful.

❯ curl -sSG "https://nic.changeip.com/nic/update"\
 -H 'Authorization: Basic d2hhdGRpZHlvdTpleHBlY3R0b2ZpbmQK'\
 --data-urlencode "set=1" --data-urlencode "ip=new_ip"
200 Successful Update

Here is the curl command with the header name lowercase: authorization: Basic blahblahblah. Notice that the request fails.

❯ curl -sSG "https://nic.changeip.com/nic/update"\
 -H 'authorization: Basic d2hhdGRpZHlvdTpleHBlY3R0b2ZpbmQK'\
 --data-urlencode "set=1" --data-urlencode "ip=new_ip"
401 badauth

Now, the RFCs governing headers (2616 & 7230) are clear that "Field names are case-insensitive." However, I cannot change ChangeIp's API. I am wondering if it is possible for reqwest to send the header with a capital A.

I can see in name.rs that if I use the enum and write .header(AUTHORIZATION, value), that the binary value is lowercase. I can also see that the basic_auth function uses the same enum value to generate the basic auth header, so the header name would be lowercase. I am surprised, though, that when I use the header function, it makes the value I put in for the header name lowercase: .header("Authorization" , format!("Basic {}", base64::encode(format!("{}:{}", username, password)))). I don't see where that happens in the code.

I don't mind using a custom header to get the functionality of having a case-sensitive header name, but I can't figure out how to do that. Am I missing something? Should this be something that gets turned into an issue and gets some work?

Again, thanks for the help!

[seanmonstar]

You can set http1_title_case_headers on the client builder.

[nb-twy]

And just like that, my problem is solved!

I would never have thought to look there. Thanks!

So, just to round things out for anyone following along or who might find this in the future, here is what the working code looks like.

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let set = 1;
    let host = "https://nic.changeip.com/nic/update";
    let username = "my_username";
    let password = "my_password";
    let new_ip = String::from("new_ip_address");

    let client = reqwest::Client::builder()
        .http1_title_case_headers()
        .build()?;

    let res = client
        .get(host)
        .query(&[("set", set.to_string()), ("ip", new_ip)])
        .basic_auth(username, Some(password)).send()
        .await?;

    println!("\n<<< Response >>>");
    println!("Status: {}", res.status());
    println!("Content-length: {}", res.content_length().unwrap());
    for (header_name, header_value) in res.headers() {
        println!("{}: {:?}", header_name, header_value);
    }
    println!("\n{}", res.text().await?);

    Ok(())
}

This is what the request looks like.

GET /nic/update?set=1&ip=new_ip_address
Authorization: Basic d2hhdGRpZHlvdTpleHBlY3R0b2ZpbmQK

Note, I didn't need the Accept or the User-Agent headers, and I can use the basic_auth function, no need to encode the credentials myself. In the end, it was super easy.

@seanmonstar Thanks a whole bunch!