How to do SSL with internal services?

[JB0925]

Hello! Before posting this, I checked the gitter channel, as it was suggested, but it seems that the channel is no longer active, so I thought I'd ask here.

My team manages an L7 reverse proxy. The reverse proxy forwards requests to various origins, but also to other internal services.

So one client call to the reverse proxy will:

• make a call to an origin server
• potentially make one or more calls to mid-tier services

We want to capture this traffic and, using plain http in a dev environment, this works well.

However, our services of course use https. We also have our own internal CA. I am finding that most times, using hoverfly to call these services, i.e.:

curl -v -x http://127.0.0.1:8500 <https://internal service>

almost always results in Got error: tls handshake failure.

I've tried many things that I've seen in the hoverfly / hoverctl documentation, but am not having any success.

I am looking for any pointers you can provide for setting this up correctly so I can successfully call our internal services on https.

Thank you!

[tommysitu]

what happens if you call curl -v <https://internal service> without hoverfly?

[JB0925]

Hey @tommysitu , thanks for the quick response.

I did some research and have a hunch that this may be related to this.

We can make one call to a downstream that does work, but many / most of the others don't.

We made the same calls through mitmproxy and it worked.

I am going to try this with an older version of Hoverfly, pre-Go 1.22.

Would you be able to point me to a version that uses something < Go 1.22? If not, I'll take a look through changelogs / releases.

Thanks again!

Edit: I should also mention that we do not have any control over what ciphers the downstream server certs are created with, unfortunately, and there's a chance that they are created with some ciphers excluded in Go 1.22+.

[tommysitu]

interesting, you can try this version which is pre go 1.22: https://github.com/SpectoLabs/hoverfly/releases/tag/v1.8.0

[JB0925]

Awesome, I'll give that a shot.

[tommysitu]

Are you sure this has nothing to do with self-signed cert being used by the internal service? Because you can tell hoverfly to skip SSL verification with this flag:

  -tls-verification
    	Turn on/off tls verification for outgoing requests (will not try to verify certificates) (default true)

[JB0925]

Thanks again, @tommysitu . That is definitely a part of it, and we do indeed have to do that to make it work with the one domain that we've had success with thus far. It's unfortunately not an option for us to add the Hoverfly cacert to our system truststore / cacert.

[JB0925]

To be clear, the other internal domains we've tried do not even work with TLS verification turned off. They do, however, return what we expect when we don't have the Hoverfly forward proxy in front of them.

[JB0925]

Hey @tommysitu , thanks again for your help. This is definitely related to the GitHub issue I posted above. I used the binary for 1.8.0 and it worked with no issues.

For us, we can probably just use an older version, but I'm not sure if you'd like to fix this going forward or not.

Overall, though, Hoverfly has been super smooth and easy to use. Thanks for a great tool!

Edit: I figured I should explain how I came to the conclusion that the above is indeed the issue.

On my system, I have a Go 1.22 binary, and its directory is on my $PATH variable. I also, for some reason, have a Go 1.19 binary in a different directory.

To start, I ran Hoverfly via hoverctl start --disable-tls && hoverctl mode capture --all-headers. I got the list of ciphers that you see from the Github Issue I mentioned above.

Then I wrote this script:

package main

import (
   "crypto/tls"
   "fmt"
   "log"
   "net/http"
   "net/url"
)

const (
   _url = "https://foo.com/"           // internal service url

)

func main() {
   proxyURL, err := url.Parse("http://127.0.0.1:8500/")           // local Hoverfly instance
   if err != nil {
       fmt.Println("Error parsing proxy URL:", err)
       return
   }

   customTransport := &http.Transport{
      //  Proxy: http.ProxyURL(proxyURL),
       TLSClientConfig: &tls.Config{
           InsecureSkipVerify: true,
           // CipherSuites: []uint16{
           //  // TLS 1.0 - 1.2 cipher suites.
           //  tls.TLS_RSA_WITH_RC4_128_SHA,
           //  tls.TLS_RSA_WITH_3DES_EDE_CBC_SHA,
           //  tls.TLS_RSA_WITH_AES_128_CBC_SHA,
           //  tls.TLS_RSA_WITH_AES_256_CBC_SHA,
           //  tls.TLS_RSA_WITH_AES_128_CBC_SHA256,
           //  tls.TLS_RSA_WITH_AES_128_GCM_SHA256,
           //  tls.TLS_RSA_WITH_AES_256_GCM_SHA384,
           //  tls.TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,
           //  tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,
           //  tls.TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,
           //  tls.TLS_ECDHE_RSA_WITH_RC4_128_SHA,
           //  tls.TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,
           //  tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,
           //  tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
           //  tls.TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,
           //  tls.TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,
           //  tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,
           //  tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
           //  tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
           //  tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
           //  tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256,
           //  tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256,
           //  // TLS 1.3 cipher suites.
           //  tls.TLS_AES_128_GCM_SHA256,
           //  tls.TLS_AES_256_GCM_SHA384,
           //  tls.TLS_CHACHA20_POLY1305_SHA256,
           // },
       },
   }

   c := &http.Client{
       Transport: customTransport,
   }

   r, err := c.Get(_url)
   if err != nil {
       log.Fatal(err)
   }
   defer r.Body.Close()

   log.Println(r.Status)
}

On Go 1.22
If I run the script exactly as above ( ciphers commented out ) , I get the error: tls: handshake failure.
If I uncomment the ciphers in the TLS config, I get the expected result and status code.

If I uncomment the Proxy config above and send the request through the latest version of Hoverfly, I get the TLS error, whether I include the ciphers or not.

On Go 1.19
I get the expected result and status code whether or not I include the ciphers. However, if I include the latest version of Hoverfly, I again get the TLS errors.

Using Hoverfly 1.8.0 and Go 1.19

I uncommented the Proxy config so that it uses Hoverfly, and I commented out the ciphers. It works as expected.

[tommysitu]

thanks for the info, I can certainly take a look at fixing it forward.