Why would this be a bad request?

[pnip]

-bash-4.1$ export PATH=$PATH:~
-bash-4.1$ hoverctl status

+------------+----------+
| Hoverfly | running |
| Admin port | 8888 |
| Proxy port | 8500 |
| Proxy type | forward |
| Mode | capture |
| Middleware | disabled |
+------------+----------+
-bash-4.1$ telnet localhost 8500
Trying ::1...
telnet: connect to address ::1: Connection refused
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GET / HTTP/1.1
Host: thecure.com
User-Agent: Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp) .default/1538949339-2
Accept: /

HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Sun, 07 Oct 2018 23:33:21 GMT
Content-Length: 64

This is a proxy server. Does not respond to non-proxy requests.
^]

telnet> quit
Connection closed.
-bash-4.1$

https://github.com/SpectoLabs/goproxy/blob/master/proxy.go#L108-L110

Why does the url has to be absolute? If a Host header is provided

[tommysitu]

@pnip I will try if I can answer your question as I don't use telnet much. Hoverfly is an HTTP proxy, in order to use it to simulate or capture your telnet traffic, you need to set up telnet to use localhost:8500 as a proxy where hoverfly listens on.

[pnip]

@tommysitu telnet is just to simulate a connection (incidently nc could be use instead) to hoverfly, once the connection is established (just like what a 1.1 browser does, I believe) be that a normal http connection or https connection with HTTP "CONNECT" handled by hoverfly + SSL negotiation is done, a 1.1 session is established, and supposedly one can request url without any specifying host nor port nor scheme in any subsequent request url.

But I think each request still need a "Host: foo.bar" header (presumable for checking for 400 bad request)

I should explain further I find that if I do the following it works as expected:

C02V81EGHTDD-lm:trafficserver pnip$ nc localhost 8500
GET http://thecure.com HTTP/1.1
Host: thecure.com

HTTP/1.1 200 OK
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Content-Length: 2705
Content-Type: text/html; charset=UTF-8
Date: Mon, 08 Oct 2018 18:24:45 GMT
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Hoverfly: Was-Here
Pragma: no-cache
Server: Apache
Set-Cookie: PHPSESSID=icgm5eqsh8vpukeciiro67o8g1; path=/
Set-Cookie: SERVERID=; Expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/
Set-Cookie: visid_incap_288692=d11uim4GRuu3k05AfNyGr+2gu1sAAAAAQUIPAAAAAADPnXLlbw45iOAtSQGSbSMc; expires=Tue, 08 Oct 2019 09:43:14 GMT; path=/; Domain=.thecure.com
Set-Cookie: incap_ses_466_288692=nHXlAde/OG7x3nr925J3Bu2gu1sAAAAAWBKrgEqzpH3o9JjIMMgmxw==; path=/; Domain=.thecure.com
Vary: Accept-Encoding
Via: 1.1 umgr44ha1che001 (squid/3.2.2)
X-Cache: MISS from umgr44ha1che001
X-Cache-Lookup: MISS from umgr44ha1che001:80
X-Cdn: Incapsula
X-Iinfo: 1-2480465-2480481 NNNN CT(13 -1 0) RT(1539023084927 192) q(0 0 0 1) r(1 1) U5

.... truncated

[tommysitu]

@pnip I'm very interested in your use case. Are you trying to setup HTTP tunnel via hoverfly proxy to use telnet? What's your end goal in doing this, to capture and simulate the telnet session? It may not work as you have pointed out as it hasn't been tested for telnet before.

Maybe you have tried this out already, but if you haven't, have a look at proxychains, it should let you configure a proxy for telnet.

[pnip]

@tommysitu nah this is not for capture/simulate telnet session but for creating an repeatable integration testing framework for our server/client stacks.

[JohnFDavenport]

From what I see @pnip you started telnet on the same port that is used by the http proxy. Telnet will use TCP/IP or similar, not http, so a http proxy will barf. If so it's a problem that has nothing to do with Hoverfly.

[JohnFDavenport]

I'm persuaded this is a bug caused by the relative path not being processed correctly.

[tommysitu]

Revisited this issue again, and GoProxy is not working properly with telnet requests. It should be able to get the destination from the Host header for HTTP 1.1 request.

[perlun]

I am seeing similar issues, trying to use Hoverfly as a MITM (man-in-the-middle) proxy between an Envoy (reverse) proxy and our application server. The use case is for recording a bunch of requests down to .json files so we can then replay them using Gatling.

Requests like this work fine:

$ telnet 192.168.1.100 8080
Trying 192.168.1.100...
Connected to 192.168.1.100.
Escape character is '^]'.
GET http://localhost/some-path HTTP/1.1
Host: localhost

HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 10534
Content-Type: text/html;charset=UTF-8
Date: Tue, 07 Jan 2020 14:11:25 GMT
Hoverfly: Was-Here
Last-Modified: Mon, 09 Dec 2019 08:13:02 GMT
Server: Apache-Coyote/1.1
Vary: Accept-Encoding

<response-data>

If I retry this with relative URL, I get the same error:

$ telnet 192.168.1.100 8080
Trying 192.168.1.100...
Connected to 192.168.1.100.
Escape character is '^]'.
GET /some-path HTTP/1.1
Host: localhost

HTTP/1.1 500 Internal Server Error
Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Tue, 07 Jan 2020 14:13:19 GMT
Content-Length: 64

This is a proxy server. Does not respond to non-proxy requests.

The problem is not in itself related to telnet per se; as @pnip described it's merely a tool that is commonly used (well, perhaps used to be more commonly used back in the days 🙂) for debugging arbitrary TCP protocols. As long as you know the (basics) of the protocol, you can debug it with telnet or nc (netcat).

The problem is more specifically this code in the upstream goproxy lib; as @tommysitu suggested, it doesn't handle non-absolute URLs. In this case, "absolute" is presumed to mean a full URL with scheme://host/path and not just a /path.

This issue on the goproxy side indicates that this by design because goproxy is a forward proxy and not a reverse proxy.

OTOH, I get the feeling from reading this example in the goproxy repo that it could actually be made to work; it's at least possible to use goproxy as a transparent (forward) proxy by overriding the NonproxyHandler: https://github.com/elazarl/goproxy/blob/master/examples/goproxy-transparent/transparent.go#L36-L44 (more details: https://github.com/elazarl/goproxy/tree/master/examples/goproxy-transparent). Does that help us make it possible to make goproxy usable as a reverse proxy? Uncertain about that part.

[tommysitu]

@perlun, that's exactly what the problem is. Getting the host from the Host header should fix it.

Hoverfly already has a reverse proxy mode, check here https://github.com/SpectoLabs/hoverfly/blob/master/core/proxy.go#L109

You can do hoverctl start webserver to enable it. But under this mode, Hoverfly doesn't do capturing.

Feel free to submit a PR if you'd like to contribute.

[perlun]

Thanks for the reply @tommysitu. 👍 Good to hear more about the reverse proxying mode. However, in our case, the capturing is the critical part, so we ended up opting for another solution for now (for reference, the Envoy tap filter described here: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_filters/tap_filter#config-http-filters-tap). Since we already use Envoy, taking that into use was easy for our particular use case.

It would however be nice to be able to to use Hoverfly in this mode, so I will support someone trying to fix it. 🙂 However, will not spend time on it myself.