Config reverse proxy #38088

Mefevre · 2024-02-12T15:02:22Z

Mefevre
Feb 12, 2024

Hello community,

I set up a Teleport cluster, added nodes and applications.
The cluster is located behind an Apache2 reverse proxy.

So far everything is going well. I am able to connect to the cluster through the reverse proxy and connect to the applications and nodes.

The problem comes from the Teleport client (tsh.exe).
I can't connect to the cluster when I use the reverse proxy. When I tap directly on the cluster, no problem, everything works normally.

Do you have any idea where the problem could be coming from?

Error send by client tsh

ERROR: connection error: desc = "transport: Error while dialing: failed to dial: failed to switch Protocols 502"

Config of apache2

<VirtualHost *:443>
    ServerName bastion.mondomaine.fr

    ErrorLog ${APACHE_LOG_DIR}/error.bastion.log
    CustomLog ${APACHE_LOG_DIR}/access.bastion.log combined

    SSLEngine on
    SSLProtocol TLSv1.3 TLSv1.2
    SSLCertificateFile /etc/letsencrypt/live/mondomaine.fr/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/mondomaine.fr/privkey.pem

    # On fait du proxy vers un autre serveur en https
    SSLProxyEngine On
    SSLProxyCheckPeerCN Off
    SSLProxyCheckPeerName Off
    SSLProxyCheckPeerExpire off
    SSLProxyVerify none

    # On transmet notre certificat au serveur distant (crt + key combiné)
    #  SSLProxyMachineCertificateFile "/etc/ssl/certs/https/client.pem"
    ProxyPass /v1/ wss://192.168.1.235/v1/
    ProxyPass / https://192.168.1.235/
    ProxyPassReverse / https://192.168.1.235/
    ProxyPreserveHost On
</VirtualHost>

Config cluster teleport

version: v3
teleport:
  nodename: bastion
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: DEBUG
    format:
      output: text
  ca_pin: ""
  diag_addr: ""
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  cluster_name: bastion.mondomain.fr
  proxy_listener_mode: multiplex
  authentication:
    second_factor: off
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:443
  public_addr: bastion.mondomain.fr:443
  https_keypairs:
  - key_file: /var/lib/teleport/mondomain.fr.key
    cert_file: /var/lib/teleport/mondomain.fr.crt
  https_keypairs_reload_interval: 0s
  acme: {}

Log in proxy

37.65.47.40 - - [12/Feb/2024:14:45:10 +0100] "POST /v1/webapi/ssh/certs HTTP/1.1" 200 13078 "-" "Go-http-client/1.1"

==> /var/log/apache2/error.bastion.log <==
[Mon Feb 12 14:45:10.593179 2024] [proxy:error] [pid 99063:tid 140279294174784] [client 37.65.47.40:47621] AH00898: Unexpected Upgrade: alpn-ping (expecting n/a) returned by /webapi/connectionupgrade

==> /var/log/apache2/access.bastion.log <==
37.65.47.40 - - [12/Feb/2024:14:45:10 +0100] "GET /webapi/connectionupgrade HTTP/1.1" 502 5088 "-" "Go-http-client/1.1"

==> /var/log/apache2/error.bastion.log <==
[Mon Feb 12 14:45:10.601255 2024] [proxy:error] [pid 99063:tid 140279285782080] [client 37.65.47.40:49406] AH00898: Unexpected Upgrade: alpn-ping (expecting n/a) returned by /webapi/connectionupgrade

Answered by webvictim

Feb 17, 2024

I got this working after a bit of trial and error. Here are my configs from an Ubuntu 22.04 server:

/etc/apache2/sites-available/001-teleport.conf:

<VirtualHost *:443>
    ServerName teleport.example.com

    ErrorLog /var/log/apache2/teleport.error.log
    CustomLog /var/log/apache2/teleport.access.log combined

    SSLEngine on
    SSLProxyEngine on
    SSLProtocol TLSv1.3 TLSv1.2
    SSLCertificateFile /etc/letsencrypt/live/teleport.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/teleport.example.com/privkey.pem

    # Replace localhost:3080 with the IP:port of your Teleport server if needed
    # The upgrade=ANY line is needed to pass custom connection upgrad…

View full answer

Mefevre · 2024-02-12T15:04:29Z

Mefevre
Feb 12, 2024
Author

version teleport : Teleport v15.0.1 git:v15.0.1-0-gd347510 go1.21.6
version tsh : Teleport v15.0.1 git: go1.21.6

0 replies

webvictim · 2024-02-14T13:55:18Z

webvictim
Feb 14, 2024
Collaborator

I'm not familiar with how to do this using Apache - it's much easier using something like nginx (guide here: #26445) or Caddy.

If you insist on using Apache, try this: https://serverfault.com/a/1123715

0 replies

Mefevre · 2024-02-17T13:48:48Z

Mefevre
Feb 17, 2024
Author

thank you for your return @webvictim !
I have already come across this suggestion. I tested it, but without success for me.

I want to stay on Apache2 because there is a lot of configuration in place. and that switching to nginx would take a lot of time.

0 replies

webvictim · 2024-02-17T21:14:05Z

webvictim
Feb 17, 2024
Collaborator

I got this working after a bit of trial and error. Here are my configs from an Ubuntu 22.04 server:

/etc/apache2/sites-available/001-teleport.conf:

<VirtualHost *:443>
    ServerName teleport.example.com

    ErrorLog /var/log/apache2/teleport.error.log
    CustomLog /var/log/apache2/teleport.access.log combined

    SSLEngine on
    SSLProxyEngine on
    SSLProtocol TLSv1.3 TLSv1.2
    SSLCertificateFile /etc/letsencrypt/live/teleport.example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/teleport.example.com/privkey.pem

    # Replace localhost:3080 with the IP:port of your Teleport server if needed
    # The upgrade=ANY line is needed to pass custom connection upgrades used by Teleport
    ProxyPass / https://localhost:3080/ upgrade=ANY
    ProxyPassReverse / https://localhost:3080/
    ProxyPreserveHost on
    ProxyRequests off
</VirtualHost>

/etc/teleport.yaml:

version: v3
teleport:
  nodename: ip-172-31-32-65
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: DEBUG
    format:
      output: text
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  proxy_listener_mode: multiplex
  cluster_name: teleport.example.com
  authentication:
    second_factor: on
    webauthn:
      rp_id: teleport.example.com
ssh_service:
  enabled: "yes"
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: "yes"
  # listen internally on port 3080
  web_listen_addr: 0.0.0.0:3080
  # public address is Apache port 443
  public_addr: teleport.example.com:443
  # trust X-Forwarded-For headers sent by reverse proxy (for correct remote IPs in Teleport logs)
  trust_x_forwarded_for: true

Enable modules mod_proxy mod_ssl mod_rewrite and site 001-teleport, then restart Apache:

sudo a2enmod proxy ssl rewrite
sudo a2ensite 001-teleport
sudo systemctl restart apache2

2 replies

webvictim Feb 21, 2024
Collaborator

This solution is also published here: #38426

jayjay3108 May 1, 2024

Thank you very much for dropping your yaml. This helped me to finally set up webauthn.

Mefevre · 2024-02-18T15:56:38Z

Mefevre
Feb 18, 2024
Author

In your tests, have you tested the connection with a tsh client ?

2 replies

webvictim Feb 18, 2024
Collaborator

Good point, I have not - I'll give it a try later. The config for arbitrary non-websocket connection upgrades is likely to be considerably harder to write!

I know that the alpn-ping test from the TLS routing troubleshooting doesn't work with this config so tsh will probably not, but there is support coming for solely using websockets soon which should help.

webvictim Feb 18, 2024
Collaborator

OK, turns out this is actually very easy as long as you're using Apache 2.4.47 or higher.

Just replace your ProxyPass directive with this:

ProxyPass / https://localhost:3080/ upgrade=ANY

The upgrade=ANY line means that the proxy will pass through any connection upgrades to the backend. You can then also remove all the mod_rewrite stuff which makes the config nicer.

Mefevre · 2024-02-19T20:21:56Z

Mefevre
Feb 19, 2024
Author

@webvictim
I confirm that the last information you sent me is the correct one.
This solved my problem with the tsh client

Great solution

ProxyPass / https://localhost:3080/ upgrade=ANY

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Config reverse proxy #38088

{{title}}

Replies: 6 comments 4 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

Config reverse proxy #38088

Mefevre Feb 12, 2024

Replies: 6 comments · 4 replies

Mefevre Feb 12, 2024 Author

webvictim Feb 14, 2024 Collaborator

Mefevre Feb 17, 2024 Author

webvictim Feb 17, 2024 Collaborator

webvictim Feb 21, 2024 Collaborator

jayjay3108 May 1, 2024

Mefevre Feb 18, 2024 Author

webvictim Feb 18, 2024 Collaborator

webvictim Feb 18, 2024 Collaborator

Mefevre Feb 19, 2024 Author

Mefevre
Feb 12, 2024

Replies: 6 comments 4 replies

Mefevre
Feb 12, 2024
Author

webvictim
Feb 14, 2024
Collaborator

Mefevre
Feb 17, 2024
Author

webvictim
Feb 17, 2024
Collaborator

webvictim Feb 21, 2024
Collaborator

Mefevre
Feb 18, 2024
Author

webvictim Feb 18, 2024
Collaborator

webvictim Feb 18, 2024
Collaborator

Mefevre
Feb 19, 2024
Author