How to disable TLS interception conditionally?

[kuefmz]

I have an example here: https://github.com/kuefmz/https-interception-proxypy/blob/main/proxy/custom_proxy.py

In the example, I have a variable called https_intercept. In case it is False I would like to let the original content go through the proxy without applying the generated SSL certificate. However, in my current code, if https_intercept is False, I get a resource not found error.

Can you please tell me if this is possible or not and if it is, how can I do it?

[abhinavsingh]

@kuefmz Wondering does do_intercept override helps your case , see

proxy.py/proxy/http/proxy/plugin.py

Lines 161 to 171 in 84c36b6

	def do_intercept(self, _request: HttpParser) -> bool:
	"""By default returns True (only) when necessary flags
	for TLS interception are passed.
	When TLS interception is enabled, plugins can still disable
	TLS interception by returning False explicitly. This hook
	will allow you to run proxy instance with TLS interception
	flags BUT only conditionally enable interception for
	certain requests.
	"""
	return tls_interception_enabled(self.flags)

[JJ-Author]

in theory it should help in practice it does not (maybe a bug?). pls see my message below.

[JJ-Author]

Hey @abhinavsingh thanks for the reply and your cool project.
Please have a look at the very simple example code again (we changed it).

The problem is when returning false in do_intercept that the request is not going through/ connection seems not established (it seems no server hello is sent / passed through).
We have written a very simple decision rule if host is www.example.org do not intercept, if host is example.org do intercept,

the issue can be reproduced with running poetry install and then poetry shell and then python proxy-test/custom_proxy.py in the root dir of https://github.com/kuefmz/https-interception-proxypy/

debug log curl on www.example.org that is not succeeding

curl -vvvx http://localhost:8899 https://www.example.org                               130 ↵  ✹ ✭main 
*   Trying 127.0.0.1:8899...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8899 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to www.example.org:443
> CONNECT www.example.org:443 HTTP/1.1
> Host: www.example.org:443
> User-Agent: curl/7.68.0
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 Connection established
< 
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.example.org:443 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.example.org:443 

debug log curl on example.org that is intercepted and succeeding

/srv/archivo/time-machine/https-interception-proxypy$ curl --cacert ca-cert.pem -vvvx http://localhost:8899 https://example.org                                35 ↵  ✹ ✭main                                                                                                                                  
*   Trying 127.0.0.1:8899...                                                                                                                                                                                                                                                                                                   
* TCP_NODELAY set                                                                                                                                                                                                                                                                                                              
* Connected to localhost (127.0.0.1) port 8899 (#0)                                                                                                                                                                                                                                                                            
* allocate connect buffer!                                                                                                                                                                                                                                                                                                     
* Establish HTTP proxy tunnel to example.org:443                                                                                                                                               
> CONNECT example.org:443 HTTP/1.1                                                                                                                                                             
> Host: example.org:443                                                                                                                                                                        
> User-Agent: curl/7.68.0                                                                                                                                                                      
> Proxy-Connection: Keep-Alive                                                                                                                                                                 
>                                                                                                                                                                                              
< HTTP/1.1 200 Connection established                                                                                                                                                          
<                                                                                                                                                                                              
* Proxy replied 200 to CONNECT request                                                                                                                                                         
* CONNECT phase completed!                                                                                                                                                                     
* ALPN, offering h2                                                                                                                                                                            
* ALPN, offering http/1.1                                                                                                                                                                      
* successfully set certificate verify locations:                                                                                                                                               
*   CAfile: ca-cert.pem                                                                                                                                                                        
  CApath: /etc/ssl/certs                                                                                                                                                                       
* TLSv1.3 (OUT), TLS handshake, Client hello (1):                                                                                                                                              
* CONNECT phase completed!                                                                                                                                                                     
* CONNECT phase completed!                                                                                                                                                                     
* TLSv1.3 (IN), TLS handshake, Server hello (2):                                                                                                               
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):                                                                                                       
* TLSv1.3 (IN), TLS handshake, Certificate (11):                                                                                                               
* TLSv1.3 (IN), TLS handshake, CERT verify (15):                                                                                                               
* TLSv1.3 (IN), TLS handshake, Finished (20):                                  
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):                                                                                                    
* TLSv1.3 (OUT), TLS handshake, Finished (20):                                 
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384                                                                                                        
* ALPN, server did not agree to a protocol                                     
* Server certificate:                                                          
*  subject: CN=www.example.org; C=US; ST=California; L=Los Angeles; O=Internet Corporation for Assigned Names and Numbers                                                                                                                                                                                                      
*  start date: Aug  2 19:26:54 2024 GMT                                        
*  expire date: Aug  2 19:26:54 2026 GMT                                       
*  subjectAltName: host "example.org" matched cert's "example.org"                                                                                             
*  issuer: CN=localhost                                                        
*  SSL certificate verify ok.                                                  
> GET / HTTP/1.1                                                               
> Host: example.org                                                            
> User-Agent: curl/7.68.0                                                      
> Accept: */*                                                                  
>                                                                              
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):                                                                                                          
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):                                                                                                          
* old SSL session ID is stale, removing                                        
* Mark bundle as not supporting multiuse                                       
< HTTP/1.1 200 OK                                                              
< Age: 316512                                                                  
< Cache-Control: max-age=604800                                                
< Content-Type: text/html; charset=UTF-8                                       
< Date: Fri, 02 Aug 2024 20:07:10 GMT                                          
< Etag: "3147526947+ident"                                                     
< Expires: Fri, 09 Aug 2024 20:07:10 GMT                                       
< Last-Modified: Thu, 17 Oct 2019 07:18:26 GMT                                 
< Server: ECAcc (nyd/D125)                                                     
< Vary: Accept-Encoding                                                        
< X-Cache: HIT                                                                 
< Content-Length: 1256                                                         
<                                                  

....

NOTE: The title of the discussion is a bit misleading. We are aware that we can not decide whether to intercept an https request or not on a URL basis since the path is sent encrypted already. but it should be possible based on a hostname/FQDN basis due to information in CONNECT request or SNI in tls handshake.

[abhinavsingh]

@JJ-Author Thanks for reporting this. I am little tied till EOW, I'll look into it over coming weekend.

[JJ-Author]

thank you. would really appreciate this.

[JJ-Author]

since I dived a little deeper into proxypy I got more confident that this is a bug.
shall I create an issue out of this?

[JJ-Author]

Just as an update: I renamed the module name of the "minimal not-working example" to not have conflicts with proxy.py so the link form @kuefmz does not resolve anymore it now is
https://github.com/kuefmz/https-interception-proxypy/blob/main/proxy-test/custom_proxy.py
I updated the command in my message accordingly

[JJ-Author]

@abhinavsingh this problem is kind of urgent since it is blocking our development.
so I would appreciate a solution or fix if needed.
do you take it from here or shall I create a bugreport?

[abhinavsingh]

Hi @JJ-Author , apologies, haven't been able to get into the root cause for this. Give me this week and I'll see what can be done. Honestly, I need to read the code flow once to understand why inspite of bypass TLS interception method, it don't get bypassed.

[abhinavsingh]

Please try

• TLS Intercept conditionally #1476

This PR provides fixes for conditional TLS interception

[JJ-Author]

great. thank you very much. that seems to solve the issue.