BIP APIs
As of the January 2024 deployment of the BIP shared microservice, VRO is only using the BIP Claims API, and while it is no longer using the BIP Claim Evidence API, the codebase for the Claims Evidence API is still present.
Information specific to one of the individual APIs can be found in the above pages. This page, in addition to providing these links, describes the details of VRO Mutual TLS implementation.
We will dive into the details of the TLS certificates momentarily, but it is first useful to understand the big picture of how these certs are used. First, there is the certificate management system that acts as the VA's primary Certificate Authority (CA). Our CA is provided by LHDI, a system called Venafi, which is deployed as a feature of our Kubernetes infrastructure. Through Venafi we provision and query for certificates and once provisioned, Venafi will manages the expiration of these certificates. Second, there is generation process itself which is a bash script that lives in the VRO repo: scripts/svc-bip-api-certgen.sh. This script pulls the certs from Venafi and combines them with the intermediate certificates which it also downloads. The certs from Venafi, along with the Intermediate certificates are then crunched through a series of openssl commands to ultimately produce an output.json file. Third, there is HashiCorp Vault, which is where we store the generated truststore, keystore and the password used to seal these stores. Forth, and finally, there are the Kubernetes secrets that are used to supply the application with the truststore, keystore and password.
Please reference the following diagram as you navigate through the remainder of this documentation:
(note: this image was created with draw.io - the png file contains the draw.io source as embedded data to the png file)
The secrets that are pertinent to these integrations can be found in HashiCorp Vault under `vro-secrets/deploy/{ENV_NAME}/VRO_SECRETS_BIP and include:
- BIP_CLAIM_SECRET
- BIP_CLAIM_USERID
- BIP_EVIDENCE_SECRET
- BIP_EVIDENCE_USERID
- BIP_KEYSTORE_BASE64
- BIP_PASSWORD
- BIP_TRUSTSTORE_BASE64
BIP APIs as well as other VA services require Mutual TLS, or mTLS. It is necessary for VRO to use a client certificate signed by a VA recognized Certificate Authority (CA) during https handshake. In addition VRO needs to use a VA recognized CA to validate BIP certificates during https handshake.
Venafi is the VA's new Internal Management System which can be used to manage the PKI certificates. Within VRO Kubernetes cluster the process is further simplified and it is possible to generate the certificates within one of VRO Kubernetes namespaces. The client certificates that are being used currently by VRO has been generated using these instructions.
The certificates tls.key, tls.crt, and va.crt has been generated in va-abd-rrd-dev namespace and can be found in va-abd-rrd-dev-va-gov-tls secrets.
The private key tls.key is not encrypted so there are no passwords needed for steps described later in this document.
Details for tls.crt can be displayed using
openssl x509 -in tls.crt -text -nooutThe certificate tls.crt expires in January 6th, 2024.
The certificate tls.crt contains both the client public key and an intermediate CA. Only the client public key is used and will be referred as tls_bip.crt.
The certificate va.crt is not currently used.
By default, certificates generated are valid for one year, and are set to auto-renew ⅔rds of the way through their lifecycle.
A second set of certificates are generated in production. All the discussion in this document is identical for production certificates. The production certificate expires in March 17th, 2024.
As discussed later in this document we do not yet know a way to use these certificates directly in the Java code and there are manual steps to make them usable. More discussions are needed with LHDI around the process on how to update them.
- The certificates will be auto-renewed. What is the trigger to perform manual steps when the switch is made?
- Should LHDI make these certificates available to containers without additional steps?
- Can we forego container level https and move that complexity to API Gateway so that it is managed by LHDI?
At this time Java cert renewal is manual and steps in this document should be followed. We have time until January 6th, 2024 to make improvement.
During VRO implementation of mTLS we were not successful using the VA CA public keys in va.crt and tls.crt.
Instead we downloaded the public keys that were used by BIP APIs from Chrome on a GFE:
- Open Chrome on your GFE
- Open one of the end points in BIP Claims API or BIP Claim Evidence API. If you use a
GETend point you should typically get ajwtmissing error - Once you get the error click on the
lock(View site information) in the url bar to bring up a pop-up window - Select "Connection is secure" menu item to bring up a second pop-up window. From this pop-up you can select "Certificate is valid" to bring up the "Certificate viewer".
- Goto "Details" tab, select the certificate for root CA and "Export". Repeat for the intermediate CA
Two certificates downloaded are
- VA-Internal-S2-RCA1-v1.crt (root)
- VA-Internal-S2-ICA4.crt (intermediate)
These public keys are concatenated in a file named va_all.crt for later use.
We have not retried to use the VA CA public keys in va.crt and tls.crt after we successfully used va_all.crt. This should be revisited in the future since it is possible that the steps we eventually ended up with could be succesful for public keys in va.crt and tls.crt as well.
VRO generates a set of self-signed certificates using a script called build-certificates.sh. These certificates are used for local development and end-to-end tests to mock the actual certificates and mTLS based https handshake.
More details and what is being generated are documented in the script itself.
As of now we do not know a simple way to consume the certificates tls.key, tls_bip.crt, and va_all.crt directly in the Java code to utilize them in https calls. Java uses either JKS files or PKCS #12 files. Since PKCS #12 files are not Java specific, we prefer them over JKS files in VRO.
To generate the PKCS #12 file keystore.p12 for client certificates we use openssl
openssl pkcs12 -export -in tls_bip.crt -out keystore.p12 -name keystore -nodes -inkey tls.keyThis commands asks for an "export password". This password is to be recorded as it needs to be later used as a secret.
To generate the PKCS #12 file truststore.p12 for VA CA certificates we use keytool
keytool -import -file va_all.crt -alias all_cas -keystore truststore.p12This command asks for a "keystore password". For simplicity VRO uses the same the value of "export password".
It should be possible either to use keystore or openssl exclusively here. But these PKCS #12 files worked and we did not try either route any further.
VRO uses three environment variables to store the content of the [PKCS #12] files and the "export password".
BIP_KEYSTOREBIP_TRUSTSTOREBIP_PASSWORD
Since PKCS #12 files are binary we convert them to Base 64 using openssl
openssl base64 -in keystore.p12 -out keystore.b64
openssl base64 -in truststore.p12 -out truststore.b64What is stored in the environment variables BIP_KEYSTORE and BIP_TRUSTSTORE are the content of the Base 64 files.
In our environments these environment variables are made available to the app container using the following secrets
- bip.bipKeystore
- bip.bipTruststore
- bip.bipPassword
In summary the manual steps to make the certificates available to the Java code are
- Convert certificates to PKCS #12 files
- Convert PKCS #12 files to Base 64 files
- Store content of the Base 64 files in Kubernetes secrets
- Restart the pod to make secret changes effective
If https handshake implementation remains in the Java code (as opposed to say moved to API Gateway) these manual steps need to be automated if certificate renewal is automated.
Java uses Keystore objects to store certificate information in PKCS #12 files. VRO
- Reads in the content of the PKCS #12 files and the password from the environment variables
BIP_KEYSTORE,BIP_TRUSTSTORE, andBIP_PASSWORDthrough application.yml - Converts Base 64 content to binary content
- Creates the keystore and trusstore objects as Keystore instances from the binary content and the password
- Creates a custom RestTemplate bean that can be used to make the
httpsrequests
The properties in application.yml that corresponds to the environment variables are
-
keystoreforBIP_KEYSTORE -
truststoreforBIP_TRUSTSTORE -
truststore_passwordforBIP_PASSWORD
The RestTemplate bean that is used to make the https requests is implemented in BipApiConfig. This file also includes the generation of the Keystore objects from the application.yml properties. This RestTemplate bean is autowired in the rest of the code using the Qualifier bipCERestTemplate.
Validation of the certificates using the Java code has been problematic since the BIP APIs are not available outside of VA Firewall
- Our GFE's typically do not have Java installed
- Extra permissions are necessary to install and maintain Java on the GFE's
- Additional security features in GFE's makes it difficult to make https calls from Java
curl has been an invaluable tool to test the validity of the certificates. In principle it should be possible to run curl in your GFE but running it from the app container in one of our environment was easier. We copied the certificates tls.key, tls_bip.crt, and va_all.crt to the container using cat <<EOF and copy&paste in the /tmp directory. You should be able to use kubectl cp with enough permissions. Once the files are in the container you can run
curl -X POST <https://......> --cacert va_all.crt --cert tls_bip.crt --key tls.key --verboseTo see the https handshake to verify the validity of the certificates. The request can be made to any end point of interest.