BIP APIs
TODO: Confirm whether this needs to be updated after ArgoCD migration.
As of the January 2024 deployment of the BIP shared microservice, VRO is only using the BIP Claims API. While it no longer uses the BIP Claim Evidence API, the codebase for the Claims Evidence API is still present.
The above pages contain information specific to one of the individual APIs. This page provides these links and describes the details of VRO Mutual TLS implementation.
We will dive into the details of the TLS certificates momentarily, but it is first helpful to understand the big picture of how these certs are used. First, there is the certificate management system that acts as the VA's primary Certificate Authority (CA). Our CA is provided by LHDI, a Venafi system deployed as a feature of our Kubernetes infrastructure. Through Venafi, we provision and query for certificates, and once provisioned, Venafi manages the expiration of these certificates. Second, there is the generation process itself which is a bash script that lives in the VRO repo: scripts/svc-bip-api-certgen.sh. This script pulls the certs from Venafi and combines them with the intermediate certificates which it also downloads. The certs from Venafi, along with the Intermediate certificates, are then crunched through a series of OpenSSL commands to produce an output.json file ultimately. Third, there is HashiCorp Vault, where we store the generated truststore, keystore, and the password used to seal these stores. Fourth and finally, the Kubernetes secrets supply the application with the trust store, key store, and password.
Please reference the following diagram as you navigate through the remainder of this documentation:
(note: this image was created with draw.io - the png file contains the draw.io source as embedded data to the png file)
The secrets that are pertinent to these integrations can be found in HashiCorp Vault under `vro-secrets/deploy/{ENV_NAME}/VRO_SECRETS_BIP and include:
- BIP_CLAIM_SECRET
- BIP_CLAIM_USERID
- BIP_KEYSTORE_BASE64
- BIP_PASSWORD
- BIP_TRUSTSTORE_BASE64
BIP APIs and other VA services require Mutual TLS, or mTLS. VRO must use a client certificate signed by a VA-recognized Certificate Authority (CA) during https handshake. In addition, VRO needs to use a VA-recognized CA to validate BIP certificates during https handshake.
Venafi is the VA's new Internal Management System, which can be used to manage the PKI certificates. Within the VRO Kubernetes cluster, the process is further simplified, and it is possible to generate the certificates within one of the VRO Kubernetes namespaces. The client certificates that VRO currently uses have been generated using these instructions.
The certificates tls.key, tls.crt, and va.crt have been generated in the va-abd-rrd-dev namespace and can be found in the va-abd-rrd-dev-va-gov-tls secrets.
The private key tls.key is not encrypted, so no passwords are needed for the steps described later in this document.
Details for tls.crt can be displayed using
openssl x509 -in tls.crt -text -nooutThe certificate tls.crt expires on January 6th, 2024.
The certificate tls.crt contains both the client public key and an intermediate CA. Only the client public key is used and referred to as tls_bip.crt.
The certificate va.crt is not currently used.
By default, certificates generated are valid for one year and are set to auto-renew ⅔rds of the way through their lifecycle.
A second set of certificates is generated in production. This document's discussion of production certificates is identical. The production certificate expires on March 17th, 2024.
As discussed later in this document, we do not yet know a way to use these certificates directly in the Java code, and there are manual steps to make them usable. More discussions are needed with LHDI around the process of updating them.
- The certificates will be auto-renewed. What is the trigger for performing manual steps when the switch is made?
- Should LHDI make these certificates available to containers without additional steps?
- Can we forego container-level https and move that complexity to API Gateway so that LHDI manages it?
At this time, Java cert renewal is manual, and the steps in this document should be followed. We have until January 6th, 2024, to improve.
During the VRO implementation of mTLS, we were not successful using the VA CA public keys in va.crt and tls.crt.
Instead, we downloaded the public keys that were used by BIP APIs from Chrome on a GFE:
- Open Chrome on your GFE
- Open one of the endpoints in BIP Claims API. If you use a
GETendpoint, you should typically get ajwtmissing error - Once you get the error, click on the
lock(View site information) in the URL bar to bring up a pop-up window Select the "Connection is secure" menu item to bring up a second pop-up window. From this pop-up, select "Certificate is valid" to bring up the "Certificate viewer." - Go to the "Details" tab, select the certificate for root CA, and "Export". Repeat for the intermediate CA
Two certificates downloaded are
- VA-Internal-S2-RCA1-v1.crt (root)
- VA-Internal-S2-ICA4.crt (intermediate)
These public keys are concatenated in a file named va_all.crt for later use.
We have not retried using the VA CA public keys in va.crt and tls.crt after we successfully used va_all.crt. This should be revisited in the future since it is possible that the steps we eventually ended up with could be successful for public keys in va.crt and tls.crt as well.
VRO generates a set of self-signed certificates using a script called build-certificates.sh. These certificates are used for local development and end-to-end tests to mock the actual certificates and mTLS based https handshake.
More details and what is being generated are documented in the script.
We do not know a simple way to consume the certificates tls.key, tls_bip.crt, and va_all.crt directly in the Java code to utilize them in https calls. Java uses either JKS files or PKCS #12 files. Since PKCS #12 files are not Java specific, we prefer them over JKS files in VRO.
To generate the PKCS #12 file keystore.p12 for client certificates we use `OpenSSL
openssl pkcs12 -export -in tls_bip.crt -out keystore.p12 -name keystore -nodes -inkey tls.keyThis command asks for an "export password." The password is to be recorded as it will later be used as a secret.
To generate the PKCS #12 file truststore.p12 for VA CA certificates we use keytool
keytool -import -file va_all.crt -alias all_cas -keystore truststore.p12This command asks for a "keystore password." For simplicity, VRO uses the same value as the "export password."
It should be possible to use keystore or openssl exclusively here. But these PKCS #12 files worked, and we did not try either route further.
VRO stores the content of the [PKCS #12] files and the "export password" in three environment variables.
BIP_KEYSTOREBIP_TRUSTSTOREBIP_PASSWORD
Since PKCS #12 files are binary, we convert them to Base 64 using openssl
openssl base64 -in keystore.p12 -out keystore.b64
openssl base64 -in truststore.p12 -out truststore.b64What is stored in the environment variables BIP_KEYSTORE and BIP_TRUSTSTORE are the content of the Base 64 files.
In our environments, these environment variables are made available to the app container using the following secrets
- bip.bipKeystore
- bip.bipTruststore
- bip.bipPassword
In summary, the manual steps to make the certificates available to the Java code are
- Convert certificates to PKCS #12 files
- Convert PKCS #12 files to Base 64 files
- Store content of the Base 64 files in Kubernetes secrets
- Restart the pod to make secret changes effective
If the https handshake implementation remains in the Java code (as opposed to being moved to API Gateway), these manual steps need to be automated if certificate renewal is automated.
Java uses Keystore objects to store certificate information in PKCS #12 files. VRO
- Reads in the content of the PKCS #12 files and the password from the environment variables
BIP_KEYSTORE,BIP_TRUSTSTORE, andBIP_PASSWORDthrough application.yml - Converts Base 64 content to binary content
- Creates the keystore and trusstore objects as Keystore instances from the binary content and the password
- Creates a custom RestTemplate bean that can be used to make the
httpsrequests
The properties in application.yml that corresponds to the environment variables are
-
keystoreforBIP_KEYSTORE -
truststoreforBIP_TRUSTSTORE -
truststore_passwordforBIP_PASSWORD
The RestTemplate bean that is used to make the https requests is implemented in BipApiConfig. This file also includes the generation of the Keystore objects from the application.yml properties. This RestTemplate bean is autowired in the rest of the code using the Qualifier bipCERestTemplate.
Validation of the certificates using the Java code has been problematic since the BIP APIs are not available outside of VA Firewall
- Our GFE's typically do not have Java installed
- Extra permissions are necessary to install and maintain Java on the GFE's
- Additional security features in GFE's make it challenging to make https calls from Java
curl has been an invaluable tool to test the validity of the certificates. In principle, it should be possible to run curl in your GFE, but running it from the app container in one of our environments was easier. We copied the certificates tls.key, tls_bip.crt, and va_all.crt to the container using cat <<EOF and copy&paste in the /tmp directory. You should be able to use kubectl cp with enough permissions. Once the files are in the container, you can run
curl -X POST <https://......> --cacert va_all.crt --cert tls_bip.crt --key tls.key --verboseTo verify the validity of the certificates, see the https handshake. The request can be made to any endpoint of interest.