Deprecated Pages
Kubernetes Clusters
From LHDI kubernetes docs, clusters are:
- ldx-nonprod-1 (DEV, SANDBOX, QA)
- ldx-prod-1 (PROD)
- ldx-preview-1 (PREVIEW)
- ldx-nonprod (NONPROD)
- ldx-nonprod-1 (NONPROD1)
- ldx-dev (DEV)
- ldx-prod-1 (PROD1)
- ldx-preview-1 (PREVIEW1) (no gateway)
- ldx-mapi-1 (MAPI-1)
DNS:
- dev.lighthouse.va.gov
- sandbox.lighthouse.va.gov
- qa.lighthouse.va.gov
- (PROD) api.lighthouse.va.gov
Output from lightkeeper list team va-abd-rrd:
{
"name": "va-abd-rrd",
"id": "...",
"description": "va-abd-rrd description",
"namespaces": [
{
"name": "va-abd-rrd-prod",
"cluster": "ldx-prod-1",
"cpu": 128,
"ram": "256G",
"provided": true,
"imageSigningEnabled": true
},
{
"name": "va-abd-rrd-qa",
"cluster": "ldx-nonprod-1",
"cpu": 128,
"ram": "256G",
"provided": true,
"imageSigningEnabled": false
},
{
"name": "va-abd-rrd-sandbox",
"cluster": "ldx-nonprod-1",
"cpu": 128,
"ram": "256G",
"provided": true,
"imageSigningEnabled": true
},
{
"name": "va-abd-rrd-dev",
"cluster": "ldx-nonprod-1",
"cpu": 128,
"ram": "256G",
"provided": true,
"imageSigningEnabled": false
}
],
"scm": null,
"artifact_store": null,
"pipeline": null,
"registry": null,
"reporting": null,
"observer": null
}See:
Release Process
The release process requires access to the VA system. Information on the release process can be found on this page of the internal wiki.
Release Tags
Review current tags and follow Semantic Versioning guidelines to choose the next version number for the release tag.
For the desired branch (usually develop), use the create-release-tag GitHub Action workflow in the public repo to update version numbers in the code and create a release tag.
- When the release tag is created, the create-gh-release GitHub Action workflow will automatically create a GitHub Release (these auto-created GitHub releases with suffix
-autoare candidates for deletion during cleanup). - Additionally, the mirror GitHub Action workflow will copy the release tag to the internal repo, where an identical GitHub Release is auto-created in the internal repo, which will trigger SecRel GitHub Action workflow to run and sign the images. The packages/images published to GHCR should be tagged with the release tag version number (
v*.*.*orrelease-*.*.*).- The Secure Release process pushes packages to the GitHub Container Repository. Note that the packages are associated with the
abd-vro-internalrepo. - You can follow progress of SecRel (Secure Release) workflow run on the internal repo's Actions.
- If the SecRel workflow is not successful, the SecRel alerts need to be addressed -- see SecRel Processing Guide.
- The Secure Release process pushes packages to the GitHub Container Repository. Note that the packages are associated with the
All GitHub releases that are annotated as a Pre-release (and its associated git tag) will be automatically deleted by the Delete old Pre-Releases and tags GitHub Action workflow. To prevent deletion, remove the Pre-release annotation.
create-release-tag workflow runs the following:
-
./gradlew releasecreates a git tag locally (until it is pushed to the remote in the last step) -
scripts/image-version.sh pinto "Pin unpinned image versions" to the release version number created in the previous step -- see Container Image Versions -
git pushupdated image versions todevelop- triggers mirror workflow to push changes to internal repo
- triggers SecRel workflow (on internal repo) but any processing is skipped due to a gate-check, which exists to avoid unnecessary processing. SecRel will be triggered in the next step.
- triggers mirror workflow to push changes to internal repo
-
git pushrelease tag- triggers Auto-create GitHub release on public repo
- triggers mirror workflow to push changes to internal repo
- triggers Auto-create GitHub release on internal repo)
- triggers SecRel workflow (on internal repo) on release tag
Periodically, the Delete old Pre-Releases and tags runs to declutter pre-releases and git release tags the public repo. These deletions will be automatically propagated to the internal repo when the mirror workflow runs, at which point:
- Git release tags (associated with GitHub pre-releases) are deleted in the internal repo
- These tag deletions will result in GitHub releases disappearing in the internal repo, but the number of releases (shown on the front page) is incorrect (i.e., there are orphaned GH releases) -- submitted bug report to GitHub.
- 8/24/2023: Bug report closed with:
Our engineering team has flagged this issue as it appears to be a bug in displaying the releases. There is no current ETA on when the fix will be available, but I recommend you keep an eye on the GitHub Blog and our social media feeds for the latest announcements about new features and fixes. Our policies prevent us from maintaining a ticket in an open state for extended amounts of time. With this I am going to resolve this ticket. I understand that this doesn't resolve the issue reported, however it will likely be some time before any changes are made.
- 8/24/2023: Bug report closed with:
CI CD Workflows
Quoting from this CI/CD page:
- Continuous integration (CI) is practice that involves developers making small changes and checks to their code. ... this process is automated to ensure that teams can build, test, and package their applications in a reliable and repeatable way.
- Continuous delivery (CD) is the automated delivery of completed code to environments like testing and development. CD provides an automated and consistent way for code to be delivered to these environments.
- Continuous deployment is the next step of continuous delivery. Every change that passes the automated tests is automatically placed in production, resulting in many production deployments.
When the develop branch is updated (e.g., due to a PR being merged), it triggers two major GH Action workflows:
- Continuous Integration: runs various tests, including linters, unit tests, end-to-end tests, and CodeQL vulnerability check.
- SecRel: builds and publishes Docker images and has SecRel sign them if the images pass SecRel checks. These images are published to GHCR (GitHub Container Registry) so they can be used in deployments to LHDI's Kubernetes environment.
At this stage, the published Dev images have the prefix dev_ and can be deployed to the dev and qa LHDI environments.
Other environments have different requirements, so non-dev images (i.e., those without the dev_ prefix) are used to clearly distinguish them to prevent accidental deletion.
Every afternoon, an automated GH Action workflow called Continuous Delivery updates the deployment in the dev LHDI environment with the latest Dev images published (by the SecRel workflow). There is a small chance that the latest published Dev images may not reflect the latest develop commit since it can take 15 minutes for the images to be built and published.
Q1: Why not deploy with every update to the develop branch?
A: If someone is testing in the dev environment, an unexpected deployment would interfere and possibly produce unexpected behavior. Predictable deployments mitigate the likelihood of this scenario.
Q2: Does that mean we have to wait until VRO is automatically deployed to do testing? A: Nope -- see the "Manual deployments" section below.
Q3: Can I deploy and test a specific version of the develop branch?
A: Yup! See the "Manual deployments" section below.
Q4: Won't GHCR be filled with Dev images? A: We have a GitHub Action workflow for that! Delete old published DEV images runs twice a month to delete Dev images older than 30 days but keeping at least 10 of the latest versions (regardless of age) for each image. This can also be run manually with different parameters.
To manually deploy to the dev environment, use the Update Deployment workflow to update containers that have changed (the DB, MQ, and Redis containers do not typically need redeployment). Click the "Run workflow" button, then:
- (Leave "Use workflow from" as
develop) - Ensure the
devis the target LHDI environment - Choose a specific version to deploy or leave it as
latest - Choose which sets of containers to update in the deployment -- the default selections are usually fine. Helm (our deployment tool) will redeploy only images that have changed (usually only the VRO App and domain services) -- this makes deployments efficient. Unselecting an option will undeploy the associated containers -- this is only needed if there is some deployment problem. The following Helm Charts can be enabled:
- Enable & update VRO Application API
- Enable & update RRD domain services
- Enable MQ
- Enable DB
- Enable Redis
- Enable VRO Console
- For "If deploy fails, rollback to previous? (Always true for prod)", leave it unchecked so that if it fails, the failure state will be available for diagnostic investigation. Let VRO DevOps know if a deploy unexpected fails. Since this is a non-production environment, it's okay that VRO is not operational so a rollback is not needed.
- "Shut down deployment completely, then redeploy" can be useful to completely reset and restart everything; however this is not typical, so keep it unselected (false). Use this option only if deployments continue to fail and diagnostics is unfruitful.
Reminders when using this workflow:
- Make sure no one is actively testing VRO in the target environment (i.e.,
dev). - Automated deployments will still run every afternoon (as part of Continuous Delivery). So if a manual deploy is done immediately before the automated deploy, the deployment may be overridden with the latest version.
We currently do not automatically deploy changes to the production environment (prod). Rigorous manual QA testing is first conducted in the qa and prod-test environments, with fake and realistic data respectively.
Before describing deployments to the sandbox, prod-test, and prod environments, let's talk about GHCR packages and signed images.
The published images are called "packages" in GHCR, as seen here:
-
mirror-*images are copied from external sources. These images are unmodified mirrors of the original and provide services used by VRO. They are added when the Publish 3rd-party image workflow is manually run. -
vro-*images are created from the VRO codebase.- These images are typically used for deployments to the
sandbox,prod-test, andprodenvironments. - New tagged versions are added as part of the SecRel workflow when run on the
mainbranch or when arelease-*is created in the abd-vro-internal repo. - New versions of images are tagged using the first 7 characters of the commit hash or the release version (if a release was created).
- These non-dev images should rarely be deleted.
- These images are typically used for deployments to the
-
dev_vro-*images are the Dev counterparts to thevro-*images.- These images are typically used for deployments to the
devandqaLHDI environments - New tagged versions are added as part of the SecRel workflow when run on the
developandqabranches.- This allows you to address vulnerabilities quickly; from changes to the image or newly discovered vulnerabilities
- This does not delay you from testing images while secrel is failing, as unsigned images are pushed to
devandqa
- New versions of images are tagged using the first 7 characters of the commit hash.
- A
dev_prefix is used to distinguish from images used inprod, so that these Dev images can be readily deleted.
- These images are typically used for deployments to the
The tags for an image in GHCR can be seen by clicking on the image, then "View and manage all versions".
To determine if an image is signed, compare the image "Digest" with the sha256-*.sig tag -- see these instructions.
LHDI environments (clusters) pull images from GHCR. LHDI enforces the following policies for deployment to each environment:
-
devandqado NOT require signed images -
sandbox,prod-test, andprodrequire signed images
Images are signed by the SecRel (Secure Release) pipeline, which runs in the the abd-vro-internal repo.
VRO's SecRel GitHub Action workflow publishes the images to GHCR and runs the SecRel pipeline on the images, which signs them if the images pass SecRel checks). If SecRel checks fail (for various reasons), the unsigned images are still available for deployment to dev and qa for testing.
Deployments to the sandbox, prod-test, and prod environments are manual and use signed non-Dev images, which are created in 2 ways:
- Create a GitHub release in the abd-vro-internal repo (Note that a release needs to be created in the abd-vro repo first to prevent the abd-vro-internal's release from being removed due to repo mirroring). The images will be tagged with the release name (e.g.,
release-1.2.3). - Update the
mainbranch. The images will be tagged with first several characters of the git commit hash.
These actions will trigger SecRel to publish and hopefully sign the images. If SecRel fails and cannot be remedied, identify the associated "SecRel scan failed" Slack message and notify @Tom to investigate.
If the SecRel workflow succeeds, run the Update Deployment workflow with the desired image tag to deploy to the desired environment -- see the "Manual deployment to Dev" above.
Possible Errors on Windows Machines
Windows users could run into some common issues when trying to build or run VRO. If you are a windows user and run into any issues that are related to using windows, please feel free to update this page with the error and your solution.
When trying to run the app on docker, I get this error message:
Error creating bean with name 'bipCERestTemplate' defined in class path resource [gov/va/vro/config/BipApiConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.client.RestTemplate]: Factory method 'getHttpsRestTemplate' threw exception; nested exception is [gov.va](http://gov.va/).vro.service.provider.bip.BipException: Failed to create SSL context.
Caused by: java.io.IOException: Keystore was tampered with, or password was incorrect
This is because in your abd-vro-dev-secrets repo the file endings are CRLF, when they need to be LF to work. To fix this:
- Open the abd-vro-dev-secrets project into an IDE. I use intelliJ for abd-vro, but for abd-vro-dev-secrets I use VS code to keep them separate.
- In the bottom right corner of the screen, you will see that it says CRLF. Click on this and change it to LF. You will need to repeat this process for every file.
- Make sure you get the files under abd-vro-dev-secrets/local as these are the files causing the issue.
- Make sure to save every file that you made the change on and then open abd-vro
- Now set your env variables using source scripts/setenv.sh
- Run the app and see that the app now starts up as expected.
Proposal to simplify abd_vro
The code collected under abd-vro serves a several purpose:
-
To act as a parent for all subprojects relating to claim processing
-
To host the Glue code (JPA, REST, and Camel) that connects the individual sub-services
-
To host the CI and deployment scripts
It's multi-purpose nature already makes this repository pretty complex. To simplify its structure, we should aim to concentrate the source code and, especially, the configurations, in a few files as possible.
In my opinion the fact that the project structure is spread over several sub-modules is problematic as it involves several unnecessary features:
- Figure out which module depends on which and try to enforce those dependencies
- Require a build script for each module
- Require spring configuration for every module, especially for testings
- Makes it very hard to run integration tests because spring requires a full configuration for @SpringBootTest
In my opinion, the solution is to simplify the project as much as possible, and this can be accomplished by combining all the sub-modules of "app" into a single module. This module will include the following sub-modules:
- api
- app
- controller
- service (and all sub-modules of service)
- persistence (and all sub-modules of persistence)
There will be one build script and one spring boot configuration (two if you count the tests) for all these modules. That's it.
Whereas it is important that the layering of the architecture should be preserved, the layering does not need to rely on submodules. In the current structure, persistence, services, controllers, etc reside in separate gradle module. It suffices that they reside in different packages within the same module. We can leverage archunit, which we are already using, to enforce the dependencies between layers.
Overall, I don’t think this application is a good candidate for breaking up into even more microservices. When refactoring into microservices, we accept the price of additional complexity in order to gain flexibility advantages. These advantages include the ability to develop parts of the application independently as well as scale different pieces independently. If no such advantage can be gained, then we just pay a complexity price for no reason.
My reasoning for not recommending microservices can be summarized by the following points:
- This application is already an orchestration layer which coordinates the actions of several microservices into a cohesive workflow.
- The same external APIs are used at every step of the workflow: Lighthouse, MAS, BIP, and the python services. This means that there is no particular component which is responsible for communicating exclusively with some type of API.
- Everytime we interface we an external service, the probability that something will go wrong increases. If we inject additional microservices, there will be two levels of indirection before calling, say, the MAS API thus augmenting the probability of error and the difficulty of debugging
- There is actually not that much code in this application.
Nevertheless, if we really want to extract microservices, the key is to identify components that can act virtually independently of one another. I can think of the following components that satisfy this criterion:
- The database/auditing framework can become a downstream service that receives claim updates and persists them in the database and generates alerts as needed
- The v1 and v2 processing streams have their own controllers and their own camel routes. This makes them good candidates for independent services. The downside of separating them is that they share a lot of the same code, so some common libraries will need to be extracted as well.
To refactor the interaction with MAS into self contained microservice, it could include the following
- Authorization flow support components in the "app folder" (props, config)
- MAS API service components(auth, service, mapper) in the "service folder" (mas)
- Abstract the call to get the MAS collection status and annotations into a single call with options to return the response from MAS asis and mapped to a common model(abdevidence)
- Abstract the call to MAS order exam
Mock MAS API
Mock MAS API is mainly developed to validate MAS related functionality in automated end-to-end tests.
Mock MAS API is a Spring Boot based REST API.
It is implemented as a Gradle subproject mocks:mock-mas-api in abd-vro project.
The primary functionality of Mock Mas API is to provide mock collections. All mock collections are specified in a resource directory called annotations as json files. A naming convention collection-ddd.json is followed where ddd stands for the collection id.
These collections are modelled after actual collection data receieved from MAS before and during end-to-end testing.
These mock collections are stored in a Java HashMap with collection id's as keys. Spring Boot Application configuration initializes the HashMap as part of initialization of the Collection Store bean.
To facilitate end-to-end tests, this mock server keeps tracks of the collections for which exams are ordered. This information is stored in a Java HashMap with collection id's as keys. The values are Boolean objects. Spring Boot Application configuration initializes the HashMap as part of initialization of the Exam Order Store bean.
End-points are directly defined and implemented in a controller.
Three end points mock MAS functionality:
- POST
/token: retrieves a jwt. - POST
/pcQueryCollectionAnnots: retrieves a collection. - POST
/pcCheckCollectionStatus: retrieves status of a collection. - POST
/pcOrderExam: orders an exam.
The remaining end points provides information about the state of the server to facilitate automated end-to-end tests:
- GET
/checkExamOrdered/{collectionsId}: checks if an exam ordered for a collection. - DELETE
/checkExamOrdered/{collectionsId}: resets exam ordered status for a collection.
The end-point /token retrieves jwt. This end-point is a pass-through; it simply calls to MAS development server for token and returns the retrieved token.
Note that this mock server itself does not validate the token when it is attached as the Authorization HTTP header in calls to other end-points. This has been left to future development primarily due to other priorities.
The end-point /pcQueryCollectionAnnots retrieves a collection. This end-points accepts a simple json object that specifies the collection's id.
Currently MAS development server provides only one collection id'ed 350. For this collection id, this end-point retrieves the collection from MAS development server and returns the the caller. For all collection id's that identify one of the mock collections, the mock collection is return. If the collection id is not 350 or does not identify one of the mock collections, Not Found (404) status code is returned.
The end-point /pcCheckCollectionStatus currently simply returns VRONOTIFIED status. Since MAS is now preparing the collection before calling to VRO, we do not expect any other status.
The end-point /pcOrderExam mocks ordering an exam. This end-point currently always returns success. In addition the collection id for which the exam is ordered for is stored in the Exam Ordered Store.
The end-point GET /checkExamOrdered returns if an exam is ordered for a collection. This information is useful in end-to-end tests. A DELETE end point is also provided so that the mock server can be reset for a particular collection id. This mackes it possible to run end-to-end multiple times after this mock api is starred.
This mock API uses the same MAS related environment variables that VRO application uses. You can find these environment variables in the application.yml mock-mas properties.
VRO Database for RRD
VRO Database is primarily used for audit purposes. It does not store any Personal Identifiable Information (PII) or Personal Health Information (PHI). The following information is available
- Claim ID (VBMS ID), disability action type, presumptive and RFD (Ready for Decision) flags
- Claim Submission data associated with the attempt to process each Claim Submission Request through the workflow
- any external vendor ID (Reference ID — used by external vendors to track the claim in their systems)
- along with the associated type (representing the specific vendor associated with the Reference ID)
- Non PII Ids for the Veteran who owns the claim
- Contentions in the claim
- Exam Order state as issued by VRO and updated from MAS
- Non PHI summary information for the assessment for each of the contention
- Non PHI summary information for each evidence PDF generated for the contention
- Various event audit logs as a claim goes through VRO system
- Database version information to support future changes through database migrations
All tables and fields are available in the Entity Relationship Diagram (ERD). This version of ERD is manually generated using DBeaver on 3/12/2023.
Note that the database is designed for multi issue claims although iMVP will not support such claims.
VRO uses PostgreSQL. Since RDS is currently not available in LHDI, a PostgresSQL Docker based container serves as the database. The subproject postgres is used to build the container.
The Dockerfile in the postgres subproject is a simple wrapper around the Docker Hub PostgresSQL image. The primary reason to use the Dockerfile instead of using the Docker Hub image directly is to run the initialization script init_db.sh. This script creates the application database and a database user to run database migrations that are different from the default database postgres and the super user for security purposes.
The version of the database can be found in the subproject Dockerfile.
VRO achieves persistance through persistent Docker Volumes. The setup the volumes can be found in the application docker-compose file for local development and in the Helm's chart templates for deployment.
VRO uses Flyway to initialize and change the database. The subproject db-init containts all database migration related artifacts. In particular all versioned migrations are SQL based and in the directory here. These migrations create all the schemas and tables. In addition an additional user with limited privileges is created. This user is used to access the database within all non migration VRO functionality.
The subproject db-init also contains a Dockerfile. The container based on this Dockerfile is used to run the migrations both in the local development docker-compose file and in application deployment.
The strategy for creating migration files is simple: Each work branch and subsequent Pull Request should be its own contained version number. Furthermore, you should create one migration file per proposed table change. Smaller, incremental changes on a per-file basis allows for better maintainability, troubleshooting, and testability.
VRO uses Spring Data JPA to access and manage data in the database from the Java code. The subproject persistance/model contains the Object-relational mapping (ORM). In particular entity files that map to the database tables are in here.
To access and manage entity data , VRO uses JPA Repositories. All the JPA repositories are in here. These JPA Repositories inherit basic CRUD methods and also contains explicit definition of more complex methods needed by the VRO applications. In either case implementation of the methods are provided by Spring Data JPA automatically.
VRO defines 3 service interfaces (Java Interfaces) in the service/spi subproject to populate and access the database. These interfaces are
- Save to Db service: This service is used within the Camel routes to store information about claims, assessments, and evidence pdf generations.
- Claim Metrics service: This service serves as the service later for the REST interface that exposes information in the database.
- Audit Event service: This is also used within the Camel routes and logs various information about events that occur as claims are processed and external systems are called.
These services are implemented (Java Implementations of Interfaces) in the service/db subproject as Spring Boot services and are autowired in the rest of the projects. Implementations use JPA Repositories in the subproject persistance/model.
The Save to Db service and Audit Event service are primarily used in Camel routes which are defined in the service/provider.
The Claim Metrics service is used in the implementation of claim-metrics and claim-info REST end points. These end points are defined in Claim Metrics resource and implemented in Claim Metrics controller.
The Entity Relationship Diagram (ERD) shows all the tables and fields used. All the tables reside in the claims schema. These are the tables in the database
-
claim: This table stores audit information about the incoming Claims themselves (as unique entities). The corresponding entity class isClaimEntity. -
claim_submission: This table stores audit information concerning each attempt to submit and process a Claim Submission through VRO. The corresponding entity class isClaimSubmissionEntity. -
veteran: This table stores audit information about the veterans in the claims. The corresponding entity class isVeteranEntity. -
contention: This table stores audit information about the contentions in the claims. The corresponding entity class isContentionEntity. -
exam_order: This table stores audit information about the contentions in the Exam Order status. The corresponding entity class isExamOrderEntity. -
assessment_result: This table stores audit information about the assessmen results for the claims. The corresponding entity class isAssessmentResultEntity. -
evidence_summary_document: This table stores audit information about the evidence documents created for the claims. The corresponding entity class isEvidenceSummaryDocumentEntity. -
audit_event: This table stores log information as claim processing progresses through the Camel routes. The corresponding entity class isAuditEventEntity. -
schema_history: This table stores migration version information and is used by Flyway database migrations. Database migration are described in Database Migrations
The tables claim, claim_submission, veteran, contention, exam_order, assessment_result and evidence_summary_document all have created_at and updated_at columns. These columns are inherited by the corresponding Entities from a BaseEntity. BaseEntity uses Spring Data JPA @CreatedAt and @LastModifiedDate annotations to implement the functionality. With these annotations Spring Data JPA automatically populates the fields without additional code in VRO.
For the same tables id column implementation is also shared in BaseEntity.
The following are the column descriptions for the claim table.
-
vbms_id: This represents the VBMS system ID for the Claim. This is not intended as the source of truth and should be paired withsubmission_dateto determine when the last known valid associated timestamp was. -
veteran_id: Foreign key to theveterantable and identifies the Veteran the claim belongs to. -
disability_action_type: i.e.INCREASE -
presumptive_flag: Represents the Claim's presumptive status for fast-tracking -
rfd_flag: Represents a Ready For Decision (RFD) state
The following are the column descriptions for the claim_submission table.
-
claim_id: Foreign-key to the corresponding ID in theclaimtable. -
reference_id: Represents an external vendor's internal system ID for tracking the Claim. Used in addition toid_typeto identify the source of the claim. -
id_type: Represents the external source, or vendor, of the Claim. Used in addition toreference_idto identify the source of the claim. This was the constantva.gov-Form526Submissionfor V1. For V2 this constant is 'mas-Form526Submission' -
incoming_status: Status of the incoming claim. This was the constantsubmissionfor V1. -
submission_source: Taken from theclaimSubmission.claimDetails.claimSubmissionDateTimeinitially sent by MAS -
submission_date: Taken from theclaimSubmission.claimDetails.claimSubmissionSourceinitially sent by MAS -
off_ramp_reason: Explanation for why the claim was off-ramped. -
in_scope: Boolean flag representing the claim is in scope of being processed. Set by VRO. Defined in #428 but potential duplicate ofoff_ramp_reason=outOfScope— we might not need this anymore
The following are the column descriptions for the veteran table.
-
icn: The Internal Control Number (ICN) for the Veteran. It uniquely identifies the Veteran in VHA systems. -
participant_id: The Participant Id for the Veteran. It uniquely identifies the Veteran in VBA systems and is actually the database ID in the CorpDb. -
icn_timestamp: Since it is possible for ICNs to change, you can tell when the ICN was last updated with this timestamp. In theory, you could also useupdated_at, but that column could possibly apply to other pieces of data here, soicn_timestampprovides a targeted "last known good time".
The following are the column descriptions for the contention table.
-
claim_id: Foreign key that links the contention to the claim it is submitted with. -
diagnostic_code: The diagnostic code for the contention. It links the contention to the VASRD codes. -
condition_name: Name of the condition to be assessed -
classification_code: Taken from theclaimSubmission.claimDetails.classificationCodeinitially sent by MAS
Note that this design assumes multiple contentions per claim for future developments although iMVP will support only one contention (Hypertension) in single issue claims.
The following are the column descriptions for the exam_order table.
-
claim_submission_id: Foreign key that links the Exam Order to the Claim Submission that issued it. -
collection_id: Collection ID associated with the Claim Submission. -
status: The current status of the Exam Order-
ORDER_SUBMITTED: initial status from VRO, which creates a record when it issues a new Exam Order -
VRO_NOTIFIED: TODO: Seek clarification on if this will continue to remain the initial submitted status follow-up from IBM/MAS -
DRAFT: This was the initially assumed initial submitted status follow-up from IBM/MAS indicating the Exam Order is in Draft State -
FINAL: status from IBM/MAS indicating the Exam Order has been completed
-
-
ordered_at: TODO: Change tostatus_atto represent the various statuses
A Claim Submission can have multiple Exam Orders.
The following are the column descriptions for the assessment_result table.
-
contention_id: Foreign key that links assessment result to contention. -
evidence_count_summary: Summary of evidence counts for the assessments. This is a JSON objects that summarizes assessment and is provided by the assessment microservice. -
sufficient_evidence_flag: Originally from #447. Represents that the Assessment Result has determined there is sufficient evidence to mark the claim as RFD.
The following are the column descriptions for the evidence_summary_document table.
-
contention_id: Foreign key that links evidence summary document to contention. -
evidence_count: Summary of evidence counts for the document. This is a JSON objects that summarizes the information shown in the document. -
document_name: Name of the document generated. -
folder_id: Represents the UUID of the folderId returned by BIP on PDF upload, in order to facilitate easier tracking down of the file in eFolder.
This is the table structure for audit events:
-
event_id: A unique id identifying the request -
route_id: The id of the camel route from which the event is issued. Example: "mas-order-exam" -
payload_type: The type of payload being processed. Example: "Automated Claim" -
details: Other details pertinent to the event, but specific to the type of processing. For Example, collectionId, offRampReason, presumptiveFlag, et cetera. -
message: A descriptive message explaining the action. Example: "Collecting evidence" -
throwable: The stacktrace of an exception, if the even indicates an error. -
event_time: Date and time the event was issued.
Simplified ER (detailed ERD can be found here: Entity Relationship Diagram (ERD)):
VRO failure runbooks
VRO is a workflow system that uses microservices to assess the claims. It connects to MAS, BIP, and LH API. MAS starts the process by sending an eligible claim, BIP and LH API are used gather evidence/order exams, the process finishes with a PDF submitted to BIP or a claim off-ramped.
The VRO Console can query Redis and DB to determine the state of the claim
- When the APIs that VRO depends on (BIP, LH API, BGS, MAS) are down, or slow, the workflow will retry multiple times, then go into a fail state and the claim is 'stuck'
- When a microservice used in the work flow fails or times out the claim is 'stuck'
- When a notice is posted that an API or Microservice will not be available, during the time they are not available, claims will be 'stuck'
- VRO does not receive confirmation to exam order status API within 24 hours of processing the claim
- BIP Claim Evidence API is down and an exam order has been requested via the MAS API
- VRO cannot successfully retrieve data from Lighthouse Patient Health API
- Slack #benfits-vro-alerts contains "Exception occurred", This shows that a claim is 'stuck' and VRO cannot complete the workflow.
- Slack notification message: "PDF upload failed after exam order requested"
- Slack notification message: "Lighthouse health data not retrieved"
Post on #benefits-vro. Use this template:
VRO CLAIM STUCK:
Claim ID: 123 was not properly processed, @your-name investigating.
Post situation on #lighthouse-delivery-platform-support Notify MAS to stop sending eligible claims. Post to #rating_automation_collaboration. Use this template:
VRO CLAIM STUCK:
VRO is not currently processing claim. Please suspend sending claims until further notice.
In order to escalate the issue to BIP use the following contacts: email: [email protected] [email protected] is the Manger in charge for further escalation. Work phone: 650-512-2936
For BGS service failure: You can reach out to [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected] For Production we should also open a SNOW or service now ticket. (Steps TBD)
Slack #benfits-vro-alerts shows an AuditEvent for each blocked claim with no exceptions
There is a special issue flag (RDR1 flag) in the VBMS database (or CorpDB to be more specific) for each claim. As the claim is moving forward the flag gets added and after finishing each step, the flag for that step is removed indicating that the step was executed successfully. If a claim is stuck, this flag will have the indicator showing the step that was failing in.
If the BIP, LH API, BGS, or MAS APIs are not working properly, a notification should be posted on the #lighthouse-delivery-platform-support channel using the provided template. The recommended action is to verify the APIs are working properly, and if not, escalate the issue to the MAS team for further investigation.
Overall, the recovery process involves identifying the root cause of the issue, resolving it, and notifying relevant teams and channels about the status of the VRO workflow.
The following steps can help to recognize where claim was stuck and resolve it to manual process:
-
Follow instructions for VRO Console, results should follow examples in wiki
- Inspect DB
- Inspect Redis
- Inspect Camel routes
-
Verify that API are working properly
- BIP is working if...
- if not post to Slack #lighthouse-delivery-platform-support
- LH API is working if status is good
- if not working
- post to Slack #lighthouse-delivery-platform-support
- BGS is working if...
- if not working
- post to Slack #lighthouse-delivery-platform-support
- MAS is working if...
- if not working
- post to Slack #lighthouse-delivery-platform-support
-
Verify that microservices are working properly
-
If microservices show exceptions in the log that they are unable to handle a claim
- rollback to previous version of VRO
-
If data dog shows a microservice is down
- restart the microservice
-
-
If claims are stuck and VRO is not able to process additional claims
- Notify MAS team to suspend the flow of claims to VRO
- Post situation on #rating_automation_colaboration with this template
VRO CLAIM STUCK:
VRO is not currently processing claims. Please suspend sending claims until further notice.
Further more, there is a ticket for development of offramp process that will remove the flag and the claim from the system to offramp that claim to manual process.
https://github.com/department-of-veterans-affairs/abd-vro/issues/1312
What are the known/likely causes of this state
Who/how to notify that system is in this state
What to do if the recovery process does not restore the system
What indicates that the system is back to good state
What indicates that system is in this state
Steps to recover...
(March 2022) Plan to Deploy to LHDI
This page provides an architectural software plan for ABD (Automated Benefits Delivery) to own and support the productization of our current prototype RRD (Rapid Ready for Decision). This reflects our current, early thinking about how RRD can be delivered as a VA product in Lighthouse Delivery Infrastructure’s (LHDI) platform, and should be expected to evolve and solidify through further discussion, research, and planning. The product will be called VRO on LHDI or simply VRO (Virtual Regional Office).TLDR:
- We’ll migrate/port specific functionalities currently implemented in va.gov’s platform to be microservices running in LHDI’s platform.
- We’ll use LHDI’s Java Starter Kit and Apache Camel (a Java library that facilitates gluing together components in various workflows) to expose REST endpoints and route requests to microservices written in any language.
- Workflows (such as those running on va.gov — see others in the “Claim submissions trigger VRO” section) will call VRO endpoints deployed on LHDI.
Features of VRO: (more may be added in the future)
- Assess Veteran’s health data from various sources
- Order an exam when additional health data is needed
- Establish a claim in VBMS/BGS
- Set special issue on the claim to fast-track it
- Generate and upload PDF for fast-tracking the claim
- Associate supplemental PDF documents to the claim to help fast-track it
Software Design for VRO: revolves around QP components and leverages EIPs
VRO implements an event-driven architecture.
- QP (Queue-Processor(s)) component acts like an internal microservice that modularizes functionalities so that it can be updated and maintained more easily
- Consists of one Queue, which has a globally unique name and simply holds items. Items can be added to the queue by anything. Queue’s contents should be persisted to restore VRO state in case of system failure.
- A Processor processes items from the Queue. They are stateless and preferably idempotent. A Processor can be implemented in practically any language (Java, Ruby, Python, etc.), as long as it can interface with a Message Queue (such as RabbitMQ or Amazon SQS). For scalability, a Processor can be replicated to process items from the Queue in parallel — shown as “instance 1” and “instance 2” in the diagram.
-
QP components will be connected together using well-tested and stable Enterprise Integration Patterns (EIP) tools (such as Apache Camel) so that we can focus on VRO functionality and less on “glue code”.
- As the software becomes more complex, using the same QP pattern for all VRO functionalities promotes low software coupling) and, as a result, simplifies debugging and maintenance.
- Using EIPs facilitate future migration of parts of the architecture to cloud-based services if needed (see Implementing EIP with AWS and with Azure), where for example a Processor could be implemented as AWS Lambda. This is mentioned to emphasize the flexibility of the architecture to evolve in case we decide to implement some subset of the QP components in those cloud services.
- Using the Apache Camel implementation of EIPs provides time-saving features and integrations with many existing tools and data formats.
- Example workflow using QP components:
-
The Router QP responds to VRO API requests and determines how the claim should be processed
- For each new claim, it quickly validates the claim and Veteran to send a response back to the API caller.
- Based on the claim characteristics and any other data it gathers, it determines how the claim should proceed and adds the claim to an appropriate Queue for processing.
-
Each hypertension/asthma/apnea QP components is expected to:
- receive an item from its Queue (either pushed or pulled)
- gathers health data by querying data sources (e.g., Lighthouse)
- decides on the claim (see details in the “VRO processing” section)
- adds a new item to the QA Queue
-
A Manual QP component can be included for discovery and research of new types of claims to fast track, or for scenarios where a claim has curious characteristics that require manual intervention.
-
The QA Processor performs quality assurance; to validate the VRO processing output and prep it for downstream processing (external to VRO). (Not shown in the diagram: another Manual QP component can be added for cases where the QA Processor finds an unsupported problem with the results.)
-
If requested as part of the original VRO API request, the Notifier Processor will execute any requested callbacks to indicate completion of VRO processing on the claim.
External Interactions:
-
vets-api’s DB to get claimant’s info and Veteran’s ICN (needed to query Lighthouse)
-
Lighthouse’s Veterans Health API to query for medical data
-
AWS S3 to temporarily store documents for uploading to eFolder
-
EVSS to submit claim (including setting special issue and uploading PDF to eFolder)
- Lighthouse is replacing EVSS interactions with a new API
-
Central Mail Portal’s API to extract structured data from unstructured eFolder docs
- (future) BAM’s Evidence DB for health data extracted from unstructured eFolder docs
-
Central Mail Portal’s API to order an exam
-
eMIS (an API for the VADIR database) to query for military service history
Example external interactions for the hypertension QP component:
Our VRO’s software design will be deployed to the Lighthouse DI (Delivery Infrastructure, a.k.a. LHDI) platform:
-
They provide a Java-based LHDI Starter Kit - Java "to build a service [on their platform] and deploy it into production hassle free" (see out-of-the-box features).
to quickly create new services for the VA without having to spend any extra cycles on tasks that are common to all development efforts. It provides lift to the development process, such that developers do not need to build their services from scratch.
-
We expect to see Starter Kits for non-Java languages in their list of repos.
-
LHDI tools: AWS EKS, AWS Parameter Store, Istio Service Mesh and Kiali, Jaeger tracing, Helm and Argo continuous deployment server, DataDog, Prometheus and Grafana monitoring, GitHub Actions, CodeClimate, OpenAPI/Swagger, Spring Boot, Flyway DB migrations, Java and Gradle
-
The Lighthouse Platform provides devops and scalability support, so that the VRO team can focus on the VRO software.
-
Lighthouse Infrastructure Topology, code repositories, and more info here.
Our Plan: Using LHDI’s Starter Kit, migrate our code to use the Queue-Processor design pattern and deploy to the LHDI platform.
Tech Stack:
- LHDI Platform (provides Kubernetes, monitoring, and devops support)
- LHDI Starter Kit - Java (creates Docker container and provides JVM for Apache Camel)
- Message Queue and Apache Camel (provides VRO’s routing pathways among QP components)
- QP components (where Processors implement VRO functionalities in Java, Groovy, Ruby, Python, etc.)
Incremental Migration to Lighthouse DI:
- Build VRO skeleton code in Lighthouse DI with one hello-world VRO API endpoint exposed via Lighthouse VRO API (This tests plumbing, network connections, etc.)
- Incrementally build out VRO in Lighthouse DI (extract/duplicate certain functionalities in the current vets-api VRO) and expose them as Lighthouse VRO API endpoints
- Incrementally change vets-api VRO to call Lighthouse VRO API endpoints
- Gradually have other applications (such as those in Central Mail) call Lighthouse VRO API endpoints
Single vs multiple repos: Initially, the VRO source code will reside in a single GitHub repository since each QP processor will be a small chunk of code. If the code becomes large and complex, we’ll have the option of extracting pieces of it into their own repositories with minimal effort due to the low coupling of the architecture.
Shared code: Within a single code repository, any common code (such as clients to query external APIs) among the processors can be easily shared. Common client classes or libraries can be created for reuse. As the software grows in size and complexity (due to incorporating more contentions and/or more external data sources), we can create QP components whose sole purpose is to act as shared services that other QP components use, for example to request data from external systems. For either scenario, the source code to request external data would be updated in one location to encourage reuse and reduce maintenance.
Collaboration: VRO source code will be publicly available by default. Any sensitive code can be extracted into a separate private repository. Other teams can contribute to the VRO code base or they can have separate code repositories to build their own QP components to be included in the VRO architecture.
Deployment: For the Lighthouse DI platform, the QP components would be deployed together as a set of Docker containers. If we wanted to deploy QP components separately, we’d have to work with Lighthouse DI to enable deployment of individual Docker containers. If components are implemented in AWS Lambdas (not currently planned), then they could be deployed separately.
The following 3 sections describe VRO input, processing, and output/artifacts.
Expose an VRO API (via Lighthouse DI) (maybe as one of Lighthouse APIs) so that an VRO API endpoint can be called from VA.gov or CMP (Central Mail Portal) for example. Following are examples of paths for claim submissions and how they trigger the VRO process (refer to “Example workflow using QP components” diagram in prior section):
Path 1: From VA.gov to VRO
- VA.gov (vets-api) receives claim submission and calls the VRO API with claimant info and claim submission data
- VRO creates claim and quickly responds back to VA.gov with a “Success” after claim establishment
Path 2: From CMP (Central Mail Portal) to VRO
- CMP receives claim submission, optionally establishing the claim, and calls the VRO API with claim submission data, along with other relevant data from CMP
- VRO creates a claim (if needed) and quickly responds back with a “Success”
Path 3: From a “New claim” VBMS Poller to VRO
- Poller queries VBMS for new claims and calls the VRO API with claim data
- VRO quickly responds back with a “Success” if there is enough info to initiate the VRO process
Path 4: From a “New claim” Subscriber to VRO
- When the Subscriber get notified about a new claim, it calls the VRO API with claim data
- VRO quickly responds back with a “Success” if there is enough info to initiate the VRO process
For all paths, after VRO responds with “Success”, VRO continues its process to collect medical data and if possible fast-track the claim — details in next sections.
For a claim submission, VRO collects relevant data and assesses the data for fast-tracking.
- If the claim is not eligible for fast tracking (or not yet supported by VRO), then skip VRO processing, else continue.
- Collect Veteran’s health and military service history data
- via Lighthouse APIs
- from OCR+NLP of docs (via MA’s processing of eFolder docs)
- from BAM’s Evidence DB
- from VADIR via eMIS
- Assess Veteran’s health and military service history data (based on disability rating rules)
- If the claim can be fast-tracked,
- Create claim in VBMS/BGS if it doesn’t already exist
- Generate and upload PDF for fast-tracking claim
- Set
RRDspecial issue on the claim for NWQ fast-track routing
- Else order an exam if appropriate
VRO Database - not necessary (given Queue contents are persisted by the Message Queue service) but having an VRO DB offers some benefits:
- Track claim status through VRO’s process
- Log and replay requests
- Produce statistics and dashboards; enables data analyses
The expected results of VRO processing include:
- Claim record in VBMS/BGS
- VRO-generated PDF in VMBS eFolder
-
RRDspecial issue for NWQ to route the claim - Exam ordered
If a notification is warranted (to inform of updates to the claim or RRD-processing status of the claim), the Notifier QP will call other APIs as needed.
CircleCI
PR #302 Remove use of CircleCI
Followed LHDI instructions and used lighthouse-di-circleci-java17-image. CircleCI was enabled in PR #16.
The configuration runs similar but not all operations as Github Actions.
-
config.yml uses
$GITHUB_USERNAMEand$GITHUB_ACCESS_TOKEN(to pull a Docker image), which are set in CircleCI's project settings (This needs to be done in the internal repo as well to get CircleCI running there.)
CircleCI does not push container images ("packages") to the Github Container Registry. We don't want to pollute it with development packages. See .
Commit c7b786c limits CircleCI runs for only the main and develop branches to reduce the time for PR checks.
MAS api spec (IBM hosted api)
{
"openapi": "3.0.3",
"info": {
"version": "1.0.0",
"title": "Mail Automation System - VRO Integration (Automated Benefits Delivery)",
"description": "Integration with VRO via MAS-hosted APIs",
"termsOfService": "",
"contact": {
"name": "IBM Dev Team",
"email": "[email protected]",
"url": ""
}
},
"servers": [{
"url": "https://viccs-api-dev.ibm-intelligent-automation.com/pca/api/dev",
"description": "(IBM - VICCS API)"
}
],
"paths": {
"/pcCheckCollectionStatus": {
"get": {
"tags": [
"pcCheckCollectionStatus"
],
"summary": "Get the status of the collection",
"description": "Get the status of the collection .i.e. whether it is OCRed, Indexed and ready to call the annotations API",
"operationId": "getCheckCollectionStatus",
"parameters": [{
"name": "Collection Identifiers",
"in": "query",
"description": "Collection Status Request",
"schema": {
"$ref": "#/components/schemas/collectionStatusReq"
}
}
],
"responses": {
"200": {
"description": "Successful operation",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/collectionStatusResp"
}
}
}
}
},
"400": {
"description": "Invalid input value"
}
},
"security": [{
"bearerAuth": []
}
]
}
},
"/pcQueryCollectionAnnots": {
"get": {
"tags": [
"pcQueryCollectionAnnots"
],
"summary": "Get the claim details",
"description": "Get the claim details",
"operationId": "pcQueryCollectionAnnots",
"parameters": [{
"name": "Collection Identifier",
"in": "query",
"description": "Get claim details Request",
"schema": {
"$ref": "#/components/schemas/collectionAnnotsReq"
}
}
],
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/collectionAnnotsResp"
}
}
}
}
},
"422": {
"description": "Invalid input value",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
},
"default": {
"description": "unexpected error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
}
},
"security": [{
"bearerAuth": []
}
]
}
},
"/pcOrderExam": {
"post": {
"description": "Request a medical exam",
"operationId": "pcOrderExam",
"requestBody": {
"description": "Request a medical exam due to insufficient medical evidence for the condition specified in the claim",
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/orderExamReq"
}
}
}
},
"responses": {
"200": {
"description": "success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/orderExamResp"
}
}
}
},
"422": {
"description": "Invalid input value",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
},
"default": {
"description": "Unexpected error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
}
},
"security": [{
"bearerAuth": []
}
]
}
}
},
"components": {
"schemas": {
"collectionStatusReq": {
"required": ["collectionId"],
"type": "object",
"properties": {
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "integer"
},
"collectionIds": {
"description": "List of unique identifiers for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "array",
"items": {
"type": "integer"
}
}
}
},
"collectionStatusResp": {
"type": "object",
"required": [
"collectionId",
"collectionStatus"
],
"properties": {
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "integer"
},
"collectionStatus": {
"description": "Status of the collection",
"type": "string",
"enum" : ["inProgress", "processed", "offramped", "vroNotified"]
}
}
},
"collectionAnnotsReq": {
"required": ["collectionId"],
"type": "object",
"properties": {
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "integer"
}
}
},
"documents": {
"required": ["eFolderVersionRefId ", "condition", "annotations"],
"type": "object",
"properties": {
"eFolderVersionRefId": {
"description": "eFolder version Reference ID",
"type": "integer"
},
"condition": {
"description": "Claims condition",
"type": "string"
},
"annotations": {
"description": "List of Annotations",
"type": "array",
"items": {
"$ref": "#/components/schemas/annotations"
}
}
}
},
"annotations": {
"type": "object",
"properties": {
"annotType": {
"description": "Annotation Type",
"type": "string"
},
"pageNum": {
"description": "Page Number",
"type": "string"
},
"annotName": {
"description": "Annotation Name",
"type": "string"
},
"annotVal": {
"description": "Annotation Value",
"type": "string"
},
"spellCheckVal": {
"description": "Spellcheck Value",
"type": "string"
},
"observationDate": {
"description": "Observation Date and Time (YYYY-MM-DDThh:mm:ss.sTZD)",
"type": "string",
"pattern": "(^\\d{4}-\\d\\d-\\d\\dT\\d\\d:\\d\\d:\\d\\d(\\.\\d+)?(([+-]\\d\\d:\\d\\d)|Z)?$)"
},
"start": {
"description": "Start Value",
"type": "integer"
},
"end": {
"description": "End Value",
"type": "integer"
},
"acdPrefName": {
"description": "Acd Pref Name",
"type": "string"
},
"relevant": {
"description": "Is it relevant",
"type": "boolean"
}
}
},
"collectionAnnotsResp": {
"type": "object",
"required": [
"vtrnFileId",
"creationDate"
],
"properties": {
"vtrnFileId": {
"description": "Veteran File Identifier",
"type": "integer"
},
"creationDate": {
"description": "Claim creation date and Time (YYYY-MM-DDThh:mm:ss.sTZD)",
"type": "string",
"pattern": "(^\\d{4}-\\d\\d-\\d\\dT\\d\\d:\\d\\d:\\d\\d(\\.\\d+)?(([+-]\\d\\d:\\d\\d)|Z)?$)"
},
"documents": {
"description": "List of documents",
"type": "array",
"items": {
"$ref": "#/components/schemas/documents"
}
}
}
},
"orderExamReq": {
"required": ["collectionId"],
"type": "object",
"properties": {
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "string"
}
}
},
"orderExamResp": {
"type": "object",
"required": [
"status"
],
"properties": {
"status": {
"description": "Order Exam Status",
"type": "string"
}
}
},
"error": {
"type": "object",
"required": [
"code",
"message"
],
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
}
}
},
"securitySchemes": {
"bearerAuth": {
"type": "http",
"scheme": "bearer",
"bearerFormat": "JWT"
}
}
}
}
MAS api spec (VRO hosted api)
{
"openapi": "3.0.0",
"info": {
"version": "1.0.0",
"title": "Mail Automation System - VRO Integration (Automated Benefits Delivery)",
"description": "Integration with Mail Automation System via VRO-hosted APIs",
"termsOfService": "",
"contact": {
"name": "ABD-VRO Maintenance Team",
"email": "[email protected]",
"url": ""
}
},
"servers": [{
"url": "http://localhost/abd-vro/v1",
"description": "(ABD-VRO API)"
}
],
"paths": {
"/automatedClaim": {
"post": {
"description": "Notify VRO of a new claim that has been forwarded to OCR and evidence gathering",
"operationId": "addclaimsNotification",
"requestBody": {
"description": "Claims Notification Request",
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/claimsNotification"
}
}
}
},
"responses": {
"200": {
"description": "Claims Notification Response",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/claimsNotificationResp"
}
}
}
},
"default": {
"description": "Unexpected error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
}
}
}
},
"/examOrderingStatus": {
"post": {
"description": "Notify health exam ordering status",
"operationId": "examOrderStatus",
"requestBody": {
"description": "Notify health exam ordering status",
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/examOrderStatus"
}
}
}
},
"responses": {
"200": {
"description": "Acknowledge Notify health exam ordering status",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/examOrderStatusResp"
}
}
}
},
"default": {
"description": "Unexpected error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
}
}
}
}
},
"components": {
"schemas": {
"claimsNotification": {
"required": ["collectionId", "veteranIdentifiers", "dob", "firstName", "lastName", "claimDetail"],
"type": "object",
"properties": {
"veteranIdentifiers": {
"$ref": "#/components/schemas/veteranIdentifiers"
},
"dob": {
"description": "Date of Birth (yyyy-mm-dd format)",
"type": "string",
"pattern": "([12]\\d{3}-(0[1-9]|1[0-2])-(0[1-9]|[12]\\d|3[01]))"
},
"firstName": {
"description": "First Name",
"type": "string"
},
"lastName": {
"description": "Last Name",
"type": "string"
},
"gender": {
"description": "Gender",
"type": "string"
},
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "string"
},
"veteranFlashIds": {
"description": "Veteran Flash IDs",
"type": "array",
"items": {
"type": "string"
}
},
"claimDetail": {
"$ref": "#/components/schemas/claimDetail"
}
}
},
"veteranIdentifiers": {
"required": ["icn", "ssn", "edipn", "veteranFileId", "participantId"],
"type": "object",
"properties": {
"icn": {
"$ref": "#/components/schemas/icn"
},
"ssn": {
"$ref": "#/components/schemas/ssn"
},
"veteranFileId": {
"$ref": "#/components/schemas/veteranFileId"
},
"edipn": {
"$ref": "#/components/schemas/edipn"
},
"participantId": {
"$ref": "#/components/schemas/participantId"
}
}
},
"ssn": {
"description": "Veteran's Social Security number (Note: pass n/a in the absence of this field)",
"type": "string",
"default" : "N/A"
},
"icn": {
"description": "Veteran's Integration Control number (Note: pass n/a in the absence of this field)",
"type": "string",
"default" : "N/A"
},
"edipn": {
"description": "Veteran's DOD EDIPN ID (Electronic Data Interchange-Personal Identifier) (Note: pass n/a in the absence of this field)",
"type": "string",
"default" : "N/A"
},
"veteranFileId": {
"description": "Veteran File ID (a.k.a. BIRLS ID or CorpDB filenumber or VBMS filenumber)\n\nBIRLS : Beneficiary Identification Records Locator Subsystem\nVBMS: Veteran Benefits Management System\nCorpDB: VA Corporate Database (Note: pass n/a in the absence of this field)",
"type": "string",
"default" : "N/A"
},
"participantId": {
"description": "Veteran's participant id",
"type": "string",
"default" : "N/A"
},
"claimDetail": {
"required": ["benefitClaimId", "claimSubmissionDateTime", "claimSubmissionSource", "veteranFileId", "conditions"],
"type": "object",
"properties": {
"benefitClaimId": {
"description": "Benefit Claim Identifier",
"type": "string"
},
"claimSubmissionDateTime": {
"description": "Claims Submission Date and Time (YYYY-MM-DDThh:mm:ss.sTZD)",
"type": "string",
"pattern": "(^\\d{4}-\\d\\d-\\d\\dT\\d\\d:\\d\\d:\\d\\d(\\.\\d+)?(([+-]\\d\\d:\\d\\d)|Z)?$)"
},
"claimSubmissionSource": {
"description": "Claims Submission Source VA.gov or MAS",
"type": "string",
"enum" : ["VA.GOV", "MAS", "OTHER"]
},
"conditions": {
"$ref": "#/components/schemas/claimCondition"
}
}
},
"claimCondition": {
"required": ["diagnosticCode"],
"type": "object",
"properties": {
"name": {
"description": "Claim Condition Name",
"type": "string"
},
"diagnosticCode": {
"description": "Claim Diagnostic Code",
"type": "string",
"enum" : ["7101"]
},
"disabilityActionType": {
"description": "Claim Disability Action Type",
"type": "string",
"enum" : ["INCREASE", "NEW"]
},
"disabilityClassificationCode": {
"description": "Claim Disability Classification Code",
"type": "string",
"enum" : ["3460", "3370"]
},
"ratedDisabilityId": {
"description": "Claim Rated Disability ID",
"type": "string"
}
}
},
"claimsNotificationResp": {
"type": "object",
"required": [
"id",
"message"
],
"properties": {
"id": {
"description": "Unique ID to identify the transaction (for audit and debug purpose)",
"type": "string"
},
"message": {
"type": "string"
}
}
},
"examOrderStatus": {
"required": ["collectionId", "collectionStatus"],
"type": "object",
"properties": {
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "string"
},
"collectionStatus": {
"description": "Claim Collection Status",
"type": "string",
"enum" : [ "DRAFT", "FINAL", "ERROR"]
},
"examOrderDateTime": {
"description": "Exam order Date and Time (YYYY-MM-DDThh:mm:ss.sTZD)",
"type": "string",
"pattern": "(^\\d{4}-\\d\\d-\\d\\dT\\d\\d:\\d\\d:\\d\\d(\\.\\d+)?(([+-]\\d\\d:\\d\\d)|Z)?$)"
}
}
},
"examOrderStatusResp": {
"type": "object",
"required": [
"id",
"message"
],
"properties": {
"id": {
"description": "Unique ID to identify the transaction (for audit and debug purpose)",
"type": "string"
},
"message": {
"type": "string"
}
}
},
"error": {
"type": "object",
"required": [
"code",
"message"
],
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
}
}
},
"securitySchemes": {
"ApiKeyAuth": {
"type": "apiKey",
"description": "X-API-KEY:valid-api-key",
"name": "X-API-KEY",
"in": "header"
}
}
},
"security": [{
"ApiKeyAuth": []
}
],
"tags": []
}PDF Generator
The PDF Generator contains all the different templates used to generate documents by providing the appropriate data in the generation requests. The following libraries are available for rendering:
- WKHTMLTOPDF: https://wkhtmltopdf.org/
- WeasyPrint: https://weasyprint.org/
The PDF Generator routes allow you to specify which library you wish to use and if it is not provided in the request, it will default to WKHTMLTOPDF
On pdfgenerator startup, the consumer will attempt to create a generate-pdf, fetch-pdf, and generate-fetch-pdf queue on a pdf-generator exchange.
- Request PDF Generation - POST /evidence-pdf
- Fetch Generated PDF - GET /evidence-pdf/{claimSubmissionID}
Both functions work off the same endpoint. The main difference being that the POST has a JSON request body while the GET uses the URL(claimSubmissionID) to find the corresponding PDF.
This endpoint is a combined version of the generate-pdf and fetch-pdf. Pass it the POST request body and it will respond with the PDF associated with the specified claimSubmissionID
Any messages passed to the generate-pdf queue will use the diagnosticCode, pdfTemplate and pdfLibrary(optional) to select the appropriate template and generate the PDF. diagnosticCode is used to translate this into a human readable diagnostic type based on the mapping in config/settings.py in the codes variable.
Using pdfTemplate like v1 along with a diagnostic type like hypertension for example, it will pull up the appropriate template and template variables in the templates and template_variables folder respectively. For example, "diagnosticCode"=7101 and "pdfTemplate"="v1"will fetch templates/hypertension-v1.html and template_variables/hypertension-v1.json.
The generator will first load the hypertension-v1.json file which is prefilled with default values for the available variables within the template and attempt to replace them with what is provided in the message. If it cannot replace the data based on what's provided in the request, it will keep the default that is defined in the JSON file.
After the HTML template is generated with the replaced variables, it uses WKHTMLTOPDF by default or the library selected by pdfLibrary to create a PDF based on the HTML file.
Once the PDF has been generated, the consumer will create a key value pair in Redis to store the data due to it containing PII information. The key being claimSubmissionId while the value is a base64 encoded string representation of the PDF.
The consumer will return a response similar to:
{
"claimSubmissionId: "1",
"status": "COMPLETE"
}
When the consumer receives a message in the fetch-pdf queue, it will use the provided claimSubmissionId to look it up on Redis.
If the PDF still hasn't been generated, you will receive a response similar to:
{
"claimSubmissionId: "1",
"status": "IN_PROGRESS"
}
but if the PDF is available then the response will be a downloadable file
ToCs are not part of the normal HTML template that gets generated for the PDF. They need to be created through a different process and merged with the main PD template
The PDF generator will check if there is a ToC file already created for the diagnosticCode that gets passed. If not found, it will generate the PDF without a ToC so you don't have to worry about having a empty section or page
-
Create a directory in
templateswhere the name will be the human readable diagnosis name used in thecodesvariable insettings.py -
Within this folder, create a
toc.xslfile. Most ToCs will follow the same format so you can just copy one from any other diagnosis if available. If you needed to create one from scratch, in the command line run the following:wkhtmltopdf --dump-default-toc-xsland copy the contents of the output to a newtoc.xslfile as stated above -
By default a ToC is generated by finding all
<h?>related tags(<h1>, <h2>, etc) so you need to modify them if you want them ignored.- To ignore a heading, it must start with
‌like this example:<h3 class="text-center">‌Test PDF Heading</h3>. Thetoc.xslfile has logic in place to skip over any headings that start with this special character. This character was used since it's an invisible character so it won't render on the PDF
- To ignore a heading, it must start with
-
The ToC page is fully customizable just like any HTML page
- The library was built using Webkit 2.2 (~2012) and QT4 (2015) so many newer HTML features are unavailable or need to be added through other ways for them to render properly
- This library renders at 96DPI but the value can be altered through the meta tags. We need to verify that the Figma design or other design software matches the proper DPI settings by making sure the resolution matches the paper size. Use the following links for proper conversions: https://a-size.com/legal-size-paper/ and https://www.papersizes.org/a-sizes-in-pixels.htm as well as https://pixelsconverter.com/inches-to-pixels
Work in Progress
- This library does not accept Javascript. At the moment, we would need to come up with a workaround by prerendering in a secondary library or just using
WKHTMLTOPDFfor Javascript specific portions but this solution has yet to be implemented. - This library renders at 96DPI and the value cannot be changed. We need to verify that the Figma design or other design software matches the proper DPI settings by making sure the resolution matches the paper size. Use the following links for proper conversions: https://a-size.com/legal-size-paper/ and https://www.papersizes.org/a-sizes-in-pixels.htm as well as https://pixelsconverter.com/inches-to-pixels
Before you can start working on the content of the PDF design, you must first match your DPI to the size dimensions of the design tool. If you skip this step, then the measurements will be all wrong and the design won't be a perfect match. This in turn will cause the developer to modify/test random values or mess with the zoom setting to get it to fit the design.
For example, if a user wants to make a new PDF, they must:
- Get the size/dimension that the design team wants to use. Not just whether its A4, Legal, etc. but the pixel dimensions of the blank design page. We will use this number to set or match the DPI accordingly
- Setup the new template to match based on the library:
-
WKHTMLTOPDF: DPI is customizable so you can use the following for proper conversions:- Legal and other sizes like A4, etc.: See what DPI setting the pixel dimensions fall under and set the meta tag to that DPI once you make the template file in the next step
-
WeasyPrint: The DPI value is set to 96 and cannot be changed. Due to this, the process is somewhat backwards. The developer needs to use the above links to get the dimensions based on the requested page size and 96 DPI and send the dimensions back to the design team so they can adjust their document to match.
- Edit the
codesdictionary inconfig/settings.pyby adding a new key, value pair for your code.- Example:
codes = { "0000": "cancer", "0001": "diabetes" //new code with human readable diagnosis } - Create a HTML version of the PDF you want to generate and save it in the
templatesfolder along with a version number- Take a look at Jinja 2 Template Documentation for a better idea of what you can do within the template files
- Every template file needs a version number. By default, the system looks for
v1if one is not specified in the request - The file name should match the name you used in Step 1. Following that example, it should be called
diabetes-v1.html
- Every template file needs a version number. By default, the system looks for
- Take a look at Jinja 2 Template Documentation for a better idea of what you can do within the template files
- Create a JSON file in
template_variablesthat will contain default values for the HTML file in case they are not provided in the RabbitMQ message- The file name should match the name you used in Step 1 and 2. Following that example, it should be called
diabetes-v1.json
- The file name should match the name you used in Step 1 and 2. Following that example, it should be called
Some diagnostic codes might need specific changes that don't need to affect other templates and instead of adding it to the assessment logic, we can use a helper function.
When generating a PDF, it will look for a helper function following this naming convention pdf_helper_0000 where 0000 would be the code we want to use. If it does not find it, it move on and then applies a pdf_helper_all that gets applied to every single template. Usually these are edits like turning date string into proper date time objects, etc that would benefit all the templates.
Currently there are 2 ways to develop/test the PDF service:
- Run
./gradlew build check dockerto build all containers and run a full test. This can be used for the testing any updates that are made to the endpoints through Swagger but it takes longer due to having to load all the containers. After the containers are built, you can take it a step further and run the containers themselves using./gradlew app:dockerComposeDown app:dcPrune app:dockerComposeUpand then heading to the Swagger page to view and run the available endpoints. - Run
python pdfgenerator/src/lib/local_pdf_test.pyfrom theservice-pythondirectory. This file calls the PDF generator while bypassing all the related RabbitMQ and Redis code. You can alter thediagnosis_nameandmessageto simulate an endpoint request and to quickly debug any template or PDF issues. Thediagnosis_nameshould be the full name including version number likehypertension-v1
MAS Integration Camel Routes
VRO v2 is using Apache Camel to coordinate several services into the workflow described by this document.
Apache Camel provides a lot of message-oriented abstractions but it is using it is own DSL (Domain Specific Language) which can be difficult to master. Familiarity with Apache Camel is a prerequisite for understanding the code. A brief overview of Apache Camel can be found here.
The MasController implements the two endpoints:
-
v2/examOrderingStatus: This endpoint does not do anything except record the fact that it has been called. -
v2/automatedClaim: This endpoint kicks off the functionality for automated claim processing.
The rest of the document will be referring the diagram below (from the VRO v2 Roadmap) and is dedicated to explaining the implementation of the pictured workflow.
The endpoint v2/automatedClaim is handled by MasController which immediately hands off to MasProcessingService.
MasProcessingService is responsible for implementing the logic contained in the topmost yellow box of the diagram, where several checks are performed to verify if the claim qualifies for automated processing. If any of these checks fails the API returns a response explaining the reason. This is the only part of the code that executes synchronously. From that point on, a message is sent and the process continues asynchronously.
The following response indicates that the claim is not in scope:
{
"id": "c9fda1ac-2422-47e1-aeff-6c0a2b08d6df",
"message": "Claim with [collection id = 351], [diagnostic code = 7101], and [disability action type = FD] is not in scope."
}The following response indicates that one of the anchor tests failed:
{
"id": "b34bc26a-68b1-4c08-bca0-ce28db9c4c98",
"message": "Claim with [collection id = 351] does not qualify for automated processing because it is missing anchors."
}The following response indicates that the claim passed the initial checks and is undergoing processing, which can potentially take a long time.
{
"id": "a14fd4e5-abe0-48b7-95df-3f9c85164a3a",
"message": "Received Claim for collection Id 350."
}The definitions of all the relevant Camel routes are in MasIntegrationRoutes. This class implements RouteBuilder which provides access to the constructs of the Camel DSL. The configure() method is the entry point and it is divided into several logical groupings of routes:
-
configureAuditing: Sets up routes for audit messages. Audit messages end up in the database and in some cases appear as Slack notifications. A full description of the Auditing framework is given in a different section of this document. This method also implements exception handling. -
configureOffRamp: This handles any claims that are off-ramped (rightmost path on the diagram) and forwards them to the "Complete Processing" step. -
configureAutomatedClaim: Integration with MAS requires that we send a request for a collection ID and then keep poling periodically for status. This is implemented by storing a message in a queue with a specific delay. Once the delay elapses, the message triggers a query for status via the MAS API. This logic is implemented in "MasPollingProcessor". If the collection ID is not ready, the message is requeued with a delay. When the collection is ready, the message is forwarded to the internal mas-processing endpoint. -
configureMasProcessing: This route orchestrates the entire workflow by delegating the processing steps to other routes. It performs the following steps:- Calls the collect-evidence endpoint to collect evidence from different sources
- Calls Health Assessment to assess the evidence (second yellow box in the diagram)
- Conditionally calls "order exam" (third yellow box)
- Generates and uploads the final PDF file (fourth yellow box)
-
configureCollectEvidence: Collect evidence from the MAS and BIP APIs. Merge the two evidence objects and call Health Assessment via RabbitMQ. -
configureUploadPDF: This route corresponds to the penultimate yellow box in the diagram. It calls the service to generate PDF via Rabbit and the calls a BIP endpoint to upload the PDF> -
configureCompleteProcessing: This is the last processing step, corresponding to the last yellow box in the diagram. Both the off-ramp claims and the ones that have been processed converge on this route. Special Issue is removed via the BIP API. If there is sufficient evidence for fast-tracking, the claim is marked as RFD also via the BIP API. -
configureOrderExamStatus: This endpoint simply records an audit event to record the fact that the REST endpoint v2/examOrderingStatus has been called.
Auditing for VRO v2 is more generic than auditing for VRO v1. Instead of having a table structure mirroring claim requests, we have a generic event data model that can contain information about any type of event.
An audit trail is created for each API request. When an entry point is invoking a UUID is created to track the request. This ID can be used to identify all events connected to the request.
Since the workflow interacts with several services, it is unavoidable that exceptional conditions will occur from time to time. Exception handling is configured as part of auditing. Every exception is caught and fed into two streams: One stream sends a notification to a slack channel, whereas the other stream posts the exception in audit_event table in the database.
A description of the data model can be found here: Audit Data Model
The auditing framework is designed to satisfy the inconsonant objectives of modeling VRO claim requests and also be completely generic. It achieves this via a layer of abstraction:
- Any object that needs to be audited in the database must implement the Auditable interface:
public interface Auditable {
String getEventId();
String getDetails();
String getDisplayName();
}Auditable objects can be converted into AuditEvent objects by means of the method AuditEvent.fromAuditable(). Audit events can be triggered asynchronously via wireTap. For example:
private void configureOrderExamStatus() {
// This route does not do anything, but an audit event is persisted
String routeId = "mas-exam-order-status";
from(ENDPOINT_EXAM_ORDER_STATUS)
.routeId(routeId)
.wireTap(VroCamelUtils.wiretapProducer(EXAM_ORDER_STATUS_WIRETAP))
.wireTap(ENDPOINT_AUDIT_WIRETAP)
.onPrepare(auditProcessor(routeId, "Exam Order Status Called"))
.log("Invoked " + routeId);
}The command wireTap(ENDPOINT_AUDIT_WIRETAP) sends a wiretap to the audit endpoint, the the onPrepare command is responsible from mapping the current object (an Auditable) to an AuditEvent.
Slack messages are also sent as part of the auditing framework, except in this case an AuditEvent is converted into a user-friendly string. Here are some examples:
AuditEvent{routeId='/automatedClaim', payloadType=Automated Claim, message='Claim with [collection id = 351] does not qualify for automated processing because it is missing anchors.}
AuditEvent{routeId='/automatedClaim', payloadType=Automated Claim, message='Claim with [collection id = 351] does not qualify for automated processing because it is missing anchors.}
AuditEvent{routeId='/automatedClaim', payloadType=Automated Claim, message='Claim with [collection id = 351], [diagnostic code = 7101], and [disability action type = FD] is not in scope.}
Exception occurred on route mas-claim-processing for Automated Claim(id = a14fd4e5-abe0-48b7-95df-3f9c85164a3a): Error in calling collection Status API .
Please check the audit store for more information.
Apache Camel makes some things easy and some things hard. One of the hard things is that Camel does not enforce type safety as a message travels from route to route. If the output type of a route does not match the input type of the next route, you are looking at a runtime exception.
I have found that the best way to debug Camel is to inject a processor between two routes and set a breakpoint to examine the contents of the message.
For example, consider the following snapshot:
.routingSlip(method(slipClaimSubmitRouter, "routeHealthSufficiency"))
.unmarshal(new JacksonDataFormat(AbdEvidenceWithSummary.class))
.process(new HealthEvidenceProcessor()) A message is processed via a routingSlip, then converted from JSON to the type AbdEvidenceWithSummary which is the input type of HealthEvidenceProcessor. Suppose this conversion does not work for some reason and we want to know why.
We can interject a processor between the two steps of interest like so:
.routingSlip(method(slipClaimSubmitRouter, "routeHealthSufficiency"))
.process(
new Processor() {
@Override
public void process(Exchange exchange) throws Exception {
var message = exchange.getMessage();
var body = message.getBody();
System.out.println(body);
}
})
.unmarshal(new JacksonDataFormat(AbdEvidenceWithSummary.class))
.process(new HealthEvidenceProcessor()) // returns MasTransferObjectand in this way we can examine the details of the exchange, the message, and the message contents.
External APIs (Partial)
Mail Automation Systems (MAS) fast-tracks claims from CMP (Central Mail Portal). The fast-tracking capability will eventually be ported to VRO.
VRO will need to trigger OCR+NLP processing of eFolder documents for veterans of eligible claims to extract health data as evidence for fast-tracking.
- Would be preferable if there was a Lighthouse API for this.
To retrieve the OCR+NLP results, VRO will query MAS directly until the Claim Evidence API is available.
VRO may want to automatically order a medical exam using MAS's exam-ordering capability.
- According to Boise RO, the value of auto-ordering exams for CFI (claim for increase) was pretty straightforward and accurate.
- Uncertain if VRO should auto-order exams for new claims.
How to access MAS endpoints
-
Call the auth server API with the required scope, grant type, and credentials to obtain the JWT token.
Token Endpoint: https://{baseurl}/pca/api/{environment}/token
scope : openid
grant_type : client_credentials
client_id: {client id}
client_secret: {client secret} -
Check the status of a collection by invoking the "collection status" API with the JWT minted above as the bearer token. https://{baseurl}/pca/api/{environment}/pcCheckCollectionStatus
Link to open api spec: https://github.com/department-of-veterans-affairs/abd-vro/wiki/MAS-api-spec-%28IBM-hosted-api%29
-
Call the "Collection Annotations" endpoint only if the collection is ready to be processed with the JWT minted above as the bearer token. https://{baseurl}/pca/api/{environment}/pcCheckCollectionStatus
Link to open api spec: https://github.com/department-of-veterans-affairs/abd-vro/wiki/MAS-api-spec-%28IBM-hosted-api%29
-
Call the "Order Exam" endpoint if the evidence is not sufficient for the claim with the JWT minted above as the bearer token. https://{baseurl}/pca/api/{environment}/pcOrderExam
Link to open api spec: https://github.com/department-of-veterans-affairs/abd-vro/wiki/MAS-api-spec-%28IBM-hosted-api%29
See EVSS
If a claim is fast-track-able, VRO will need to upload a generated PDF to eFolder and associate the PDF to the claim.
- VRO should use LH's new EVSS-replacement API (timeline? suitability for VRO?)
The following is ordered according to when it will be needed.
- ✅ health evidence data - LH Patient Health API (FHIR)
- mark claim as being assessed for fast-tracking so that users don't work on the claim
- mark using a VBMS/BGS "station 398" and "Rating Decision Review - Level 1" special issue? Paul Shute says it's not a good long-term solution
- LH doesn't currently have this capability
- The
RRD(Rapid Ready for Decision) special issue also needs to be set.
- upload PDF to eFolder and associate the PDF to the claim
- LH is working on a new EVSS-replacement API (file upload service) to do this: timeline? suitability for VRO?
- claim details (diagnostic codes) using a claimID
- LH Benefits Claims API response is missing diagnostic codes
- veteran details (SSN, name, DOB) to generate a PDF
- LH Veteran Verification APIs doesn't return veteran information like SSN, name, DOB
- MAS claim notification to VRO (for VRO 2.0)
- Expose VRO endpoints as a Lighthouse API for MAS to use?
- query VBMS and BGS to verify some things (MAS does this for CMP-submitted claims)
- TBD: Need to determine MAS features that will be ported to VRO. Can we do the queries via Lighthouse?
- request OCR processing from MAS and retrieve OCR results from MAS
- VRO will connect directly to MAS
- listen for BAM/BIA contention event notifications (for VRO 3.0)
- VRO will need to set up a event subscriber directly with BIA's Kafka
- retrieve OCR results from BAM's Claims Evidence API
- VRO will connect directly to CE API
- query API to map given veteran identifier into an ICN, which is needed to query LH health data
- LH Benefits Claims API v2 accesses MPI to do this, but the veteran's SSN, name, and DOB is needed. Spoke with Derek but will need new "SR" to access MPI FHIR API to enable lookup by file number (a.k.a., BIRLS ID)
- mark claim as Ready for Decision (RFD)
- This is not required for single issue CFI, but will be needed for handling multi-issue claims. We should be guided by the AIM project on when this is needed.
Not yet incorporated into ordered list above:
-
veteran service history for presumptives
- perhaps the Veteran Verification APIs, which uses eMIS, which is an API for VADIR (VA/DoD Identity Repository)
- Pact Act CorpBD flashes
- requested to be added to LH Veteran Verification API but it doesn't currently support CCG (machine-to-machine) authentication
Initial Roadmap for VRO's RRD Implementation
This pages answers questions like: Where is the software heading? How will we get there and in what order? What dependencies need to be satisfied?
High-level visualization:
- version 0.1: Minimally functional software; receives request, routes to dummy processor, returns response
- includes (Ruby) PDF generator from RRD
- (subsequent versions can be created to migrate other RRD functionalities to VRO (e.g., claim establishment, uploading docs for the claim)
- assess-data: Assess fast-tracking eligibility given all health data (as part of the request)
- assess-claim: Assess fast-tracking eligibility given claim (VRO must query for health data)
- use VRO from vets-api: RRD code (in vets-api) uses VRO API (deployed in prod)
- save-to-db: Save VRO claims processing to DB for reporting and diagnostics
- version 1.0: VRO actively used by va.gov-submitted claims
- (subsequent version will depend on readiness of presumptive claims, OCR data, CMP claims, and VBMS notifications)
- Software approach/architecture: Plan to Deploy to Lighthouse DI
- Software Design Document
- CamelApp demonstrates using Apache Camel for microservices to prove out the architecture
- VRO v2 Roadmap - iMVP (Integrated MVP)
- VRO v1 Roadmap - MVP
Architecture diagram (DrawIO diagram source)
What is the Future State of VRO?
It depends. There are several external factors outside of the VRO team that influence what VRO will look like. One possibility is:
Track this evolution in ticket #92.
Amida Hand-off Resources
Links to documents, videos, and any resources regarding the Amida hand-off to OCTO.
- SD Elements + cATO handoff video / folder
- Camel Route video session 1 / video session 2 / folder
- BIP API video / folder
- Lighthouse API video / folder
- VRO Environment Cross walk Spreadsheet: VRO Environment Crosswalk
- MAS API video / folder
- RRD logic and PDF Generator video 1 / video 2 / video 2 (slack) / 101 notes (sharepoint) / 101 notes (google) / folder
Problems being solved
(Much of the following was extracted from ABD Vision slides, drafted May 2022.)
- The initial VA.gov Benefits Automation prototype was built inside vets-api to prioritize speed to production and ability to learn/iterate quickly
- This wasn’t a long term home. Scaling inside vets-api would burden the normal 526 submission pathway and add unnecessary complexity. Benefits automation should be a separate, VA-owned service called by vets-api.
- The MAS Benefits Automation logic is currently a managed service (owned by IBM contractors)
- This creates a dependency with a proprietary codebase and reduces the ability to collaborate and learn.
- It is costly and confusing to build Benefits Automation processing in separate prototypes when they should be as consistent as possible regardless of claim source (i.e., mail, fax, or online).
We will
- Dedicate resources to architecting and building a shared service that processes claims faster regardless of intake source (i.e., this VRO software)
- Cut off prototype duplication (i.e., stop feature development on the RRD prototype)
- Ensure all benefits automation logic exists in the shared service, which is open-source by default and owned by VA (i.e., VRO will have an API, hosted by LHDI)
Conceptual vision:
(Note that VRO may be notified of claims from CMP, VA.gov, and other claim sources by VBMS event Pub-Sub mechanism.)
Roadmap
- Deferring too much until after migration to Lighthouse DI. Existing pain points and areas for enhancement grow the longer we discuss and work toward the “long term solution.” Can we identify smaller, incremental steps that can reduce problems, add value, and be shipped sooner?
- Doing this architectural migration work at the same time we’re trying to ship as many presumptive features as possible w/ potential legislation implementation deadlines. When we migrate existing functionality to the new platform, it will temporarily slow development of new functionality.
- Scaling teams and cross-team operations in tandem with delivering the new platform alongside key new functionality to support presumptive claims. Can we clearly define areas of ownership and quickly align on standard ways of working through collaboration to establish a "one team" culture?
POCs
Communication is key!
VA Slack channel: #benefits-virtual-regional-office
(Refer to Roadmap for details on particular topics)
- Yoom - VRO vision and architecture
- relevant skills: Java, Ruby, Groovy, Gradle, Spring, Docker, Bash
- Seth - full stack, SecRel
- relevant skills: Python, Java, Kotlin, react, JS, CSS, Docker, AWS, FHIR/HL7, etc.
- Yang - full stack, vets-api
- relevant skills: Python, Ruby, JS, Bash
- Mason - devops, full stack
- relevant skills: Java, Python, Docker, AWS, Kubernetes
- Ilya - backend SDE for VRO(?)
- relevant skills: Java Spring, Node.js, Ruby/Rails
- Cheng - full stack
- relevant skills: Java, Spring, Javascript/Typescript, Docker, AWS
- Derek - backend
- relevant skills: Java, Spring, AWS, Python
- Teja - full stack, devops, architecture
- relevant skills: Typescript/JS, AWS, Python, Java, Ruby/Rails, NodeJS, Docker, React, etc
Current Software State (Oct. 2022)
This page answers questions like: In what state is the software? What can the current software do? Where's the changelog?
Please add new significant changes and capabilities in reverse chronological order below. The headings below refer to the blocks illustrated in the Roadmap.
- Integrate with PDF generation end points (TBD).
- Save
assess-claimoutput to DB for reporting and diagnostics (TBD). - Pull request #172 integrated with
assess-claimend point on the input level. Wheneverassess-claimend point is called the veteran and claim information is now saved to the database for reporting and diagnostics. - Pull request #166 integrated the flyway scripts and db initiation to the code base.
- Pull request #123 started to integrate
save-to-dbfunctionality into the codebase. - Save to DB integrated in camel route "access claim" (multiple PRs)
- The end-point
\health-data-assessmentis now ready for v1.0 and can be tried out from Swagger UI (local only until devops make dev and qa environments ready) for patients in Lighthouse Health API Sandbox. All the json field are also available from Swagger UI and matches fields used by va.gov. The following are active fixes and improvements.- Tests and linting for Data Access service (which is the code base for the Lighthouse Health API interaction) are not integrated to abd-vro build.
- Since current Hypertension PDF includes medications we added medications to health data for diagnostic code 7101. Pull request #181
- VRO Swagger documentation is updated in #169 and #170. The end point can be tried out from Swagger UI.
- Pull request #174 corrected a few fields in medication query to match fields used by va.gov.
- Queries for health data from Lighthouse Health API are integrated to the code base in #163. Authorization, blood pressure query for diagnostic code 7101, and medication query for diagnostic code 6602 are implemented. Conditions and procedures are also included but are preliminary. Blood pressure fields are identical to fields used by va.gov.
- The end-points
\evidence-pdf(POST and GET) are now ready for v1.0 and can be tried out from Swagger UI (local only until devops make dev and qa environments ready). All the json field are also available from Swagger UI and matches to assess-claim. You can use output of assess-claim end-point \health-data-assessment to generate input for\evidence-pdf. - JSON inputs to the endpoints are improved, Swagger documentation is added and full JSON object integration with
assess-claimis achieved in Pull Requests #172 and #178. - Python service code structure is improved and code coverage is added in Pull Request #168.
- PDF Generation started to use Redis instead of S3 and integrated to demo end points in Pull Request #143.
- Testing is integrated to CI/CD pipeline in Pull Request #137.
- V0.1 Generator is disabled in Pull Request #131.
- A series of Pull Requests (#126, #127, #128, #129, #130) integrated PDF generation into the codebase. They implement a stand alone Python service that generates PDFs for Hypertension and Asthma and saves it to S3.
This provides further assessment calculations on the data provided by assess-claims (such as predominant blood pressure reading). Is not necessary for V1.0 since it is not part of the current va.gov functinality.
- Creates a queue to which claim data is passed and responds with a decision on fast track-eligibility and supporting evidence. Messages to the queue must be routed with their VASRD DC code connected to the claim, which in this case is "7101". #112
- Initial VRO API with
generate_summary_pdfandassess_dataendpoints (PRs #71 and #109) - Deployed RabbitMQ and Ruby microservice containers to LHDI's
devenvironment (PR #69) - Ruby PDF generator (and health data assessor) ported from RRD prototype (#63)
- Added RabbitMQ and initial Camel routes (PR #50)
- Deployed to LHDI's
devenvironment (PR #47, along with #46 and #44) - Enabled CircleCI checks to new pull requests (PR #16)
- Used LHDI's Java Starter Kit to populate the initial codebase (PR #8) using Java (AdoptOpenJDK) 17 and Gradle 7.4
See all merged PRs.
VRO Console (May 2024)
To investigate and recover from errors particularly in the production environment, a VRO Console container was implemented and took inspiration from Rails Console.
The VRO Console container (or simply Console) facilitates diagnostics, such as examining the processing state of the claim by looking at Camel routes and the DB contents, and realtime updates to VRO's state in Redis and VRO's DB.
-
export COMPOSE_PROFILES=debug-- the console container only starts if the docker-composedebugprofile is enabled. - Start VRO. Note the
vro-console-1container; check out the logs:docker logs vro-console-1. - Attach to the container:
docker attach vro-console-1
If you need to restart the console container:
cd app/src/docker
docker-compose up -d console
docker attach vro-console-1
In order to connect to Console container deployed in LHDI, set up kubectl using the Lightkeeper tool.
Next to connect to the DEV deployment:
# For convenience
❯ alias kc='kubectl -n va-abd-rrd-dev'
❯ kc get pods
NAME READY STATUS RESTARTS AGE
vro-api-645dc44c64-w95mw 0/6 Init:1/3 0 7m52s
vro-api-postgres-559c5bddbb-7rm2r 1/1 Running 0 7m53s
vro-api-rabbit-mq-74bd4c5bfc-lxb7v 1/1 Running 0 7m53s
vro-api-redis-555446854-jwfqg 1/1 Running 0 7m53s
# The console container is in the pod with several containers
❯ kc exec -i -t vro-api-645dc44c64-w95mw -c abd-vro-console -- sh -c "java -jar vro-console.jar"For other deployment environment, adjust the kubectl namespace in the alias.
More details in PR #695.
Added in PR #531.
- On the
groovy:000>prompt, try the following:? # display help // Note the printJson (alias pj) custom command :show variables // Note the `claimsT` variable, which can be used to query the claims DB table claimsT.findAll().collect{ it.claimSubmissionId } c = getAnyClaim() // Different ways to do the same thing: printJson c printJson getAnyClaim() pj c // exit # This will stop the container // Instead, press Ctrl-p Ctrl-q to detach from the container without stopping it
- See https://groovy-lang.org/groovysh.html for other built-in console commands
Added in PR #614.
groovy:000> :show variables
// Note the redis and redisT variables
groovy:000> redis.keys "*"
===> [claim-1234]
groovy:000> redis.hlen("claim-1234")
===> 2
groovy:000> redis.hkeys("claim-1234")
===> ["type", "pdf"]
groovy:000> redis.hget("claim-1234", "type")
===> hypertension
groovy:000> redis.hget("claim-1234", "pdf")
===> JVBERi0xLjQKMSAwIG9iago8PAovVGl0bGUgKP7/KQovQ3JlYX ... (truncated base64 encoding of the generated pdf)
// Using RedisTemplate redisT
groovy:000> ops=redisT.opsForValue()
groovy:000> ops.hget("claim-1234", "type")
===> hypertensionListen to messages at certain predefined wireTap Camel endpoints.
Added in PR #597
groovy:000> :show variables
// Note the camel variable
// Check out http://localhost:15672/#/queues for current queues and see how `console-*` queues are added as the following commands are run.
// Initially no routes in the CamelContext of the Console container
groovy:000> camel.routes
===> []
groovy:000> wireTap claim-submitted
// Now, submit a claim using Swagger
// Expect to see a log message
groovy:000> wireTap generate-pdf
// Now, generate a pdf using Swagger
// Expect to see a log message
groovy:000> camel.routes
===> [
Route[rabbitmq://tap-generate-pdf?exchangeType=topic&queue=console-generate-pdf -> null],
Route[rabbitmq://tap-claim-submitted?exchangeType=topic&queue=console-claim-submitted -> null]
]To submit a message from the VRO Console into a Camel Route endpoint:
// Create the request as a JSON String
req="""{
"resourceId": "123444",
"diagnosticCode": "A"
}"""
// Create the Camel endpoint URI
exchangeName="v3"
routingKey="postResource"
uri="rabbitmq:" + exchangeName + "?skipQueueBind=true&routingKey=" + routingKey
// Inject the message -- see CamelEntry for alternatives ways to inject
resp=pt.requestBody(uri, req, String)
===> {"resourceId":"123444","diagnosticCode":"A","status":"PROCESSING","reason":null}This requires that the Camel Route endpoint be exposed outside of the JVM, e.g., the endpoint uses rabbitmq: and not direct: or seda:. The VRO Console has access to RabbitMQ queues, not the internal JVM queues or endpoints.
See the PrintJson and Wiretap classes.
Add to the console/.../groovysh.rc file.
BIP Claims Evidence API (May 2024)
VRO uses BIP Claim Evidence API to upload evidence PDFs to E-Folder.Access to BIP Claims API is available only within VA firewall.
BIP Claim Evidence API uses mTLS. VRO's mTLS implementation is detailed in BIP-APIs.
- VRO uploads PDF documents to E-Folder
- VRO specifies meta data about the document
BIP Claim Evidence API's Open API Specification is available from the Swagger page.
The only end point used from BIP Claim Evidence API is the /files end-point.
It should be possible to activate the Swagger page for the Mock BIP Claim Evidence API similar to Mock BIP Claims API easily. But that has not been yet done.
BIP requires a Bearer JWT for access. Following claims are used
- Subject (sub)
- User Id (userID): Custom - VRO system user
- Issuer (iss)
- Station Id (stationID): Custom - VRO system user facility (?)
- Application Id (applicationID): Custom - must be equivalent to Issuer per documentation
- Expiration (exp)
- Issued At (iat)
The JWT is created before each API call in BipCEApiService createJwt method.
Subject claim is hard-coded in createJwt. Expiration and Issued At claims are dynamicaly created in createJwt. The other claims are made available to the application with environment variables through application.properties.
- User Id:
BIP_EVIDENCE_USERIDthroughbip.evidenceClientId - Issuer:
BIP_EVIDENCE_ISSthroughbip.evidenceIssuer - Station Id:
BIP_STATION_IDthroughbip.stationId - Application Id:
BIP_APPLICATION_IDthroughbip.applicationId
JWT is signed by a secret provided by the BIP API team. The secret is made available to the application with the environment variable BIP_EVIDENCE_SECRET through application.properties bip.evidenceSecret setting.
In the VRO Kubernetes environment the related Kubernetes secrets for the BIP environment variables are
-
BIP_EVIDENCE_USERID:bip.bipEvidenceUserId -
BIP_STATION_ID:bip.bipStationId -
BIP_APPLICATION_ID:bip.bipApplicationId -
BIP_EVIDENCE_SECRET:bip.bipEvidenceSecret
A set of BIP environment variables are available for local development by sourcing the setenv.sh script. There were attempts to move these to application-local.properties but failed. Please see the note in setenv.sh script.
The only end-point that is used is
- POST
/files
The base URL is made available to the application with the environment variable BIP_EVIDENCE_URL through application.properties bip.evidenceBaseURL setting. The corresponding Kubernetes secret is bip.bipEvidenceUrl.
For local development and testing a Mock Server is available in docker compose with host name mock-bip-ce-api.
The API call is implemented in Bip Claim Evidence API Service. Bip Claim Evidence API Service uses the custom RestTemplate bean (qualifier: bipCERestTemplate) described in BIP-APIs.
Bip Claim Evidence Api Service is available to rest of the application as a Spring service. The only current customer is Bip Claim Service which uses it through Bip Claim Evidence Api Service Interface. This is mostly for historical progression of the implementation but also makes it possible to unit test Bip Claim Service more easily as Bip Api Service Interface is overridden in a test configuration.
BIP Claim Evidence API related functionality is provided to the rest of the application through BIP Claim Service. The public method uploadPdf is self explanatory.