Update data pipeline for LEAR table renaming and additional changes

[esune]

The source database for LEAR is undergoing a refactoring that will require updates to the data pipeline responsible for ingesting records to be processed.

There are a few references of the businesses table in the code: https://github.com/search?q=repo%3Abcgov%2Fvon-bc-registries-agent%20businesses&type=code

Required changes include:

• renaming the businesses table to entities
• other?

[argush3]

I don't think the change will be as simple as renaming businesses to legal_entities.

I remember being asked about how LEAR versions data in the past so I am assuming that OrgBook processes are querying LEAR versioning tables. The new data model has adopted a new way of versioning data.

I've created a document that contains some of the details around the LEAR data model changes.

LEAR Data Model Changes.pdf

[ianco]

@argush3 what's the timeline on these changes being available in the dev or test environments (and expected timeline to promote into prod)?

[argush3]

We are trying to wrap up development by the end of the month at which point we'll start integration testing.

When integration testing gets to a good enough place, we'll move things into the sandbox environment for a period of time that is still TBD(traditionally it's been 30 days). After sandbox we'll go to Prod.

Just for some additional context, we are in the process of moving all our LEAR BE pieces(including the database) to GCP. We already have an initial version of LEAR db in the GCP dev environment but we still need to get other BE services up and running.

We'll need to figure out the timing and how access to the new database will be given. I'll need to get back to you on this.

Also, the data conversion from old to new LEAR db still needs some work so it probably makes sense to wait until the data is closer to the intended end state to avoid re-work on your side.

So still a lot of moving pieces and things will change depending on how things play out based off of the things I mentioned above.

cc: @davemck513

[esune]

Thank you @argush3. Please keep us posted so we can coordinate when to start development on our end.

[esune]

@argush3 just a quick nudge to check whether you have a tentative timeline for this, as we'll need to coordinate our efforts/planning. If possible, if we could get access to a DB with the new data/structure sooner rather than later to start developing against it would be great.

[argush3]

We haven't been able to finish up the legal name updates and data fixes yet. I had to pivot a bit with other priorities so the timeline has extended out a bit. Will need to see when I can get back to the legal name work.

I'll talk to the team to see about providing access to the DB with the assumption that there will still be some model and data tweaks.

[esune]

We haven't been able to finish up the legal name updates and data fixes yet. I had to pivot a bit with other priorities so the timeline has extended out a bit. Will need to see when I can get back to the legal name work.

I'll talk to the team to see about providing access to the DB with the assumption that there will still be some model and data tweaks.

Great, that would be very helpful - thanks @argush3 !

[swcurran]

Comment from @esune on ZenHub issue:

Access to the new LEAR db should be possible from the BC Registries Agent namespace, using a command like the following: psql -h gcp-lear-db-backup-dev.73c567-dev.svc -p 5432 -d legal-entities -U readonly.

TO be assessed whether we can connect to the db for schema exploration as-is, or we need additional permissions to the registries namespace.

[esune]

On hold until data changes and migration is in a stable state, allowing us to work through the changes.