Removing jq because we no longer need it for transformations

Removing this script because we no longer need the gh CLI

Added this because we now need to reference it directly within the codebase, instead of indirectly like we did previously with the gh CLI

Since we are in this file, can we rename DBSettings to something more accurate

Prevents tests from failing if someone hasn't set their GitHub token locally.

Removed these because they've been removed in the latest version of ruff

Same with this one

This files is basically a complete refactor, but preserves the existing helper functions for the export to prevent this PR from getting bigger than it already is.

Removes this because we no longer need it

Changed this because the tests were failing locally if there was a period in the username, e.g. billy.daly

I don't understand why we need safe_pluck. If there's a bunch of fields missing, I would rather the code raise a keyerror, instead of getting us bad (eg. mostly null) data.

I can see the concern, but it's technically valid for all of these attributes to be empty in GitHub, except issue_title, issue_url and issue_opened_at. For example this issue has issue_type and issue_status but everything else is blank (e.g. sprint, parent, points, etc.)

We're currently validating the output data using the IssueMetada pydantic class when we parse these items in this step

We could have the non-nullable fields fail with a KeyError at this step, but the pydantic validation gives us better debugging output and allows us to gracefully continue with exporting and transforming the rest of the issues.

If we want to be more strict with what we consider "valid" data, I could see us requiring issue_type and issue_status as well.

Although since the logs are only retained for a limited amount of time, having issues without a type or status get "silently" dropped is often less helpful than having them with null data in Metabase.

The broader strategy around data quality and effectively handling a "dead letter queue" of bad data is the subject of this epic on data quality checks.

I'd definitely welcome your thoughts, though, on other potential strategies here as an intermediate step to implementing more robust data quality checks!

I think the basic things we're trying to achieve in the transform step are:

• Prevent "bad" data from being inserted into the database (i.e. data that is missing required columns, or data that is missing optional columns because of a bug in the ETL -- the latter one is harder to check for)
• Support inserts of data that are missing optional columns, when they are valid
• Prevent failures of a subset of data from blocking loads of the remaining valid data

Typically I've achieved these goals by using tools like Great Expectations or Anomalo which run on the entire data set to check for quality issues or anomalies, but there might be immediate steps we can take right now to block bad data or catch more programming errors upfront.

Since we are already using Pydantic, we can use it's nested models feature:

https://stackoverflow.com/questions/70302056/define-a-pydantic-nested-model

Then drop this translation and safe_pluck layer entirely, and rely entirely on Pydantic to do the validation and null data transformation