Upgrade to Python 3.11 / Ubuntu Mantic Minotaur (Recidiviz/recidiviz-…

…data#26243)

## Description of the change
This upgrades us to Ubuntu Mantic Minotaur, the latest release that does
not reproduce psf/requests#6560

Additionally, it moves us on to Python 3.11 as the deadsnakes PPA is not
available for non-LTS releases.

`pylint` has been removed from our virtualenv as `apache-beam` has a
strict version band requirement on `dill` . See
apache/beam#22893

This PR includes lots of whitespace changes caused by Black due to the
upgrade of `astroid`.
## Type of change

> All pull requests must have at least one of the following labels
applied (otherwise the PR will fail):

| Label | Description |
|-----------------------------
|-----------------------------------------------------------------------------------------------------------
|
| Type: Bug | non-breaking change that fixes an issue |
| Type: Feature | non-breaking change that adds functionality |
| Type: Breaking Change | fix or feature that would cause existing
functionality to not work as expected |
| Type: Non-breaking refactor | change addresses some tech debt item or
prepares for a later change, but does not change functionality |
| Type: Configuration Change | adjusts configuration to achieve some end
related to functionality, development, performance, or security |
| Type: Dependency Upgrade | upgrades a project dependency - these
changes are not included in release notes |

## Related issues
Closes Recidiviz/recidiviz-data#20615
Closes Recidiviz/recidiviz-data#24244
Closes Recidiviz/recidiviz-data#4287
Related to Recidiviz/recidiviz-data#12469

## Checklists

### Development

**This box MUST be checked by the submitter prior to merging**:
- [x] **Double- and triple-checked that there is no Personally
Identifiable Information (PII) being mistakenly added in this pull
request**

These boxes should be checked by the submitter prior to merging:
- [ ] Tests have been written to cover the code changed/added as part of
this pull request

### Code review

These boxes should be checked by reviewers prior to merging:

- [ ] This pull request has a descriptive title and information useful
to a reviewer
- [ ] Potential security implications or infrastructural changes have
been considered, if relevant

---------

Co-authored-by: Helper Bot <[email protected]>
GitOrigin-RevId: 2a71759c1bd091be8e383ae45e7314ee0ad42a27

.github/actions/python-setup/action.yml

@@ -5,7 +5,7 @@ runs:
     - uses: actions/setup-python@v2
       id: setup-python
       with:
-        python-version: "3.9.12"
+        python-version: "3.11.6"
     - name: Install pipenv
       run: pip install pipenv
       shell: bash

.github/workflows/ci.yml

@@ -68,6 +68,8 @@ jobs:
           fetch-depth: 0
       - name: Python Setup
         uses: ./.github/actions/python-setup
+      - name: Install Pylint
+        run: pipenv run pip install pylint
       - name: Pylint
         run: pipenv run ./recidiviz/tools/lint/run_pylint.sh
 

Dockerfile

@@ -1,4 +1,4 @@
-FROM ubuntu:focal-20221130 AS recidiviz-init
+FROM ubuntu:mantic-20231011 AS recidiviz-init
 ENV DEBIAN_FRONTEND noninteractive
 # NOTE: It is is extremely important that we do not delete this
 # variable. One of our dependencies, dateparser, seems to require
@@ -7,18 +7,14 @@ ENV DEBIAN_FRONTEND noninteractive
 # return None in a large set of circumstances which is of course,
 # unideal.
 ENV TZ America/New_York
-# Add a package repo to get archived python versions.
-RUN apt update -y && \
-    apt install -y software-properties-common && \
-    add-apt-repository ppa:deadsnakes/ppa
 RUN apt update -y && \
     apt install -y \
     locales \
     git \
     libxml2-dev libxslt1-dev \
-    python3.9-dev python3.9-distutils python3-pip \
     default-jre \
     libpq-dev \
+    python3.11-dev pipenv/mantic \
     curl
 RUN locale-gen en_US.UTF-8
 ENV LC_ALL en_US.UTF-8
@@ -29,7 +25,6 @@ RUN ln -fs /usr/share/zoneinfo/America/New_York /etc/localtime
 # Make stdout/stderr unbuffered. This prevents delay between output and cloud
 # logging collection.
 ENV PYTHONUNBUFFERED 1
-RUN pip3 install pipenv
 # In order to use this Dockerfile with Cloud Run, PIPENV_VENV_IN_PROJECT must be set.
 # If not, Cloud Run will try to "helpfully" create a new virtualenv for us which will not match our
 # expected set of dependencies.

Pipfile

@@ -11,20 +11,18 @@ pytest = "*"
 # TODO(#23317): pytest-xdist changed its parallelization method after this result
 # This resulted in our ModuleCollectorMixin encountering a condition where some modules were collected twice
 pytest-xdist = "< 3.0"
-pylint = "*"
 pytest-cov = "*"
 mypy = "*"
 bandit = "*"
 coveralls = "*"
 # This should always be pinned to the latest version officially supported by Dataflow: https://cloud.google.com/dataflow/docs/support/sdk-version-support-status#python
 # If this version is bumped then the beam version in recidiviz/pipelines/Dockerfile.pipelines must be as well.
-apache-beam = { extras = ["gcp", "test"], version = "==2.50.0" }
+apache-beam = { extras = ["gcp", "test"], version = "==2.52.0" }
 fakeredis = "*"
 freezegun = "*"
 ipdb = "*"
 jupyter = "*"
-# TODO(#20615): Unpin orjson dependency when the installation failure is debugged.
-orjson = "==3.8.10"
+orjson = "*"
 pytest-alembic = "*"
 pipenv = "*"
 pygit2 = "*"
@@ -151,13 +149,10 @@ cssselect = "*"
 grpcio = "*"
 gunicorn = "*"
 gevent = ">=20.5.1"
-google-api-core = ">=2.12.0"
-google-cloud-core = ">=2.3.0"
-# TODO(#4287): Versions >= 3.0.0 create namespace conflicts with other cloud libraries
-#  (and resulting mypy errors). We should revisit upgrade when we've updated all
-#  google-cloud* libraries to >= 2.0.0.
-google-cloud-bigquery = "<3.0.0"
-google-cloud-bigquery-datatransfer = "<3.0.0"
+google-api-core = "*"
+google-cloud-core = "*"
+google-cloud-bigquery = "*"
+google-cloud-bigquery-datatransfer = "*"
 google-cloud-bigquery-storage = "*"
 google-cloud-build = "*"
 google-cloud-dataflow-client = "*"
@@ -205,8 +200,7 @@ seaborn = "*"
 statsmodels = "*"
 openpyxl = "*"
 analytics-python = "*"
-# TODO(#24244): Remove pin once TryExcept attribute errors are fixed
-astroid = "2.15.7"
+astroid = "*"
 # TODO(#11759): Remove version pin once fakeredis supports 4.2.0+
 redis = "<4.6.0"
 pyarrow = "*"
@@ -244,7 +238,7 @@ xlrd = "*"
 
 [requires]
 # If you change this, please update https://github.com/Recidiviz/recidiviz-research at the same time!
-python_version = "3.9"
+python_version = "3.11.6"
 
 [scripts]
 # Run via `pipenv run <name>`

frontends/admin-panel/src/recidiviz/admin_panel/models/validation_pb.js

@@ -13,7 +13,13 @@
 
 var jspb = require('google-protobuf');
 var goog = jspb;
-var global = Function('return this')();
+var global =
+    (typeof globalThis !== 'undefined' && globalThis) ||
+    (typeof window !== 'undefined' && window) ||
+    (typeof global !== 'undefined' && global) ||
+    (typeof self !== 'undefined' && self) ||
+    (function () { return this; }).call(null) ||
+    Function('return this')();
 
 var google_protobuf_timestamp_pb = require('google-protobuf/google/protobuf/timestamp_pb.js');
 goog.object.extend(proto, google_protobuf_timestamp_pb);
@@ -1415,7 +1421,8 @@ proto.recidiviz.admin_panel.models.SamenessPerViewValidationResultDetails.Partit
  */
 proto.recidiviz.admin_panel.models.SamenessPerViewValidationResultDetails.PartitionCounts.prototype.clearColumnCountsMap = function() {
   this.getColumnCountsMap().clear();
-  return this;};
+  return this;
+};
 
 
 /**

recidiviz/admin_panel/ingest_operations_store.py

@@ -114,7 +114,6 @@ def set_cache(
             Dict[DirectIngestInstance, Optional[DataflowPipelineMetadataResponse]],
         ],
     ) -> None:
-
         jobs_dict = {
             state_code.value: {
                 instance.value: attr_to_json_dict(
@@ -240,13 +239,15 @@ def _verify_clean_ingest_view_state(
         ingest_view_contents = InstanceIngestViewContentsImpl(
             self.bq_client, state_code.value, instance, dataset_prefix=None
         )
-        dataset_id = ingest_view_contents.results_dataset()
+        dataset_ref = self.bq_client.dataset_ref_for_id(
+            ingest_view_contents.results_dataset()
+        )
         if (
-            self.bq_client.dataset_exists(dataset_id)
-            and len(list(self.bq_client.list_tables(dataset_id))) > 0
+            self.bq_client.dataset_exists(dataset_ref)
+            and len(list(self.bq_client.list_tables(dataset_ref.dataset_id))) > 0
         ):
             raise DirectIngestInstanceError(
-                f"There are ingest view results in {dataset_id} that have not been"
+                f"There are ingest view results in {dataset_ref} that have not been"
                 f"cleaned up. Cannot proceed with ingest rerun."
             )