[IA-3699] Update Django, Python & backend dependencies

[tdethier]

Update Django, Python & backend dependencies.

Related JIRA tickets : IA-3699

Self proofreading checklist

• Did I use eslint and black formatters
• Is my code clear enough and well documented
• Are my typescript files well typed
• New translations have been added or updated if new strings have been introduced in the frontend
• My migrations file are included
• Are there enough tests
• Documentation has been included (for new feature)

TODO

Update files with new versions:

• Update Dockerfiles for
• Update requirements.txt
• Update requirements-dev.txt
• Update github actions
• Update actions inside github actions?

Fix code:

• Fix logout
• Fix Polio
• Fix other broken tests
  • Passing 'crs' is not supported with the 'pyogrio' engine. in geopandas
  • pyxform (see ⬇️)
  • wrong counts in Instances
  • wrong freq in Instances
  • "no x matches the given query" errors became "Not found"
  • error simplified geom 😵‍💫
  • flaky tests 😵‍💫 :
    • payment lot task (on non fresh database)
    • crash in Instance deletion
    • all issues with IDs offset by one
• Find a stable solution for pyxform & get_definition_data()
  • Discussion about similar topic - here on ODK forum

Testing:

• Test setuper
• Test web app
  • Test admin pages
  • Test GroupSet
  • Test WFP auth
  • Test logout
  • Test InstanceLock
  • Test routes with changed basename
  • Test refresh_lqas_data.py (see requirements.txt)
  • Test export + reimport of geopackage
  • Test payment lots
• Test storage
  • local
  • S3
  • Polio local
  • Polio S3
• Test Trypelim
• Test mobile app
  • Test request.headers.get("user-agent", "")

Other:

• Wait for 03/12/2024 to get Django 5.1.4
• Plan some upgrade checks in the release cycle (every x release? monthly?)
• Wait for January 2025 for Python 3.13 to be available in AWS beanstalk?
  

Doc

• django.contrib.gis.admin.GeoModelAdmin was removed in Django 5.0.0
• Support for logging out via GET requests in the django.contrib.auth.views.LogoutView was removed in Django 5.0.0
• Using QuerySet.iterator() on a queryset that prefetches related objects without providing the chunk_size argument is now forbidden since Django 5.0.0
• django.utils.timezone.utc was removed in Django 5.0.0
• Using QuerySets as arguments of conditions inside a FilteredRelation() has been forbidden since Django 5.0.2
• settings.DEFAULT_FILE_STORAGE was removed in Django 5.1.0
• Enums (model.TextChoices) member values became a tuple of arguments in Django 5.1.0 (not in change logs)
• allauth.account.middleware.AccountMiddleware middleware has been required to be present in settings.MIDDLEWARE since allauth 0.56.0
• Pandas' frequency alias M has been deprecated since Pandas 2.2.0
• geopandas' I/O engine now defaults to Pyogrio which is now installed with GeoPandas instead of Fiona, which prevents from passing a crs, since geopandas 1.0.0
• pyxform's API for non-path input was changed in pyxform 2.1.0

Changes

• update to geopackage import/export
• update to Django storage
• update to geo data in Django admin
• update to instance locks
• update to Auth middleware
• update to form versions

How to test

Run tests, check the web application, check the mobile application.

Print screen / video

/

Notes

Thanks to Marc, I used django-upgrade to update the code to match Django 5.1, then I manually edited to files to fix the remaining issues.
I used pip-review to update all backend dependencies.

This PR will need to be thoroughly tested before deployment, as there are a lot of updates and breaking changes due to these updates. This is even more true for the features that were edited.

Some of the test issues showed up because there are many tests that fetch lists, but do not include an order parameter. If there is no order_by on the queryset, then of course the result can't be 100% determined.

Below is the result of flaky tests, after a local initial (successful) run:

=====================================================================
FAIL: test_create_payment_lot (iaso.tests.api.payments.test_payment_lots.PaymentLotsViewSetAPITestCase.test_create_payment_lot)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thibault/dev/iaso/iaso/iaso/tests/api/payments/test_payment_lots.py", line 98, in test_create_payment_lot
    self.runAndValidateTask(task, "SUCCESS")
  File "/home/thibault/dev/iaso/iaso/iaso/tests/tasks/task_api_test_case.py", line 17, in runAndValidateTask
    return self.assertValidTaskAndInDB(response.json(), new_status)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/thibault/dev/iaso/iaso/iaso/tests/tasks/task_api_test_case.py", line 20, in assertValidTaskAndInDB
    self.assertEqual(task_dict["status"], status, task_dict)
AssertionError: 'ERRORED' != 'SUCCESS'
- ERRORED
+ SUCCESS
 : {'id': 224, 'created_at': 1732805610.866682, 'started_at': 1732805610.878912, 'ended_at': 1732805610.889754, 'progress_value': 0, 'end_value': 0, 'launcher': {'first_name': '', 'last_name': '', 'username': 'user'}, 'result': {'result': 'ERRORED', 'message': 'Error while creating Payment Lot', 'stack_trace': 'Traceback (most recent call last):\n  File "/home/thibault/dev/iaso/iaso/beanstalk_worker/__init__.py", line 32, in wrapper\n    func(*args, task=the_task, **kwargs)\n  File "/home/thibault/dev/iaso/iaso/iaso/tasks/create_payment_lot.py", line 100, in create_payment_lot\n    end_task_and_delete_payment_lot(\n  File "/home/thibault/dev/iaso/iaso/iaso/tasks/create_payment_lot.py", line 60, in end_task_and_delete_payment_lot\n    raise Exception("Error while creating Payment Lot")\nException: Error while creating Payment Lot\n', 'last_progress_message': 'Searching for Payments to modify'}, 'status': 'ERRORED', 'name': 'create_payment_lot', 'should_be_killed': False, 'progress_message': 'Error while creating Payment Lot', 'polio_notification_import_id': None}



======================================================================
FAIL: test_can_retrieve_submissions_list_in_csv_format (iaso.tests.api.test_instances.InstancesAPITestCase.test_can_retrieve_submissions_list_in_csv_format)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thibault/dev/iaso/iaso/iaso/tests/api/test_instances.py", line 996, in test_can_retrieve_submissions_list_in_csv_format
    self.assertEqual(row_to_test, expected_row)
AssertionError: Lists differ: ['547', '', 'E8TjFaDNl9W', '', '', '', '', '2[154 chars], ''] != ['546', '', 'Vzhn0nceudr', '', '', '', '', '2[156 chars], '']

First differing element 0:
'547'
'546'

- ['547',
?     ^

+ ['546',
?     ^

   '',
-  'E8TjFaDNl9W',
+  'Vzhn0nceudr',
   '',
   '',
   '',
   '',
-  '202002',
?        ^

+  '202001',
?        ^

   '2020-02-01 00:00:05',
   '2020-02-01 00:00:05',
-  'guest',
-  'DUPLICATED',
+  'yoda (Yo Da)',
+  'READY',
   '',
   'Coruscant Jedi Council',
   '40',
   'jedi_council_corruscant_ref',
   '',
   '',
   '',
   '']



======================================================================
FAIL: test_instances_list_user (iaso.tests.api.test_instances.InstancesAPITestCase.test_instances_list_user)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thibault/dev/iaso/iaso/iaso/tests/api/test_instances.py", line 1698, in test_instances_list_user
    self.assertEqual(self.instance_4.id, instances[1].get("id"))
AssertionError: 549 != 553



======================================================================
FAIL: test_instances_modification_date (iaso.tests.api.test_instances.InstancesAPITestCase.test_instances_modification_date)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thibault/dev/iaso/iaso/iaso/tests/api/test_instances.py", line 1838, in test_instances_modification_date
    self.assertEqual(self.instance_5.id, instances[0].get("id"))
AssertionError: 550 != 551



======================================================================
FAIL: test_instances_sent_date (iaso.tests.api.test_instances.InstancesAPITestCase.test_instances_sent_date)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/thibault/dev/iaso/iaso/iaso/tests/api/test_instances.py", line 1778, in test_instances_sent_date
    self.assertEqual(self.instance_1.id, instances[0].get("id"))
AssertionError: 546 != 547

Follow the Conventional Commits specification

The merge message of a pull request must follow the Conventional Commits specification.

This convention helps to automatically generate release notes.

Use lowercase for consistency.

Example:

fix: empty instance pop up

Refs: IA-3665

Note that the Jira reference is preceded by a line break.

Both the line break and the Jira reference are entered in the Add an optional extended description… field.