Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump utils to 91.1.0 #169

Open
wants to merge 14 commits into
base: main
Choose a base branch
from
Open

Bump utils to 91.1.0 #169

wants to merge 14 commits into from

Conversation

quis
Copy link
Member

@quis quis commented Jul 30, 2024

This had some performance problems last time we tried. Probably related to the Werkzueg 2 → 3 bump. Putting it up here so we can try again some time.

91.1.0

  • bump all libs in requirements_for_test_common.in

91.0.4

  • Don’t copy config when bumping utils version

91.0.3

  • Fix validating supported international country codes for phone numbers without a leading "+" that look a bit like UK numbers

91.0.1

  • Adds public utility method to check if a phonenumber is international (with correct handling for OFCOM TV numbers)
  • Updates PhoneNumber.get_international_phone_info to return the correct info for OFCOM TV numbers

91.0.0

  • BREAKING CHANGE: All standalone functions for validating, normalising and formatting phone numbers has been removed from notifications_utils/recipient_validation/phone_number.py, and are replaced and encapsualated entirely with a single PhoneNumber class. All code that relies on validation, normalisation or fomratting of a phone number must be re-written to use instances of PhoneNumber instead.

89.2.0

  • Use github API rather than fetching from their CDN in version_tools

89.1.0

  • Change version_tools to a package that uses importlib to grab common reqs/config, rather than fetching common files from github. This repo's usage of those common files are now symlinks to the sources of truth found within notification_utils/version_tools/

89.0.1

  • Raise an exception if we cant fetch remote github files in version_tools.py

89.0.0

  • requirements_for_test_common.txt is now requirements_for_test_common.in. Apps should freeze this into a local requirements file for fully reproducible dependencies

88.1.1

  • Bug fix: applies minor version bump of PhoneNumbers to v8.13.48 to apply a fix to the UG metadata which was causing validation to fail in error

88.1.0

  • logging: slightly redesign filters behaviour to allow log-call-supplied request_id, span_id, user_id, service_id to be used if they are not available by normal means. Supply these manually for the streaming-response-closed log message.

88.0.1

  • logging: don't use current_app in _log_response_closed

88.0.0

  • Removes template.EmailPreviewTemplate (only used in admin app)

87.1.0

  • logging: don't calculate_content_length() for after_request logs of streaming responses - this caused attempted streaming responses to be eagerly consumed, negating the point of streaming responses. Instead emit the after_request log when status code and header is returned and a separate log message when a streaming response is closed.

87.0.0

  • Reintroduce changes to AntivirusClient and ZendeskClient from 83.0.0

86.2.0

  • Adds asset_fingerprinter.AssetFingerprinter to replace the versions duplicated across our frontend apps

86.1.0

  • Add EventletTimeoutMiddleware

  • BREAKING CHANGE: The Phonenumber class now accepts a flag allow_landline, which defaults to False. This changes the previous default behaviour, allowing landlines.

85.0.0

  • Removes SerialisedModel.ALLOWED_PROPERTIES in favour of annotations syntax

84.3.0

  • Reverts 84.1.0

84.2.0

  • The Zendesk client takes a new optional argument, user_created_at which populates a new field on the Notify Zendesk form if provided.

84.1.1

  • Remove GIR 0AA from valid postcodes

84.1.0

  • Bump versions of common test dependencies (run make bootstrap to copy these into your app)

84.0.0

  • AntivirusClient and ZendeskClient have returned to their behaviour as of 82.x.x to allow the 83.0.1 fix to go out to apps without the required changes.

  • Updates phone_numbers to 8.13.45 to apply a fix to the metadata for phone numbers that was discovered causing a subset of valid Jersey numbers to be incorrectly invalidated

83.0.0

  • AntivirusClient and ZendeskClient are no longer thread-safe because they now use persistent requests sessions. Thread-local instances should be used in place of any global instances in any situations where threading is liable to be used
  • AntivirusClient and ZendeskClient have had their init_app(...) methods removed as this is an awkward pattern to use for initializing thread-local instances. It is recommended to use LazyLocalGetter to construct new instances on-demand, passing configuration parameters via the constructor arguments in a factory function.

82.7.0

  • Add expected_type mechanism to LazyLocalGetter

82.6.1

  • Adds validation check to PhoneNumber so that it returns the expected error message TOO_SHORT if an empty string is passed. This has caused issues with users of the v2 API getting inconsistent error messages

82.6.0

  • Add LazyLocalGetter class for lazily-initialized context-local resources

82.5.0

  • Add support for validation of UK landlines for services with sms_to_uk_landlines enabled using CSV flow

82.4.0

  • Add support for sending SMS to more international numbers. Sending to Eritrea, Wallis and Futuna, Niue, Kiribati, and Tokelau is now supported.

82.3.0

  • Extends the validation logic for the PhoneNumber class to disallow premium rate numbers

82.2.1

  • Add fix to recipient_validation/phone_number.py to raise correct error if a service tries to send to an international number without that permission

82.2.0

  • Add unsubscribe_link argument to email templates

82.1.2

  • Write updated version number to requirements.txt if no requirements.in file found

82.1.1

  • Fix the way we log the request_size. Accessing the data at this point can trigger a validation error early and cause a 500 error

82.1.0

  • Adds new logging fields to request logging. Namely environment_name, request_size and response_size

82.0.0

  • Change PostalAddress to add has_no_fixed_abode_address method. No fixed abode addresses are now considered invalid.

81.1.1

81.1.0

  • introduce new validation class - PhoneNumber, that we will use for services that want to send sms to landline (and in the future this new code can be extended for all phone number validation)
  • in this new class, we use phonenumbers library for validating phone numbers, instead of our custom valdiation code

81.0.0

  • BREAKING CHANGE: The constructor for notification_utils.recipient_validation.errors.InvalidPhoneError
    • When raising InvalidPhoneError, instead of supplying a custom message, you must create an error by supplying a code from the InvalidPhoneError.Codes enum
  • InvalidPhoneError.code will contain this machine-readable code for an exception if you need to examine it later
  • InvalidPhoneError.get_legacy_v2_api_error_message returns a historical error message for use on the public v2 api

80.0.1

  • Reduces minimum required Gunicorn version for compatibility

80.0.0

  • Copies additional config files from utils into repos
  • Renames version_tools.copy_pyproject_yaml to version_tools.copy_config

79.0.1

  • Update the send_ticket_to_zendesk method of the ZendeskClient to return the ID of the ticket that was created.

79.0.0

  • Switches on Pyupgrade and a bunch of other more opinionated linting rules

78.2.0

  • Bumped minimum versions of select subdependencies

78.1.0

  • Restrict postcodes to valid UK postcode zones

78.0.0

  • BREAKING CHANGE: recipient validation code has all been moved into separate files in a shared folder. Functionality is unchanged.
    • Email address validation can be found in notifications_utils.recipient_validation.email_address
    • Phone number validation can be found in notifications_utils.recipient_validation.phone_number
    • Postal address validation can be found in notifications_utils.recipient_validation.postal_address
  • BREAKING CHANGE: InvalidPhoneError and InvalidAddressError no longer extend InvalidEmailError.
    • if you wish to handle all recipient validation errors, please use notifications_utils.recipient_validation.errors.InvalidRecipientError

77.2.1

  • Change redis delete behaviour to error, rather than end up with stale data, if Redis is unavailable.

77.2.0

  • NotifyTask: include pid and other structured fields in completion log messages

Complete changes: alphagov/notifications-utils@77.1.1...89.2.0


Fixes #154

Fixes #153


🚨⚠️ This will be deployed automatically all the way to production when you click merge ⚠️🚨

For more information, including how to check this deployment on preview or staging first before it goes to production, see our team wiki section on deployment

@quis quis force-pushed the bump-utils-82.1.1 branch from 2cb051f to bf19d49 Compare July 30, 2024 09:18
quis added 7 commits November 5, 2024 11:01
 ## 82.1.1
*  Fix the way we log the request_size. Accessing the data at this point can trigger a validation error early and cause a 500 error

 ## 82.1.0
*  Adds new logging fields to request logging. Namely environment_name, request_size and response_size

 ## 82.0.0

* Change `PostalAddress` to add `has_no_fixed_abode_address` method. No fixed abode addresses are now considered invalid.

 ## 81.1.1
*  Adds condition to validation to allow TV Numbers (https://www.ofcom.org.uk/phones-and-broadband/phone-numbers/numbers-for-drama/) for UK mobiles

 ## 81.1.0
* introduce new validation class - `PhoneNumber`, that we will use for services that want to send sms
to landline (and in the future this new code can be extended for all phone number validation)
* in this new class, we use `phonenumbers` library for validating phone numbers, instead of our custom valdiation code

 ## 81.0.0

* BREAKING CHANGE: The constructor for `notification_utils.recipient_validation.errors.InvalidPhoneError`
  - When raising InvalidPhoneError, instead of supplying a custom message, you must create an error by supplying a code from the `InvalidPhoneError.Codes` enum
* `InvalidPhoneError.code` will contain this machine-readable code for an exception if you need to examine it later
* `InvalidPhoneError.get_legacy_v2_api_error_message` returns a historical error message for use on the public v2 api

 ## 80.0.1

* Reduces minimum required Gunicorn version for compatibility

 ## 80.0.0

* Copies additional config files from utils into repos
* Renames `version_tools.copy_pyproject_yaml` to `version_tools.copy_config`

 ## 79.0.1

* Update the `send_ticket_to_zendesk` method of the ZendeskClient to return the ID of the ticket that was created.

 ## 79.0.0

* Switches on Pyupgrade and a bunch of other more opinionated linting rules

 ## 78.2.0

* Bumped minimum versions of select subdependencies

 ## 78.1.0

* Restrict postcodes to valid UK postcode zones

 ## 78.0.0

* BREAKING CHANGE: recipient validation code has all been moved into separate files in a shared folder. Functionality is unchanged.
  - Email address validation can be found in `notifications_utils.recipient_validation.email_address`
  - Phone number validation can be found in `notifications_utils.recipient_validation.phone_number`
  - Postal address validation can be found in `notifications_utils.recipient_validation.postal_address`
* BREAKING CHANGE: InvalidPhoneError and InvalidAddressError no longer extend InvalidEmailError.
  - if you wish to handle all recipient validation errors, please use `notifications_utils.recipient_validation.errors.InvalidRecipientError`

 ## 77.2.1

* Change redis delete behaviour to error, rather than end up with stale data, if Redis is unavailable.

 ## 77.2.0

* `NotifyTask`: include pid and other structured fields in completion log messages

***

Complete changes: alphagov/notifications-utils@77.1.1...82.1.1
We excluded this file from version control because it’s copied
automatically from utils, so we only need to modify it and track changes
there. In theory at least.

In practice Dependabot cannot:
- cope with having a requirements file which isn’t in the repo
- be configured to only look at certain requirements files

So I think the best thing to do is keep this file in version control. It
already has an automatically generated comment to suggest it shouldn’t
be edited manually.

Dependabot may try to raise PRs against it, but we can resolve these by
updating the upstream version in utils. We can then roll out several
version bumps in a single PR per app, rather than several.
@quis quis force-pushed the bump-utils-82.1.1 branch from 0fe9379 to 9cf378f Compare November 5, 2024 11:01
quis added 2 commits November 18, 2024 10:59
So anyone running the tests in an app will have all the same
dependencies as someone else, not just the same app dependencies.
 ## 89.2.0

* Use github API rather than fetching from their CDN in version_tools

 ## 89.1.0

* Change version_tools to a package that uses importlib to grab common reqs/config, rather than fetching common files from github. This repo's usage of those common files are now symlinks to the sources of truth found within notification_utils/version_tools/

 ## 89.0.1

* Raise an exception if we cant fetch remote github files in version_tools.py

 ## 89.0.0

* `requirements_for_test_common.txt` is now `requirements_for_test_common.in`. Apps should freeze this into a local requirements file for fully reproducible dependencies

 ## 88.1.1

* Bug fix: applies minor version bump of PhoneNumbers to v8.13.48 to apply a fix to the UG metadata which was causing validation to fail in error

 ## 88.1.0

* logging: slightly redesign filters behaviour to allow log-call-supplied request_id, span_id, user_id, service_id to be used if they are not available by normal means. Supply these manually for the streaming-response-closed log message.

 ## 88.0.1

* logging: don't use current_app in _log_response_closed

 ## 88.0.0

* Removes `template.EmailPreviewTemplate` (only used in admin app)

 ## 87.1.0

* logging: don't calculate_content_length() for after_request logs of streaming responses - this caused attempted streaming responses to be eagerly consumed, negating the point of streaming responses. Instead emit the after_request log when status code and header is returned and a separate log message when a streaming response is closed.

 ## 87.0.0

* Reintroduce changes to `AntivirusClient` and `ZendeskClient` from 83.0.0

 ## 86.2.0

* Adds `asset_fingerprinter.AssetFingerprinter` to replace the versions duplicated across our frontend apps

 ## 86.1.0

* Add `EventletTimeoutMiddleware`

* BREAKING CHANGE: The `Phonenumber` class now accepts a flag `allow_landline`, which defaults to False. This changes the previous default behaviour, allowing landlines.

 ## 85.0.0

* Removes `SerialisedModel.ALLOWED_PROPERTIES` in favour of annotations syntax

 ## 84.3.0

* Reverts 84.1.0

 ## 84.2.0

* The Zendesk client takes a new optional argument, `user_created_at` which populates a new field on the Notify Zendesk form if provided.

 ## 84.1.1

* Remove GIR 0AA from valid postcodes

 ## 84.1.0

* Bump versions of common test dependencies (run `make bootstrap` to copy these into your app)

 ## 84.0.0

* `AntivirusClient` and `ZendeskClient` have returned to their behaviour as of 82.x.x to allow the 83.0.1 fix to go out to apps without the required changes.

* Updates phone_numbers to 8.13.45 to apply a fix to the metadata for phone numbers that was discovered causing a subset of valid Jersey numbers to be incorrectly invalidated

 ## 83.0.0

* `AntivirusClient` and `ZendeskClient` are no longer thread-safe because they now use persistent requests sessions. Thread-local instances should be used in place of any global instances in any situations where threading is liable to be used
* `AntivirusClient` and `ZendeskClient` have had their `init_app(...)` methods removed as this is an awkward pattern to use for initializing thread-local instances. It is recommended to use `LazyLocalGetter` to construct new instances on-demand, passing configuration parameters via the constructor arguments in a `factory` function.

 ## 82.7.0

* Add `expected_type` mechanism to `LazyLocalGetter`

 ## 82.6.1

* Adds validation check to `PhoneNumber` so that it returns the expected error message `TOO_SHORT` if an empty string is passed. This has caused issues with users of the v2 API getting inconsistent error messages

 ## 82.6.0

* Add `LazyLocalGetter` class for lazily-initialized context-local resources

 ## 82.5.0

* Add support for validation of UK landlines for services with sms_to_uk_landlines enabled using CSV flow

 ## 82.4.0

* Add support for sending SMS to more international numbers. Sending to Eritrea, Wallis and Futuna, Niue, Kiribati, and Tokelau is now supported.

 ## 82.3.0
* Extends the validation logic for the `PhoneNumber` class to disallow premium rate numbers

 ## 82.2.1

* Add fix to recipient_validation/phone_number.py to raise correct error if a service tries to send to an international number without that permission

 ## 82.2.0
* Add `unsubscribe_link` argument to email templates

 ## 82.1.2
* Write updated version number to `requirements.txt` if no `requirements.in` file found

 ## 82.1.1
***

Complete changes: alphagov/notifications-utils@82.1.1...89.2.0
@quis quis changed the title Bump utils to 82.1.1 Bump utils to 89.2.0 Nov 18, 2024
quis added 5 commits December 11, 2024 10:24
 ## 91.1.0

* bump all libs in requirements_for_test_common.in

 ## 91.0.4

* Don’t copy config when bumping utils version

 ## 91.0.3

* Fix validating supported international country codes for phone numbers without a leading "+" that look a bit like UK numbers

 ## 91.0.1

* Adds public utility method to check if a phonenumber is international (with correct handling for OFCOM TV numbers)
* Updates PhoneNumber.get_international_phone_info to return the correct info for OFCOM TV numbers

 ## 91.0.0

* BREAKING CHANGE: All standalone functions for validating, normalising and formatting phone numbers has been removed from notifications_utils/recipient_validation/phone_number.py, and are replaced and encapsualated entirely with a single `PhoneNumber` class. All code that relies on validation, normalisation or fomratting of a phone number must be re-written to use instances of `PhoneNumber` instead.
uv is:
- between 10× and 100× faster than pip
- a drop-in replacement for `pip install`, `pip compile`, etc
In order to do this we need to install uv inside the Docker container.
We don't want it being installed in the venv that we use for production.
So we need to do this as a separate step.

Part of the reason uv is faster is because it uses a cache of packages.
By default it uses `$HOME/.cache/uv` as the location for this cache
(even if you use --no-cache, it makes a temp folder in there)[^1].

Our docker image doesn't have permission to write there, so we override
the default and set the cache to be in /tmp/ instead

[^1] https://docs.astral.sh/uv/concepts/cache/#cache-directory
> By default, uv does not compile Python (.py) files
> to bytecode (__pycache__/*.pyc); instead, compilation
> is performed lazily the first time a module is imported.
> For use-cases in which start time is critical, such as
> CLI applications and Docker containers, this option can
> be enabled to trade longer installation times for faster
> start times.

Without this we will see increased startup times for apps which can in
turn break the dev envs, where instances have reduced cpu and can’t
compile all their bytecode in time to return a healthcheck.
@quis quis changed the title Bump utils to 89.2.0 Bump utils to 91.1.0 Dec 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants