Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: add async_rest extra for async rest dependencies #2195

Merged
merged 17 commits into from
Sep 30, 2024
Merged

chore: add async_rest extra for async rest dependencies #2195

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
a6c1789
feat: add async rest option in setup
ohmayr Sep 24, 2024
17a1c5f
address PR feedback
ohmayr Sep 25, 2024
a468255
update exception statements
ohmayr Sep 25, 2024
3ec9135
reorder imports
ohmayr Sep 25, 2024
df3ff6f
minor fix in import statement
ohmayr Sep 25, 2024
7c4679e
address PR feedback
ohmayr Sep 27, 2024
4247f57
update constraints for async rest
ohmayr Sep 27, 2024
50dc716
update noxfile
ohmayr Sep 27, 2024
a75bfce
add whitespace
ohmayr Sep 27, 2024
fcc4982
remove constraints file
ohmayr Sep 27, 2024
1160ace
revert noxfile
ohmayr Sep 27, 2024
1214787
use extra to install the required deps
ohmayr Sep 27, 2024
4607716
update noxfile for gapics
ohmayr Sep 27, 2024
56e9f05
avoid conflict with 3.7 constraints
ohmayr Sep 27, 2024
715926a
update noxfile
ohmayr Sep 27, 2024
6533c77
address PR feedback
ohmayr Sep 27, 2024
9a370b5
address PR feedback
ohmayr Sep 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .../templates/%namespace/%name_%version/%sub/services/%service/transports/rest_asyncio.py.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ try:
from google.api_core import rest_streaming_async # type: ignore
{# NOTE: `pragma: NO COVER` is needed since the coverage for presubmits isn't combined. #}
except ImportError as e: # pragma: NO COVER
raise ImportError("`rest_asyncio` transport requires the library to be installed with the `async_rest` extra. Install the library with the `async_rest` extra using `pip install <insert_library_name>[async_rest]`") from e
raise ImportError("`rest_asyncio` transport requires the library to be installed with the `async_rest` extra. Install the library with the `async_rest` extra using `pip install {{ api.naming.warehouse_package_name }}[async_rest]`") from e

from google.auth.aio import credentials as ga_credentials_async # type: ignore

Expand Down
7 changes: 7 additions & 0 deletions gapic/templates/setup.py.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
{# TODO(https://github.com/googleapis/gapic-generator-python/issues/2121): Remove the following variable (and the condition later in this file) for async rest transport once support for it is GA. #}
{% set rest_async_io_enabled = api.all_library_settings[api.naming.proto_package].python_settings.experimental_features.rest_async_io_enabled %}
{% extends '_base.py.j2' %}
{% from '_pypi_packages.j2' import pypi_packages %}
{% block content %}
Expand Down Expand Up @@ -46,12 +48,14 @@ dependencies = [
{% endif %}
{% endfor %}
]
{% if rest_async_io_enabled %}
extras = {
"async_rest": [
"google-api-core[grpc] >= 2.20.0, < 3.0.0dev",
"google-auth[aiohttp] >= 2.35.0, <3.0.0dev"
],
}
{% endif %}
vchudnov-g marked this conversation as resolved.
Show resolved Hide resolved
url = "https://github.com/googleapis/google-cloud-python/tree/main/packages/{{ api.naming.warehouse_package_name }}"

package_root = os.path.abspath(os.path.dirname(__file__))
Expand Down Expand Up @@ -94,6 +98,9 @@ setuptools.setup(
packages=packages,
python_requires=">=3.7",
install_requires=dependencies,
{% if rest_async_io_enabled %}
extras_require=extras,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So if this wasn't there before, why weren't some tests failing? This would suggest a failure in our tests. At the GAPIC level, the test should silently pass if the GAPIC was generated without REST transport or installed without the extra. But at the generator level, we want to ensure that the tests with the extras pass, so how can we be notified if we run a test suite where we expect the extra and the associated tests, but it turns out the extra wasn't installed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests weren't failing because we're not using the async_rest extra to install the dependencies. We've explicitly defined each required dependency within the constraints file.

{% endif %}
include_package_data=True,
zip_safe=False,
)
Expand Down
22 changes: 22 additions & 0 deletions gapic/templates/testing/_default_constraints-async-rest.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
{# TODO(https://github.com/googleapis/gapic-generator-python/issues/2121): Remove the following variable (and the condition later in this file) for async rest transport once support for it is GA. #}
{% set rest_async_io_enabled = api.all_library_settings[api.naming.proto_package].python_settings.experimental_features.rest_async_io_enabled %}
{% if rest_async_io_enabled %}
{% from '_pypi_packages.j2' import pypi_packages %}
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
google-auth
# from google-auth[aiohttp]
aiohttp
requests
proto-plus
protobuf
{% for package_tuple, package_info in pypi_packages.items() %}
{# Quick check to make sure the package is different from this setup.py #}
{% if api.naming.warehouse_package_name != package_info.package_name %}
{% if api.requires_package(package_tuple) %}
{{ package_info.package_name }}
{% endif %}
{% endif %}
{% endfor %}
{% endif %}{# if rest_async_io_enabled #}
8 changes: 0 additions & 8 deletions gapic/templates/testing/_default_constraints.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,7 @@
{# TODO(https://github.com/googleapis/gapic-generator-python/issues/2121): Remove the following variable (and the condition later in this file) for async rest transport once support for it is GA. #}
{% set rest_async_io_enabled = api.all_library_settings[api.naming.proto_package].python_settings.experimental_features.rest_async_io_enabled %}
{% from '_pypi_packages.j2' import pypi_packages %}
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
{% if rest_async_io_enabled %}
google-auth
# from google-auth[aiohttp]
aiohttp
requests
{% endif %}
proto-plus
protobuf
{% for package_tuple, package_info in pypi_packages.items() %}
Expand Down
10 changes: 0 additions & 10 deletions gapic/templates/testing/constraints-3.7.txt.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,22 +1,12 @@
{# TODO(https://github.com/googleapis/gapic-generator-python/issues/2121): Remove the following variable (and the condition later in this file) for async rest transport once support for it is GA. #}
{% set rest_async_io_enabled = api.all_library_settings[api.naming.proto_package].python_settings.experimental_features.rest_async_io_enabled %}
{% from '_pypi_packages.j2' import pypi_packages %}
# This constraints file is used to check that lower bounds
# are correct in setup.py
# List all library dependencies and extras in this file.
# Pin the version to the lower bound.
# e.g., if setup.py has "google-cloud-foo >= 1.14.0, < 2.0.0dev",
# Then this file should have google-cloud-foo==1.14.0
{% if rest_async_io_enabled %}
google-api-core==2.20.0
google-auth==2.35.0
# from google-auth[aiohttp]
aiohttp==3.6.2
requests==2.20.0
{% else %}
google-api-core==1.34.1
google-auth==2.14.1
{% endif %}
proto-plus==1.22.3
protobuf==3.20.2
{% for package_tuple, package_info in pypi_packages.items() %}
Expand Down
8 changes: 8 additions & 0 deletions gapic/templates/testing/constraints-async-rest-3.10.txt.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{# TODO(https://github.com/googleapis/gapic-generator-python/issues/2121): Remove the following variable (and the condition later in this file) for async rest transport once support for it is GA. #}
{% set rest_async_io_enabled = api.all_library_settings[api.naming.proto_package].python_settings.experimental_features.rest_async_io_enabled %}
{% if rest_async_io_enabled %}
# -*- coding: utf-8 -*-
{% block constraints %}
{% include "testing/_default_constraints-async-rest.j2" %}
{% endblock %}
{% endif %}{# if rest_async_io_enabled #}
8 changes: 8 additions & 0 deletions gapic/templates/testing/constraints-async-rest-3.11.txt.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{# TODO(https://github.com/googleapis/gapic-generator-python/issues/2121): Remove the following variable (and the condition later in this file) for async rest transport once support for it is GA. #}
{% set rest_async_io_enabled = api.all_library_settings[api.naming.proto_package].python_settings.experimental_features.rest_async_io_enabled %}
{% if rest_async_io_enabled %}
# -*- coding: utf-8 -*-
{% block constraints %}
{% include "testing/_default_constraints-async-rest.j2" %}
{% endblock %}
{% endif %}{# if rest_async_io_enabled #}
8 changes: 8 additions & 0 deletions gapic/templates/testing/constraints-async-rest-3.12.txt.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{# TODO(https://github.com/googleapis/gapic-generator-python/issues/2121): Remove the following variable (and the condition later in this file) for async rest transport once support for it is GA. #}
{% set rest_async_io_enabled = api.all_library_settings[api.naming.proto_package].python_settings.experimental_features.rest_async_io_enabled %}
{% if rest_async_io_enabled %}
# -*- coding: utf-8 -*-
{% block constraints %}
{% include "testing/_default_constraints-async-rest.j2" %}
{% endblock %}
{% endif %}{# if rest_async_io_enabled #}
26 changes: 26 additions & 0 deletions gapic/templates/testing/constraints-async-rest-3.7.txt.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{# TODO(https://github.com/googleapis/gapic-generator-python/issues/2121): Remove the following variable (and the condition later in this file) for async rest transport once support for it is GA. #}
{% set rest_async_io_enabled = api.all_library_settings[api.naming.proto_package].python_settings.experimental_features.rest_async_io_enabled %}
{% if rest_async_io_enabled %}
{% from '_pypi_packages.j2' import pypi_packages %}
# This constraints file is used to check that lower bounds
# are correct in setup.py
# List all library dependencies and extras in this file.
# Pin the version to the lower bound.
# e.g., if setup.py has "google-cloud-foo >= 1.14.0, < 2.0.0dev",
# Then this file should have google-cloud-foo==1.14.0
google-api-core==2.20.0
google-auth==2.35.0
# from google-auth[aiohttp]
aiohttp==3.6.2
requests==2.20.0
proto-plus==1.22.3
protobuf==3.20.2
{% for package_tuple, package_info in pypi_packages.items() %}
{# Quick check to make sure the package is different from this setup.py #}
{% if api.naming.warehouse_package_name != package_info.package_name %}
{% if api.requires_package(package_tuple) %}
{{ package_info.package_name }}=={{ package_info.lower_bound }}
{% endif %}
{% endif %}
{% endfor %}
{% endif %}{# if rest_async_io_enabled #}
8 changes: 8 additions & 0 deletions gapic/templates/testing/constraints-async-rest-3.8.txt.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{# TODO(https://github.com/googleapis/gapic-generator-python/issues/2121): Remove the following variable (and the condition later in this file) for async rest transport once support for it is GA. #}
{% set rest_async_io_enabled = api.all_library_settings[api.naming.proto_package].python_settings.experimental_features.rest_async_io_enabled %}
{% if rest_async_io_enabled %}
# -*- coding: utf-8 -*-
{% block constraints %}
{% include "testing/_default_constraints-async-rest.j2" %}
{% endblock %}
{% endif %}{# if rest_async_io_enabled #}
8 changes: 8 additions & 0 deletions gapic/templates/testing/constraints-async-rest-3.9.txt.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
{# TODO(https://github.com/googleapis/gapic-generator-python/issues/2121): Remove the following variable (and the condition later in this file) for async rest transport once support for it is GA. #}
{% set rest_async_io_enabled = api.all_library_settings[api.naming.proto_package].python_settings.experimental_features.rest_async_io_enabled %}
{% if rest_async_io_enabled %}
# -*- coding: utf-8 -*-
{% block constraints %}
{% include "testing/_default_constraints-async-rest.j2" %}
{% endblock %}
{% endif %}{# if rest_async_io_enabled #}
3 changes: 2 additions & 1 deletion noxfile.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -300,8 +300,9 @@ def showcase_library(
# supported version of a dependency.
# This is needed to recreate the issue reported in
# https://github.com/googleapis/google-cloud-python/issues/12254
constraints_type = "async-rest-" if rest_async_io_enabled else ""
constraints_path = str(
f"{tmp_dir}/testing/constraints-{session.python}.txt"
f"{tmp_dir}/testing/constraints-{constraints_type}{session.python}.txt"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A note: unlike what I remarked on another comment thread in this PR, where we're using the constraints files as -c arguments to PIP (which hence do not control whether those dependencies are installed), here we are passing the file via -r (three lines below) thus making it into a requirements file where the listed dependencies are installed. AFAICT, we do not use -r for the (same??!) constraints files under templates/.

We should be clear about the semantics and how we are using the files. I'd suggest for the purposes of this file, creating proper requirements files that do list what will be installed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the constraints file.

We're infact installing the constraints file with a -r flag which is why this wasn't an issue here (at least at the generator layer). Anyways, this has been addressed by installing the deps with the extra.

)
# Install the library with a constraints file.
session.install("-e", tmp_dir, "-r", constraints_path)
Expand Down
6 changes: 0 additions & 6 deletions tests/integration/goldens/asset/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,12 +49,6 @@
"google-cloud-os-config >= 1.0.0, <2.0.0dev",
"grpc-google-iam-v1 >= 0.12.4, <1.0.0dev",
]
extras = {
"async_rest": [
"google-api-core[grpc] >= 2.20.0, < 3.0.0dev",
"google-auth[aiohttp] >= 2.35.0, <3.0.0dev"
],
}
url = "https://github.com/googleapis/google-cloud-python/tree/main/packages/google-cloud-asset"

package_root = os.path.abspath(os.path.dirname(__file__))
Expand Down
6 changes: 0 additions & 6 deletions tests/integration/goldens/eventarc/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,6 @@
"protobuf>=3.20.2,<6.0.0dev,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5",
"grpc-google-iam-v1 >= 0.12.4, <1.0.0dev",
]
extras = {
"async_rest": [
"google-api-core[grpc] >= 2.20.0, < 3.0.0dev",
"google-auth[aiohttp] >= 2.35.0, <3.0.0dev"
],
}
url = "https://github.com/googleapis/google-cloud-python/tree/main/packages/google-cloud-eventarc"

package_root = os.path.abspath(os.path.dirname(__file__))
Expand Down
6 changes: 0 additions & 6 deletions tests/integration/goldens/logging/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,6 @@
"proto-plus >= 1.22.3, <2.0.0dev",
"protobuf>=3.20.2,<6.0.0dev,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5",
]
extras = {
"async_rest": [
"google-api-core[grpc] >= 2.20.0, < 3.0.0dev",
"google-auth[aiohttp] >= 2.35.0, <3.0.0dev"
],
}
url = "https://github.com/googleapis/google-cloud-python/tree/main/packages/google-cloud-logging"

package_root = os.path.abspath(os.path.dirname(__file__))
Expand Down
2 changes: 1 addition & 1 deletion ...ation/goldens/redis/google/cloud/redis_v1/services/cloud_redis/transports/rest_asyncio.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
from google.auth.aio.transport.sessions import AsyncAuthorizedSession # type: ignore
from google.api_core import rest_streaming_async # type: ignore
except ImportError as e: # pragma: NO COVER
raise ImportError("`rest_asyncio` transport requires the library to be installed with the `async_rest` extra. Install the library with the `async_rest` extra using `pip install <insert_library_name>[async_rest]`") from e
raise ImportError("`rest_asyncio` transport requires the library to be installed with the `async_rest` extra. Install the library with the `async_rest` extra using `pip install google-cloud-redis[async_rest]`") from e

from google.auth.aio import credentials as ga_credentials_async # type: ignore

Expand Down
1 change: 1 addition & 0 deletions tests/integration/goldens/redis/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@
packages=packages,
python_requires=">=3.7",
install_requires=dependencies,
extras_require=extras,
include_package_data=True,
zip_safe=False,
)
4 changes: 0 additions & 4 deletions tests/integration/goldens/redis/testing/constraints-3.10.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,5 @@
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
google-auth
# from google-auth[aiohttp]
aiohttp
requests
proto-plus
protobuf
4 changes: 0 additions & 4 deletions tests/integration/goldens/redis/testing/constraints-3.11.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,5 @@
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
google-auth
# from google-auth[aiohttp]
aiohttp
requests
proto-plus
protobuf
4 changes: 0 additions & 4 deletions tests/integration/goldens/redis/testing/constraints-3.12.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,5 @@
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
google-auth
# from google-auth[aiohttp]
aiohttp
requests
proto-plus
protobuf
7 changes: 2 additions & 5 deletions tests/integration/goldens/redis/testing/constraints-3.7.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,7 @@
# Pin the version to the lower bound.
# e.g., if setup.py has "google-cloud-foo >= 1.14.0, < 2.0.0dev",
# Then this file should have google-cloud-foo==1.14.0
google-api-core==2.20.0
google-auth==2.35.0
# from google-auth[aiohttp]
aiohttp==3.6.2
requests==2.20.0
google-api-core==1.34.1
google-auth==2.14.1
proto-plus==1.22.3
protobuf==3.20.2
4 changes: 0 additions & 4 deletions tests/integration/goldens/redis/testing/constraints-3.8.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,5 @@
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
google-auth
# from google-auth[aiohttp]
aiohttp
requests
proto-plus
protobuf
4 changes: 0 additions & 4 deletions tests/integration/goldens/redis/testing/constraints-3.9.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,5 @@
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
google-auth
# from google-auth[aiohttp]
aiohttp
requests
proto-plus
protobuf
10 changes: 10 additions & 0 deletions tests/integration/goldens/redis/testing/constraints-async-rest-3.10.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# -*- coding: utf-8 -*-
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
google-auth
# from google-auth[aiohttp]
aiohttp
requests
proto-plus
protobuf
10 changes: 10 additions & 0 deletions tests/integration/goldens/redis/testing/constraints-async-rest-3.11.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# -*- coding: utf-8 -*-
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
google-auth
# from google-auth[aiohttp]
aiohttp
requests
proto-plus
protobuf
10 changes: 10 additions & 0 deletions tests/integration/goldens/redis/testing/constraints-async-rest-3.12.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# -*- coding: utf-8 -*-
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
google-auth
# from google-auth[aiohttp]
aiohttp
requests
proto-plus
protobuf
13 changes: 13 additions & 0 deletions tests/integration/goldens/redis/testing/constraints-async-rest-3.7.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# This constraints file is used to check that lower bounds
# are correct in setup.py
# List all library dependencies and extras in this file.
# Pin the version to the lower bound.
# e.g., if setup.py has "google-cloud-foo >= 1.14.0, < 2.0.0dev",
# Then this file should have google-cloud-foo==1.14.0
google-api-core==2.20.0
google-auth==2.35.0
# from google-auth[aiohttp]
aiohttp==3.6.2
requests==2.20.0
proto-plus==1.22.3
protobuf==3.20.2
10 changes: 10 additions & 0 deletions tests/integration/goldens/redis/testing/constraints-async-rest-3.8.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# -*- coding: utf-8 -*-
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
google-auth
# from google-auth[aiohttp]
aiohttp
requests
proto-plus
protobuf
10 changes: 10 additions & 0 deletions tests/integration/goldens/redis/testing/constraints-async-rest-3.9.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# -*- coding: utf-8 -*-
# This constraints file is required for unit tests.
# List all library dependencies and extras in this file.
google-api-core
google-auth
# from google-auth[aiohttp]
aiohttp
requests
proto-plus
protobuf