[Failing Test]: Python PostCommit is failing due to exceeded rate limits

[AnandInguva]

What happened?

https://ci-beam.apache.org/job/beam_PostCommit_Python37/6753/testReport/junit/apache_beam.io.gcp.bigquery_write_it_test/BigQueryWriteIntegrationTests/test_big_query_write_without_schema/

https://ci-beam.apache.org/job/beam_PostCommit_Python311/240/testReport/junit/apache_beam.io.gcp.bigquery_read_it_test/ReadAllBQTests/test_read_queries/

Python 3.7 is red and python 3.11 just started to be in red. The error seems the same but for different tests in the apache_bem/io/

Stack trace:

apache_beam/io/gcp/bigquery_write_it_test.py:305: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
apache_beam/io/gcp/bigquery_write_it_test.py:115: in create_table
    self.bigquery_client.client.tables.Insert(request)
apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py:955: in Insert
    return self._RunMethod(config, request, global_params=global_params)
../../build/gradleenv/1398941891/lib/python3.7/site-packages/apitools/base/py/base_api.py:731: in _RunMethod
    return self.ProcessHttpResponse(method_config, http_response, request)
../../build/gradleenv/1398941891/lib/python3.7/site-packages/apitools/base/py/base_api.py:737: in ProcessHttpResponse
    self.__ProcessHttpResponse(method_config, http_response, request))
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <apache_beam.io.gcp.internal.clients.bigquery.bigquery_v2_client.BigqueryV2.TablesService object at 0x7f59aaaff410>
method_config = <ApiMethodInfo
 relative_path: 'projects/{projectId}/datasets/{datasetId}/tables'
 method_id: 'bigquery.tables.insert'...ype_name: 'BigqueryTablesInsertRequest'
 response_type_name: 'Table'
 request_field: 'table'
 supports_download: False>
http_response = Response(info={'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Thu, 0...ogleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_write_to_table_1685625959d97c5f/tables?alt=json')
request = <BigqueryTablesInsertRequest
 datasetId: 'python_write_to_table_1685625959d97c5f'
 projectId: 'apache-beam-testing'
 t...setId: 'python_write_to_table_1685625959d97c5f'
 projectId: 'apache-beam-testing'
 tableId: 'python_no_schema_table'>>>

    def __ProcessHttpResponse(self, method_config, http_response, request):
        """Process the given http response."""
        if http_response.status_code not in (http_client.OK,
                                             http_client.CREATED,
                                             http_client.NO_CONTENT):
            raise exceptions.HttpError.FromResponse(
>               http_response, method_config=method_config, request=request)
E           apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing <https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/datasets/python_write_to_table_1685625959d97c5f/tables?alt=json>: response: <{'vary': 'Origin, X-Origin, Referer', 'content-type': 'application/json; charset=UTF-8', 'date': 'Thu, 01 Jun 2023 13:25:59 GMT', 'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0', 'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'status': '403', 'content-length': '564', '-content-encoding': 'gzip'}>, content <{
E             "error": {
E               "code": 403,
E               "message": "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see [https://cloud.google.com/bigquery/docs/troubleshoot-quotas",](https://cloud.google.com/bigquery/docs/troubleshoot-quotas)
E               "errors": [
E                 {
E                   "message": "Exceeded rate limits: too many api requests per user per method for this user_method. For more information, see [https://cloud.google.com/bigquery/docs/troubleshoot-quotas",](https://cloud.google.com/bigquery/docs/troubleshoot-quotas)
E                   "domain": "usageLimits",
E                   "reason": "rateLimitExceeded"
E                 }
E               ],
E               "status": "PERMISSION_DENIED"
E             }
E           }

Issue Failure

Failure: Test is flaky

Issue Priority

Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[AnandInguva]

cc: @johnjcasey

[kennknowles]

3.7 is gone and 3.11 was green but has other things making it red.