Easier system test setup/teardown

[duggelz]

Feature request: A single command I (as an authenticated user, or CI system) run that:

1. Creates an ephemeral google cloud project based on the billing, permissions and name of an existing project
2. Enables all the necessary APIs
3. Creates and pulls down credentials and secrets
4. Runs the system tests
   [Optional] 5) Tear down and delete everything

For example system_test/setup cloud-python-runtime-qa run by user [email protected] creates a project cloud-python-runtime-qa-1234, populates the datastore with sample data, downloads the json credentials in some reasonable way.

I don't actually know if this level of automation is actually possible with the platform yet.

[daspecster]

I don't think that 2 and 3 are really possible now. I don't know that it would be good practice to pull down overall credentials. There is kind of a public/private key model right now that is secure.

I'm not really sure what the overall goal that you're trying to achieve is. Maybe you can clarify that a little bit?

[duggelz]

I want to qualify new Python runtime images, by running all the google-cloud-python system tests inside the new image, before it's been released. I have set up a project to run these tests in, with the populated sample data. But a few problems crop up:

1. Concurrency: Ideally, this qualification test would be run by both CI and developers, often concurrently. So if they all use the same project name and/or storage, tests will stomp on each other whenever they perform mutating actions, which they presumably do.

2. Robust Setup: I got 7 out of 10 system test targets working, and left it at that, because I couldn't figure out what I needed to enable for the other three. I ever need to recreate the project, or create a second project for CI or whatever, it will be a pain, and probably won't be identical to the first project. Non-reproducible test environments are bad for everyone.

This is what I'm doing right now, from the command line, locally:

GoogleCloudPlatform/python-runtime@9aeedd3

Any suggestions on improvements are welcome. I want to get this running on the internal CI, but handling the credentials and api keys in a safe way is challenging.

[dhermes]

Thanks for the pointer to GoogleCloudPlatform/python-runtime#39, it gives a bit (a lotta bit?) more context than your GitHub profile page.

1. Our CONTRIBUTING doc should explain how to set up the system tests, if it doesn't let's try to fix that.
2. The actual tests should fail with a helpful error explaining the missing config values (again, if this isn't the case, let's fix it)
3. Creating short-lived credentials is outside (way outside?) of our scope, but again, maybe we could bring the help messages nearer to devs (e.g. in error messages). Someday the gcloud CLI might do this, but even that is doubtful
4. The concurrency issue is "handled" by our use of unique_resource_id (e.g.). If you have code paths which result in test collisions, please let us know which they are, we certainly don't intend for that to be the case.

[duggelz]

I added a company name and snazzy picture to my GitHub profile :) If the tests are already setup to play nicely within a single project, that is excellent, and removes the need for 1, 3 and 5. I just assumed it was a problem without actually checking.

That leaves the list of APIs that need to be enabled, and a nicer way to setup data and indexes that doesn't require installing the cloud SDK.

After some more work, I got everything working except for logging (see error message below).

This is the list of APIs I enabled in my project (cloud-python-runtime-qa), which should be explicitly listed in the CONTRIBUTING doc.

Google Cloud Storage JSON API
Bigquery API
Google Cloud Pub/Sub API
Cloud Bigtable Admin API
Stackdriver Logging API
Stackdriver Monitoring API
Google Cloud Natural Language API
Google Translate API
Cloud Bigtable API (Not sure if this one is necessary)

logging

======================================================================
ERROR: test_create_sink_bigquery_dataset (logging_.TestLogging)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/vmagent/app/google-cloud-python/system_tests/logging_.py", line 400, in test_create_sink_bigquery_dataset
    sink.create()
  File "/home/vmagent/app/google-cloud-python/.tox/system-tests/local/lib/python2.7/site-packages/google/cloud/logging/sink.py", line 119, in create
    self.project, self.name, self.filter_, self.destination)
  File "/home/vmagent/app/google-cloud-python/.tox/system-tests/local/lib/python2.7/site-packages/google/cloud/logging/_gax.py", line 210, in sink_create
    self._gax_api.create_sink(parent, sink_pb, options=options)
  File "/home/vmagent/app/google-cloud-python/.tox/system-tests/local/lib/python2.7/site-packages/google/cloud/gapic/logging/v2/config_service_v2_api.py", line 303, in create_sink
    return self._create_sink(request, options)
  File "/home/vmagent/app/google-cloud-python/.tox/system-tests/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 481, in inner
    return api_caller(api_call, this_settings, request)
  File "/home/vmagent/app/google-cloud-python/.tox/system-tests/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 469, in base_caller
    return api_call(*args)
  File "/home/vmagent/app/google-cloud-python/.tox/system-tests/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 434, in inner
    errors.create_error('RPC failed', cause=exception))
  File "/home/vmagent/app/google-cloud-python/.tox/system-tests/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 430, in inner
    return a_func(*args, **kwargs)
  File "/home/vmagent/app/google-cloud-python/.tox/system-tests/local/lib/python2.7/site-packages/google/gax/api_callable.py", line 64, in inner
    return a_func(*updated_args, **kwargs)
  File "/home/vmagent/app/google-cloud-python/.tox/system-tests/local/lib/python2.7/site-packages/grpc/_channel.py", line 481, in __call__
    return _end_unary_response_blocking(state, False, deadline)
  File "/home/vmagent/app/google-cloud-python/.tox/system-tests/local/lib/python2.7/site-packages/grpc/_channel.py", line 432, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
GaxError: GaxError(RPC failed, caused by <_Rendezvous of RPC that terminated with (StatusCode.PERMISSION_DENIED, The caller does not have permission)>)

Also, monitoring produces these errors, which I'm not sure are benign:

400 The provided filter doesn't refer to any known metric. (GET https://monitoring.googleapis.com/v3/projects/cloud-python-runtime-qa/timeSeries/?filter=metric.type+%3D+%22custom.googleapis.com%2Ftmp%2Fsystem_test_example_1478206668779%22&interval.endTime=2016-11-03T20%3A57%3A00.000000Z&interval.startTime=2016-11-03T20%3A52%3A00.000000Z), Trying again in 1 seconds...
400 The provided filter doesn't refer to any known metric. (GET https://monitoring.googleapis.com/v3/projects/cloud-python-runtime-qa/timeSeries/?filter=metric.type+%3D+%22custom.googleapis.com%2Ftmp%2Fsystem_test_example_1478206668779%22&interval.endTime=2016-11-03T20%3A57%3A00.000000Z&interval.startTime=2016-11-03T20%3A52%3A00.000000Z), Trying again in 2 seconds...
400 The provided filter doesn't refer to any known metric. (GET https://monitoring.googleapis.com/v3/projects/cloud-python-runtime-qa/timeSeries/?filter=metric.type+%3D+%22custom.googleapis.com%2Ftmp%2Fsystem_test_example_1478206668779%22&interval.endTime=2016-11-03T20%3A57%3A00.000000Z&interval.startTime=2016-11-03T20%3A52%3A00.000000Z), Trying again in 4 seconds...
_has_timeseries. Trying again in 1 seconds...
400 The provided filter doesn't refer to any known metric. (GET https://monitoring.googleapis.com/v3/projects/cloud-python-runtime-qa/timeSeries/?filter=metric.type+%3D+%22custom.googleapis.com%2Ftmp%2Fsystem_test_example_1478206668779%22&interval.endTime=2016-11-03T20%3A57%3A00.000000Z&interval.startTime=2016-11-03T20%3A52%3A00.000000Z), Trying again in 8 seconds

[dhermes]

• We'd be happy to have the datastore setup happen in Python code instead of through the SDK, but AFAIK there isn't a public API for updating indexes (also, it's a lot of churn to create and throw away a project every time you run the tests. If you have a long-lived project, the index creation is a one-shot cost.)
• The logging error you see is because some bucket or other object needs to add [email protected] to an ACL. This is something we CAN improve in the system test itself, you want to file an issue or should I?
• I'm happy to add a list of APIs that need to be enabled, but I'm fairly certain it would go stale due to human error. The only Google APIs I know of that has a way check project info is the Resource Manager API but the "get project" payload doesn't list enabled APIs
• Believe it or not the monitoring messages (not errors) are benign. The API using a 400 when it should use a 404. This is just eventual consistency of a query rearing it's head.

[duggelz]

Filed #2740 for the logging error. If the rest is won't/can't fix, which seems reasonable, then go ahead and close this.