Metrics prefetcher script (#30327)

* Adding github workflow data prefetcher script

* added workflow runs data for alerting

* Add provider config, squeduler and invoker account. Also added chagnes
to the dashboard

* remove whitespace

* bucket name adjust

* replace test table names with proper names

* rework GA Post-Commits dashboard query for consistency

* remove old sync script

* adjust thresholds

* adjust schedule for every 3 hrs

* remove test paramaters

---------

Co-authored-by: Vlado Djerek <[email protected]>
Co-authored-by: Andrey Devyatkin <[email protected]>

.gitignore

@@ -147,3 +147,7 @@ playground/cloudfunction.zip
 
 # Ignore .test-infra/mock-apis related files
 .test-infra/mock-apis/**/charts/
+
+# Ignore .test-infra/metrics/github_runs_prefetcher/code.zip
+# as its generated with terraform
+.test-infra/metrics/sync/github/github_runs_prefetcher/code.zip

.test-infra/metrics/sync/github/github_runs_prefetcher/README.md

@@ -0,0 +1,31 @@
+<!--
+    Licensed to the Apache Software Foundation (ASF) under one
+    or more contributor license agreements.  See the NOTICE file
+    distributed with this work for additional information
+    regarding copyright ownership.  The ASF licenses this file
+    to you under the Apache License, Version 2.0 (the
+    "License"); you may not use this file except in compliance
+    with the License.  You may obtain a copy of the License at
+
+      http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing,
+    software distributed under the License is distributed on an
+    "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+    KIND, either express or implied.  See the License for the
+    specific language governing permissions and limitations
+    under the License.
+-->
+# Github Workflows Prefetcher
+Github Workflows Prefetcher is a script used to fetch workflow run data and store it in CloudSQL for further use.
+It is used by Grafana (https://metrics.beam.apache.org) to display status dashboards and for alerting which is further used for failing test issue creation.
+
+## Setup
+
+Initialize the terraform state with
+```
+terraform init -backend-config="bucket=beam-arc-state"
+```
+
+
+Then execute `terraform apply`

.test-infra/metrics/sync/github/github_runs_prefetcher/bucket.tf

@@ -0,0 +1,28 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+resource "google_storage_bucket" "bucket" {
+  name     = "github_workflow_prefetcher_function"
+  location = "US"
+
+}
+
+resource "google_storage_bucket_object" "archive" {
+  name   = "${data.archive_file.code.output_md5}.zip"
+  bucket = google_storage_bucket.bucket.name
+  source = "code.zip"
+}
+
+

.test-infra/metrics/sync/github/github_runs_prefetcher/code.tf

@@ -0,0 +1,21 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+data "archive_file" "code" {
+  type        = "zip"
+  output_path = "code.zip"
+
+  source_dir = "code"
+}