Skip to content

Commit

Permalink
cloud: Load submissions into PostgreSQL only, faster
Browse files Browse the repository at this point in the history
Remove loading submissions into BigQuery until we come up with a way to
increase throughput (likely pulling chunks from PostgreSQL). This should
help us deal with the backlog in production submission queue.

We might also need to either switch to direct triggering of Cloud
Functions by messages from the queue, to reduce latency, or simply
switch to a persistent Cloud Run service.

Concerns: #541
  • Loading branch information
spbnick committed Jul 5, 2024
1 parent 3ac8ce8 commit 91e3025
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 9 deletions.
9 changes: 1 addition & 8 deletions cloud
Original file line number Diff line number Diff line change
Expand Up @@ -217,14 +217,7 @@ function execute_command() {
fi
declare -a -r bigquery_args

declare -r database=$(
echo -n "mux: "
echo -n "$psql_kcidb_db" | escape_whitespace
echo -n " "
echo -n "$bigquery_kcidb_db" | escape_whitespace
echo -n " "
echo -n "$bigquery_sample_kcidb_db" | escape_whitespace
)
declare -r database="$psql_kcidb_db"

if "$test"; then
declare -r sqlite_clean_test_file="$TMPDIR/clean.sqlite3"
Expand Down
2 changes: 1 addition & 1 deletion kcidb/cloud/functions.sh
Original file line number Diff line number Diff line change
Expand Up @@ -228,7 +228,7 @@ function functions_deploy() {
--env-vars-file "$env_yaml_file" \
--trigger-topic "${load_queue_trigger_topic}" \
--memory 1024MB \
--max-instances=1 \
--max-instances=4 \
--timeout 540
# Remove the environment YAML file
rm "$env_yaml_file"
Expand Down

0 comments on commit 91e3025

Please sign in to comment.