Skip to content

Commit

Permalink
ingest/fetch-ncov-global-case-counts: follow redirection
Browse files Browse the repository at this point in the history
Our case counts updates resulted in an empty file¹, which lead to
the failure of our model pipelines.² Turns out the OWID endpoint is
redirected to another endpoint.

```
$ curl -I https://covid.ourworldindata.org/data/owid-covid-data.csv
HTTP/2 302
date: Fri, 27 Oct 2023 17:34:37 GMT
location: https://covid-19.nyc3.digitaloceanspaces.com/public/owid-covid-data.csv
...
```

curl does not follow HTTP redirections by default, so we have to tell
curl to follow redirection with `--location` option.

¹ https://github.com/nextstrain/forecasts-ncov/actions/runs/6656776373/job/18090077156#step:5:11
² https://bedfordlab.slack.com/archives/C03BY4MPNCS/p1698393532650979
  • Loading branch information
joverlee521 committed Oct 27, 2023
1 parent b04040d commit 8ffa11e
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion ingest/bin/fetch-ncov-global-case-counts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ set -euo pipefail

# Fetch CSV from Our World in Data
curl https://covid.ourworldindata.org/data/owid-covid-data.csv \
--fail --silent --show-error \
--fail --silent --show-error --location \
--header 'User-Agent: https://github.com/nextstrain/counts ([email protected])' |
# Only keep the date, location, and new_cases columns
csvtk cut -f location,date,new_cases |
Expand Down

0 comments on commit 8ffa11e

Please sign in to comment.