Backup of data on kg-hub s3 bucket

[justaddcoffee]

We have enough data and projects to where we should strongly consider a backup solution.

Describe the desired behavior

Backup existing data on s3://kg-hub-public-data/ to a non-AWS location on an ongoing basis - monthly?

Additional context

Google Storage might be a good solution:
https://cloud.google.com/storage/docs/storage-classes

[justaddcoffee]

@caufieldjh @kltm

[caufieldjh]

One strategy for GCP: https://cloud.google.com/architecture/transferring-data-from-amazon-s3-to-cloud-storage-using-vpc-service-controls-and-storage-transfer-service

[caufieldjh]

Discussion in BBOP meeting today (Nov 11 2023):

• Set up KG-Hub community on Zenodo
• Automate upload to Zenodo (try sandbox.zenodo.org) - see also https://github.com/cthoyt/zenodo-client and the GO's client
• Check with @kltm for any assistance

Note that we're trying to ensure both data security/integrity (i.e., backups) and accessibility/findability (i.e., can users download graph releases and have specific DOIs)

[justaddcoffee]

Sorry to have missed the discussion at group meeting (I was traveling)

@caufieldjh should we hack on this on our KG construction calls, and/or our M/W/F hackathons?

[caufieldjh]

@caufieldjh should we hack on this on our KG construction calls, and/or our M/W/F hackathons?

Probably a good subject for the KG construction calls, especially if we can record the details in a best practices doc

[justaddcoffee]

+1 @caufieldjh sounds good!