transferring data from HPC storage (e.g., NCAR campaign store) to LEAP-Pangeo

[josephko91]

Hello. I'm starting a general thread about transferring data from HPC systems to LEAP-Pangeo cloud buckets. @jbusecke mentioned that multiple people have asked about this, so consolidating this topic into a central discussion here,

[jbusecke]

Fantastic. Thanks for taking the time to raise this. I think the basic problem here is that our current approach to ingest data is to 'pull' the data using pangeo-forge. This fundamentally requires the data to be available to download (via http, ftp, ...).

This is often not a supported pattern in many HPC centers.

So in lieu of this preferred method I think we have a few options:

• Run the ingestion from inside the HPC center
  • pros: Probably fairly easy to implement on the short term
  • cons: needs somebody with credentials to run whatever pipeline we use, might break in the future..
• Move data out of HPC storage in a batch fashion to cloud storage (e.g. using globus), then ingesting the data from there.
  • pros: overall pipeline is more reproducible
  • cons: still needs 'insider' to run the transfer (less knowledge required than above)

[jbusecke]

Also pinging @TomNicholas here, who might have overlapping needs outside of LEAP.

[josephko91]

@jbusecke This might be a bit of a tangential question, but what about data that might not neatly fit into the xarray data structure? For example, I have trajectory outputs from Lagrangian LES simulations that are not formatted as netcdf files. They are just a bunch of tabular text files, organized by processor id and time step. Of course this is just one example, but I'm sure there are other data out there that might not be a good fit for the pangeo-forge framework, e.g., any tabular data that isn't multi-dimensional. Any thoughts on how to handle other data structures like this? Thanks!

[jbusecke]

Are they actually tabular (e.g. same number of rows? then you could use parquet?) or are they 'ragged' (different numbers of rows per column?

[josephko91]

They are tabular (same number of rows). So I'm guessing the general route would be to convert them to parquet and then transfer from NCAR -> GCP buckets using Globus (as outlined below by Negin).

My original question was more directly related to the suggested approach of using pangeo-forge, but I found the following in the pangeo-forge FAQ:

Pangeo Forge is currently focused primarily on multidimensional array data; i.e., data that fit the NetCDF data model and can be opened by Xarray. This includes NetCDF, GRIB, HDF4/5, Zarr, OPeNDAP, and many other obscure formats. Pangeo Forge is not currently focused on tabular data; i.e., data that can be opened by Pandas, CSV, SQL, Parquet, Arrow, etc. This data model and these formats are very well supported by other ETL tools in the modern data stack.

Just leaving this here for reference.

[josephko91]

Response from Negin Sobhani at NCAR CISL:

Here are two methods I suggest for this:

1. To transfer files from NCAR HPC to GCP buckets, one efficient method is using Globus. We have a Globus endpoint on Glade that you can easily access. While I do not have read access to LEAP's GCP buckets, if they have a Globus endpoint on GCP, you can seamlessly transfer data directly to those buckets.

Here's how to locate and activate the NCAR GLADE endpoint using the command line:

> gc_glade=$(globus endpoint search "[email protected] NCAR GLADE" --jq "DATA[0].id" --format UNIX)
> globus endpoint activate $gc_glade

Alternatively, you can use the Globus GUI for a more user-friendly interface.

For this to work, LEAP needs to install Google Cloud Storage Connector which allows Globus Connect Server to access Google Cloud Storage buckets associated with Google accounts. Here is the link to Globus Google Cloud Bucket connector: https://docs.globus.org/premium-storage-connectors/v5/google-cloud-storage/

2. Another method to consider is using the gsutil tool from Google Cloud. After installing it in a Conda environment, you can transfer files using the following command: https://cloud.google.com/storage/docs/gsutil

For example, after installing it in a conda environment, you can use:

> gsutil -v cp /glade/u/home/.... gs://your-bucket-name/

[negin513]

Thanks, @josephko91, for adding my response here.

[josephko91]

thank you for the help! @negin513

[jbusecke]

Thanks for all this info @negin513 and @josephko91. I am still very keen to pursue globus, but also just heard about pangeo-forge/pangeo-forge-recipes#753 (comment).

@josephko91 do you think we could access the files via ssh?

[negin513]

I just wanted to follow up on this. You can indeed use scp to move files directly from and to /glade (NCAR HPC filesystem). For example,

scp -r  [email protected]:/glade/work/joeko/this_folder . 

But you need to have an account on the system (for example @josephko91 has one).

You can setup your ssh config to avoid 2-step completely as well. Please let me know if you need help with that.

[negin513]

@josephko91 Here are our docs for using scp : https://ncar-hpc-docs.readthedocs.io/en/latest/storage-systems/data-transfer/scp-and-sftp/?h=scp#scp-transfer

[negin513]

@jbusecke , I think using Globus might offer better performance. I think for creating an endpoint on LEAP Google Cloud bucket, you need to use Google storage connector: https://docs.globus.org/premium-storage-connectors/v5.4/google-cloud-storage/#google_cloud_storage_virtual_filesystem

Once there is an endpoint on the bucket (and there is already one on glade), data transfer should be relatively straightforward and performant. Please let me know if I can help with this. 😄

[jbusecke]

Thanks for that info @negin513. I agree that long term globus is the better way here, but I am curious to see if this can unblock folks here to ingest datasets with an alternate method (getting thise endpoints/connectors set up seems to be a medium sized hassle feom what I can tell 😭, but I am working on that).

[negin513]

Agreed @jbusecke !
I think in the meantime using gsutils as an alternative would work fine and you can point them to this. For example after installing they install on their conda environment they can run:

> gsutil -v cp /glade/u/home/....  gs://your-bucket-name/

[jbusecke]

leap-stc/data-management#118 is having the same issue. We just got our OSN pod comissioned and I will try to find at least a temporary solution to do this!