can't use opendap on a datasetScan S3 object

[WeatherGod]

Working off of 5.5-SNAPSHOT docker image, I am trying to utilize the new datasetScan for S3 feature. It seems to be working fairly well, except that it appears that opendap is getting the "delimiter=/" fragment and maybe getting confused?

The response I got trying to access thredds/dodsC/gfs_noaa/gfs.20230503/00/atmos/gfs.t00z.pgrb2.0p25.f000.html

Error {
    code = 500;
    message = "java.lang.IllegalStateException: No records found in dataset gfs.t00z.pgrb2.0p25.f000#delimiter=";
};

The relevant portion of the catalog file:

  <datasetScan name="GFS NOAA Open Bucket" ID="gfs_noaa" path="gfs_noaa" 
    location="cdms3:noaa-gfs-bdp-pds#delimiter=/">
    <metadata inherited="true">
    <serviceName>all</serviceName>
    <dataType>Grid</dataType>
    </metadata>
    <filter>
        <include wildcard="*gfs.t00z.pgrb2.0p25.f???"/>
    </filter>
  </datasetScan>

[dopplershift]

I'm not shocked at all here since this is GRIB2 data. It works fine here: https://thredds-aws.unidata.ucar.edu/thredds/catalog/nexrad/level2/S3/2005/catalog.html (well, it did. Right now that server is timing out). With the need for creating indexes for GRIB data, I'm not sure that's all been properly worked out (i.e. having the indexes separate from the data). (If anyone knows better, by all means correct my ignorance.)

[WeatherGod]

Possible relevant info from the threddsServlet.log:

2023-05-08T19:12:35.809 +0000 [   1569923][     167] INFO  - threddsServlet - Remote host: 10.199.2.137 - Request: "GET /thredds/ncml/gfs_noaa/gfs.20230503/00/atmos/gfs.t00z.pgrb2.0p25.f000 HTTP/1.1"
2023-05-08T19:12:36.201 +0000 [   1570315][     167] ERROR - ucar.nc2.grib.collection.Grib2Iosp - Grib2CollectionBuilder gfs.t00z.pgrb2.0p25.f000#delimiter= : reading/Creating gbx9 index for file cdms3:noaa-gfs-bdp-pds?gfs.20230503/00/atmos/gfs.t00z.pgrb2.0p25.f000#delimiter= failed
java.io.FileNotFoundException: cdms3:noaa-gfs-bdp-pds?gfs.20230503/00/atmos/gfs.t00z.pgrb2.0p25.f000#delimiter= (No such file or directory)

[dopplershift]

That would seem to confirm my suspicion. There's also a line that looks in the same vein of (but different to) the problem @tdrwenski fixed in Unidata/netcdf-java#1175.

[tdrwenski]

Thanks for testing @WeatherGod! Ryan is correct that things don't yet work for creating grib indexes or feature collection indexes for S3. I will make a GitHub issue for that so that hopefully the current status of the remaining S3 issues is clearer.

[tdrwenski]

I made this issue for S3 grib files: Unidata/netcdf-java#1177

I will close this issue as a duplicate. Let us know if you encounter any other S3 issues!