Issue running icenet_data_sic with 2020 date range

[bnubald]

• IceNet version: 0.2.6
• Python version: 3.8.18
• Operating System: Windows (WSL2 Ubuntu 22.04.3 LTS)

Description

Attempting to download Sea Ice Concentration dataset for southern hemisphere from OSI SAF between 2020-1-1 and 2020-1-30 causes FTP error warning, and does not download any files to data/ directory as expected.

Steps taken are based on 01.cli_demonstration notebook in icenet-notebooks.

What I Did

Commands run:

$ icenet_data_masks south
$ icenet_data_era5 south -d --vars uas,vas,tas,zg --levels ',,,500|250' 2020-1-1 2020-1-30
$ icenet_data_sic south -d 2020-1-1 2020-1-30

The last line results in the following error:

[26-10-23 22:20:44 :INFO    ] - OSASIF-SIC Data Downloading
[26-10-23 22:20:44 :INFO    ] - Downloading SIC datafiles to .temp intermediates...
[26-10-23 22:20:45 :INFO    ] - FTP opening
[26-10-23 22:20:45 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_01
[26-10-23 22:20:45 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_02
[26-10-23 22:20:45 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_03
[26-10-23 22:20:45 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_04
[26-10-23 22:20:45 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_05
[26-10-23 22:20:45 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_06
[26-10-23 22:20:45 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_07
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_08
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_09
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_10
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_11
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_12
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_13
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_14
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_15
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_16
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_17
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_18
[26-10-23 22:20:46 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_19
[26-10-23 22:20:47 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_20
[26-10-23 22:20:47 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_21
[26-10-23 22:20:47 :WARNING ] - FTP error, possibly missing month chdir for 2020_01_22

Note: Changing command's date range to 2021 year, i.e. between 2021-1-1 and 2021-1-30 downloads files as expected without warnings.

[bnubald]

Relevant code section:

icenet/icenet/data/sic/osisaf.py

Line 404 in 4608624

chdir_path = chdir_path.format(el.year, el.month)

Checking the chdir_path variable in above code with this date range shows it is searching for files in the following path on the OSI SAF ftp server:

ftp://osisaf.met.no/reprocessed/ice/conc-cont-reproc/v3p0/2020/01/*

But, 2020 directory does not exist in v3p0/.

Snapshot of ftp://osisaf.met.no/reprocessed/ice/conc-cont-reproc/v3p0/ path

Possible causes

• OSI SAF dataset section shows OSI-450 and OSI-430-b as superseded, has path changed since archiving, or is path incorrect?

• OSI SAF OSI-430-b shows data directory (for 01/01/2016 to 30/04/2023) as conc-cont-reproc/v2p0, not conc-cont-reproc/v3p0, so, it is searching under OSI-430-a instead.

Checking conc-cont-reproc/v2p0 path on remote server:

Snapshot of ftp://osisaf.met.no/reprocessed/ice/conc-cont-reproc/v2p0/ path

[bnubald]

OSI-430-a shows start date for dataset as 01/01/2021, but start date set to 18/11/2018 here:

icenet/icenet/data/sic/osisaf.py

Line 325 in 4608624

osi430a_start = dt.date(2018, 11, 18)

Is this the correct start date?

This results in OSI-430-a remote directory being set instead of OSI-430-b for date range (between 2020-1-1 and 2020-1-30) specified in original post, and the data not being found:

icenet/icenet/data/sic/osisaf.py

Lines 400 to 402 in 4608624

	chdir_path = self._ftp_osi450 \
	if el < osi430b_start else self._ftp_osi430b \
	if el < osi430a_start else self._ftp_osi430a

[bnubald]

Might be redundant issue, with v3.0.0 moving to a separate data downloader module (download-toolbox), and with changes in chdir_path.

Refer to #186

Downloader module (download-toolbox)