Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Stable version numbering (ie 1.x.x) #230

Closed
charles-turner-1 opened this issue Oct 24, 2024 · 22 comments
Closed

Stable version numbering (ie 1.x.x) #230

charles-turner-1 opened this issue Oct 24, 2024 · 22 comments
Assignees
@rbeucher @marc-white @charles-turner-1
Labels
enhancement New feature or request

Comments

@charles-turner-1
Copy link
Collaborator

I think we're probably at a point where we don't want to be altering the behaviour of the catalog in a way that will confuse users and/or change workflows substantially.

To reflect this, should we consider updating our version number to a 1.x.x release to indicate that the library interface should be stable?
@charles-turner-1 charles-turner-1 added the enhancement New feature or request label Oct 24, 2024
@marc-white
Copy link
Collaborator

marc-white commented Oct 24, 2024 •
edited by charles-turner-1
Loading

I agree we're just about there, but I think there are a few things we need to get lined up before pulling that trigger. In particular:

@charles-turner-1
Copy link
Collaborator Author

charles-turner-1 commented Oct 24, 2024 •
edited by rbeucher
Loading

I think that would be a good target in terms of features. I think we could potentially also think about adding translators for the datasets with open data requests (as of today) - this process is generally pretty straightforward & fast.

@rbeucher
Copy link
Member

I agree. Let's do that and we can make a stable release.

@charles-turner-1
Copy link
Collaborator Author

I've edited the features list to add a two parter for Take2 - I'm going to leave #153 open until we've got an E2E build test on a catalog subset.

This was referenced Oct 29, 2024
Merged
Closed
@charles-turner-1
Copy link
Collaborator Author

@rbeucher @marc-white Did we decide last week to leave #188 out of this release?

@rbeucher
Copy link
Member

rbeucher commented Nov 3, 2024

Yes, pb for later :-)

@charles-turner-1 charles-turner-1 mentioned this issue Nov 5, 2024
Open
5 tasks
@charles-turner-1
Copy link
Collaborator Author

I agree we're just about there, but I think there are a few things we need to get lined up before pulling that trigger. In particular:

We've ticked these all off, so unless there are any more issues that we think are crucial to get fixed, I think we can get ready for a release chaps @rbeucher @marc-white.

@rbeucher
Copy link
Member

We have a new request on the Hive:
ROMS Ice Shelf Request.
https://forum.access-hive.org.au/t/roms-iceshelf/3931?u=rbeucher

@AndyHoggANU also asked me yesterday during TL how much effort this would involve.

Currently, they are still deciding where to store the data. My suggestion is that we proceed with a release as planned and then start incorporating the Intake-ESM data stores from NCI.

Once the location of the ROMS data is confirmed, we can explore adding it to the catalog.
@marc-white

@rbeucher
Copy link
Member

One thing that would be good to have is an updated version of the instructions on requesting data to be added to the catalog.

@marc-white
Copy link
Collaborator

Pretty sure that's already been done? I just built the documentation from main, it seems to be all updated w.r.t. the new catalog structure.

@rbeucher
Copy link
Member

We need to ensure that any references to or dependencies on hh5 are removed. With CLEX shutting down, hh5 will no longer be updated after December. Could you please verify that the documentation does not mention hh5?

@marc-white marc-white mentioned this issue Nov 27, 2024
Closed
@marc-white
Copy link
Collaborator

@rbeucher this is now being blocked by #273 , which is in turn being blocked by ACCESS-NRI/ACCESS-Analysis-Conda#123.

@rbeucher
Copy link
Member

Considering we are taking over analysis3.
Maybe we should use the mirrored copy I have in xp65.

@marc-white
Copy link
Collaborator

I tried to load up analysis3 via xp65 in an ARE session, but there seems to be an issue:

Script starting...
Waiting for Jupyter Notebook server to open port 62931...
TIMING - Starting wait at: Wed Nov 27 16:55:13 AEDT 2024
TIMING - Starting main script at: Wed Nov 27 16:55:13 AEDT 2024
Loading conda/analysis3-24.07
  Loading requirement: singularity
/g/data/xp65/public/apps/med_conda_scripts/analysis3-24.07.d/bin/jupyter
/g/data/xp65/public/apps/med_conda_scripts/analysis3-24.07.d/bin/jupyter
TIMING - Starting jupyter at: Wed Nov 27 16:55:14 AEDT 2024
+ ssh-agent -- jupyter lab --config=/home/120/mcw120/ondemand/data/sys/dashboard/batch_connect/sys/jupyter/ncigadi/output/70084362-b188-48ed-b450-656c0efaafe3/config.py --FileContentsManager.root_dir=/home/120/mcw120/.jupyter-root
[I 2024-11-27 16:55:24.067 ServerApp] dask_labextension | extension was successfully linked.
[I 2024-11-27 16:55:24.067 ServerApp] jupyter_lsp | extension was successfully linked.
[I 2024-11-27 16:55:24.067 ServerApp] jupyter_resource_usage | extension was successfully linked.
[I 2024-11-27 16:55:24.074 ServerApp] jupyter_server_mathjax | extension was successfully linked.
[I 2024-11-27 16:55:24.074 ServerApp] jupyter_server_proxy | extension was successfully linked.
[I 2024-11-27 16:55:24.079 ServerApp] jupyter_server_terminals | extension was successfully linked.
[W 2024-11-27 16:55:24.083 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-11-27 16:55:24.083 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-11-27 16:55:24.083 LabApp] 'port_retries' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-11-27 16:55:24.083 LabApp] 'password' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-11-27 16:55:24.083 LabApp] 'base_url' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-11-27 16:55:24.084 LabApp] 'base_url' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-11-27 16:55:24.084 LabApp] 'allow_origin' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-11-27 16:55:24.084 LabApp] 'notebook_dir' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-11-27 16:55:24.084 LabApp] 'disable_check_xsrf' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2024-11-27 16:55:24.086 ServerApp] ServerApp.password config is deprecated in 2.0. Use PasswordIdentityProvider.hashed_password.
[W 2024-11-27 16:55:24.086 ServerApp] notebook_dir is deprecated, use root_dir
[I 2024-11-27 16:55:24.086 ServerApp] jupyterlab | extension was successfully linked.
[I 2024-11-27 16:55:24.086 ServerApp] nbdime | extension was successfully linked.
[I 2024-11-27 16:55:24.092 ServerApp] notebook | extension was successfully linked.
[I 2024-11-27 16:55:24.633 ServerApp] notebook_shim | extension was successfully linked.
[I 2024-11-27 16:55:24.634 ServerApp] panel.io.jupyter_server_extension | extension was successfully linked.
Traceback (most recent call last):
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-24.07/bin/jupyter-lab", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-24.07/lib/python3.11/site-packages/jupyter_server/extension/application.py", line 616, in launch_instance
    serverapp = cls.initialize_server(argv=args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-24.07/lib/python3.11/site-packages/jupyter_server/extension/application.py", line 586, in initialize_server
    serverapp.initialize(
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-24.07/lib/python3.11/site-packages/traitlets/config/application.py", line 118, in inner
    return method(app, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-24.07/lib/python3.11/site-packages/jupyter_server/serverapp.py", line 2755, in initialize
    self.init_configurables()
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-24.07/lib/python3.11/site-packages/jupyter_server/serverapp.py", line 2058, in init_configurables
    self.kernel_spec_manager = self.kernel_spec_manager_class(
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-24.07/lib/python3.11/site-packages/nb_conda_kernels/manager.py", line 127, in __init__
    "nb_conda_kernels | enabled, %s kernels found.", len(self._conda_kspecs)
                                                         ^^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-24.07/lib/python3.11/site-packages/nb_conda_kernels/manager.py", line 389, in _conda_kspecs
    for name, info in self._all_specs().items():
                      ^^^^^^^^^^^^^^^^^
  File "/g/data/xp65/public/apps/med_conda/envs/analysis3-24.07/lib/python3.11/site-packages/nb_conda_kernels/manager.py", line 267, in _all_specs
    conda_prefix = self._conda_info['conda_prefix']
                   ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
KeyError: 'conda_prefix'
+ die 'ERROR: Jupyter Lab exited with non-zero code (1)'
+ MSG='ERROR: Jupyter Lab exited with non-zero code (1)'
+ '[' '!' -z 'ERROR: Jupyter Lab exited with non-zero code (1)' ']'
+ echo '<div class="error-msg">ERROR: Jupyter Lab exited with non-zero code (1)</div>'
+ kill -9 4009395
-/bin/bash: line 1: 4009395 Killed                  /local/spool/pbs/mom_priv/jobs/129602447.gadi-pbs.SC
++ date
+ echo 'TIMING - Finished jupyter at: Wed Nov 27 16:55:27 AEDT 2024'
TIMING - Finished jupyter at: Wed Nov 27 16:55:27 AEDT 2024

======================================================================================
                  Resource Usage on 2024-11-27 16:55:30:
   Job Id:             129602447.gadi-pbs
   Project:            tm70
   Exit Status:        137 (Linux Signal 9 SIGKILL Kill, unblockable)
   Service Units:      0.17
   NCPUs Requested:    28                     NCPUs Used: 28              
                                           CPU Time Used: 00:00:07        
   Memory Requested:   126.0GB               Memory Used: 535.82MB        
   Walltime requested: 01:00:00            Walltime Used: 00:00:17        
   JobFS requested:    100.0MB                JobFS used: 0B              
======================================================================================

@rbeucher
Copy link
Member

I just tried. It works fine for me

`Image

@marc-white
Copy link
Collaborator

That was my setup as well, still getting the same issue. Will come back to it again tomorrow.

@marc-white
Copy link
Collaborator

OK that environment is now suddenly working for me this morning, must have been a ghost in the machine. Cracking on with updating the notebooks...

@marc-white
Copy link
Collaborator

Oops, spoke to soon. Putting conda/analysis3 in as "Modules" starts up OK, but doesn't give me the right Jupyter kernel (just the system Python3 one). Putting in conda/analysis3-24.07 gives the same error I got yesterday:
Image
Image

@rbeucher
Copy link
Member

OK I am looking into it

@rbeucher
Copy link
Member

Hi @marc-white, @charles-turner-1,

I’ve successfully updated and fixed the conda environment. The new environment is named conda/analysis3-24.12.

Key differences from the current hh5 environment:

  • Upgraded Python: Now using Python >= 3.11.
  • Removed packages:
    • cdms2
    • intake-thredds
  • Cleaned up Jupyter Lab extensions: Removed some problematic or unnecessary extensions.

Please give it a try and let me know how it performs or if you encounter any issues.

@charles-turner-1
Copy link
Collaborator Author

Just noting the issue requiring openmpi to be loaded as a module separately when using conda/analysis3-24.12 here.

I don't think this should affect the release at all (although we might need to rebuild the environment after the release with the updated version of this package), so I think we're ready to publish the release now?

@marc-white @rbeucher

This was referenced Dec 3, 2024
Merged
Open
@charles-turner-1
Copy link
Collaborator Author

Marking this as completed.

@github-project-automation github-project-automation bot moved this from In Progress to Done in Model Evaluation & Diagnostics Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rbeucher rbeucher

@marc-white marc-white

@charles-turner-1 charles-turner-1

Labels
enhancement New feature or request
Projects
Model Evaluation & Diagnostics
Status: Done
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rbeucher @marc-white @charles-turner-1