disable our netapp S3 objectstore temporarily in the objectstore conf

[sanjaysrikakulam]

We seem to have hit the following error for the upload jobs since last night at 12:00. Removing both the S3 stores from the object store conf locally, syncing the conf, and restarting the handlers seem to have fixed the error for now, and the upload jobs are running without any errors.

Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: galaxy.tool_util.output_checker INFO 2024-07-02 08:13:19,462 [pN:handler_sn06_0,p:2950044,tN:CondorRunner.work_thread-1] Job error detected, failing job. Reasons are [{'type': 'exit_code', 'desc': 'Fatal error: Exit code 1 ()', 'exit_code': 1, 'code_desc': '', 'error_level': 3}]
Jul 02 08:13:19 sn06.galaxyproject.eu python[3952921]: galaxy.jobs.runners.pulsar DEBUG 2024-07-02 08:13:19,474 [pN:handler_sn06_5,p:3952921,tN:PulsarJobRunner.monitor_thread] (71326023) Received status update: running
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: galaxy.jobs ERROR 2024-07-02 08:13:19,475 [pN:handler_sn06_0,p:2950044,tN:CondorRunner.work_thread-1] problem importing job outputs. stdout [] stderr [Traceback (most recent call last):
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/tools/data_fetch.py", line 20, in <module>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     from galaxy.datatypes.registry import Registry
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/datatypes/registry.py", line 29, in <module>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     from . import (
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/datatypes/binary.py", line 43, in <module>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     from galaxy.datatypes import metadata
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/datatypes/metadata.py", line 8, in <module>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     from galaxy.model.metadata import (
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/model/__init__.py", line 143, in <module>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     from galaxy.objectstore import ObjectStore
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/objectstore/__init__.py", line 1395, in <module>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     class ConcreteObjectStoreModel(BaseModel):
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 205, in __new__
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     complete_model_class(
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 534, in complete_model_class
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = cls.__get_pydantic_core_schema__(cls, handler)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/main.py", line 642, in __get_pydantic_core_schema__
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     return handler(source)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:            ^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 83, in __call__
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = self._handler(source_type)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 512, in generate_schema
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = self._generate_schema_inner(obj)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 784, in _generate_schema_inner
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     return self._model_schema(obj)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:            ^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 591, in _model_schema
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 591, in <dictcomp>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 947, in _generate_md_field_schema
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     common_field = self._common_field_schema(name, field_info, decorators)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1134, in _common_field_schema
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = self._apply_annotations(
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1890, in _apply_annotations
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = get_inner_schema(source_type)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 83, in __call__
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = self._handler(source_type)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1972, in new_handler
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = metadata_get_schema(source, get_inner_schema)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_std_types_schema.py", line 327, in __get_pydantic_core_schema__
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     constrained_schema = core_schema.list_schema(items_schema, **metadata)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: TypeError: list_schema() got an unexpected keyword argument 'fail_fast'
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: Traceback (most recent call last):
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/data/jwd05e/main/071/343/71343472/metadata/set.py", line 5, in <module>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     from galaxy_ext.metadata.set_metadata import set_metadata; set_metadata()
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy_ext/metadata/set_metadata.py", line 20, in <module>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     from galaxy.metadata.set_metadata import set_metadata
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/metadata/__init__.py", line 9, in <module>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     import galaxy.model
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/model/__init__.py", line 143, in <module>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     from galaxy.objectstore import ObjectStore
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/objectstore/__init__.py", line 1395, in <module>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     class ConcreteObjectStoreModel(BaseModel):
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 205, in __new__
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     complete_model_class(
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 534, in complete_model_class
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = cls.__get_pydantic_core_schema__(cls, handler)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/main.py", line 642, in __get_pydantic_core_schema__
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     return handler(source)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:            ^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 83, in __call__
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = self._handler(source_type)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 512, in generate_schema
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = self._generate_schema_inner(obj)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 784, in _generate_schema_inner
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     return self._model_schema(obj)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:            ^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 591, in _model_schema
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 591, in <dictcomp>
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     {k: self._generate_md_field_schema(k, v, decorators) for k, v in fields.items()},
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 947, in _generate_md_field_schema
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     common_field = self._common_field_schema(name, field_info, decorators)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1134, in _common_field_schema
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = self._apply_annotations(
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1890, in _apply_annotations
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = get_inner_schema(source_type)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_schema_generation_shared.py", line 83, in __call__
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = self._handler(source_type)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 1972, in new_handler
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     schema = metadata_get_schema(source, get_inner_schema)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/venv/lib/python3.11/site-packages/pydantic/_internal/_std_types_schema.py", line 327, in __get_pydantic_core_schema__
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     constrained_schema = core_schema.list_schema(items_schema, **metadata)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: TypeError: list_schema() got an unexpected keyword argument 'fail_fast'
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: ]
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: Traceback (most recent call last):
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/jobs/__init__.py", line 1916, in finish
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     import_model_store.perform_import(history=job.history, job=job)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/model/store/__init__.py", line 404, in perform_import
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     datasets_attrs = self.datasets_properties()
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/model/store/__init__.py", line 1540, in datasets_properties
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     datasets_attrs = load(open(datasets_attrs_file_name))
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: FileNotFoundError: [Errno 2] No such file or directory: '/data/jwd05e/main/071/343/71343472/metadata/outputs_populated/datasets_attrs.txt'
Jul 02 08:13:19 sn06.galaxyproject.eu python[3952921]: galaxy.jobs.runners.pulsar DEBUG 2024-07-02 08:13:19,478 [pN:handler_sn06_5,p:3952921,tN:PulsarJobRunner.monitor_thread] (71339080) Received status update: running
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: galaxy.jobs.runners ERROR 2024-07-02 08:13:19,484 [pN:handler_sn06_0,p:2950044,tN:CondorRunner.work_thread-1] (71343472/51455731) Job wrapper finish method failed
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: Traceback (most recent call last):
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/jobs/runners/__init__.py", line 676, in _finish_or_resubmit_job
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     job_wrapper.finish(
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/jobs/__init__.py", line 1916, in finish
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     import_model_store.perform_import(history=job.history, job=job)
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/model/store/__init__.py", line 404, in perform_import
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     datasets_attrs = self.datasets_properties()
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:   File "/opt/galaxy/server/lib/galaxy/model/store/__init__.py", line 1540, in datasets_properties
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:     datasets_attrs = load(open(datasets_attrs_file_name))
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: FileNotFoundError: [Errno 2] No such file or directory: '/data/jwd05e/main/071/343/71343472/metadata/outputs_populated/datasets_attrs.txt'
Jul 02 08:13:19 sn06.galaxyproject.eu python[2950044]: galaxy.jobs DEBUG 2024-07-02 08:13:19,499 [pN:handler_sn06_0,p:2950044,tN:CondorRunner.work_thread-1] fail(): Moved /data/jwd05e/main/071/343/71343472/outputs/dataset_5a2aecee-0e4a-4325-963b-c1bfb3734aa1.dat to /data/dnb10/galaxy_db/files/5/a/2/dataset_5a2aecee-0e4a-4325-963b-c1bfb3734aa1.dat

I hope there will not be any parse error when this gets deployed. Please check carefully.

[mira-miracoli]

could line 248 (nested comment) lead to an issue when commenting out?
(not an expert in xml)

[mira-miracoli]

https://stackoverflow.com/questions/1324821/nested-comments-in-xml

[sanjaysrikakulam]

Thank you @mira-miracoli!