Geonode 4.0.2 with geonode-importer does not upload

[fabiomonticelli23]

Expected Behavior

After changing the settings.py with the geonode.importer celery tasks it should use the new API and upload but it returns an error

Actual Behavior

After calling the API and passing a simple tiff file it return this error

Steps to Reproduce the Problem

1.installed Geonode 4.0.2 using docker compose
2. added in dockerfile after pip install requirements -> RUN pip install -e git+https://github.com/geosolutions-it/[email protected]#egg=geonode_importer
3.Added at the end of settings.py

INSTALLED_APPS += ('dynamic_models', 'importer', 'importer.handlers',)

CELERY_TASK_QUEUES += (
    Queue('importer.import_orchestrator', GEONODE_EXCHANGE, routing_key='importer.import_orchestrator'),
    Queue('importer.import_resource', GEONODE_EXCHANGE, routing_key='importer.import_resource', max_priority=8),
    Queue('importer.publish_resource', GEONODE_EXCHANGE, routing_key='importer.publish_resource', max_priority=8),
    Queue('importer.create_geonode_resource', GEONODE_EXCHANGE, routing_key='importer.create_geonode_resource', max_priority=8),
    Queue('importer.import_with_ogr2ogr', GEONODE_EXCHANGE, routing_key='importer.import_with_ogr2ogr', max_priority=10),
    Queue('importer.import_next_step', GEONODE_EXCHANGE, routing_key='importer.import_next_step', max_priority=3),
    Queue('importer.create_dynamic_structure', GEONODE_EXCHANGE, routing_key='importer.create_dynamic_structure', max_priority=10),
    Queue('importer.copy_geonode_resource', GEONODE_EXCHANGE, routing_key='importer.copy_geonode_resource', max_priority=0),
    Queue('importer.copy_dynamic_model', GEONODE_EXCHANGE, routing_key='importer.copy_dynamic_model'),
    Queue('importer.copy_geonode_data_table', GEONODE_EXCHANGE, routing_key='importer.copy_geonode_data_table'),
    Queue('importer.copy_raster_file', GEONODE_EXCHANGE, routing_key='importer.copy_raster_file'),
    Queue('importer.rollback', GEONODE_EXCHANGE, routing_key='importer.rollback'),
)

DATABASE_ROUTERS = ["importer.db_router.DatastoreRouter"]

SIZE_RESTRICTED_FILE_UPLOAD_ELEGIBLE_URL_NAMES += ('importer_upload',)

IMPORTER_HANDLERS = os.getenv('IMPORTER_HANDLERS', [
    'importer.handlers.gpkg.handler.GPKGFileHandler',
    'importer.handlers.geojson.handler.GeoJsonFileHandler',
    'importer.handlers.shapefile.handler.ShapeFileHandler',
    'importer.handlers.kml.handler.KMLFileHandler',
    'importer.handlers.csv.handler.CSVFileHandler',
    'importer.handlers.geotiff.handler.GeoTiffFileHandler'
])
FILE_UPLOAD_DIRECTORY_PERMISSIONS = 0o777
FILE_UPLOAD_PERMISSIONS = 0o777

Specifications

• GeoNode version: 4.0.2
• Installation type (vanilla, geonode-project):
• Installation method (manual, docker): docker
• Platform:
• Additional details:

[mattiagiupponi]

geonode-importer was introduced for GeoNode>=4.1.x if i'm not wrong ( #10474 ) so is not guarantee that works with Geonode 4.0.2

Aside this, the importer logs more info in the log when something happen. Please try to re-upload the tif and attach the celery.log file from the celery container. The log is located under /var/log/celery.log

Plus the dockerfile between the introduction of the two versions has changed as you can see from this:
GeoNode/geonode-project@7a3d31f#diff-dd2c0eb6ea5cfc6c4bd4eac30934e2d5746747af48fef6da689e85b752f39557R5

So i suppose that gdal-bin is missing in your dockerfile as dependency

[fabiomonticelli23]

this is the content of celery.log as you can see it doesnt find the db['ENGINE'] value

[2024-09-30 08:34:02,080: ERROR/ForkPoolWorker-10] Task importer.publish_resource[00e97085-7ed9-4a69-a0fb-b1ada77e3add] raised unexpected: PublishResourceException()
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/geonode/geoserver/helpers.py", line 1421, in create_geoserver_db_featurestore
    raise FailedRequestError
geoserver.catalog.FailedRequestError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/importer/celery_tasks.py", line 236, in publish_resource
    _publisher.publish_resources(data)
  File "/usr/local/lib/python3.10/site-packages/importer/publisher.py", line 63, in publish_resources
    self.get_or_create_store()
  File "/usr/local/lib/python3.10/site-packages/importer/publisher.py", line 93, in get_or_create_store
    self.store = create_geoserver_db_featurestore(
  File "/usr/local/lib/python3.10/site-packages/geonode/geoserver/helpers.py", line 1427, in create_geoserver_db_featurestore
    db_engine = "postgis" if "postgis" in db["ENGINE"] else db["ENGINE"]
KeyError: 'ENGINE'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 451, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/celery/app/trace.py", line 734, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/importer/celery_tasks.py", line 282, in publish_resource
    raise PublishResourceException(detail=error_handler(e, execution_id))
importer.api.exception.PublishResourceException: ENGINE. Request: 44cb6eae-51a6-4396-b051-79744bd5a3b6
[2024-09-30 08:34:02,080: INFO/MainProcess] Task importer.rollback[c792a367-e0fc-4716-a292-a0cb3c6b6ea0] received
[2024-09-30 08:34:02,081: DEBUG/MainProcess] TaskPool: Apply <function fast_trace_task at 0x7f6913d2b640> (args:('importer.rollback', 'c792a367-e0fc-4716-a292-a0cb3c6b6ea0', {'lang': 'py', 'task': 'importer.rollback', 'id': 'c792a367-e0fc-4716-a292-a0cb3c6b6ea0', 'shadow': None, 'eta': None, 'expires': None, 'group': None, 'group_index': None, 'retries': 0, 'timelimit': [None, None], 'root_id': '7f2484a1-113f-4104-88ef-6de239a09500', 'parent_id': 'a5b2cd4a-9547-40bd-90f4-06b8eb546cd6', 'argsrepr': "('44cb6eae-51a6-4396-b051-79744bd5a3b6', 'importer.rollback', 'test_local_machine_5_aoi', 'test_local_machine_5_aoi', 'importer.handlers.geojson.handler.GeoJsonFileHandler', 'rollback')", 'kwargsrepr': "{'kwargs': {'kwargs': {}, 'previous_action': 'import', 'error': 'ENGINE. Request: 44cb6eae-51a6-4396-b051-79744bd5a3b6'}}", 'origin': 'gen178@ff61406f96fd', 'ignore_result': True, 'properties': {'content_type': 'application/json', 'content_encoding': 'utf-8', 'application_headers': {'lang': 'py', 'task': 'importer.rollback', 'id': 'c792a367-e0fc-4716-a292-a0cb3c6b6ea0', 'shadow': None, 'eta': None, 'expires': None, 'group':... kwargs:{})
[2024-09-30 08:34:02,082: INFO/ForkPoolWorker-10] Calling rollback for execution_id 44cb6eae-51a6-4396-b051-79744bd5a3b6 in progress
[2024-09-30 08:34:02,088: INFO/ForkPoolWorker-12] Task importer.import_orchestrator[a5b2cd4a-9547-40bd-90f4-06b8eb546cd6] succeeded in 0.054907314000047336s: None
[2024-09-30 08:34:02,097: WARNING/ForkPoolWorker-10] Starting rollback for execid: 44cb6eae-51a6-4396-b051-79744bd5a3b6 resource published was: test_local_machine_5_aoi
[2024-09-30 08:34:02,097: INFO/ForkPoolWorker-10] Rollback publishing step in progress for execid: 44cb6eae-51a6-4396-b051-79744bd5a3b6 resource published was: test_local_machine_5_aoi
[2024-09-30 08:34:02,145: INFO/ForkPoolWorker-10] Rollback dynamic model & ogr2ogr step in progress for execid: 44cb6eae-51a6-4396-b051-79744bd5a3b6 resource published was: test_local_machine_5_aoi
[2024-09-30 08:34:02,145: INFO/ForkPoolWorker-10] Dynamic model does not exists, removing ogr2ogr table in progress
[2024-09-30 08:34:02,166: INFO/ForkPoolWorker-10] table "test_local_machine_5_aoi" does not exist

for gdal-bin i modified the dockerfile to install it

[mattiagiupponi]

it looks like an issue with the generation of the geonode_data store in geoserver, can you try to create it manually?