must has ad attribute odd_metadata

[Maegor]

Hello,

I have set up several MySQL (5.x) databases that are on the same server. The collector collects data from some of them but not from others.

In the log I see messages like this:

sep 28 12:36:38 yyy-xxx-server-gv-1 bash[12663]: 2023-09-28 12:36:38.065 | ERROR | odd_collector_sdk.utils.metadata:extract_metadata:62 - Couldn't extract metadata, Entity Column(table_catalog='def', table_schema='bd_real', table_name='dx_table', name='instrument', type='varchar', is_nullable='NO', default=None, comment='', is_primary_key=False, is_sort_key=False, metadata={'ordinal_position': 1, 'character_maximum_length': 25, 'character_octet_length': 25, 'numeric_precision': None, 'numeric_scale': None, 'datetime_precision': None, 'character_set_name': 'latin1', 'collation_name': 'latin1_swedish_ci', 'column_type': 'varchar(25)', 'column_key': 'PRI', 'extra': '', 'privileges': 'select', 'generation_expression': ''}) must has ad attribute odd_metadata

What does "must has ad attribute odd_metadata" mean?

I'm not sure what version of odd-collector I have. I deployed about 30-40 days ago, how can I check my version?

[Vixtir]

Hi, @Maegor 👋🏻
Due to bug in mapping layer you saw that error, we have fixed that issue with commit, and changed log level to warning.
You just need to pull the latest image and restart collector.

The collector collects data from some of them but not from others.

Was metadata loaded only for one database?

[Maegor]

Hi @Vixtir ,

I have just upgraded to the latest version. The error message has changed, now it shows the error: "Could not ingest data. Reason: 500, message='Internal Server Error', url=URL('http://xxxxxxx/ingestion/entities')"

The BD Server is MySQL5. Other schemas from the same server are read correctly. Could it be a permissions problem? I have tried launching SQL queries to read metadata and there seems to be no problems.

The complete log is
sep 29 06:53:12 yyyyyy-gv-1 systemd[1]: Started ODD Collector. sep 29 06:53:13 yyyyyy-gv-1 bash[6402]: 2023-09-29 06:53:13.472 | WARNING | odd_collector_sdk.utils.print_version:print_versions:16 - Could not get version for: odd_collector. Reason: No package sep 29 06:53:13 yyyyyy-gv-1 bash[6402]: 2023-09-29 06:53:13.474 | INFO | odd_collector_sdk.domain.collector_config:load_config:37 - Start reading config sep 29 06:53:13 yyyyyy-gv-1 bash[6402]: 2023-09-29 06:53:13.624 | SUCCESS | odd_collector_sdk.load_adapter:load_adapters:60 - Loaded 1 adapters! sep 29 06:53:13 yyyyyy-gv-1 bash[6402]: 2023-09-29 06:53:13.637 | INFO | odd_collector_sdk.collector:run:131 - Config interval interval=60 sep 29 06:53:13 yyyyyy-gv-1 bash[6402]: 2023-09-29 06:53:13.657 | INFO | odd_collector_sdk.job:log_execution:22 - [BDRReal] collecting metadata started. sep 29 06:53:14 yyyyyy-gv-1 bash[6402]: 2023-09-29 06:53:14.254 | ERROR | odd_collector_sdk.job:log_execution:26 - [BDRReal] failed. sep 29 06:53:14 yyyyyy-gv-1 bash[6402]: Could not ingest data. Reason: 500, message='Internal Server Error', url=URL('http://xxxxxxx/ingestion/entities')

Thank you!"!

[Vixtir]

Hi, @Maegor !
Collector gathered data but sent some inconsistent or empty ( can be a permissions problem) data.
Could please share logs for an odd-platform? It will help me to clarify the reason

[Maegor]

Hi @Vixtir
I attach the full log at the end of the message. I see this trace in the log:

docker-odd-platform-1 | 2023-09-29T09:15:57.629Z ERROR 1 --- [actor-tcp-nio-1] o.o.o.c.exception.ControllerAdvice : Internal server error docker-odd-platform-1 | docker-odd-platform-1 | docker-odd-platform-1 | java.lang.IllegalStateException: Duplicate key //mysql/host/connection.my.db/databases/bd_real/tables/EMT/columns/TICKET (attempted merging values DatasetFieldIngestionDto[field=DatasetFieldPojo (null, TICKET, //mysql/host/connection.my.db/databases/bd_real/tables/EMT/columns/TICKET, null, 0, {}, {"is_nullable":false,"logical_type":"int","type":"TYPE_INTEGER"}, null, null, , null, false, false, None, null), labels=[], enumValues=null, metadata={ordinal_position=1, numeric_precision=10, numeric_scale=0, column_type=int(11), column_key=PRI, extra=, privileges=select, generation_expression=}] and DatasetFieldIngestionDto[field=DatasetFieldPojo (null, TICKET, //mysql/host/connection.my.db/databases/bd_real/tables/EMT/columns/TICKET, null, 0, {}, {"is_nullable":false,"logical_type":"int","type":"TYPE_INTEGER"}, null, null, , null, false, false, None, null), labels=[], enumValues=null, metadata={ordinal_position=1, numeric_precision=10, numeric_scale=0, column_type=int(11), column_key=PRI, extra=, privileges=select, generation_expression=}]) docker-odd-platform-1 | at java.base/java.util.stream.Collectors.duplicateKeyException(Collectors.java:135) ~[na:na] docker-odd-platform-1 | Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: docker-odd-platform-1 | Error has been observed at the following site(s): docker-odd-platform-1 | *__checkpoint ⇢ Handler org.opendatadiscovery.oddplatform.controller.IngestionController#postDataEntityList(Mono, ServerWebExchange) [DispatcherHandler]

odd_platform_log.txt

Thank you very much!!

[Vixtir]

Reproduced, will fix asap!

[Vixtir]

Hi @Maegor! Could you try the latest release!

[Maegor]

Hello @Vixtir ,

with the latest version (0.1.43) the problem has been fixed 👍

Thank you very much!