Pipeline fails silently with status code 429

[cristianvasquez]

While testing some mappings an API call in fetch_by_date_notice_from_ted fails with code 429.

The task continues running and notify a SUCCESS without notices processed.

Log:

[2024-08-20T08:53:32.227+0000] {log.py:232} WARNING - [2024-08-20 08:53:32] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 53, 32, 224323), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 53, 32, 224310), 'ended_at': datetime.datetime(2024, 8, 20, 8, 53, 32, 224311), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:53:37.243+0000] {log.py:232} WARNING - [2024-08-20 08:53:37] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 53, 37, 240654), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 53, 37, 240641), 'ended_at': datetime.datetime(2024, 8, 20, 8, 53, 37, 240642), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:53:51.383+0000] {log.py:232} WARNING - [2024-08-20 08:53:51] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 53, 51, 380316), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 53, 51, 380304), 'ended_at': datetime.datetime(2024, 8, 20, 8, 53, 51, 380305), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:53:55.545+0000] {log.py:232} WARNING - [2024-08-20 08:53:55] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 53, 55, 542699), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 53, 55, 542686), 'ended_at': datetime.datetime(2024, 8, 20, 8, 53, 55, 542687), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:54:13.033+0000] {log.py:232} WARNING - [2024-08-20 08:54:13] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 54, 13, 31074), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 54, 13, 31062), 'ended_at': datetime.datetime(2024, 8, 20, 8, 54, 13, 31063), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:54:15.208+0000] {log.py:232} WARNING - [2024-08-20 08:54:15] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 54, 15, 205460), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 54, 15, 205447), 'ended_at': datetime.datetime(2024, 8, 20, 8, 54, 15, 205448), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:54:22.516+0000] {log.py:232} WARNING - [2024-08-20 08:54:22] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 54, 22, 513762), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 54, 22, 513748), 'ended_at': datetime.datetime(2024, 8, 20, 8, 54, 22, 513749), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:54:26.064+0000] {log.py:232} WARNING - [2024-08-20 08:54:26] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 54, 26, 61876), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 54, 26, 61861), 'ended_at': datetime.datetime(2024, 8, 20, 8, 54, 26, 61862), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:54:28.331+0000] {log.py:232} WARNING - [2024-08-20 08:54:28] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 54, 28, 328814), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 54, 28, 328802), 'ended_at': datetime.datetime(2024, 8, 20, 8, 54, 28, 328802), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:54:37.667+0000] {log.py:232} WARNING - [2024-08-20 08:54:37] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 54, 37, 664447), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 54, 37, 664434), 'ended_at': datetime.datetime(2024, 8, 20, 8, 54, 37, 664435), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:54:46.550+0000] {log.py:232} WARNING - [2024-08-20 08:54:46] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 54, 46, 547826), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 54, 46, 547814), 'ended_at': datetime.datetime(2024, 8, 20, 8, 54, 46, 547814), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:54:48.281+0000] {log.py:232} WARNING - [2024-08-20 08:54:48] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 54, 48, 278828), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 54, 48, 278817), 'ended_at': datetime.datetime(2024, 8, 20, 8, 54, 48, 278818), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:54:51.740+0000] {log.py:232} WARNING - [2024-08-20 08:54:51] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 54, 51, 737993), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 54, 51, 737980), 'ended_at': datetime.datetime(2024, 8, 20, 8, 54, 51, 737981), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:54:55.569+0000] {log.py:232} WARNING - [2024-08-20 08:54:55] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 54, 55, 566578), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 54, 55, 566566), 'ended_at': datetime.datetime(2024, 8, 20, 8, 54, 55, 566567), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:55:05.861+0000] {log.py:232} WARNING - [2024-08-20 08:55:05] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 55, 5, 858766), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 55, 5, 858749), 'ended_at': datetime.datetime(2024, 8, 20, 8, 55, 5, 858750), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:55:17.768+0000] {log.py:232} WARNING - [2024-08-20 08:55:17] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 55, 17, 765549), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 55, 17, 765536), 'ended_at': datetime.datetime(2024, 8, 20, 8, 55, 17, 765537), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:55:20.615+0000] {log.py:232} WARNING - [2024-08-20 08:55:20] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 55, 20, 612918), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 55, 20, 612906), 'ended_at': datetime.datetime(2024, 8, 20, 8, 55, 20, 612907), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:55:32.786+0000] {log.py:232} WARNING - [2024-08-20 08:55:32] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 55, 32, 783980), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 55, 32, 783968), 'ended_at': datetime.datetime(2024, 8, 20, 8, 55, 32, 783969), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:55:34.711+0000] {log.py:232} WARNING - [2024-08-20 08:55:34] - ConsoleHandler:ROOT - WARNING - EventMessage :: {'message': 'Request returned status code 429, retrying in 1 seconds!', 'created_at': datetime.datetime(2024, 8, 20, 8, 55, 34, 708913), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 30, 'caller_name': 'log_warning', 'started_at': datetime.datetime(2024, 8, 20, 8, 55, 34, 708890), 'ended_at': datetime.datetime(2024, 8, 20, 8, 55, 34, 708902), 'duration': None, 'metadata': None, 'kwargs': None}
[2024-08-20T08:56:27.430+0000] {log.py:232} WARNING - [2024-08-20 08:56:27] - ConsoleHandler:DAG - DEBUG - TechnicalEventMessage :: {'message': 'fetch_notice_from_ted', 'created_at': datetime.datetime(2024, 8, 20, 8, 40, 22, 204916), 'year': 2024, 'month': 8, 'day': 20, 'severity_level': 10, 'caller_name': 'fetch_by_date_notice_from_ted', 'started_at': datetime.datetime(2024, 8, 20, 8, 40, 22, 757516), 'ended_at': datetime.datetime(2024, 8, 20, 8, 56, 27, 430549), 'duration': 964.673033, 'metadata': {'process_type': 'DAG', 'process_name': 'fetch_notices_by_date', 'process_id': None, 'process_context': None}, 'kwargs': None}
[2024-08-20T08:56:27.450+0000] {python.py:177} INFO - Done. Returned value was: None
[2024-08-20T08:56:27.462+0000] {taskinstance.py:1318} INFO - Marking task as SUCCESS. dag_id=fetch_notices_by_date, task_id=fetch_by_date_notice_from_ted, execution_date=20240820T083900, start_date=20240820T084022, end_date=20240820T085627
[2024-08-20T08:56:27.529+0000] {local_task_job.py:208} INFO - Task exited with return code 0
[2024-08-20T08:56:27.559+0000] {taskinstance.py:2578} INFO - 1 downstream tasks scheduled from follow-on schedule check