Rest authorizer cannot fetch projects sometimes

[Bdegraaf1234]

On some occasions the rest-authorizer frontend cannot get the projects from the backend

curl --location 'https://radar-k3s-test.thehyve.net/rest-sources/backend/projects' --header 'Authorization: Bearer {mytoken}

Responds with

503 timeout

At the same time, it seems to be able to get the source clients

curl --location 'https://radar-k3s-test.thehyve.net/rest-sources/backend/source-clients' --header 'Authorization: Bearer {mytoken}

The problem occurs and resolves at random, but can be fixed by restarting the RSA pods. One observation was that the managementportal complained about a missing refresh token

[Bdegraaf1234]

sample logs from the RSA backend. Note the successful source-clients request in the middle. This may be due to cached results.

token requests also fail, likely due to the same reason

[2024-03-13 08:54:37,959] ERROR - [503] GET users/3/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:54:37,960] HTTP  - [503] GET users/3/token -- <application/json;charset=utf-8> 
[2024-03-13 08:54:37,966] AUTH  - GET users/3/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:54:44,944] ERROR - [503] GET users/38/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:54:44,944] HTTP  - [503] GET users/38/token -- <application/json;charset=utf-8> 
[2024-03-13 08:54:44,950] AUTH  - GET users/38/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:54:47,098] ERROR - [503] GET users/9/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:54:47,098] HTTP  - [503] GET users/9/token -- <application/json;charset=utf-8> 
[2024-03-13 08:54:47,105] AUTH  - GET users/9/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:54:50,055] ERROR - [503] GET users/9/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:54:50,055] HTTP  - [503] GET users/9/token -- <application/json;charset=utf-8> 
[2024-03-13 08:54:50,062] AUTH  - GET users/9/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:54:50,067] ERROR - [503] GET users/4/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:54:50,068] HTTP  - [503] GET users/4/token -- <application/json;charset=utf-8> 
[2024-03-13 08:54:50,073] AUTH  - GET users/8/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:54:50,297] ERROR - [503] GET users/9/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:54:50,297] HTTP  - [503] GET users/9/token -- <application/json;charset=utf-8> 
[2024-03-13 08:54:50,302] AUTH  - GET users/9/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:54:51,190] ERROR - [503] GET users/27/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:54:51,191] HTTP  - [503] GET users/27/token -- <application/json;charset=utf-8> 
[2024-03-13 08:54:51,196] AUTH  - GET users/76/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:54:52,445] ERROR - [503] GET users/8/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:54:52,445] HTTP  - [503] GET users/8/token -- <application/json;charset=utf-8> 
[2024-03-13 08:54:52,452] AUTH  - GET users/4/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:54:54,453] ERROR - [503] GET users/8/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:54:54,454] HTTP  - [503] GET users/8/token -- <application/json;charset=utf-8> 
[2024-03-13 08:54:54,459] AUTH  - GET users/4/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:54:55,900] ERROR - [503] GET users/9/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:54:55,900] HTTP  - [503] GET users/9/token -- <application/json;charset=utf-8> 
[2024-03-13 08:54:55,905] AUTH  - GET users/9/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:54:56,168] ERROR - [503] GET users/8/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:54:56,168] HTTP  - [503] GET users/8/token -- <application/json;charset=utf-8> 
[2024-03-13 08:54:56,178] AUTH  - GET users/4/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:55:07,332] ERROR - [503] GET projects - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:55:07,333] HTTP  - [503] GET projects -- <application/json;charset=utf-8> 
[2024-03-13 08:55:07,967] ERROR - [503] GET users/3/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:55:07,967] HTTP  - [503] GET users/3/token -- <application/json;charset=utf-8> 
[2024-03-13 08:55:07,975] AUTH  - GET users/3/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:55:09,109] AUTH  - GET projects - @admin - GRANTED PROJECT.READ
[2024-03-13 08:55:09,114] AUTH  - GET source-clients - @admin - GRANTED SOURCETYPE.READ
[2024-03-13 08:55:09,116] HTTP  - [200] GET source-clients -- <application/json> 
[2024-03-13 08:55:14,951] ERROR - [503] GET users/38/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:55:14,951] HTTP  - [503] GET users/38/token -- <application/json;charset=utf-8> 
[2024-03-13 08:55:14,957] AUTH  - GET users/38/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:55:17,105] ERROR - [503] GET users/9/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:55:17,105] HTTP  - [503] GET users/9/token -- <application/json;charset=utf-8> 
[2024-03-13 08:55:17,111] AUTH  - GET users/9/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:55:20,062] ERROR - [503] GET users/9/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:55:20,062] HTTP  - [503] GET users/9/token -- <application/json;charset=utf-8> 
[2024-03-13 08:55:20,069] AUTH  - GET users/9/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:55:20,073] ERROR - [503] GET users/8/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)
[2024-03-13 08:55:20,073] HTTP  - [503] GET users/8/token -- <application/json;charset=utf-8> 
[2024-03-13 08:55:20,078] AUTH  - GET users/4/token - radar_fitbit_connector - GRANTED MEASUREMENT.CREATE
[2024-03-13 08:55:20,302] ERROR - [503] GET users/9/token - timeout: no message (HttpApplicationExceptionMapper.kt:30)