Catch API exception and serve helpful error response

[mrchrisadams]

This PR is intended to solve issue #459, where we were seeing a lot of invalid API requests causing 500 errors in Sentry.

This was triggered when people use the old directory to look up what information we have about a given provider, and it looks like someone starting hitting that API pretty hard - triggering hundreds of exceptions in the last few days, from the same client.

With this merged in, we instead serve a 406 response, making it clear to consumers how this endpoint is intended to be used.

#459

This also tidies up the code with Ruff, removing unused variables and so on

[tortila]

Thanks for delivering this fix @mrchrisadams! Logic-wise it looks good. However, I would like to suggest a different response code. 406 Not Acceptable is a good fit for when the response of a requested format cannot be produced. In this specific case, I think either 400 Bad Request or 422 Unprocessable Entity would be more precise.

I understand that this fix intends to keep Sentry usable for us while the "abusing" of the API is taking place. But I'm wondering: is there any additional action we should take, e.g. introducing rate-limiting?

[mrchrisadams]

Thanks Oliwia!

You're right - I wasn't sure about 406 either, so thanks for the pointer. I think I'll go with 400.

On the rate limiting front - I think we have some rate limiting in front of django with nginx. At least I wrote it up when we set it up a while back in 2019

https://rtl.chrisadams.me.uk/2019/12/how-to-rate-limit-punks-with-nginx/

We don't use any special rate limiting in Django or DRF right now, and it might be worth revisiting again.

Although to be fair, if our reverse proxy server is solving the rate limiting issue before it reaches django, it should allow us to keep the codebase smaller.

I've created an issue below for any further discussion

#462

But for now, I'll add the other commit now to update the status code so we can merge in.

It's a small change - you're happy for me to do so without pinging you again?