automate spam detection for Job, Event, Codebase and MemberProfile using an external LLM service

[asuworks]

This PR attempts to automate the spam detection process for Job, Event, Codebase and MemberProfile objects using an external LLM service.

LLM Spam Detection Process

1. A SpamModeration record with status SCHEDULED_FOR_CHECK is stored on every Job, Event, Codebase, submission and User (SpamModeration object is attached to the associated MemberProfile) creation.
2. A decoupled external service asuworks/comses.spamcheck queries for these SpamModeration objects (api/spam/get-latest-batch/), analyzes them for spam and submits a spam report to api/spam/update for each one of them.
3. The handler for the api/spam/update on the CoMSES side updates the corresponding SpamModeration object according to the LLM report from the external service.

Starting the LLM Spam Detection Process

The external service asuworks/comses.spamcheck is deployed on an existing JetStream2 instance which is unshelved before the spam check workflow is triggered and shelved automatically after it is done by the following management command:

./manage.py  curator_llm_spam_check

# Following flags are available for troubleshooting purposes:
--skip-changing-instance-state # will skip changing the status of the JetStream2 instance
--skip-shelving-when-done # will start the JetStream2 instance, execute the `CheckSpamWorkflow`, but not shelve it

Environment & Secrets

Following environment variables must be set:

LLM_SPAM_CHECK_API_URL=http://<JetStream2 instance IP>:8001
LLM_SPAM_CHECK_JETSTREAM_SERVER_ID=<JetStream2 instance ID>

JetStream2 Credentials

can be found here: https://js2.jetstream-cloud.org/identity/application_credentials/

secrets/llm_spam_check_jetstream_os_application_credential_secret
secrets/llm_spam_check_jetstream_os_application_credential_id

X-API-Key header for the API

Access to api/spam/update and api/spam/get-latest-batch routes is protected by the X-API-Key header verification.
The key should be set in secrets/llm_spam_check_api_key

ALLOWED_HOSTS

The IP of the JetStream2 instance must be added to Django's ALLOWED_HOSTS

[asuworks]

we would need to add the jetstream instance ip to ALLOWED_HOSTS

[sgfost]

just missing the migration for SpamModeration.status, which I think should turn anything with "unreviewed" status into "spam_likely" status.

I'll read through it again but it seemed all good on a first pass, besides having a way to kick off the process

[asuworks]

Regarding the way to start the process from CoMSES side:

1. start the instance with openstack-cli
2. poll the service until HEALTHY
3. keep polling the service untill the workflow is COMPLETED
4. shutdown instance with openstack-cli

something like this?

[sgfost]

@asuworks I just remembered there was some additional cleanup I wanted to do eventually with the spam stuff. This might be a good place to get that done if you are up for it. comses/planning#249. Namely the second point (refactoring the serializer mixin to actually be just a mixin)