Opmon protobuf microservice

[mroda88]

This PR creates a new microservice that uses the new protobuf opmon stream and puts it into influxdb. Eventually this will replace the old stream, but for now this only deploys the new one.

[jcpunk]

I'd be tempted to run the python code through the black autoformatter.

[mroda88]

Format with black is done

[jcfreeman2]

Basic code structure in opmon-protobuf-dbwriter/ looks correct. There's a trailing whitespace in the string passed to opmon-protobuf-dbwriter/entrypoint.sh's ensure_required_variables, but it shouldn't matter.

[jcfreeman2]

If I want to actually try running this microservice, what are some reasonable values to set OPMON_DBWRITER_KAFKA_GROUP , etc., to? And are these documented anywhere?

Possibly relevant here is Issue #104

[mroda88]

Hi John, so good values for this micro service are set in the yaml file in the same directory. Those are working values based on previous tests with opmon and influx.

But running it effectively outside the kubernetes cluster is impossible. The influx server is configured in such a way that it has to be contacted from within the cluster. That's why for now this is the initial PR only and the test will be done once beam is off next week and we can tamper with the cluster freely.

But if you want to, we can sit down tomorrow and see what we can do.

[mroda88]

ok, I think I addressed all the comments. Now if you do something like

docker run --rm -e MICROSERVICE=opmon-protobuf-dbwriter -e OPMON_DBWRITER_KAFKA_BOOTSTRAP_SERVER=dummyval -e OPMON_DBWRITER_KAFKA_GROUP=dummyval -e OPMON_DBWRITER_SUBSCRIBER_TIMEOUT_MS=10000  -e OPMON_DBWRITER_TOPIC=dummyval -e OPMON_DBWRITER_INFLUX_HOST=dummyval -e OPMON_DBWRITER_INFLUX_PORT=12345 -e OPMON_DBWRITER_TABLE=dummyval -e OPMON_DBWRITER_BATCH_SIZE_MS=1000 ghcr.io/dune-daq/microservices:mroda-opmon_protobuf

You get the correct failures due to connecting issues to the database.

[jcpunk]

Cool! Do you have a snippet I could use to create the missing database?

[mroda88]

Ah, but that's the nice thing. This micro service will create the database if not already available. So we don't have to do anything. When we will test, we just deploy this and that's it. 😉

[jcpunk]

Shouldn't this be opmon?

[mroda88]

Indeed!