Add Pipeline controller and Kafka JDBC driver

[ryannedolan]

Summary

• Added pipeline-controller.
• Added basic Kafka JDBC driver.
• Added quidem integration tests.

Details

The new pipeline-controller is significantly simpler than the existing subscription-controller, since it does not need to plan and deploy everything. So far, the pipeline controller is responsible for checking and reporting status of the pipeline. We need both implementations for now, but we plan to get rid of the subscription controller eventually.

The new catalog API is based on JDBC, which works well for most data sources. I've included a simple JDBC driver for Kafka, with just the minimal functionality to get integration tests working.

Our integration tests are now just unit tests tagged with integration, rather than full K8s pods. This should reduce our build complexity a bit, but we still need to wait for Kafka to come up before most integration tests will pass.

I have added a couple integration tests using quidem. I've included a text fixture for running quidem scripts.

Testing

We can create a Kafka changestream using our "demo" db:

0: Hoptimator> create or replace materialized view kafka.test1 as select * from ads.ad_clicks;

The resulting pipeline can be seen reaching Ready. state:

$ kubectl get pipelines
NAME            SQL                                                                                                                                                                   STATUS
ads-audience2   CREATE TABLE IF NOT EXISTS `PAGE_VIEWS` (`PAGE_URN` VARCHAR, `MEMBER_URN` VARCHAR) WITH ('database'='ads-database', 'connector'='demo', 'table'='PAGE_VIEWS');...     Ready.
ads-target2     CREATE TABLE IF NOT EXISTS `PAGE_VIEWS` (`PAGE_URN` VARCHAR, `MEMBER_URN` VARCHAR) WITH ('database'='ads-database', 'connector'='demo', 'table'='PAGE_VIEWS');...     Ready.
kafka-target    CREATE TABLE IF NOT EXISTS `PAGE_VIEWS` (`PAGE_URN` VARCHAR, `MEMBER_URN` VARCHAR) WITH ('database'='ads-database', 'connector'='demo', 'table'='PAGE_VIEWS');...     Ready.
kafka-test1     CREATE TABLE IF NOT EXISTS `AD_CLICKS` (`CAMPAIGN_URN` VARCHAR, `MEMBER_URN` VARCHAR) WITH ('database'='ads-database', 'connector'='demo', 'table'='AD_CLICKS');...   Ready.

The physical Kafka topic is created by the Strimzi topic operator:

$ kubectl get kafkatopics.kafka.strimzi.io -n kafka
NAME                    CLUSTER   PARTITIONS   REPLICATION FACTOR   READY
existing-topic          one       1            1                    True
existing-topic-1        one       1            1                    True
existing-topic-2        one       1            1                    True
kafka-database-test1    one       1            1                    True