kafka-docker-playground

Playground for Kafka/Confluent Docker experimentations...

---

ℹ️ By default Confluent Platform version 5.5.0 is used, but you can test with another version (greater or equal to 5.0.0) simply by exporting TAG environment variable:

Example:

export TAG=5.3.2

---

Table of Contents

• 🔗 Kafka Connectors
  • ↘️ Source
  • ↗️ Sink
• ☁️ Confluent Cloud
  • Confluent Cloud Demo
  • 🔗 Kafka Connectors connected to Confluent Cloud
  • Other
• 🔄 Confluent Replicator and Mirror Maker 2
• 🔐 Environments
• 🎓 Kafka Tutorials
• Confluent Commercial
• CP-Ansible Playground
• 👾 Other Playgrounds
• 📚 Useful Resources

🔗 Kafka Connectors

Quick start examples from Confluent docs but in Docker version for ease of use.

↘️ Source

• Hadoop
  • HDFS 2
  • HDFS 3
• AWS
  • S3
  • Kinesis
  • SQS
    • using SASL_SSL
    • using SSL
  • CloudWatch Logs
  • AWS Redshift (using JDBC)
• Debezium
  • MySQL
  • PostgreSQL
  • MongoDB
  • SQL Server
• IBM MQ
• Solace
• ActiveMQ
• TIBCO EMS
• Syslog
• JDBC
  • MySQL
  • Oracle 11
  • Oracle 12
  • PostGreSQL
  • SQL Server
  • AWS Redshift
• MQTT
• JMS TIBCO EMS
• InfluxDB
• Splunk
• RabbitMQ
• Spool Dir
• GCP
  • Pub/Sub
  • GCS
  • Firebase
• Couchbase
• SFTP
• MongoDB
• Kudu
• SNMP
• ServiceNow
• Data Diode
• Azure
  • Blob Storage
  • Event Hubs
  • Service Bus

↗️ Sink

• Hadoop
  • HDFS 2
  • HDFS 3
• AWS
  • S3
  • Redshift
  • DynamoDB
  • Lambda
  • CloudWatch Metrics
• Elasticsearch
• HTTP
• GCP
  • BigQuery
  • Cloud Functions
  • GCS
    • using SASL_SSL
    • using SSL
    • using Kerberos GSSAPI
    • using LDAP Authorizer SASL/PLAIN
    • using RBAC environment SASL/PLAIN
  • Firebase
  • Spanner
• Solace
• Splunk
• TIBCO EMS
• IBM MQ
• MQTT
• InfluxDB
• Cassandra
• JDBC
  • MySQL
  • Oracle 11
  • Oracle 12
  • PostGreSQL
  • SQL Server
  • Vertica
  • Hive
• ActiveMQ
• OmniSci
• JMS
  • ActiveMQ
  • Solace
  • TIBCO EMS
• Azure
  • Blob Storage
  • Data Lake Gen1
  • Data Lake Gen2
  • SQL Data Warehouse
  • Search
• Neo4j
• Couchbase
• SFTP
• MongoDB
• HBase
• Redis
• Kudu
• Vertica
• ServiceNow
• Minio
• Prometheus
• Snowflake

☁️ Confluent Cloud

Confluent Cloud Demo

• How to connect your components to Confluent Cloud
• How to monitor your Confluent Cloud cluster
• How to restrict access
• etc...

🔗 Kafka Connectors connected to Confluent Cloud

• ServiceNow source
• ServiceNow sink
• MongoDB source
• Firebase

Other

• Using cp-ansible with Confluent Cloud
• Demo using dabz/ccloudexporter in order to pull Metrics API data from Confluent Cloud cluster and export it to Prometheus (Grafana dashboard is also available)
• .NET client (producer/consumer)
• Go client (producer/consumer)
• kafka-admin Managing topics and ACLs using matt-mangia/kafka-admin
• Confluent Replicator OnPrem to cloud and Cloud to Cloud examples
• Multi-Cluster Schema Registry with hybrid configuration (onprem/confluent cloud)

🔄 Confluent Replicator and Mirror Maker 2

Using Multi-Data-Center setup with US 🇺🇸 and EUROPE 🇪🇺 clusters.

• Using Confluent Replicator as connector
  • Using PLAINTEXT
  • Using SASL_PLAIN
  • Using Kerberos
• 👾 Using Confluent Replicator as executable
  • Using PLAINTEXT
  • Using SASL_PLAIN
  • Using Kerberos
• Using Mirror Maker 2
  • Using PLAINTEXT

🔐 Environments

Single cluster:

• PLAINTEXT: no security
• SASL_PLAIN: no SSL encryption, SASL/PLAIN authentication
• SASL/SCRAM no SSL encryption, SASL/SCRAM-SHA-256 authentication
• SASL_SSL: SSL encryption, SASL/PLAIN authentication
• 2WAY_SSL: SSL encryption, SSL authentication
• Kerberos: no SSL encryption, Kerberos GSSAPI authentication
• SSL_Kerberos SSL encryption, Kerberos GSSAPI authentication
• LDAP Authorizer with SASL/PLAIN no SSL encryption, SASL/PLAIN authentication, LDAP Authorizer for ACL authorization
• RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication

Multi-Data-Center setup:

• PLAINTEXT: no security
• SASL_PLAIN: no SSL encryption, SASL/PLAIN authentication
• Kerberos: no SSL encryption, Kerberos GSSAPI authentication

🎓 Kafka Tutorials

This is just the excellent examples in Kafka tutorial but in Docker version for ease of use.

• Apply a function to data
  • KSQL
    • Transform a stream of events
    • Flatten deeply nested events
    • Filter a stream of events
    • Rekey a stream with a value
    • Rekey a stream with a function
    • Convert a stream's serialization format
    • Split a stream of events into substreams
    • Merge many streams into one stream
    • Apply a custom, User-Defined Function
  • Kafka Streams
    • Transform a stream of events
    • Filter a stream of events
    • Convert a stream's serialization format
    • Split a stream of events into substreams
    • Merge many streams into one stream
    • Finding distinct events
• Aggregate data
  • KSQL
    • Count a stream of events
    • Sum a stream of events
    • Find the min/max in a stream of events
  • Kafka Streams
    • Count a stream of events
    • Sum a stream of events
    • Find the min/max in a stream of events
• Join data
  • KSQL
    • Join a stream and a table together
    • Join a stream and a stream together
    • Join a table and a table together
  • Kafka Streams
    • Join a stream and a table together
    • Join two tables with a foreign key
• Collect data over time
  • KSQL
    • Create tumbling windows
    • Create hopping windows
  • Kafka Streams
    • Create tumbling windows
• Connect data sources & sinks.
  • Kafka Streams
    • Add a key to a stream from a JDBC source

Confluent Commercial

• Control Center
  • Control Center in "Read-Only" mode
  • Configuring Control Center with LDAP authentication
• Tiered Storage
  • Tiered storage with AWS S3
  • Tiered storage with Minio (unsupported)
• Confluent Rebalancer
• JMS Client
• RBAC with SASL/PLAIN RBAC with no SSL encryption, SASL/PLAIN authentication
• Audit Logs

CP-Ansible Playground

Easily play with Confluent Platform Ansible playbooks by using Ubuntu based Docker images generated daily from this cp-ansible-playground repository

There is also a Confluent Cloud version available here

👾 Other Playgrounds

• Confluent Replicator [also with SASL_SSL and 2WAY_SSL]
• Testing Separate principals (connector.client.config.override.policy) for Source connector (SFTP source)
• Testing Separate principals (connector.client.config.override.policy) for Source connector (SFTP sink)
• How to write logs to files when using docker-compose
• Publish logs to kafka with Elastic Filebeat
• .NET basic producer

📚 Useful Resources

• A Kafka Story 📖: A step by step guide to use Kafka ecosystem (Kafka Connect, KSQL, Java Consumers/Producers, etc..) with Docker
• Kafka Boom Boom 💥: An attempt to break kafka
• Kafka Security playbook 🔒: demonstrates various security configurations with Docker
• MDC and single views 🌍: Multi-Data-Center setup using Confluent Replicator
• Kafka Platform Prometheus 📊: Simple demo of how to monitor Kafka Platform using Prometheus and Grafana.