Express + HTTP Types + Kafka + Meeshkan

An example express server using Meeshkan express middleware to record HTTP traffic to Kafka.

The full tutorial associated with this repository is available on the Meeshkan blog: Building a real-time HTTP traffic stream with Apache Kafka.

Prerequisites

1. Node.js >= 8.0 and optionally yarn
2. Either Docker or Kafka installation
3. Optional: Python 3.6+ and pip if building an OpenAPI specification with our HTTP Mocking Toolkit (HMT)

Instructions

Setting up the server

First, clone the repository and navigate inside it:

git clone https://github.com/meeshkan/meeshkan-express-kafka-demo.git
cd meeshkan-express-kafka-demo

Install Express server dependencies:

# If you're using npm
$ npm install

# If you're using yarn
$ yarn

Preparing Kafka

If you have Docker, start a local Kafka cluster with one Zookeeper and Kafka node by using Docker Compose:

$ docker-compose up -d

Note: You can find the configuration for this in our docker-compose.yml file.

Alternatively, if you have Kafka installed, start Zookeeper and Kafka as instructed in Kafka documentation.

Create the destination topic:

# If you're using Docker
$ bin/kafka-topics.sh --bootstrap-server localhost:9092 --topic http_recordings --create --partitions 3 --replication-factor 1

# If you have Kafka installed
$ docker exec kafka1 kafka-topics --bootstrap-server localhost:9092 --topic http_recordings --create --partitions 3 --replication-factor 1

Starting the server

Start the Express server:

# If you're using npm
$ npm run start

# If you're using yarn
$ yarn start

Working with Kafka

Start console consumer to read messages from Kafka:

# If you're using Docker
$ docker exec kafka1 kafka-console-consumer --bootstrap-server localhost:9092 --topic http_recordings --from-beginning

# If you have Kafka installed
$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic http_recordings --from-beginning

Alternatively, if you use kafkacat:

$ kafkacat -b localhost:9092 -t http_recordings -C

Start making calls to the server via curl and see traffic being recorded in Kafka:

# Create a user with POST /users
$ curl -X POST -d '{"name": "Kimmo", "email": "[email protected]" }' -H "Content-Type: application/json" http://localhost:3000/users

# Get user with GET /users/{id}
$ curl http://localhost:3000/users/5720862f-9534-4d97-8afe-c07f38a728b7

Note: The recorded traffic logs are in the HTTP Types format.

Creating an OpenAPI specification

To create an OpenAPI specification from recordings, install hmt:

$ pip install hmt

Then, run the following command:

$ hmt build --source kafka -o my_spec

Note: This instructs hmt to write the OpenAPI specification to the my_spec/ directory.

Shutting down the server and Kafka cluster

Finally, stop the Express server and close down the Kafka cluster:

$ docker-compose down

Miscellaneous

Using Kafka CLI locally

Create the topic:

$ kafka-topics.sh --bootstrap-server localhost:9092 --topic http_recordings --create --partitions 3 --replication-factor 1

List topics:

$ kafka-topics.sh --bootstrap-server localhost:9092 --list

Delete the topic:

$ kafka-topics.sh --bootstrap-server localhost:9092 --topic http_recordings --delete

Print consumer groups:

$ kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --all-groups