Extend Kafka Topic Modeling with Diffusion

Introduction to Diffusion Real-Time Event Stream through a simple application using Diffusion Cloud and Apache Kafka.

A set of simple projects, illustrating production and consumption of foreign exchange (fx) data streams to and from Kafka clusters and Diffusion Cloud instance via the use of our Kafka Adapter.

These JavaScript code examples will help you publish fx data on real-time from a front end app to a Kafka cluster, consume from it and transform data on-the-fly via our powerful Topic Views feature. You can also use other programming languages from our SDKs, including iOS, Android, C, .NET, and more.

Lesson 1: Publish event streams to Kafka cluster

diffusion-kafka-app-L1 introduces the concept of our Kafka Adapter and its configuration, to consume fx topic data from your Diffusion server and publish it to Kafka cluster in specific Kafka topic.

Lesson 2: Fine-grained distribution of Kafka event firehose with Topic Views

diffusion-kafka-app-L2 introduces the concept of Topic Views, a dynamic mechanism to map part of a server's Topic Tree (data structure) to another. This enables real-time data transformation before sending it to subscribed clients as well as to create dynamic data models based on on-the-fly data (eg: Kafka firehose data).

Lesson 3: Security with Topic-level access control

diffusion-kafka-app-L3 introduces the concept of Security and topic path permissions for fine-grained security management of your data structure. It also shows how to Subscribe to fx data using Diffusion Topic Views in order to consume what you need, not all the Kafka stream.

The code in action

Pre-requisites

• Download our code examples or clone them to your local environment:

 git clone https://github.com/pushtechnology/tutorials/

• A Diffusion service (Cloud or On-Premise), version 6.6 (update to latest preview version) or greater. Create a service here.
• Follow our Quick Start Guide and get your service up in a minute!

Setup

Make sure to add Diffusion library to your code. For JavaScript, we have added the following line in our public/kafka2diffusion.html and public/diffusion2kafka.html:

<script src='https://download.pushtechnology.com/clients/6.5.1/js/diffusion-6.5.1.js'></script>

Update public/js/producerApp.js and public/js/subscriberApp.js to the hostname of your Diffusion Cloud service, which you can find in your service dashboard. You can also leave the default values and connect to our sandbox service:

* host: 'kafkagateway.us.diffusion.cloud'
* user: 'user'
* password: 'password'