Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blog Post for PR: Prototype a MQTT Source #7919 #6062

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions blog/config/nav.yml
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ nav:
- articles/knative-v0-3-autoscaling-a-love-story.md
- articles/ko-fast-kubernetes-microservice-development-in-go.md
- articles/build-deploy-manage-modern-serverless-workloads-using-knative-on-kubernetes.md
- articles/introduce-mqtt-source.md
ctmphuongg marked this conversation as resolved.
Show resolved Hide resolved
- Steering Committee:
- steering/2024-02-14-trademark-committee-dissolved.md
- steering/toc-2022-election-announcement.md
Expand Down
106 changes: 106 additions & 0 deletions blog/docs/articles/introduce-mqtt-source.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# Introducing MQTT Source with ContainerSource CRD

**Author: Phuong Cao...**
ctmphuongg marked this conversation as resolved.
Show resolved Hide resolved

Knative Eventing is taking a step forward in IoT integration with the introduction of an MQTT Source. This new feature contributes to addressing a growing need in the IoT space, allowing developers to seamlessly transform MQTT messages into CloudEvents and ingest them into the Knative Eventing system.

## Why MQTT Source?

With the proliferation of IoT devices and edge computing, there's an increasing number of devices emitting MQTT messages to brokers. Until now, processing these messages within Knative Eventing required additional steps. The new MQTT Source bridges this gap, enabling direct ingestion and processing of MQTT messages in a cloud-native event-driven architecture.

## Feature Description:

Instead of developing a full-fledged controller from scratch, we've taken a simpler and more efficient approach. The MQTT Source is implemented as a container image that can be used with the existing ContainerSource CRD. This approach is similar to what we've done for the WebSocket protocol, allowing us to transform MQTT messages received into CloudEvents and send them to specified brokers.

## Installations:

This feature uses mqtt_paho to receive and send MQTT messages as CloudEvents. The Mosquitto Eclipse package is used to send test MQTT messages, so it can be disregarded when the feature is complete and MQTT messages are sent from another source.

```shell
go get github.com/eclipse/paho.golang/paho
go get github.com/cloudevents/sdk-go/v2
go get github.com/cloudevents/sdk-go/protocol/mqtt_paho/v2
```
ctmphuongg marked this conversation as resolved.
Show resolved Hide resolved

## Using the feature:

1. Set up an MQTT broker (for testing, you can use Mosquitto):

```shell
docker run -it --rm --name mosquitto -p 1883:1883 eclipse-mosquitto:2.0 mosquitto -c /mosquitto-no-auth.conf
```

2. Set up sink:
The MQTT Source needs a sink to receive the CloudEvent message it sends. For testing purposes, we can use an event display service to receive and display those messages. Add this event_display.yaml file.

```
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
name: event-display
namespace: default
spec:
template:
spec:
containers:
- # This corresponds to
# https://github.com/knative/eventing/tree/main/cmd/event_display/main.go
image: gcr.io/knative-releases/knative.dev/eventing/cmd/event_display
```

Then keep this event_display running:

```shell
kubectl apply event-display.yaml
```

## Deploy the MQTT Source

We will first need to build the Container Source using command:

```shell
ko build cmd/mqttsource/main.go
```
Comment on lines +49 to +53
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here, instead of using a local dev command, can we point users to a released image?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just resolved the other comments! However, I'm not too sure how I can get the released image here...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pierDipi @Leo6Leo any ideas on how to get a nightly image for this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/knative/docs/blob/main/.github/workflows/knative-docs-image.yaml

We use the GitHub action to build and publish the image to Knative's release image registry.

Copy link
Member

@Cali0707 Cali0707 Jul 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Leo6Leo we need it in eventing, not docs sadly...

Would you be interested in looking into getting the job in eventing to build+publish the mqtt source image?

Copy link
Member

@pierDipi pierDipi Aug 22, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To get the image built and released we need to add mqttsource to the list here (in Eventing) https://github.com/knative/eventing/blob/e4b6d6861c5b6b3696ec8d9ab82ec73e04e0a781/hack/generate-yamls.sh#L57


Then, use the following yaml file in config/tools/mqtt_source. Here, you can specify a different topic and event source by adding args. The default source would be localhost:1883, assuming you have started the mosquitto MQTT broker, and the default topic would be mqtt-topic

```
apiVersion: sources.knative.dev/v1
kind: ContainerSource
metadata:
name: mqttsource
spec:
template:
spec:
containers:
- image: ko://knative.dev/eventing/cmd/mqttsource
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here, instead of using the ko image url, can we point them to an actual registry url?

securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsNonRoot: true
capabilities:
drop:
- ALL
seccompProfile:
type: RuntimeDefault


sink:
ref:
apiVersion: serving.knative.dev/v1
kind: Service
name: event-display
```

```shell
ko apply -f config/tools/mqttsource/mqttsource.yaml
```

4. Send testing messages:
Use this command to trigger the mosquitto broker to send a MQTT message to the MQTT Source.

```shell
mosquitto_pub -t '<topic-name>' -m '{"specversion" : "1.0","type" :"com.example.someevent", "id" : "1234-1234-1234","source" : "/mycontext/subcontext","data":{"msg":<message>"}}' -D PUBLISH user-property Content-Type application/cloudevents+json; charset=utf-8
```

The MQTT Source will receive this message, transform it into a CloudEvent, and send it to the specified sink in your Knative Eventing system. The event received will also be logged to the console. You can view the message if it is successfully sent by viewing the logs of event_display.
Loading