This project aims to use Apache Kafka in order to ingest Formula 1 telemetry data from the F1 202x game (by CodeMasters/EA Sports) running on Microsoft Xbox. It uses different products, projects and technologies:
- F1 202x games (by CodeMasters/EA Sports) for getting the telemetry data via UDP;
- Apache Camel project for routing telemetry events from UDP to Apache Kafka and to InfluxDB;
- Apache Kafka as the core project for ingesting the telemetry events on different topics;
- Kubernetes for deploying most of the components to run in the cloud;
- Strimzi for deploying easily the Apache Kafka on Kubernetes;
- InfluxDB for storing the telemetry time series as data source for dashboards;
- Grafana for providing dashboards showing the real time telemetry data;
The formula 1 telemetry library, used to decode the UDP packets, works with the F1 2020 game format. It's possible to have the project working with newer game editions by selecting the "2020" UDP packet format in the Telemetry settings on the game.
It is possible to select the season for teams, drivers and tracks by setting the F1_TELEMETRY_SEASON environment variable when running the different applications.
By default, it uses 2020 season data. For more information see here.
Following an overall picture of how these technologies are used together. A couple of short introduction videos are available here and here.
You can also find some material from conference presentations:
- Let's code a motorsport telemetry processing pipeline - slides demo video
- Formula 1 telemetry processing using Kafka Streams - slides demo video
- Formula 1 telemetry processing using Quarkus and OpenShift Streams for Apache Kafka - slides demo video
In order to ingest the telemetry events into Apache Kafka, the Apache Camel project is used with:
- a route getting the raw UDP packets from the F1 202x game (by CodeMasters/EA Sports) on Microsoft Xbox and dispatching these events to three more routes;
- a route just forwarding the raw UDP packets to a corresponding Apache Kafka topic;
- a route to filter only the
EVENTtype raw UDP packets and forwarding them to a corresponding Apache Kafka topic; - a route to aggregate the raw UDP packets data for producing drivers related data and forwarding them to a corresponding Apache Kafka topic;
The Kafka Streams API based application shows an example of real-time analytics on the telemetry data. The application processes the average speed in the last 5 seconds.
It has a source node reading from the topic with drivers related messages, filtering the ones not containing valid telemetry data. Extract and group the driver's speed by corresponding driver's id using in a tumbling window of 5 seconds. Next, it sums speeds and counts them in order to process the corresponding average value. Finally, the sink node writes to the destination topic.
In order to provide the telemetry data to Grafana dashboards, InfluxDB is used as data source and the telemetry events are stored through Apache Camel with:
- a route getting drivers related data for storing telemetry, motion and car status data;
- a route getting the
EVENTtype raw UDP packets for storing fastest lap and speedtrap events;
Contains different components for ingesting and handling Formula 1 202x game (by CodeMasters/EA Sports) telemetry data through Apache Kafka.
- udp-kafka: Apache Camel application bridging the telemetry packets got on UDP from the Formula 1 202x game to Apache Kafka to different topics:
- f1-telemetry-packets contains the raw
Packet(s); - f1-telemetry-events contains only the raw
Packet(s) ofEVENTtype; - f1-telemetry-drivers contains the
Drivermessages as result of aggregatingPacket(s) in the same frame with telemetry data for all drivers;
- f1-telemetry-packets contains the raw
- consumer: Apache Kafka client application consuming
Drivermessages from Apache Kafka; - streams-avg-speed: Apache Kafka Streams API based application getting raw
Driver(s), processing them in real time to get average speed in a 5 seconds window and writing to Apache Kafka to the f1-telemetry-streams-avg-speed topic; - streams-laps: Apache Kafka Streams API based application getting raw
Driver(s), processing them in real time to get the best overall times in the three track sectors and writing to Apache Kafka to the f1-telemetry-best-overall-sector topic; - common: common library providing model classes and related Apache Kafka serializer/deserializer;
- kafka-influxdb: Apache Camel application writing driver/car's telemetry data to InfluxDB as time series;
- webui: A sample Web application showing the race ranking in real time getting
Driver(s) from Apache Kafka; - streams-processing: Apache Kafka Streams API based application getting raw
Driver(s) and extracting some telemetry in JSON format; - dashboard: folder containing Grafana dashboard showing driver/car's telemetry data;
- deployment: folder containing Kubernetes deployments for all the provided applications;
- documentation: folder containing documentation about how to deploy the entire solution;
The Formula 1 2020 game UDP packets specification is here. The library used for decoding the packets is here.
You can find more information about building, deploying and running the solution in the documentation here
The telemetry dashboard shows information like speed, engine (rpm), throttle and brake.
The motion dashboard shows information about the car in motion like the G-force.
The car status dashboard shows information mostly related to the status of each car, for example wings damages and fuel in tank.
The events dashboard shows race events like max speed trap and fastest lap.
The driver dashboard allows to get data for a single driver, selecting one through the corresponding drop down list. It has a top bar with information about current lap, position, distance and lap times.
It also has some graphs showing specific driver data related to throttle/brake, engine/speed, gear/clutch, brakes and steering.
It also shows specific information about tyres like, for example, the compound, the age laps, wear, surface temperature and damage
The streams dashboard shows data about processed telemetry data through Kafka Streams application. The first one is about the average speed during the last 5 seconds.
