Merge pull request #11 from flightaware/BCK-5076

Create API to retrieve positions; Display positions on a Google Static map on the frontend

.env-sample

@@ -1,3 +1,4 @@
 FH_USERNAME=user
 FH_APIKEY=key
 INIT_CMD_ARGS=events "flightplan departure arrival cancellation position"
+GOOGLE_MAPS_API_KEY=key

README.md

@@ -55,13 +55,15 @@ The connector then forwards Firehose messages to its own clients.
 
 ### db-updater
 The db-updater service receives Firehose messages from the connector and
-maintains a database table of flights based on their contents. The service is
-only capable of handling so-called "flifo" (flight info) messages currently; in
-the future, position messages will also be handled. The default database
-configuration writes to a sqlite database in a named volume, but PostgreSQL is
-also supported. Other databases could potentially be supported with little
-effort. To prevent bloat, flights older than 48 hours are automatically
-dropped from the table.
+maintains a database table based on their contents. The service is capable of
+handling so-called "flifo" (flight info) messages and airborne position messages.
+Two db-updater containers are configured by default - one handles flight info and
+updates a "flights" table, and the other handles airborne positions and updates
+a "positions" table. The flight info db-updater uses sqlite by default (but has been
+tested with PostgreSQL), and the position db-updater uses TimescaleDB which is
+based on PostgreSQL. Other databases could potentially be supported with little
+effort. To prevent bloat, flights and positions older than 48 hours are
+automatically dropped from the table.
 
 ### fids
 The sample application is a webapp backed by the flights database. You can use
@@ -72,6 +74,23 @@ be used in a production environment, this sample application should only be
 considered a demonstration of what can be built using the data from Firehose.
 It should *not* be used in a production environment.
 
+### fids with Google Maps
+Now, you can see positions displayed on a static Google Maps image on each
+flight info page! In order to enable this feature, you need to configure your
+own Google Maps API key.
+Instructions:
+https://developers.google.com/maps/documentation/maps-static/get-api-key
+Once you get your API key, just specify it in your .env file as
+GOOGLE_MAPS_API_KEY. Then you will see static maps with a flight track on your
+flight info pages. Note that you may need to enter your payment information.
+The Google Maps API is a paid service with a limited free tier.
+Pricing information:
+https://developers.google.com/maps/documentation/maps-static/usage-and-billing
+You can see that you currently get "a $200 USD Google Maps Platform credit"
+each month, and each query 0-100,000 is 0.002 USD each. So that means that you
+will get 100,000 free queries per month. Since this is a demo and not meant for
+production, that should be fine.
+
 ### kafka/zookeeper
 We are using kafka as a message queue between the connector and the db-updater.
 Kafka depends on zookeeper to coordinate some important tasks, so we included
@@ -96,6 +115,5 @@ different group names will each consume all messages in a given topic, and
 consumers with the same group name will split messages from that topic between
 them.
 
-
 Check out [the roadmap](./ROADMAP.md) to see what components are coming in the
 future!

ROADMAP.md

@@ -9,6 +9,11 @@ Firestarter v2 will introduce a robust queueing component between the connector
 and its clients, allowing for efficient data fan-out.
 
 ### v3
-Firestarter v3 will support processing and storage of positional data from
-Firehose. It will likely also include a sample application for viewing such
+Firestarter v3 adds support for processing and storage of airborne position data
+from Firehose. It will likely also include a sample application for viewing such
 data.
+
+### v4
+Firestarter v4 will support processing and storage of surface position data from
+Firehose. It will likely also include a sample application for viewing such
+data.

docker-compose.yml

@@ -94,7 +94,7 @@ services:
     environment:
       # URL to database that will be updated based on Firehose contents.
       # Documentation at https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls
-      - DB_URL=${POSITION_DB_URL:-postgresql://postgres:positions@timescaledb:5432}
+      - DB_URL=${POSITIONS_DB_URL:-postgresql://postgres:positions@timescaledb:5432}
       - PYTHONUNBUFFERED=1
       # Same kafka topic name as the producer of the feed that you want to consume
       - KAFKA_TOPIC_NAME=position_feed1
@@ -116,6 +116,8 @@ services:
     build:
       context: .
       dockerfile: fids/Dockerfile
+      args:
+        - GOOGLE_MAPS_API_KEY=${GOOGLE_MAPS_API_KEY:-}
     init: true
     ports:
       # Port upon which to serve webapp
@@ -125,7 +127,8 @@ services:
     environment:
       # URL to database that is being updated by db-updater.
       # Documentation at https://docs.sqlalchemy.org/en/13/core/engines.html#database-urls
-      - DB_URL=${FLIGHTS_DB_URL:-sqlite:///db/flights.db}
+      - FLIGHTS_DB_URL=${FLIGHTS_DB_URL:-sqlite:///db/flights.db}
+      - POSITIONS_DB_URL=${POSITIONS_DB_URL:-postgresql://postgres:positions@timescaledb:5432}
       - PYTHONUNBUFFERED=1
     volumes:
       - data:/home/firestarter/app/db

docs/database.md

@@ -6,10 +6,14 @@ https://docs.sqlalchemy.org/en/13/dialects/index.html). By default, db-updater
 uses a sqlite database located on a Docker named volume. This allows the
 database file (located at `/home/firestarter/db/flights.db`) to persist between
 container restarts and allows sharing the database file with the fids
-component. db-updater has also been tested against PostgreSQL.
+component. Db-updater uses TimescaleDB for positions. TimescaleDB is a
+time-series database, so it is very effecient in handling time-series data like
+positions. It is also an extention of PostgreSQL, which has been fully tested
+in db-updater.
 
-When starting db-updater, it checks the database it's connected to for a table
-named "flights". If no such table exists, it is created with the schema found
+When starting db-updater, it checks the database it's connected to ensure that
+the "flights" or "positions" table exists (depending on what it is intended to
+update). If no such table exists, it is created with the schema found
 [here](../db-updater/main.py).
 
 ## Customizing the database connection

fids/Dockerfile

@@ -1,5 +1,8 @@
 FROM python:3.8-slim-buster
 
+ARG GOOGLE_MAPS_API_KEY
+ENV REACT_APP_GOOGLE_MAPS_API_KEY=$GOOGLE_MAPS_API_KEY
+
 RUN apt-get update && \
 	apt-get install -y libpq-dev gcc npm make && \
 	npm install npm@latest -g

fids/app.py

@@ -9,14 +9,32 @@
 from sqlalchemy.sql import union, select, func, and_, or_  # type: ignore
 
 # pylint: disable=invalid-name
-engine = sa.create_engine(os.getenv("DB_URL"), echo=True)
-meta = sa.MetaData()
-insp = sa.inspect(engine)
-while "flights" not in insp.get_table_names():
+flights_engine = sa.create_engine(os.getenv("FLIGHTS_DB_URL"), echo=True)
+flights_meta = sa.MetaData()
+flights_insp = sa.inspect(flights_engine)
+while "flights" not in flights_insp.get_table_names():
     print("Waiting for flights table to exist before starting")
-    insp.info_cache.clear()
+    flights_insp.info_cache.clear()
     time.sleep(3)
-flights = sa.Table("flights", meta, autoload_with=engine)
+flights = sa.Table("flights", flights_meta, autoload_with=flights_engine)
+
+positions_engine = sa.create_engine(os.getenv("POSITIONS_DB_URL"), echo=True)
+
+while True:
+    try:
+        positions_engine.connect()
+        break
+    except sa.exc.OperationalError as error:
+        print(f"Can't connect to the database ({error}), trying again in a few seconds")
+        time.sleep(3)
+
+positions_meta = sa.MetaData()
+positions_insp = sa.inspect(positions_engine)
+while "positions" not in positions_insp.get_table_names():
+    print("Waiting for positions table to exist before starting")
+    positions_insp.info_cache.clear()
+    time.sleep(3)
+positions = sa.Table("positions", positions_meta, autoload_with=positions_engine)
 
 app = Flask(__name__, template_folder="frontend/build", static_folder="frontend/build/static")
 # Uncomment to enable serving the frontend separately (when testing, perhaps)
@@ -34,6 +52,17 @@ def catch_all(path):
     return render_template("index.html")
 
 
+@app.route("/positions/<flight_id>")
+def get_positions(flight_id: str) -> dict:
+    """Get positions for a specific flight_id"""
+    result = positions_engine.execute(
+        positions.select().where(positions.c.id == flight_id).order_by(positions.c.time.desc())
+    )
+    if result is None:
+        abort(404)
+    return jsonify([dict(e) for e in result])
+
+
 @app.route("/flights/")
 @app.route("/flights/<flight_id>")
 def get_flight(flight_id: Optional[str] = None) -> dict:
@@ -48,7 +77,7 @@ def get_flight(flight_id: Optional[str] = None) -> dict:
         )
     else:
         whereclause = flights.c.id == flight_id
-    result = engine.execute(flights.select().where(whereclause)).first()
+    result = flights_engine.execute(flights.select().where(whereclause)).first()
     if result is None:
         abort(404)
     return dict(result)
@@ -61,7 +90,7 @@ def get_busiest_airports() -> Response:
     since = datetime.fromtimestamp(int(request.args.get("since", 0)), tz=UTC)
     query = request.args.get("query")
     if query:
-        result = engine.execute(
+        result = flights_engine.execute(
             union(
                 select([flights.c.origin]).distinct().where(flights.c.origin.like(f"%{query}%")),
                 select([flights.c.destination])
@@ -76,7 +105,7 @@ def get_busiest_airports() -> Response:
     return jsonify(
         [
             row.origin
-            for row in engine.execute(
+            for row in flights_engine.execute(
                 select([flights.c.origin])
                 .where(func.coalesce(flights.c.actual_off, flights.c.actual_out) > since)
                 .group_by(flights.c.origin)
@@ -92,7 +121,7 @@ def airport_arrivals(airport: str) -> Response:
     """Get a list of arrivals for a certain airport"""
     airport = airport.upper()
     dropoff = datetime.now(tz=UTC) - timedelta(hours=5)
-    result = engine.execute(
+    result = flights_engine.execute(
         flights.select().where(
             and_(
                 flights.c.destination == airport,
@@ -113,7 +142,7 @@ def airport_departures(airport: str) -> Response:
     """Get a list of departures for a certain airport"""
     airport = airport.upper()
     dropoff = datetime.now(tz=UTC) - timedelta(hours=5)
-    result = engine.execute(
+    result = flights_engine.execute(
         flights.select().where(
             and_(
                 flights.c.origin == airport,
@@ -135,7 +164,7 @@ def airport_enroute(airport: str) -> Response:
     airport = airport.upper()
     past_dropoff = datetime.now(tz=UTC) - timedelta(hours=5)
     future_dropoff = datetime.now(tz=UTC) + timedelta(hours=6)
-    result = engine.execute(
+    result = flights_engine.execute(
         flights.select().where(
             and_(
                 flights.c.destination == airport,
@@ -158,7 +187,7 @@ def airport_scheduled(airport: str) -> Response:
     airport = airport.upper()
     past_dropoff = datetime.now(tz=UTC) - timedelta(hours=5)
     future_dropoff = datetime.now(tz=UTC) + timedelta(hours=6)
-    result = engine.execute(
+    result = flights_engine.execute(
         flights.select().where(
             and_(
                 flights.c.origin == airport,

fids/frontend/client/src/components/FlightInfo/FlightInfo.js

@@ -2,14 +2,17 @@ import React, { Component } from 'react';
 import { Container, Row, Col, Card } from 'react-bootstrap';
 import axios from 'axios';
 import * as FS from '../../controller.js';
+import * as helpers from '../../helpers.js';
 
 export default class FlightInfo extends Component {
     constructor(props) {
         super(props);
 
         this.state = {
             data: {},
-            loading: true
+            positions: {},
+            loading_flight: true,
+            loading_positions: true,
         }
     }
 
@@ -18,6 +21,7 @@ export default class FlightInfo extends Component {
         const { match: { params } } = this.props;
 
         this.fetchFlightInfo(params.flight);
+        this.fetchPositions(params.flight);
 
         console.log(params.flight);
 
@@ -28,15 +32,28 @@ export default class FlightInfo extends Component {
             .then(response => {
                 console.log(response.data)
                 this.setState({
-                    loading: false,
+                    loading_flight: false,
                     data: response.data
                 });
             });
     }
 
+    fetchPositions(flightID) {
+        if (flightID) {
+            axios.get(`/positions/${flightID}`)
+                .then(response => {
+                    console.log(response)
+                    this.setState({
+                        loading_positions: false,
+                        positions: response.data
+                    });
+                });
+        }
+    }
+
     render() {
 
-        const { data, loading } = this.state;
+        const { data, positions, loading_flight, loading_positions } = this.state;
 
 
 
@@ -107,10 +124,32 @@ export default class FlightInfo extends Component {
             }
         }
 
+        const getMapImage = () => {
+            // do not display a map if there are no positions
+            if (positions.length == 0) {
+                return <div></div>
+            }
+
+            // Get aircraft image bearing
+            let image_bearing = 0;
+            if (positions.length > 5) {
+                image_bearing = helpers.getClosestDegreeAngle(helpers.getBearingDegrees(positions[4].latitude, positions[4].longitude, positions[0].latitude, positions[0].longitude));
+            }
+
+            // build latlon list
+            let latlon = "";
+            for(let i = 0; i < positions.length; i++) {
+                let obj = positions[i];
+                latlon +=  "|" + obj.latitude + "," + obj.longitude;
+            }
+
+            return <div className="d-flex align-items-center"><img src={`https://maps.googleapis.com/maps/api/staticmap?size=640x400&markers=anchor:center|icon:https://github.com/flightaware/firestarter/raw/master/fids/images/aircraft_${image_bearing}.png|${positions[0].latitude},${positions[0].longitude}&path=color:0x0000ff|weight:5${latlon}&key=${process.env.REACT_APP_GOOGLE_MAPS_API_KEY}`}/></div>
+        }
+
         return (
             <Container className="flight-info-wrapper">
                 {
-                !loading ?
+                !loading_flight ?
                     <>
                     <Container className="p-3 flight-number">
                         <Row lg={3}>
@@ -161,6 +200,9 @@ export default class FlightInfo extends Component {
                             </Col>
                         </Row>
                     </Container>
+                    {!loading_positions && process.env.REACT_APP_GOOGLE_MAPS_API_KEY != "" ?
+                    <div className="d-flex justify-content-center">{getMapImage()}</div>
+                    : <Container></Container>}
                     <Container>
                         <Card className="detail-card">
                             <Row className="detail-card-title">

fids/frontend/client/src/helpers.js

@@ -0,0 +1,38 @@
+const degreesToRadians = (degrees) => {
+	return (degrees * Math.PI / 180);
+}
+
+const radiansToDegrees = (radians) => {
+	return (radians * 180 / Math.PI);
+}
+
+export const getBearingDegrees = (d_lat1, d_lon1, d_lat2, d_lon2) => {
+	let r_lat1 = degreesToRadians(d_lat1);
+	let r_lon1 = degreesToRadians(d_lon1);
+	let r_lat2 = degreesToRadians(d_lat2);
+	let r_lon2 = degreesToRadians(d_lon2);
+	let dLon = (r_lon2 - r_lon1);
+
+	let y = Math.sin(dLon) * Math.cos(r_lat2);
+	let x = Math.cos(r_lat1) * Math.sin(r_lat2) - Math.sin(r_lat1)
+			* Math.cos(r_lat2) * Math.cos(dLon);
+
+	let brng = Math.atan2(y, x);
+
+	brng = radiansToDegrees(brng);
+	brng = (brng + 360) % 360;
+
+	return brng;
+}
+
+export const getClosestDegreeAngle = (input_degrees) => {
+	if ((input_degrees >= 315 && input_degrees < 360) || (input_degrees >= 0 && input_degrees < 45)) {
+		return 0;
+	} else if (input_degrees >= 45 && input_degrees < 135) {
+		return 90;
+	} else if (input_degrees >= 135 && input_degrees < 225) {
+		return 180;
+	} else {
+		return 270;
+	}
+}

fids/requirements/base.in

@@ -1,3 +1,4 @@
 sqlalchemy==1.3.16
 Flask==1.1.2
 flask-cors==3.0.8
+psycopg2-binary==2.8.5