Replicate our Pipeline

📄️ NATS AWS Setup

💡 In the AWS Setup Section, you will be guided through creating a new EKS Cluster, connecting to it, starting a NATS Server, connecting to it, and finally creating a Network Load Balancer.

Now that we have the backbone of the pipeline set up it is ready to start accepting data. Set up the Client program component in the pipeline, which is responsible for reading raw spectrum data packets from a radio device, and unpacking/processing them into a more human-readable format. The client program is highly customizable to your own application. In the case of our airplane tracker application, it specifically processes ADS-B packets using the dump1090 software which decodes the data into a more human-readable format, in this case a JSON file.

📄️ Annotator

The Annotator is a module within our data pipeline that is responsible for enriching the processed radio data with additional meaningful information. For our application, we downloaded data from the FAA’s (Federal Aviation Administration) Releasable Aircraft Database and used it to add additional information to each packet.

📄️ Storage & Search Engine

ElasticSearch (ES) is a distributed search and analytics engine that provides database-like functionality to store, search, and/or analyze real-time data. This is a core component in our pipeline that comes sequentially after the Annotator. It consumes all the annotated data from the pipeline and serves multiple purposes–including monitoring the Client status, storing the data, and finally serving as a persistent database to the backend flask server.

📄️ Backend

Overview

📄️ Frontend

Introduction

📄️ Dashboards

1. Check Out Our Dashboard for the Airplane Tracker Example

📄️ Stress Test

This page will show you how to stress test our data pipeline

📄️ Overview