A kafkaesque solution
Streaming Realtime Vehicle Positions: City of Calgary
In this article, I am going to define a problem and propose a solution for it with the new technologies thst are the defacto buzzwords in the tech industry: Kafka, Cloud, Microservices, Docker, Python and more! This will not only help myself to develop a solution based on my newly acquired skills, but also help my viewers learn by my learning my thought process. Problem: Recently after a long time since my last ride with the city of Calgary's transportation services, while I was waiting for my bus, I was trying to find out how I could get some info from their website about any arrival estimates. Of course one option was to text some number and get some info but I was not successful! After some time I got home and decided to browse the City's website from my computer to crawl some data. alas, I found a page that updates a file every 30 seconds with new positions for all their fleet.
</div>
The Solution: In order to solve any problems, we need to:: - first identify the problem - a big picture solution - bridging the gap
What We know: - We want to have a "file-downloader" that automatically get the data from the new file - The file is in Google's Protobuf file format - We are going to use Python, so we need solutions that can download a file periodically, and parse the data into readable format, insert these data into a database, have an ORM like SQLAlchemy to interact with the databse through python, Have a system to stream these data: This is Where Kafka comes into play! Have an automated system that with newer updates (inserts into the database), will automatically post these (and only these) data points into our streaming service (Debezium) and let Kafka take care of the rest. - finally, have a frontend to display these and showcase all these amazing techniques!
A quick note
Thanks for reading!