Build a Docker + Airflow application to fetch daily stock price
This summer my friend showed me a cool python application he has developed. On a high level it was a containerized python application that fetched the status of pollen in air, current weather conditions and couple of other useful related data related to pollen allergy. The application was scheduled to run periodically within Airflow. I found it inspiring and I decided to embark on a short journey to learn building docker application and running it in Airflow.
When one downloads and runs containerized images of databases (like MySQL) or workflow tools like Airflow or ones own containerized code, it becomes as simple as executing start and stop commands. One does not have to get into complex task of installing multiple dependencies based on your platform. This makes it easy to run applications on different types of machines and remove them if not needed.
Here I am sharing my learning journey, that may help you too. These series of chapters help you build your hands-on docker knowledge from scratch to a full-fledged application that uses airflow + mysql database to fetch daily end of day stock price and store it in MySQL database.
These series of lessons are a very good accompaniment to some YouTube tutorial available for Docker and Airflow (two separate tutorial topics). These tutorials will likely explain what is Docker, why Docker, what is Airflow etc.
Pre-requisite
- Python knowledge
- Basis Docker knowledge (what is docker and why it’s a great idea to build docker based application)
- Basic Airflow knowledge (what is Airflow, what can be achieved with Airflow)
Note: The python applications created are small and simple, no explanation of the code logic has been provided though with the inline comments they should be easy to read and understand. Same is true for the DAG code (i.e. the Airflow scheduler).
The github repository has multiple directories. One can simply execute the commands described in each directories Readme file to get training on docker commands. Each subsequent chapter builds new skills from previous chapter.
Here is an overview of all the chapters. The actual commands to execute and more explanation is in the Readme of each chapter (directory)
docker001
- Get familiar with basic Docker commands e.g. pulling a docker image, running a container, checking status
- Build a custom image on top of ubuntu and install python libraries and save it in a docker image
- User your custom docker image to run your python programs (note: at this stage python program is not part of image itself)
docker002
- Create an image using Docker file
- Build a simple python based docker application that echoes the command line input parameter.
- Run docker container with different parameter options.
Note: unlike previous chapter here the application is shipped as a part of your docker image.
docker003
- Build a first version of python application that fetches the end-of-day stock price for input number of days. The output is of this python application is displayed on console or written to file based on command line parameters
- Build a custom docker application containing above end-of-day stock pricer
- Learn to run docker image to save the output to local directory
docker004
- Download MySQL server docker image. Learn to start and stop the MySQL containers.
- Connect to MySQL DB from command line and create tables manually.
- Learn about docker-compose.yaml that helps to define and run multi-container Docker applications.
Note: Here we take a break from python application and learn only about the MySQL docker container
docker005
- Enhance the python application to write the fetched end-of-day prices and write to database (besides console and file)
- Create docker-compose.yaml to (download+)run MYSQL server and build python image docker application
- Run docker app that fetches end-of-day stock price and writes to DB
- View the different running configurations and data output
- Learn how to pass environment variables to from .env file to applications via docker-compose
Note: docker005 combines and builds upon the learnings of chapters docker003 and docker004.
docker006
- Download, configure, run and stop Airflow docker container
- Run Airflow examples and understand the output
Note: Here we take a break from python application and learn only about Airflow
docker007
Use multiple docker compose files that
a. launches MySQL server,
b. launches Airflow server,
c. creates application docker image if it does not exist
- Learn to start, check status and stop containers specified in multiple docker-compose images
- Test the docker application to check output to console, file in a directory and in the database
- Automatically run the docker python application at configured time and verify the output
Note: docker007 combines and builds upon the learnings of chapters docker005 (which itself is built upon 003 and 004) and docker006.
What can be done next?
This application base can be further enhanced as follows:
- Instead of providing a single ticker (e.g. GOOG for google), provide an input file containing list of tickers
- Enhance the program to scrape (or via API) get earnings results and other parameters from company website or other source
- Use business intelligence tools like PowerBI, Tableau to build analytics
