Skip to content

Latest commit

 

History

History
65 lines (45 loc) · 2.13 KB

README.md

File metadata and controls

65 lines (45 loc) · 2.13 KB
Raw

Weather Data Pipeline from API

Objective

Create a pipeline to fetch weather data from an API, process it, and store it in a database for further analysis and visualization.

You can check my presentation with this LINK

Technologies Involved

  • Programming Language: Python (for scripting and data manipulation)
  • API: OpenWeatherMap API
  • Database: PostgreSQL
  • Tools: Docker (for containerization), Airflow (for scheduling and orchestrating the pipeline), Grafana (for visualization)

How to run

Step 1: docker-compose up -d

  • Dockerize your environment to ensure consistency and portability.
  • Set up containers for PostgreSQL (for data storage), Airflow (for orchestration), and Grafana (for visualization).
  • {make sure the db is auto-build and the coordinates data is already in db}
  • {how to append collected data in development}

Step 2: add your API key

  • You can Sign Up on Weather API to get your API keys
  • {How to set user-own API}

Step 3: unpause the DAGs

  • {installing airflow}
  • {make sure the DAGs is already have}
  • {make sure connected to postgres}

Step 4: Grafana

  • {How to grafana}
  • {make sure db is configurated}
  • {ensure dashboard template is exist}

Code Explained

DAGs

Response

Wrangling and Cleaning

Inserting to Database

Additional Considerations

  • Security: Ensure API keys and sensitive data are securely managed within your Docker/local environment.

Conclusion

By building this pipeline, I gained practical experience in fetching API, scheduling DAGs, Processing raw data, also evaluate and monitoring data by visualization in dashboard

Recommendation

  • Make improvement about Data Governance in the database
  • Adding more API or use real hardware so it can be streamed with real-time data and much more source data

Updates Notice

(16/07/24)

  • Fixing README.md

TODO :

  • Make sure docker and build makefile to initialize container
  • Explaining how the code works
  • Make sure the image, database, and API key to smoothly runs