Skip to content

EuroPython 2021 - High Performance Data Processing using Python, Elasticsearch and Kafka.

License

Notifications You must be signed in to change notification settings

harshit98/Retail-Updates-Streamer

Repository files navigation

Retail Updates Streamer

EuroPython 2021 - High Performance Data Processing using Python, Kafka and Elasticsearch.

Prerequisites

  • Python >= 3.6
  • FastAPI
  • Kafka
  • Elasticsearch

Workflow Architecture

Overall Flow:

workflow-architecture

Code Explanation:

workflow-kafka

Setup

  1. Make sure Python >= 3.6 is installed on your local machine and create a virtual environment.

    python3 -m venv retail-updates-streamer
  2. Activate your virtual environment.

    source retail-updates-streamer/bin/activate
  3. Install application packages.

    pip install -r requirements.txt
  4. Add your producer-consumer related config in a .env file.

  5. For docker setup, add global environment constant in your zshrc or bashrc depending on shell.

    export DOCKER_KAFKA_HOST = $(ipconfig getifaddr en0)
  6. Please note that I've commented out elasticsearch and kibana in docker-compose.yml as they need a lot of virtual environment memory. You can run these two stacks on local by downloading them as zip from official elastic site.

Sample CURLs

Producer

  1. Health Check

    Request

    curl --location --request GET 'http://0.0.0.0:8001/ping'

    Response

    {
      "ping": "pong!"
    }
  2. Produce message to Kafka

    Request

    curl --location --request POST 'http://127.0.0.1:8001/producer/retail-product' \
    --header 'Content-Type: application/json' \
    --data-raw '{
       "name": "Funny Farm House Ketchup",
       "category" : "Dips and Ketchups",
       "price" : 15,
       "stock" : 3,
       "product_id": 10,
       "timestamp": ""
    }'

    Response

    {
      "name": "Funny Farm House Ketchup",
      "message_id": "Funny Farm House Ketchup_e5c0f1b0-ac1e-44c7-92c1-1f86728a36dc",
      "topic": "retail-product",
      "timestamp": "2021-07-12 15:17:20.314894"
    }

Consumer

  1. Health Check

    Request

    curl --location --request GET 'http://0.0.0.0:8000/ping'

    Response

    {
      "ping": "pong!"
    }
  2. Consume messages from Kafka

    Request

    curl --location --request GET 'http://127.0.0.1:8000/consumer/retail-product'

    Response

    {
      "topic": "retail-product",
      "timestamp": "2021-07-12 20:47:21.067802",
      "product_name": "Funny Farm House Ketchup",
      "product_id": 10,
      "success": true
    }

Author

👤 Harshit Prasad

Twitter: @HarshitPrasad8
Github: @harshit98
Website: harshitprasad.com
LinkedIn: harshit-prasad

🤝 Contributing

Contributions, issues and feature requests are welcome!

Show your support

Give a ⭐️ if you think this project is awesome!

📝 License

Copyright © 2021 Harshit Prasad
This project is Apache License licensed.

About

EuroPython 2021 - High Performance Data Processing using Python, Elasticsearch and Kafka.

Topics

Resources

License

Stars

Watchers

Forks