News websites and applications are gradually being replaced by user-centric news applications. One of the major reasons behind this change is that these apps provide a personalized experience to the users where they can get their desired content without having to go through the long list of articles.
-
Branch: airflow - link
- Web Scraping DAG
- Great Expectation Report DAG
- Data Processing Script
- Docker Compose Config
├── DAG_generator.py ├── DAG_template.py ├── dags │ ├── NYT_Scraping_arts.py │ ├── NYT_Scraping_automobiles.py │ ├── NYT_Scraping_books.py │ ├── NYT_Scraping_business.py │ ├── NYT_Scraping_climate.py │ ├── NYT_Scraping_education.py │ ├── NYT_Scraping_fashion.py │ ├── NYT_Scraping_food.py │ ├── NYT_Scraping_health.py │ ├── NYT_Scraping_science.py │ ├── NYT_Scraping_sports.py │ ├── NYT_Scraping_technology.py │ ├── NYT_Scraping_travel.py │ ├── NYT_Scraping_universal.py │ ├── NYT_Scraping_world.py │ └── ge-report.py ├── data_processing.py └── docker-compose.yaml -
Branch: great-expectations - link
- Great Expectation Checkpoint Source Code
└── great_expectations ├── checkpoints │ └── nyt_raw_data_suite_checkpoint_v1 │ ├── 0.yml │ └── 1.yml ├── data ├── expectations │ └── nyt_raw_data_suite.json ├── great_expectations.yml ├── plugins │ └── custom_data_docs │ ├── renderers │ ├── styles │ │ └── data_docs_custom_styles.css │ └── views └── profilers -
Branch: fastapi - link
- Backend Data Service Endpoints
├── Dockerfile ├── __init__.py ├── custom_functions.py ├── database.py ├── iam.db ├── keys ├── main.py ├── models.py ├── requirements.txt ├── routers │ ├── __init__.py │ ├── authentication.py │ ├── mongodb.py │ ├── news_feeds.py │ ├── oaut2.py │ ├── token.py │ ├── users.py │ └── whatsapp.py ├── schemas.py ├── test_cases │ ├── dev_server.html │ ├── prod_server.html │ └── test.py └── ui ├── assets ├── components.html └── index.html -
Branch: streamlit - link
- User Application Source Code
├── Dockerfile ├── main.py └── requirements.txt
| Developer | Deliverables |
|---|---|
| Jui | Streamlit Component 1 |
| Jui | FastAPI Endpoint - Feeds |
| Jui | Technical Documentation |
| Jui | UI Testing |
| Piyush | Data Ingestion pipeline |
| Piyush | Data Processing pipeline |
| Piyush | Streamlit Component 2 and 3 |
| Piyush | FastAPI Endpoint - Database and IM |
| Piyush | Technical Documentation |
| Piyush | Infra Deployment |
WE ATTEST THAT WE HAVEN’T USED ANY OTHER STUDENTS’ WORK IN OUR ASSIGNMENT AND ABIDE BY THE POLICIES LISTED IN THE STUDENT HANDBOOK
Contribution: 🤝
- Jui :
40% - Piyush :
60%