SorrTask762_start_downloading_data

[tkpratardan]

@sonaalKant @gpsaggese
I have developed an initial version of the code to download data from the specified sources mention in #762. However, we have a few bottlenecks with the free API:

1. goperigon.com -> limited API request (150)
2. newsdata.io -> daily limit of 10 articles
3. mediastack.com -> I haven't implemented a pipeline for this datasource because the free
   version requires payment info.
4. thenewsapi.com -> limited API requests per day and some access to some endpoints is
   blocked. Example, crypto.
5. marketaux.com -> limited API requests and access to some endpoints is blocked.

Review:

1. Is this according to expectations?
2. Do we need to implement data formatting and processing? LIke converting JSON responses to a dataframe and cleaning the texts? or is the data processing implemented in another script?

[sonaalKant]

Good start, Lets get some example data points from each API and log them in some doc so that we quickly see and analyse the next steps.

[samarth9008]

No need for OOPs programming inside notebooks. Over here, you can try interacting with API directly. Download the data and get a sense about it.

[samarth9008]

But lets follow what @sonaalKant says. This is just my opinion

[gpsaggese]

My take is to do whatever is faster. This is exploration. We want a dataframe with data to look at

[tkpratardan]

I kept the OOP method because I feel like this way gives us more functionality through controlling attributes of class instances. This could come quite handy while extracting data with different params/endpoints, making it faster, IMO. Let me know if this is okay.

[tkpratardan]

@gpsaggese @sonaalKant @samarth9008
I have made changes to the notebook, including some preliminary preprocessing and text cleaning for better readability.
Since the JSON responses were nested, I unpacked them by keeping the hierarchy intact. For ex:

We can interpret this as different keys under keywords and same index of the list across the columns form the corresponding values to the keys. Let me know if this makes sense.