-
-
Notifications
You must be signed in to change notification settings - Fork 63
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Adding the API call script to fetch all rows of data for a given year #1257
Changes from 3 commits
83066b2
e1f1d3a
7b44f35
98d522f
b8cafdf
eb31154
785c8fe
6162e85
7439568
8e8a2dd
85c1c3f
059d4b4
3844426
fac94d8
e334bd4
559a7f6
8060ec0
aad1ac5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,35 @@ | ||
# -*- coding: utf-8 -*- | ||
""" | ||
Created on Sat Jun 4 17:49:23 2022 | ||
|
||
@author: AdithiPriya | ||
|
||
""" | ||
# import the necessary packages... | ||
|
||
import pandas as pd | ||
import requests | ||
|
||
data_2021= [] | ||
skip=0 # this denotes the page number | ||
priyakalyan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
limit= 10000 | ||
# since there are no next_url option or information about the total number of records, we will create a while loop... | ||
while True: | ||
url=f'https://dev-api.311-data.org/requests?start_date=2021-01-01&end_date=2021-12-31&skip={skip}&limit={limit}' | ||
priyakalyan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
#print('Requesting', url) | ||
priyakalyan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
response= requests.get(url) | ||
data= response.json() | ||
# Are there any more page left??? | ||
if data==[]: # if not, exit the loop... | ||
break | ||
# If there are pages left, add them to the variable data_2021 and move on to the next loop... | ||
data_2021.extend(data) | ||
skip=skip+limit | ||
data_2021_df = pd.DataFrame(data_2021) | ||
|
||
data_2021_df=data_2021_df.sort_values(by='createdDate') # sorting the data frame by created date... | ||
|
||
data_2021_df=data_2021_df.reset_index(drop=True) # reindexing... | ||
|
||
data_2021_df.to_csv('clean_311_data_2021.csv') # saving the dataframe as a csv file... | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Please make this an argument to the binary using https://abseil.io/docs/python/guides/flags. This will require you to create a main() method and and such, as in the example from the link. You can create a helper function that encapsulates the code that you've already written. Then, in main(), you can call that helper function with the absl flags. |
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can get rid of these comments