Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added partial update capability to sqlIngest #204

Merged
merged 9 commits into from
Jan 26, 2020
Merged

Added partial update capability to sqlIngest #204

merged 9 commits into from
Jan 26, 2020

Commits on Jan 15, 2020

  1. Added sqlIngest documentation notebook

    @ryanmswan
    ryanmswan committed Jan 15, 2020
    Configuration menu
    Copy the full SHA
    dbe33a4 View commit details
    Browse the repository at this point in the history

  2. Minor changes to fetchSocrataFull method 

    --Now able to specify page size, allowing larger pages if necessary
    @ryanmswan
    ryanmswan committed Jan 15, 2020
    Configuration menu
    Copy the full SHA
    49c4200 View commit details
    Browse the repository at this point in the history

Commits on Jan 19, 2020

  1. Merge pull request #8 from hackforla/dev 

    Bring dev up to date with upstream
    @ryanmswan
    ryanmswan authored Jan 19, 2020
    Configuration menu
    Copy the full SHA
    024bba4 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2020

  1. Updated sqlIngest pipeline 

    --orm configuration files now in databaseOrm.py package
--updateDatabase method of DataHandler now allows partial updates of records in database
  --runs roughly 90 seconds per 1000 records
  --around 5000 new records can be expected per day
--fixed fetchSocrata method of DataHandler to correctly grab number of records specified in querySize parameter
    @ryanmswan
    ryanmswan committed Jan 25, 2020
    Configuration menu
    Copy the full SHA
    843a2ea View commit details
    Browse the repository at this point in the history

Commits on Jan 26, 2020

  1. Fixing conflicts in sqlIngest before merge

    @ryanmswan
    ryanmswan committed Jan 26, 2020
    Configuration menu
    Copy the full SHA
    d0f3338 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'hackforla-dev' into dev 

    Bringing up to date with main repo
    @ryanmswan
    ryanmswan committed Jan 26, 2020
    Configuration menu
    Copy the full SHA
    784a285 View commit details
    Browse the repository at this point in the history

  3. Updated sqlIngest documentation 

    --documented updateDatabase method
--added a couple syntax/linting fixes to sqlIngest file
    @ryanmswan
    ryanmswan committed Jan 26, 2020
    Configuration menu
    Copy the full SHA
    a61e163 View commit details
    Browse the repository at this point in the history

  4. Fixed linting issues

    @ryanmswan
    ryanmswan committed Jan 26, 2020
    Configuration menu
    Copy the full SHA
    9dd9c52 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'dev' into dev

    @sellnat77
    sellnat77 authored Jan 26, 2020
    Configuration menu
    Copy the full SHA
    9850837 View commit details
    Browse the repository at this point in the history