Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove client secrets from core code #981

Merged
merged 10 commits into from
Aug 4, 2017
Merged

Remove client secrets from core code #981

merged 10 commits into from
Aug 4, 2017

Conversation

noahpresler
Copy link
Owner

@noahpresler noahpresler commented Aug 3, 2017

Advanced Configuration

Semester.ly makes use of several secrets which allow it to interact securely with third party software providers. These providers include Facebook (for oauth and social graph), Google (oauth), and university APIs.

In order for Semester.ly to run out of the box, we have included credentials to test Google and Facebook applications for development purposes. We override these keys for production use thereby keeping our client secrets... well, secrets! These provided credentials can be found in semesterly/dev_credentials.py::

SECRETS = {
    #Credentials for a test application for Semester.ly (+ Google/Facebook)
    'SECRET_KEY': ...,
    'HASHING_SALT': ...,
    'GOOGLE_API_KEY': ...,
    'SOCIAL_AUTH_GOOGLE_OAUTH2_KEY': ...,
    'SOCIAL_AUTH_GOOGLE_OAUTH2_SECRET': ...,
    'SOCIAL_AUTH_FACEBOOK_KEY': ...,
    'SOCIAL_AUTH_FACEBOOK_SECRET': ...,
    'FB_TEST_EMAIL': ...,
    'FB_TEST_PASS': ...,

    #Not essential for testing, but can be filled in for advanced usage
    ...
}

Overriding/Setting Secrets

However, if you wish to override these credentials or add login credentials for a school which requires a client secret, you may add your key/value pair to semesterly/sensitive.py. This file is gitignored and will be kept private so you can safely store the private information you wish within this file. It should have a format indentical to SECRETS above and in semesterly/dev_credentials.py.

Using Secrets

In order to properly access a secret from anywhere within the code, simply import the get_secret function and use it to access the secret by key::

from semesterly.settings import get_secret
hashids = Hashids(salt=get_secret('HASHING_SALT'))

This will check the following locations for the secret (in order, using the first value it finds), throwing an error if it does not find the key at all:

1. Check OS environment variables
2. Check ``semesterly/sensitive.py``
3. Default to ``semesterly/dev_credentials.py``
4. Error

@noahpresler noahpresler changed the base branch from master to staging August 3, 2017 20:20
@noahpresler noahpresler changed the title Secrets removal Remove client secrets from core code Aug 4, 2017
Copy link
Collaborator

@felixzhuologist felixzhuologist left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe just clean up settings.py first, but other than that looks good

@felixzhuologist
Copy link
Collaborator

lgtm

@felixzhuologist
Copy link
Collaborator

felixzhuologist commented Aug 4, 2017

LGTM

@noahpresler noahpresler merged commit 0ded903 into staging Aug 4, 2017
@noahpresler noahpresler deleted the secrets_removal branch August 4, 2017 17:03
@noahpresler noahpresler mentioned this pull request Aug 5, 2017
noahpresler added a commit that referenced this pull request Aug 5, 2017
* implemented course vectorizers under searches/utils

* implemented course vectorizers under searches/utils

* implemented relevance scoring

* implemented relevance scoring

* add get_relevant_search() method returning course objs

* add get_relevant_search() method returning course objs

* implemented stemming / ngram

* implemented stemming / ngram

* test case added

* stemming/acronym/weighting implemented

* evaluation functions implemented

* evaluation functions implemented

* objectified search utility functions

* objectified search utility functions

* Course clustering with LDA

* Course clustering with LDA

* optimized advance search

* optimized advance search

* refactoring codes

* refactoring codes

* rebased from staging

* rebased from staging

* satisfied linting

* satisfied linting

* ingested/digested jhu and modified search flow

* temp

* rebased from staging origin, still need vectorized search migration

* implemented course vectorizers under searches/utils

* implemented course vectorizers under searches/utils

* implemented relevance scoring

* implemented relevance scoring

* add get_relevant_search() method returning course objs

* add get_relevant_search() method returning course objs

* implemented stemming / ngram

* implemented stemming / ngram

* test case added

* stemming/acronym/weighting implemented

* stemming/acronym/weighting implemented

* evaluation functions implemented

* evaluation functions implemented

* objectified search utility functions

* objectified search utility functions

* Course clustering with LDA

* Course clustering with LDA

* optimized advance search

* optimized advance search

* refactoring codes

* refactoring codes

* rebased from staging

* rebased from staging

* satisfied linting

* satisfied linting

* ingested/digested jhu and modified search flow

* temp

* rebased from staging origin, still need vectorized search migration

* do not store count vectorizer obj

* removed pickle file

* revised unit test

* revised unit test

* prevented from crashing when ingesting

* rebased from master

* removed non-using part of the code for now

* modified requirements and cv loader

* updated upon review

* fixed to render searcher obj at app initialization

* resolved migration issue

* rebased and print exception

* modified scoring / strip() query

* resolved dependency issue

* handle duplicates

* changed to use static english dictionary count vectorizer for search.

* added docstrings

* modify license description, formatting and searches.rst description

* merge migrations

* fix tests for vectorized search

* expand on documentation

* cleanup webpack

* cleanup scripts

* fix update_semesterfield importing in migrations

* cleanup root

* update package.json

* provide cause of internal errors by setting debug to true

* fix tests

* remove documentation legacy

* move dict out of root

* umich parser pep8 compliance

* salisbury pep8

* JHU pep8

JHU course evals access db directly and are not pep8 compliant.
80 char limit is not enforced.

* UMD pep8 compliance and refactor

* gw instructor list limit

* vandy pep8 compliance and refactor, UNCOMPLETE

* Validator pep8 compliance

Almost full compliance. Still debating on whether 79 char line limit is worth
making 99 char limit.

* Style compliance for Queens parser

* style complianace for digestor, uncomplete

* working on refactor

* Conitnuing with refactor and pep8 compliance

:plane: to Israel!

* working on dictionary term/semester filtering

* fixed parenthesis syntax error in ingestor

* removed ingest_offerings and changed to ingest_meeting

* removed ingest_offering

* Ingestor refactor

Made detailed docstrings,
created internal _get function and removed getchain
Changed ALL_KEYS to _ALL_KEYS

* Cleaned ingestor cleandict and renamed to clean

+ general refactor of validator but not complete yet

* method reordering in gw parser

* add pep8 newline

* Various refactors

- filter years and terms command line arg and filtering

* spaces instead of tabs in data pipeline json schemas

* removed unneccessary import

* fixed issue created by new years and terms filtering scheme

* style update for commands argparser file

* working on conforming jhu evals to data pipeline

* fixed pep8 line overflow error in hacky way

* changed wording in comment

* Added eval to ingestion command

NOTE: digest command is temporarily broken.

- removed skip_shallow_duplicates from ingest cmd args
-- NOTE: this could very well cause unforseen issues with peoplesoft courses hiding/overriden classes!
- created --type cmd arg for ingestion
-- had to hack in a way to use new_{course, textbook, eval}_parsers, to be changed on deprecation of school_mappers.py
- changed hide_progress_bar to progress_bar (logic flip)

* hopkins evals to pipeline

* Added Vandy/Hopkins evals to data pipeline

NOTE: digestion command is still broken due to argparse errors

- Added evals to ingestor + validator
- tweaked ingestion cmd args some more
- pep8 compliance

* removed extra lines from supervisord config file

* tabs to spaces in logger

* created new exceptions modules, moved some functionality to utils module

* function out complex resolutions

* moved and documented external_utils to utils

* removed invalid reference to internal_utils

* some library reference refactors

* Created JSONWriter in logger but not yet used in code base

* JHU ingestion starts without error

* Refactor Tracker

- Seperated Viewer and Tracker by file and made better distinction.
- used @Property to clean up Tracker class
- Factored out Counter and TimeDistribution into their own Viewer classes
- Create Bus-like protocol so Viewers can filter Tracker broadcasts

* track status and better docstrings

* cleaned up validatoin schema loading

* load_schemas once only in Validator

* Refactor and cleanup validator

* changed parser_library to library

* The big move 🏠 🚚💨 🏠

- moved all parsing functionality into its own module
- created global variable to hold parsing directory in settings.py
- updated school mappers
- BROKEN: uoft was not moved and must be dealt with later, digestion command is still untested

* delete deprecated/moved/redone files b/c git is dumb

* moved makeschool to parsing module

* Trimmed names of files and parsers ✂️

- changed <school}_<pytpe>.py to <ptype>.py
- updated school_mappers to handle new format
- no longer need each type of parser in school directory if unused
- some pep8 things

* tabs to spaces in parsing commands

* Refactored pipeline commands

- use format strings to handle defaults in args
- changed format of progress bar and cleaned up
- started refactor of digestor

* ❗❗ remove parsing files from scripts again... 😡

* Added vectorizer back to digest command

- would like to move this to its own command in due course.

* partial refactor of digestor + bugfix in digest diff

* uncomment .eslintrc.js

* added dummy VALID_SCHOOLS to school_mappers to pass tests

* added uoft to VALID_SCHOOLS even though parser is not active in order to pass tests

* Sort'em 🔤

This should help with conflicts and duplications in these files

* License and registration please 👮

Added Open GPL3 license to parsing directory files with # comment style.

* Fix npm run watch

* add/update testing/contrib docs (#979)

* added everything back (#980)

* Remove client secrets from core code (#981)

* move hashids to local_settings.py

* move all secrets to sensitive.py

* add encrypted secrets as environment variables for travis

* Correct global vars indentation in travis.yml

* use environment variables via travisci web client instead

* handle secret key before imports

* provide dev credentials

* use SemesterlyTest Facebook App credentials

* add docs for secrets

* uncomment out try catch for error printing on get_secret

* Parsing/docs edit (#983)

* Added DataUpdate object to parsing models

- Created DataUpdate object to track last updated status per schools per semester
- Modified SchoolList view and commented out assertNone in associated test
- Used JSONStreamWriter to write ingestor and save meta data for digestor
to read and be able to update DataUpdate objects per semester
- General cleanup as well
- Created Hoarder as tracker viewer to accumulate term, dept, instr data through parse

* removed Updates completely and updated tests to use DataUpdate

* doc and docstring edits
noahpresler added a commit that referenced this pull request Aug 5, 2017
* implemented course vectorizers under searches/utils

* implemented course vectorizers under searches/utils

* implemented relevance scoring

* implemented relevance scoring

* add get_relevant_search() method returning course objs

* add get_relevant_search() method returning course objs

* implemented stemming / ngram

* implemented stemming / ngram

* test case added

* stemming/acronym/weighting implemented

* evaluation functions implemented

* evaluation functions implemented

* objectified search utility functions

* objectified search utility functions

* Course clustering with LDA

* Course clustering with LDA

* optimized advance search

* optimized advance search

* refactoring codes

* refactoring codes

* rebased from staging

* rebased from staging

* satisfied linting

* satisfied linting

* ingested/digested jhu and modified search flow

* temp

* rebased from staging origin, still need vectorized search migration

* implemented course vectorizers under searches/utils

* implemented course vectorizers under searches/utils

* implemented relevance scoring

* implemented relevance scoring

* add get_relevant_search() method returning course objs

* add get_relevant_search() method returning course objs

* implemented stemming / ngram

* implemented stemming / ngram

* test case added

* stemming/acronym/weighting implemented

* stemming/acronym/weighting implemented

* evaluation functions implemented

* evaluation functions implemented

* objectified search utility functions

* objectified search utility functions

* Course clustering with LDA

* Course clustering with LDA

* optimized advance search

* optimized advance search

* refactoring codes

* refactoring codes

* rebased from staging

* rebased from staging

* satisfied linting

* satisfied linting

* ingested/digested jhu and modified search flow

* temp

* rebased from staging origin, still need vectorized search migration

* do not store count vectorizer obj

* removed pickle file

* revised unit test

* revised unit test

* prevented from crashing when ingesting

* rebased from master

* removed non-using part of the code for now

* modified requirements and cv loader

* updated upon review

* fixed to render searcher obj at app initialization

* resolved migration issue

* rebased and print exception

* modified scoring / strip() query

* resolved dependency issue

* handle duplicates

* changed to use static english dictionary count vectorizer for search.

* added docstrings

* modify license description, formatting and searches.rst description

* merge migrations

* fix tests for vectorized search

* expand on documentation

* cleanup webpack

* cleanup scripts

* fix update_semesterfield importing in migrations

* cleanup root

* update package.json

* provide cause of internal errors by setting debug to true

* fix tests

* remove documentation legacy

* move dict out of root

* umich parser pep8 compliance

* salisbury pep8

* JHU pep8

JHU course evals access db directly and are not pep8 compliant.
80 char limit is not enforced.

* UMD pep8 compliance and refactor

* gw instructor list limit

* vandy pep8 compliance and refactor, UNCOMPLETE

* Validator pep8 compliance

Almost full compliance. Still debating on whether 79 char line limit is worth
making 99 char limit.

* Style compliance for Queens parser

* style complianace for digestor, uncomplete

* working on refactor

* Conitnuing with refactor and pep8 compliance

:plane: to Israel!

* working on dictionary term/semester filtering

* fixed parenthesis syntax error in ingestor

* removed ingest_offerings and changed to ingest_meeting

* removed ingest_offering

* Ingestor refactor

Made detailed docstrings,
created internal _get function and removed getchain
Changed ALL_KEYS to _ALL_KEYS

* Cleaned ingestor cleandict and renamed to clean

+ general refactor of validator but not complete yet

* method reordering in gw parser

* add pep8 newline

* Various refactors

- filter years and terms command line arg and filtering

* spaces instead of tabs in data pipeline json schemas

* removed unneccessary import

* fixed issue created by new years and terms filtering scheme

* style update for commands argparser file

* working on conforming jhu evals to data pipeline

* fixed pep8 line overflow error in hacky way

* changed wording in comment

* Added eval to ingestion command

NOTE: digest command is temporarily broken.

- removed skip_shallow_duplicates from ingest cmd args
-- NOTE: this could very well cause unforseen issues with peoplesoft courses hiding/overriden classes!
- created --type cmd arg for ingestion
-- had to hack in a way to use new_{course, textbook, eval}_parsers, to be changed on deprecation of school_mappers.py
- changed hide_progress_bar to progress_bar (logic flip)

* hopkins evals to pipeline

* Added Vandy/Hopkins evals to data pipeline

NOTE: digestion command is still broken due to argparse errors

- Added evals to ingestor + validator
- tweaked ingestion cmd args some more
- pep8 compliance

* removed extra lines from supervisord config file

* tabs to spaces in logger

* created new exceptions modules, moved some functionality to utils module

* function out complex resolutions

* moved and documented external_utils to utils

* removed invalid reference to internal_utils

* some library reference refactors

* Created JSONWriter in logger but not yet used in code base

* JHU ingestion starts without error

* Refactor Tracker

- Seperated Viewer and Tracker by file and made better distinction.
- used @Property to clean up Tracker class
- Factored out Counter and TimeDistribution into their own Viewer classes
- Create Bus-like protocol so Viewers can filter Tracker broadcasts

* track status and better docstrings

* cleaned up validatoin schema loading

* load_schemas once only in Validator

* Refactor and cleanup validator

* changed parser_library to library

* The big move 🏠 🚚💨 🏠

- moved all parsing functionality into its own module
- created global variable to hold parsing directory in settings.py
- updated school mappers
- BROKEN: uoft was not moved and must be dealt with later, digestion command is still untested

* delete deprecated/moved/redone files b/c git is dumb

* moved makeschool to parsing module

* Trimmed names of files and parsers ✂️

- changed <school}_<pytpe>.py to <ptype>.py
- updated school_mappers to handle new format
- no longer need each type of parser in school directory if unused
- some pep8 things

* tabs to spaces in parsing commands

* Refactored pipeline commands

- use format strings to handle defaults in args
- changed format of progress bar and cleaned up
- started refactor of digestor

* ❗❗ remove parsing files from scripts again... 😡

* Added vectorizer back to digest command

- would like to move this to its own command in due course.

* partial refactor of digestor + bugfix in digest diff

* uncomment .eslintrc.js

* added dummy VALID_SCHOOLS to school_mappers to pass tests

* added uoft to VALID_SCHOOLS even though parser is not active in order to pass tests

* Sort'em 🔤

This should help with conflicts and duplications in these files

* License and registration please 👮

Added Open GPL3 license to parsing directory files with # comment style.

* Fix npm run watch

* add/update testing/contrib docs (#979)

* added everything back (#980)

* Remove client secrets from core code (#981)

* move hashids to local_settings.py

* move all secrets to sensitive.py

* add encrypted secrets as environment variables for travis

* Correct global vars indentation in travis.yml

* use environment variables via travisci web client instead

* handle secret key before imports

* provide dev credentials

* use SemesterlyTest Facebook App credentials

* add docs for secrets

* uncomment out try catch for error printing on get_secret

* Parsing/docs edit (#983)

* Added DataUpdate object to parsing models

- Created DataUpdate object to track last updated status per schools per semester
- Modified SchoolList view and commented out assertNone in associated test
- Used JSONStreamWriter to write ingestor and save meta data for digestor
to read and be able to update DataUpdate objects per semester
- General cleanup as well
- Created Hoarder as tracker viewer to accumulate term, dept, instr data through parse

* removed Updates completely and updated tests to use DataUpdate

* doc and docstring edits

* dedupe list of dictionaries correctly (#990)

* dedupe list of dictionaries correctly

* fix typo
noahpresler added a commit that referenced this pull request Aug 5, 2017
* move hashids to local_settings.py

* move all secrets to sensitive.py

* add encrypted secrets as environment variables for travis

* Correct global vars indentation in travis.yml

* use environment variables via travisci web client instead

* handle secret key before imports

* provide dev credentials

* use SemesterlyTest Facebook App credentials

* add docs for secrets

* uncomment out try catch for error printing on get_secret
noahpresler added a commit that referenced this pull request Aug 5, 2017
* implemented course vectorizers under searches/utils

* implemented course vectorizers under searches/utils

* implemented relevance scoring

* implemented relevance scoring

* add get_relevant_search() method returning course objs

* add get_relevant_search() method returning course objs

* implemented stemming / ngram

* implemented stemming / ngram

* test case added

* stemming/acronym/weighting implemented

* evaluation functions implemented

* evaluation functions implemented

* objectified search utility functions

* objectified search utility functions

* Course clustering with LDA

* Course clustering with LDA

* optimized advance search

* optimized advance search

* refactoring codes

* refactoring codes

* rebased from staging

* rebased from staging

* satisfied linting

* satisfied linting

* ingested/digested jhu and modified search flow

* temp

* rebased from staging origin, still need vectorized search migration

* implemented course vectorizers under searches/utils

* implemented course vectorizers under searches/utils

* implemented relevance scoring

* implemented relevance scoring

* add get_relevant_search() method returning course objs

* add get_relevant_search() method returning course objs

* implemented stemming / ngram

* implemented stemming / ngram

* test case added

* stemming/acronym/weighting implemented

* stemming/acronym/weighting implemented

* evaluation functions implemented

* evaluation functions implemented

* objectified search utility functions

* objectified search utility functions

* Course clustering with LDA

* Course clustering with LDA

* optimized advance search

* optimized advance search

* refactoring codes

* refactoring codes

* rebased from staging

* rebased from staging

* satisfied linting

* satisfied linting

* ingested/digested jhu and modified search flow

* temp

* rebased from staging origin, still need vectorized search migration

* do not store count vectorizer obj

* removed pickle file

* revised unit test

* revised unit test

* prevented from crashing when ingesting

* rebased from master

* removed non-using part of the code for now

* modified requirements and cv loader

* updated upon review

* fixed to render searcher obj at app initialization

* resolved migration issue

* rebased and print exception

* modified scoring / strip() query

* resolved dependency issue

* handle duplicates

* changed to use static english dictionary count vectorizer for search.

* added docstrings

* modify license description, formatting and searches.rst description

* merge migrations

* fix tests for vectorized search

* expand on documentation

* cleanup webpack

* cleanup scripts

* fix update_semesterfield importing in migrations

* cleanup root

* update package.json

* provide cause of internal errors by setting debug to true

* fix tests

* remove documentation legacy

* move dict out of root

* umich parser pep8 compliance

* salisbury pep8

* JHU pep8

JHU course evals access db directly and are not pep8 compliant.
80 char limit is not enforced.

* UMD pep8 compliance and refactor

* gw instructor list limit

* vandy pep8 compliance and refactor, UNCOMPLETE

* Validator pep8 compliance

Almost full compliance. Still debating on whether 79 char line limit is worth
making 99 char limit.

* Style compliance for Queens parser

* style complianace for digestor, uncomplete

* working on refactor

* Conitnuing with refactor and pep8 compliance

:plane: to Israel!

* working on dictionary term/semester filtering

* fixed parenthesis syntax error in ingestor

* removed ingest_offerings and changed to ingest_meeting

* removed ingest_offering

* Ingestor refactor

Made detailed docstrings,
created internal _get function and removed getchain
Changed ALL_KEYS to _ALL_KEYS

* Cleaned ingestor cleandict and renamed to clean

+ general refactor of validator but not complete yet

* method reordering in gw parser

* add pep8 newline

* Various refactors

- filter years and terms command line arg and filtering

* spaces instead of tabs in data pipeline json schemas

* removed unneccessary import

* fixed issue created by new years and terms filtering scheme

* style update for commands argparser file

* working on conforming jhu evals to data pipeline

* fixed pep8 line overflow error in hacky way

* changed wording in comment

* Added eval to ingestion command

NOTE: digest command is temporarily broken.

- removed skip_shallow_duplicates from ingest cmd args
-- NOTE: this could very well cause unforseen issues with peoplesoft courses hiding/overriden classes!
- created --type cmd arg for ingestion
-- had to hack in a way to use new_{course, textbook, eval}_parsers, to be changed on deprecation of school_mappers.py
- changed hide_progress_bar to progress_bar (logic flip)

* hopkins evals to pipeline

* Added Vandy/Hopkins evals to data pipeline

NOTE: digestion command is still broken due to argparse errors

- Added evals to ingestor + validator
- tweaked ingestion cmd args some more
- pep8 compliance

* removed extra lines from supervisord config file

* tabs to spaces in logger

* created new exceptions modules, moved some functionality to utils module

* function out complex resolutions

* moved and documented external_utils to utils

* removed invalid reference to internal_utils

* some library reference refactors

* Created JSONWriter in logger but not yet used in code base

* JHU ingestion starts without error

* Refactor Tracker

- Seperated Viewer and Tracker by file and made better distinction.
- used @Property to clean up Tracker class
- Factored out Counter and TimeDistribution into their own Viewer classes
- Create Bus-like protocol so Viewers can filter Tracker broadcasts

* track status and better docstrings

* cleaned up validatoin schema loading

* load_schemas once only in Validator

* Refactor and cleanup validator

* changed parser_library to library

* The big move 🏠 🚚💨 🏠

- moved all parsing functionality into its own module
- created global variable to hold parsing directory in settings.py
- updated school mappers
- BROKEN: uoft was not moved and must be dealt with later, digestion command is still untested

* delete deprecated/moved/redone files b/c git is dumb

* moved makeschool to parsing module

* Trimmed names of files and parsers ✂️

- changed <school}_<pytpe>.py to <ptype>.py
- updated school_mappers to handle new format
- no longer need each type of parser in school directory if unused
- some pep8 things

* tabs to spaces in parsing commands

* Refactored pipeline commands

- use format strings to handle defaults in args
- changed format of progress bar and cleaned up
- started refactor of digestor

* ❗❗ remove parsing files from scripts again... 😡

* Added vectorizer back to digest command

- would like to move this to its own command in due course.

* partial refactor of digestor + bugfix in digest diff

* uncomment .eslintrc.js

* added dummy VALID_SCHOOLS to school_mappers to pass tests

* added uoft to VALID_SCHOOLS even though parser is not active in order to pass tests

* Sort'em 🔤

This should help with conflicts and duplications in these files

* License and registration please 👮

Added Open GPL3 license to parsing directory files with # comment style.

* Fix npm run watch

* add/update testing/contrib docs (#979)

* added everything back (#980)

* Remove client secrets from core code (#981)

* move hashids to local_settings.py

* move all secrets to sensitive.py

* add encrypted secrets as environment variables for travis

* Correct global vars indentation in travis.yml

* use environment variables via travisci web client instead

* handle secret key before imports

* provide dev credentials

* use SemesterlyTest Facebook App credentials

* add docs for secrets

* uncomment out try catch for error printing on get_secret

* Parsing/docs edit (#983)

* Added DataUpdate object to parsing models

- Created DataUpdate object to track last updated status per schools per semester
- Modified SchoolList view and commented out assertNone in associated test
- Used JSONStreamWriter to write ingestor and save meta data for digestor
to read and be able to update DataUpdate objects per semester
- General cleanup as well
- Created Hoarder as tracker viewer to accumulate term, dept, instr data through parse

* removed Updates completely and updated tests to use DataUpdate

* doc and docstring edits
noahpresler added a commit that referenced this pull request Aug 5, 2017
* implemented course vectorizers under searches/utils

* implemented course vectorizers under searches/utils

* implemented relevance scoring

* implemented relevance scoring

* add get_relevant_search() method returning course objs

* add get_relevant_search() method returning course objs

* implemented stemming / ngram

* implemented stemming / ngram

* test case added

* stemming/acronym/weighting implemented

* evaluation functions implemented

* evaluation functions implemented

* objectified search utility functions

* objectified search utility functions

* Course clustering with LDA

* Course clustering with LDA

* optimized advance search

* optimized advance search

* refactoring codes

* refactoring codes

* rebased from staging

* rebased from staging

* satisfied linting

* satisfied linting

* ingested/digested jhu and modified search flow

* temp

* rebased from staging origin, still need vectorized search migration

* implemented course vectorizers under searches/utils

* implemented course vectorizers under searches/utils

* implemented relevance scoring

* implemented relevance scoring

* add get_relevant_search() method returning course objs

* add get_relevant_search() method returning course objs

* implemented stemming / ngram

* implemented stemming / ngram

* test case added

* stemming/acronym/weighting implemented

* stemming/acronym/weighting implemented

* evaluation functions implemented

* evaluation functions implemented

* objectified search utility functions

* objectified search utility functions

* Course clustering with LDA

* Course clustering with LDA

* optimized advance search

* optimized advance search

* refactoring codes

* refactoring codes

* rebased from staging

* rebased from staging

* satisfied linting

* satisfied linting

* ingested/digested jhu and modified search flow

* temp

* rebased from staging origin, still need vectorized search migration

* do not store count vectorizer obj

* removed pickle file

* revised unit test

* revised unit test

* prevented from crashing when ingesting

* rebased from master

* removed non-using part of the code for now

* modified requirements and cv loader

* updated upon review

* fixed to render searcher obj at app initialization

* resolved migration issue

* rebased and print exception

* modified scoring / strip() query

* resolved dependency issue

* handle duplicates

* changed to use static english dictionary count vectorizer for search.

* added docstrings

* modify license description, formatting and searches.rst description

* merge migrations

* fix tests for vectorized search

* expand on documentation

* cleanup webpack

* cleanup scripts

* fix update_semesterfield importing in migrations

* cleanup root

* update package.json

* provide cause of internal errors by setting debug to true

* fix tests

* remove documentation legacy

* move dict out of root

* umich parser pep8 compliance

* salisbury pep8

* JHU pep8

JHU course evals access db directly and are not pep8 compliant.
80 char limit is not enforced.

* UMD pep8 compliance and refactor

* gw instructor list limit

* vandy pep8 compliance and refactor, UNCOMPLETE

* Validator pep8 compliance

Almost full compliance. Still debating on whether 79 char line limit is worth
making 99 char limit.

* Style compliance for Queens parser

* style complianace for digestor, uncomplete

* working on refactor

* Conitnuing with refactor and pep8 compliance

:plane: to Israel!

* working on dictionary term/semester filtering

* fixed parenthesis syntax error in ingestor

* removed ingest_offerings and changed to ingest_meeting

* removed ingest_offering

* Ingestor refactor

Made detailed docstrings,
created internal _get function and removed getchain
Changed ALL_KEYS to _ALL_KEYS

* Cleaned ingestor cleandict and renamed to clean

+ general refactor of validator but not complete yet

* method reordering in gw parser

* add pep8 newline

* Various refactors

- filter years and terms command line arg and filtering

* spaces instead of tabs in data pipeline json schemas

* removed unneccessary import

* fixed issue created by new years and terms filtering scheme

* style update for commands argparser file

* working on conforming jhu evals to data pipeline

* fixed pep8 line overflow error in hacky way

* changed wording in comment

* Added eval to ingestion command

NOTE: digest command is temporarily broken.

- removed skip_shallow_duplicates from ingest cmd args
-- NOTE: this could very well cause unforseen issues with peoplesoft courses hiding/overriden classes!
- created --type cmd arg for ingestion
-- had to hack in a way to use new_{course, textbook, eval}_parsers, to be changed on deprecation of school_mappers.py
- changed hide_progress_bar to progress_bar (logic flip)

* hopkins evals to pipeline

* Added Vandy/Hopkins evals to data pipeline

NOTE: digestion command is still broken due to argparse errors

- Added evals to ingestor + validator
- tweaked ingestion cmd args some more
- pep8 compliance

* removed extra lines from supervisord config file

* tabs to spaces in logger

* created new exceptions modules, moved some functionality to utils module

* function out complex resolutions

* moved and documented external_utils to utils

* removed invalid reference to internal_utils

* some library reference refactors

* Created JSONWriter in logger but not yet used in code base

* JHU ingestion starts without error

* Refactor Tracker

- Seperated Viewer and Tracker by file and made better distinction.
- used @Property to clean up Tracker class
- Factored out Counter and TimeDistribution into their own Viewer classes
- Create Bus-like protocol so Viewers can filter Tracker broadcasts

* track status and better docstrings

* cleaned up validatoin schema loading

* load_schemas once only in Validator

* Refactor and cleanup validator

* changed parser_library to library

* The big move 🏠 🚚💨 🏠

- moved all parsing functionality into its own module
- created global variable to hold parsing directory in settings.py
- updated school mappers
- BROKEN: uoft was not moved and must be dealt with later, digestion command is still untested

* delete deprecated/moved/redone files b/c git is dumb

* moved makeschool to parsing module

* Trimmed names of files and parsers ✂️

- changed <school}_<pytpe>.py to <ptype>.py
- updated school_mappers to handle new format
- no longer need each type of parser in school directory if unused
- some pep8 things

* tabs to spaces in parsing commands

* Refactored pipeline commands

- use format strings to handle defaults in args
- changed format of progress bar and cleaned up
- started refactor of digestor

* ❗❗ remove parsing files from scripts again... 😡

* Added vectorizer back to digest command

- would like to move this to its own command in due course.

* partial refactor of digestor + bugfix in digest diff

* uncomment .eslintrc.js

* added dummy VALID_SCHOOLS to school_mappers to pass tests

* added uoft to VALID_SCHOOLS even though parser is not active in order to pass tests

* Sort'em 🔤

This should help with conflicts and duplications in these files

* License and registration please 👮

Added Open GPL3 license to parsing directory files with # comment style.

* Fix npm run watch

* add/update testing/contrib docs (#979)

* added everything back (#980)

* Remove client secrets from core code (#981)

* move hashids to local_settings.py

* move all secrets to sensitive.py

* add encrypted secrets as environment variables for travis

* Correct global vars indentation in travis.yml

* use environment variables via travisci web client instead

* handle secret key before imports

* provide dev credentials

* use SemesterlyTest Facebook App credentials

* add docs for secrets

* uncomment out try catch for error printing on get_secret

* Parsing/docs edit (#983)

* Added DataUpdate object to parsing models

- Created DataUpdate object to track last updated status per schools per semester
- Modified SchoolList view and commented out assertNone in associated test
- Used JSONStreamWriter to write ingestor and save meta data for digestor
to read and be able to update DataUpdate objects per semester
- General cleanup as well
- Created Hoarder as tracker viewer to accumulate term, dept, instr data through parse

* removed Updates completely and updated tests to use DataUpdate

* doc and docstring edits

* dedupe list of dictionaries correctly (#990)

* dedupe list of dictionaries correctly

* fix typo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants