-
Notifications
You must be signed in to change notification settings - Fork 179
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove client secrets from core code #981
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
felixzhuologist
approved these changes
Aug 4, 2017
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe just clean up settings.py first, but other than that looks good
lgtm |
LGTM |
Merged
noahpresler
added a commit
that referenced
this pull request
Aug 5, 2017
* implemented course vectorizers under searches/utils * implemented course vectorizers under searches/utils * implemented relevance scoring * implemented relevance scoring * add get_relevant_search() method returning course objs * add get_relevant_search() method returning course objs * implemented stemming / ngram * implemented stemming / ngram * test case added * stemming/acronym/weighting implemented * evaluation functions implemented * evaluation functions implemented * objectified search utility functions * objectified search utility functions * Course clustering with LDA * Course clustering with LDA * optimized advance search * optimized advance search * refactoring codes * refactoring codes * rebased from staging * rebased from staging * satisfied linting * satisfied linting * ingested/digested jhu and modified search flow * temp * rebased from staging origin, still need vectorized search migration * implemented course vectorizers under searches/utils * implemented course vectorizers under searches/utils * implemented relevance scoring * implemented relevance scoring * add get_relevant_search() method returning course objs * add get_relevant_search() method returning course objs * implemented stemming / ngram * implemented stemming / ngram * test case added * stemming/acronym/weighting implemented * stemming/acronym/weighting implemented * evaluation functions implemented * evaluation functions implemented * objectified search utility functions * objectified search utility functions * Course clustering with LDA * Course clustering with LDA * optimized advance search * optimized advance search * refactoring codes * refactoring codes * rebased from staging * rebased from staging * satisfied linting * satisfied linting * ingested/digested jhu and modified search flow * temp * rebased from staging origin, still need vectorized search migration * do not store count vectorizer obj * removed pickle file * revised unit test * revised unit test * prevented from crashing when ingesting * rebased from master * removed non-using part of the code for now * modified requirements and cv loader * updated upon review * fixed to render searcher obj at app initialization * resolved migration issue * rebased and print exception * modified scoring / strip() query * resolved dependency issue * handle duplicates * changed to use static english dictionary count vectorizer for search. * added docstrings * modify license description, formatting and searches.rst description * merge migrations * fix tests for vectorized search * expand on documentation * cleanup webpack * cleanup scripts * fix update_semesterfield importing in migrations * cleanup root * update package.json * provide cause of internal errors by setting debug to true * fix tests * remove documentation legacy * move dict out of root * umich parser pep8 compliance * salisbury pep8 * JHU pep8 JHU course evals access db directly and are not pep8 compliant. 80 char limit is not enforced. * UMD pep8 compliance and refactor * gw instructor list limit * vandy pep8 compliance and refactor, UNCOMPLETE * Validator pep8 compliance Almost full compliance. Still debating on whether 79 char line limit is worth making 99 char limit. * Style compliance for Queens parser * style complianace for digestor, uncomplete * working on refactor * Conitnuing with refactor and pep8 compliance :plane: to Israel! * working on dictionary term/semester filtering * fixed parenthesis syntax error in ingestor * removed ingest_offerings and changed to ingest_meeting * removed ingest_offering * Ingestor refactor Made detailed docstrings, created internal _get function and removed getchain Changed ALL_KEYS to _ALL_KEYS * Cleaned ingestor cleandict and renamed to clean + general refactor of validator but not complete yet * method reordering in gw parser * add pep8 newline * Various refactors - filter years and terms command line arg and filtering * spaces instead of tabs in data pipeline json schemas * removed unneccessary import * fixed issue created by new years and terms filtering scheme * style update for commands argparser file * working on conforming jhu evals to data pipeline * fixed pep8 line overflow error in hacky way * changed wording in comment * Added eval to ingestion command NOTE: digest command is temporarily broken. - removed skip_shallow_duplicates from ingest cmd args -- NOTE: this could very well cause unforseen issues with peoplesoft courses hiding/overriden classes! - created --type cmd arg for ingestion -- had to hack in a way to use new_{course, textbook, eval}_parsers, to be changed on deprecation of school_mappers.py - changed hide_progress_bar to progress_bar (logic flip) * hopkins evals to pipeline * Added Vandy/Hopkins evals to data pipeline NOTE: digestion command is still broken due to argparse errors - Added evals to ingestor + validator - tweaked ingestion cmd args some more - pep8 compliance * removed extra lines from supervisord config file * tabs to spaces in logger * created new exceptions modules, moved some functionality to utils module * function out complex resolutions * moved and documented external_utils to utils * removed invalid reference to internal_utils * some library reference refactors * Created JSONWriter in logger but not yet used in code base * JHU ingestion starts without error * Refactor Tracker - Seperated Viewer and Tracker by file and made better distinction. - used @Property to clean up Tracker class - Factored out Counter and TimeDistribution into their own Viewer classes - Create Bus-like protocol so Viewers can filter Tracker broadcasts * track status and better docstrings * cleaned up validatoin schema loading * load_schemas once only in Validator * Refactor and cleanup validator * changed parser_library to library * The big move 🏠 🚚💨 🏠 - moved all parsing functionality into its own module - created global variable to hold parsing directory in settings.py - updated school mappers - BROKEN: uoft was not moved and must be dealt with later, digestion command is still untested * delete deprecated/moved/redone files b/c git is dumb * moved makeschool to parsing module * Trimmed names of files and parsers ✂️ - changed <school}_<pytpe>.py to <ptype>.py - updated school_mappers to handle new format - no longer need each type of parser in school directory if unused - some pep8 things * tabs to spaces in parsing commands * Refactored pipeline commands - use format strings to handle defaults in args - changed format of progress bar and cleaned up - started refactor of digestor * ❗❗ remove parsing files from scripts again... 😡 * Added vectorizer back to digest command - would like to move this to its own command in due course. * partial refactor of digestor + bugfix in digest diff * uncomment .eslintrc.js * added dummy VALID_SCHOOLS to school_mappers to pass tests * added uoft to VALID_SCHOOLS even though parser is not active in order to pass tests * Sort'em 🔤 This should help with conflicts and duplications in these files * License and registration please 👮 Added Open GPL3 license to parsing directory files with # comment style. * Fix npm run watch * add/update testing/contrib docs (#979) * added everything back (#980) * Remove client secrets from core code (#981) * move hashids to local_settings.py * move all secrets to sensitive.py * add encrypted secrets as environment variables for travis * Correct global vars indentation in travis.yml * use environment variables via travisci web client instead * handle secret key before imports * provide dev credentials * use SemesterlyTest Facebook App credentials * add docs for secrets * uncomment out try catch for error printing on get_secret * Parsing/docs edit (#983) * Added DataUpdate object to parsing models - Created DataUpdate object to track last updated status per schools per semester - Modified SchoolList view and commented out assertNone in associated test - Used JSONStreamWriter to write ingestor and save meta data for digestor to read and be able to update DataUpdate objects per semester - General cleanup as well - Created Hoarder as tracker viewer to accumulate term, dept, instr data through parse * removed Updates completely and updated tests to use DataUpdate * doc and docstring edits
noahpresler
added a commit
that referenced
this pull request
Aug 5, 2017
* implemented course vectorizers under searches/utils * implemented course vectorizers under searches/utils * implemented relevance scoring * implemented relevance scoring * add get_relevant_search() method returning course objs * add get_relevant_search() method returning course objs * implemented stemming / ngram * implemented stemming / ngram * test case added * stemming/acronym/weighting implemented * evaluation functions implemented * evaluation functions implemented * objectified search utility functions * objectified search utility functions * Course clustering with LDA * Course clustering with LDA * optimized advance search * optimized advance search * refactoring codes * refactoring codes * rebased from staging * rebased from staging * satisfied linting * satisfied linting * ingested/digested jhu and modified search flow * temp * rebased from staging origin, still need vectorized search migration * implemented course vectorizers under searches/utils * implemented course vectorizers under searches/utils * implemented relevance scoring * implemented relevance scoring * add get_relevant_search() method returning course objs * add get_relevant_search() method returning course objs * implemented stemming / ngram * implemented stemming / ngram * test case added * stemming/acronym/weighting implemented * stemming/acronym/weighting implemented * evaluation functions implemented * evaluation functions implemented * objectified search utility functions * objectified search utility functions * Course clustering with LDA * Course clustering with LDA * optimized advance search * optimized advance search * refactoring codes * refactoring codes * rebased from staging * rebased from staging * satisfied linting * satisfied linting * ingested/digested jhu and modified search flow * temp * rebased from staging origin, still need vectorized search migration * do not store count vectorizer obj * removed pickle file * revised unit test * revised unit test * prevented from crashing when ingesting * rebased from master * removed non-using part of the code for now * modified requirements and cv loader * updated upon review * fixed to render searcher obj at app initialization * resolved migration issue * rebased and print exception * modified scoring / strip() query * resolved dependency issue * handle duplicates * changed to use static english dictionary count vectorizer for search. * added docstrings * modify license description, formatting and searches.rst description * merge migrations * fix tests for vectorized search * expand on documentation * cleanup webpack * cleanup scripts * fix update_semesterfield importing in migrations * cleanup root * update package.json * provide cause of internal errors by setting debug to true * fix tests * remove documentation legacy * move dict out of root * umich parser pep8 compliance * salisbury pep8 * JHU pep8 JHU course evals access db directly and are not pep8 compliant. 80 char limit is not enforced. * UMD pep8 compliance and refactor * gw instructor list limit * vandy pep8 compliance and refactor, UNCOMPLETE * Validator pep8 compliance Almost full compliance. Still debating on whether 79 char line limit is worth making 99 char limit. * Style compliance for Queens parser * style complianace for digestor, uncomplete * working on refactor * Conitnuing with refactor and pep8 compliance :plane: to Israel! * working on dictionary term/semester filtering * fixed parenthesis syntax error in ingestor * removed ingest_offerings and changed to ingest_meeting * removed ingest_offering * Ingestor refactor Made detailed docstrings, created internal _get function and removed getchain Changed ALL_KEYS to _ALL_KEYS * Cleaned ingestor cleandict and renamed to clean + general refactor of validator but not complete yet * method reordering in gw parser * add pep8 newline * Various refactors - filter years and terms command line arg and filtering * spaces instead of tabs in data pipeline json schemas * removed unneccessary import * fixed issue created by new years and terms filtering scheme * style update for commands argparser file * working on conforming jhu evals to data pipeline * fixed pep8 line overflow error in hacky way * changed wording in comment * Added eval to ingestion command NOTE: digest command is temporarily broken. - removed skip_shallow_duplicates from ingest cmd args -- NOTE: this could very well cause unforseen issues with peoplesoft courses hiding/overriden classes! - created --type cmd arg for ingestion -- had to hack in a way to use new_{course, textbook, eval}_parsers, to be changed on deprecation of school_mappers.py - changed hide_progress_bar to progress_bar (logic flip) * hopkins evals to pipeline * Added Vandy/Hopkins evals to data pipeline NOTE: digestion command is still broken due to argparse errors - Added evals to ingestor + validator - tweaked ingestion cmd args some more - pep8 compliance * removed extra lines from supervisord config file * tabs to spaces in logger * created new exceptions modules, moved some functionality to utils module * function out complex resolutions * moved and documented external_utils to utils * removed invalid reference to internal_utils * some library reference refactors * Created JSONWriter in logger but not yet used in code base * JHU ingestion starts without error * Refactor Tracker - Seperated Viewer and Tracker by file and made better distinction. - used @Property to clean up Tracker class - Factored out Counter and TimeDistribution into their own Viewer classes - Create Bus-like protocol so Viewers can filter Tracker broadcasts * track status and better docstrings * cleaned up validatoin schema loading * load_schemas once only in Validator * Refactor and cleanup validator * changed parser_library to library * The big move 🏠 🚚💨 🏠 - moved all parsing functionality into its own module - created global variable to hold parsing directory in settings.py - updated school mappers - BROKEN: uoft was not moved and must be dealt with later, digestion command is still untested * delete deprecated/moved/redone files b/c git is dumb * moved makeschool to parsing module * Trimmed names of files and parsers ✂️ - changed <school}_<pytpe>.py to <ptype>.py - updated school_mappers to handle new format - no longer need each type of parser in school directory if unused - some pep8 things * tabs to spaces in parsing commands * Refactored pipeline commands - use format strings to handle defaults in args - changed format of progress bar and cleaned up - started refactor of digestor * ❗❗ remove parsing files from scripts again... 😡 * Added vectorizer back to digest command - would like to move this to its own command in due course. * partial refactor of digestor + bugfix in digest diff * uncomment .eslintrc.js * added dummy VALID_SCHOOLS to school_mappers to pass tests * added uoft to VALID_SCHOOLS even though parser is not active in order to pass tests * Sort'em 🔤 This should help with conflicts and duplications in these files * License and registration please 👮 Added Open GPL3 license to parsing directory files with # comment style. * Fix npm run watch * add/update testing/contrib docs (#979) * added everything back (#980) * Remove client secrets from core code (#981) * move hashids to local_settings.py * move all secrets to sensitive.py * add encrypted secrets as environment variables for travis * Correct global vars indentation in travis.yml * use environment variables via travisci web client instead * handle secret key before imports * provide dev credentials * use SemesterlyTest Facebook App credentials * add docs for secrets * uncomment out try catch for error printing on get_secret * Parsing/docs edit (#983) * Added DataUpdate object to parsing models - Created DataUpdate object to track last updated status per schools per semester - Modified SchoolList view and commented out assertNone in associated test - Used JSONStreamWriter to write ingestor and save meta data for digestor to read and be able to update DataUpdate objects per semester - General cleanup as well - Created Hoarder as tracker viewer to accumulate term, dept, instr data through parse * removed Updates completely and updated tests to use DataUpdate * doc and docstring edits * dedupe list of dictionaries correctly (#990) * dedupe list of dictionaries correctly * fix typo
noahpresler
added a commit
that referenced
this pull request
Aug 5, 2017
* move hashids to local_settings.py * move all secrets to sensitive.py * add encrypted secrets as environment variables for travis * Correct global vars indentation in travis.yml * use environment variables via travisci web client instead * handle secret key before imports * provide dev credentials * use SemesterlyTest Facebook App credentials * add docs for secrets * uncomment out try catch for error printing on get_secret
noahpresler
added a commit
that referenced
this pull request
Aug 5, 2017
* implemented course vectorizers under searches/utils * implemented course vectorizers under searches/utils * implemented relevance scoring * implemented relevance scoring * add get_relevant_search() method returning course objs * add get_relevant_search() method returning course objs * implemented stemming / ngram * implemented stemming / ngram * test case added * stemming/acronym/weighting implemented * evaluation functions implemented * evaluation functions implemented * objectified search utility functions * objectified search utility functions * Course clustering with LDA * Course clustering with LDA * optimized advance search * optimized advance search * refactoring codes * refactoring codes * rebased from staging * rebased from staging * satisfied linting * satisfied linting * ingested/digested jhu and modified search flow * temp * rebased from staging origin, still need vectorized search migration * implemented course vectorizers under searches/utils * implemented course vectorizers under searches/utils * implemented relevance scoring * implemented relevance scoring * add get_relevant_search() method returning course objs * add get_relevant_search() method returning course objs * implemented stemming / ngram * implemented stemming / ngram * test case added * stemming/acronym/weighting implemented * stemming/acronym/weighting implemented * evaluation functions implemented * evaluation functions implemented * objectified search utility functions * objectified search utility functions * Course clustering with LDA * Course clustering with LDA * optimized advance search * optimized advance search * refactoring codes * refactoring codes * rebased from staging * rebased from staging * satisfied linting * satisfied linting * ingested/digested jhu and modified search flow * temp * rebased from staging origin, still need vectorized search migration * do not store count vectorizer obj * removed pickle file * revised unit test * revised unit test * prevented from crashing when ingesting * rebased from master * removed non-using part of the code for now * modified requirements and cv loader * updated upon review * fixed to render searcher obj at app initialization * resolved migration issue * rebased and print exception * modified scoring / strip() query * resolved dependency issue * handle duplicates * changed to use static english dictionary count vectorizer for search. * added docstrings * modify license description, formatting and searches.rst description * merge migrations * fix tests for vectorized search * expand on documentation * cleanup webpack * cleanup scripts * fix update_semesterfield importing in migrations * cleanup root * update package.json * provide cause of internal errors by setting debug to true * fix tests * remove documentation legacy * move dict out of root * umich parser pep8 compliance * salisbury pep8 * JHU pep8 JHU course evals access db directly and are not pep8 compliant. 80 char limit is not enforced. * UMD pep8 compliance and refactor * gw instructor list limit * vandy pep8 compliance and refactor, UNCOMPLETE * Validator pep8 compliance Almost full compliance. Still debating on whether 79 char line limit is worth making 99 char limit. * Style compliance for Queens parser * style complianace for digestor, uncomplete * working on refactor * Conitnuing with refactor and pep8 compliance :plane: to Israel! * working on dictionary term/semester filtering * fixed parenthesis syntax error in ingestor * removed ingest_offerings and changed to ingest_meeting * removed ingest_offering * Ingestor refactor Made detailed docstrings, created internal _get function and removed getchain Changed ALL_KEYS to _ALL_KEYS * Cleaned ingestor cleandict and renamed to clean + general refactor of validator but not complete yet * method reordering in gw parser * add pep8 newline * Various refactors - filter years and terms command line arg and filtering * spaces instead of tabs in data pipeline json schemas * removed unneccessary import * fixed issue created by new years and terms filtering scheme * style update for commands argparser file * working on conforming jhu evals to data pipeline * fixed pep8 line overflow error in hacky way * changed wording in comment * Added eval to ingestion command NOTE: digest command is temporarily broken. - removed skip_shallow_duplicates from ingest cmd args -- NOTE: this could very well cause unforseen issues with peoplesoft courses hiding/overriden classes! - created --type cmd arg for ingestion -- had to hack in a way to use new_{course, textbook, eval}_parsers, to be changed on deprecation of school_mappers.py - changed hide_progress_bar to progress_bar (logic flip) * hopkins evals to pipeline * Added Vandy/Hopkins evals to data pipeline NOTE: digestion command is still broken due to argparse errors - Added evals to ingestor + validator - tweaked ingestion cmd args some more - pep8 compliance * removed extra lines from supervisord config file * tabs to spaces in logger * created new exceptions modules, moved some functionality to utils module * function out complex resolutions * moved and documented external_utils to utils * removed invalid reference to internal_utils * some library reference refactors * Created JSONWriter in logger but not yet used in code base * JHU ingestion starts without error * Refactor Tracker - Seperated Viewer and Tracker by file and made better distinction. - used @Property to clean up Tracker class - Factored out Counter and TimeDistribution into their own Viewer classes - Create Bus-like protocol so Viewers can filter Tracker broadcasts * track status and better docstrings * cleaned up validatoin schema loading * load_schemas once only in Validator * Refactor and cleanup validator * changed parser_library to library * The big move 🏠 🚚💨 🏠 - moved all parsing functionality into its own module - created global variable to hold parsing directory in settings.py - updated school mappers - BROKEN: uoft was not moved and must be dealt with later, digestion command is still untested * delete deprecated/moved/redone files b/c git is dumb * moved makeschool to parsing module * Trimmed names of files and parsers ✂️ - changed <school}_<pytpe>.py to <ptype>.py - updated school_mappers to handle new format - no longer need each type of parser in school directory if unused - some pep8 things * tabs to spaces in parsing commands * Refactored pipeline commands - use format strings to handle defaults in args - changed format of progress bar and cleaned up - started refactor of digestor * ❗❗ remove parsing files from scripts again... 😡 * Added vectorizer back to digest command - would like to move this to its own command in due course. * partial refactor of digestor + bugfix in digest diff * uncomment .eslintrc.js * added dummy VALID_SCHOOLS to school_mappers to pass tests * added uoft to VALID_SCHOOLS even though parser is not active in order to pass tests * Sort'em 🔤 This should help with conflicts and duplications in these files * License and registration please 👮 Added Open GPL3 license to parsing directory files with # comment style. * Fix npm run watch * add/update testing/contrib docs (#979) * added everything back (#980) * Remove client secrets from core code (#981) * move hashids to local_settings.py * move all secrets to sensitive.py * add encrypted secrets as environment variables for travis * Correct global vars indentation in travis.yml * use environment variables via travisci web client instead * handle secret key before imports * provide dev credentials * use SemesterlyTest Facebook App credentials * add docs for secrets * uncomment out try catch for error printing on get_secret * Parsing/docs edit (#983) * Added DataUpdate object to parsing models - Created DataUpdate object to track last updated status per schools per semester - Modified SchoolList view and commented out assertNone in associated test - Used JSONStreamWriter to write ingestor and save meta data for digestor to read and be able to update DataUpdate objects per semester - General cleanup as well - Created Hoarder as tracker viewer to accumulate term, dept, instr data through parse * removed Updates completely and updated tests to use DataUpdate * doc and docstring edits
noahpresler
added a commit
that referenced
this pull request
Aug 5, 2017
* implemented course vectorizers under searches/utils * implemented course vectorizers under searches/utils * implemented relevance scoring * implemented relevance scoring * add get_relevant_search() method returning course objs * add get_relevant_search() method returning course objs * implemented stemming / ngram * implemented stemming / ngram * test case added * stemming/acronym/weighting implemented * evaluation functions implemented * evaluation functions implemented * objectified search utility functions * objectified search utility functions * Course clustering with LDA * Course clustering with LDA * optimized advance search * optimized advance search * refactoring codes * refactoring codes * rebased from staging * rebased from staging * satisfied linting * satisfied linting * ingested/digested jhu and modified search flow * temp * rebased from staging origin, still need vectorized search migration * implemented course vectorizers under searches/utils * implemented course vectorizers under searches/utils * implemented relevance scoring * implemented relevance scoring * add get_relevant_search() method returning course objs * add get_relevant_search() method returning course objs * implemented stemming / ngram * implemented stemming / ngram * test case added * stemming/acronym/weighting implemented * stemming/acronym/weighting implemented * evaluation functions implemented * evaluation functions implemented * objectified search utility functions * objectified search utility functions * Course clustering with LDA * Course clustering with LDA * optimized advance search * optimized advance search * refactoring codes * refactoring codes * rebased from staging * rebased from staging * satisfied linting * satisfied linting * ingested/digested jhu and modified search flow * temp * rebased from staging origin, still need vectorized search migration * do not store count vectorizer obj * removed pickle file * revised unit test * revised unit test * prevented from crashing when ingesting * rebased from master * removed non-using part of the code for now * modified requirements and cv loader * updated upon review * fixed to render searcher obj at app initialization * resolved migration issue * rebased and print exception * modified scoring / strip() query * resolved dependency issue * handle duplicates * changed to use static english dictionary count vectorizer for search. * added docstrings * modify license description, formatting and searches.rst description * merge migrations * fix tests for vectorized search * expand on documentation * cleanup webpack * cleanup scripts * fix update_semesterfield importing in migrations * cleanup root * update package.json * provide cause of internal errors by setting debug to true * fix tests * remove documentation legacy * move dict out of root * umich parser pep8 compliance * salisbury pep8 * JHU pep8 JHU course evals access db directly and are not pep8 compliant. 80 char limit is not enforced. * UMD pep8 compliance and refactor * gw instructor list limit * vandy pep8 compliance and refactor, UNCOMPLETE * Validator pep8 compliance Almost full compliance. Still debating on whether 79 char line limit is worth making 99 char limit. * Style compliance for Queens parser * style complianace for digestor, uncomplete * working on refactor * Conitnuing with refactor and pep8 compliance :plane: to Israel! * working on dictionary term/semester filtering * fixed parenthesis syntax error in ingestor * removed ingest_offerings and changed to ingest_meeting * removed ingest_offering * Ingestor refactor Made detailed docstrings, created internal _get function and removed getchain Changed ALL_KEYS to _ALL_KEYS * Cleaned ingestor cleandict and renamed to clean + general refactor of validator but not complete yet * method reordering in gw parser * add pep8 newline * Various refactors - filter years and terms command line arg and filtering * spaces instead of tabs in data pipeline json schemas * removed unneccessary import * fixed issue created by new years and terms filtering scheme * style update for commands argparser file * working on conforming jhu evals to data pipeline * fixed pep8 line overflow error in hacky way * changed wording in comment * Added eval to ingestion command NOTE: digest command is temporarily broken. - removed skip_shallow_duplicates from ingest cmd args -- NOTE: this could very well cause unforseen issues with peoplesoft courses hiding/overriden classes! - created --type cmd arg for ingestion -- had to hack in a way to use new_{course, textbook, eval}_parsers, to be changed on deprecation of school_mappers.py - changed hide_progress_bar to progress_bar (logic flip) * hopkins evals to pipeline * Added Vandy/Hopkins evals to data pipeline NOTE: digestion command is still broken due to argparse errors - Added evals to ingestor + validator - tweaked ingestion cmd args some more - pep8 compliance * removed extra lines from supervisord config file * tabs to spaces in logger * created new exceptions modules, moved some functionality to utils module * function out complex resolutions * moved and documented external_utils to utils * removed invalid reference to internal_utils * some library reference refactors * Created JSONWriter in logger but not yet used in code base * JHU ingestion starts without error * Refactor Tracker - Seperated Viewer and Tracker by file and made better distinction. - used @Property to clean up Tracker class - Factored out Counter and TimeDistribution into their own Viewer classes - Create Bus-like protocol so Viewers can filter Tracker broadcasts * track status and better docstrings * cleaned up validatoin schema loading * load_schemas once only in Validator * Refactor and cleanup validator * changed parser_library to library * The big move 🏠 🚚💨 🏠 - moved all parsing functionality into its own module - created global variable to hold parsing directory in settings.py - updated school mappers - BROKEN: uoft was not moved and must be dealt with later, digestion command is still untested * delete deprecated/moved/redone files b/c git is dumb * moved makeschool to parsing module * Trimmed names of files and parsers ✂️ - changed <school}_<pytpe>.py to <ptype>.py - updated school_mappers to handle new format - no longer need each type of parser in school directory if unused - some pep8 things * tabs to spaces in parsing commands * Refactored pipeline commands - use format strings to handle defaults in args - changed format of progress bar and cleaned up - started refactor of digestor * ❗❗ remove parsing files from scripts again... 😡 * Added vectorizer back to digest command - would like to move this to its own command in due course. * partial refactor of digestor + bugfix in digest diff * uncomment .eslintrc.js * added dummy VALID_SCHOOLS to school_mappers to pass tests * added uoft to VALID_SCHOOLS even though parser is not active in order to pass tests * Sort'em 🔤 This should help with conflicts and duplications in these files * License and registration please 👮 Added Open GPL3 license to parsing directory files with # comment style. * Fix npm run watch * add/update testing/contrib docs (#979) * added everything back (#980) * Remove client secrets from core code (#981) * move hashids to local_settings.py * move all secrets to sensitive.py * add encrypted secrets as environment variables for travis * Correct global vars indentation in travis.yml * use environment variables via travisci web client instead * handle secret key before imports * provide dev credentials * use SemesterlyTest Facebook App credentials * add docs for secrets * uncomment out try catch for error printing on get_secret * Parsing/docs edit (#983) * Added DataUpdate object to parsing models - Created DataUpdate object to track last updated status per schools per semester - Modified SchoolList view and commented out assertNone in associated test - Used JSONStreamWriter to write ingestor and save meta data for digestor to read and be able to update DataUpdate objects per semester - General cleanup as well - Created Hoarder as tracker viewer to accumulate term, dept, instr data through parse * removed Updates completely and updated tests to use DataUpdate * doc and docstring edits * dedupe list of dictionaries correctly (#990) * dedupe list of dictionaries correctly * fix typo
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Advanced Configuration
Semester.ly makes use of several secrets which allow it to interact securely with third party software providers. These providers include Facebook (for oauth and social graph), Google (oauth), and university APIs.
In order for Semester.ly to run out of the box, we have included credentials to test Google and Facebook applications for development purposes. We override these keys for production use thereby keeping our client secrets... well, secrets! These provided credentials can be found in
semesterly/dev_credentials.py
::Overriding/Setting Secrets
However, if you wish to override these credentials or add login credentials for a school which requires a client secret, you may add your key/value pair to
semesterly/sensitive.py
. This file is gitignored and will be kept private so you can safely store the private information you wish within this file. It should have a format indentical to SECRETS above and insemesterly/dev_credentials.py
.Using Secrets
In order to properly access a secret from anywhere within the code, simply import the
get_secret
function and use it to access the secret by key::This will check the following locations for the secret (in order, using the first value it finds), throwing an error if it does not find the key at all: