Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor footprint changes #167

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

iantei
Copy link
Contributor

@iantei iantei commented Oct 11, 2024

Handle Future cleanups mentioned in #152 Changes:

  1. Introduce better exception logging
future fix: I would suggest logging what the exception was (ideally using logging.exception) so that we can check and fix as needed. We may also want to add an entry into the client stats of how many trips were affected. This will allow us to check on why the builds are failing
  1. Type fixed from relecant to relevant.

  2. Refactor extract_co2 and extract_kwh to a single consolidated parameterized function extract_footprint()

future fix: it seems like this should be combinable into a single parameterized function
  1. Changed the labels for Stacked Bar Chart's Inferred label from "Labeled and Inferred by OpenPATH" to "Inferred from prior labels"
future fix: I think I flagged this somewhere else, but it would be good to change this to "inferred from prior labels" here because there are multiple inference algorithms in OpenPATH

…into a single parameterized function extract_footprint(footprint_dict, footprint_key). Removed the print missing kwh/co2, footprint_dict since it was polluting the cell execution, and in all these cases footprint_dict was empty.
…he actual exception error reason, also while accounting the total number of trips where there were errors.
@iantei
Copy link
Contributor Author

iantei commented Oct 11, 2024

Testing Scenario:

Dataset used: cortezebikes

Execution of notebooks- generic_metrics, mode_specific_metrics - to lookup for inferred label changes
energy_calculations - to lookup for footprint calculation changes

Detailed Execution of notebooks using generate_plots.py script

(emission) root@31c91ddf8794:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py generic_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-10-11T20:12:14.132517+00:00 with args Namespace(plot_notebook='generic_metrics.ipynb', program='default', date=None) for range (<Arrow [2023-06-01T00:00:00+00:00]>, <Arrow [2024-10-01T00:00:00+00:00]>)
Running at 2024-10-11T20:12:14.172297+00:00 with params [Parameter('year', int), Parameter('month', int), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:12:24.509729+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:12:31.283995+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:12:38.615102+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:12:45.967215+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:12:53.272719+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:13:00.616589+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=11), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:13:07.474202+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=12), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:13:14.436243+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=1), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:13:20.742465+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=2), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:13:27.058960+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=3), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:13:32.466903+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=4), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:13:37.850492+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=5), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:13:43.130289+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:13:48.653394+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:13:53.974055+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:13:59.467121+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
Running at 2024-10-11T20:14:04.991671+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True), Parameter('sensed_algo_prefix', str, value='cleaned'), Parameter('survey_info', dict, value={'surveys': {'UserProfileSurvey': {'formPath': 'json/demo-survey-v2.json', 'version': 1, 'compatibleWith': 1, 'dataKey': 'manual/demographic_survey', 'labelTemplate': {'en': 'Answered', 'es': 'Contestada'}}}, 'trip-labels': 'MULTILABEL'})]
(emission) root@31c91ddf8794:/usr/src/app/saved-notebooks# 




(emission) root@31c91ddf8794:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py mode_specific_metrics.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-10-11T20:14:43.716544+00:00 with args Namespace(plot_notebook='mode_specific_metrics.ipynb', program='default', date=None) for range (<Arrow [2023-06-01T00:00:00+00:00]>, <Arrow [2024-10-01T00:00:00+00:00]>)
Running at 2024-10-11T20:14:43.753000+00:00 with params [Parameter('year', int), Parameter('month', int), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:14:50.994066+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:14:56.255125+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:01.821568+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:07.079884+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:12.496401+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:17.846748+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=11), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:22.654577+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=12), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:26.979411+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=1), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:31.839309+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=2), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:36.648695+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=3), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:40.951585+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=4), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:45.208036+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=5), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:49.401527+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:53.630460+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:15:57.848482+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:16:02.359828+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:16:06.632476+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
(emission) root@31c91ddf8794:/usr/src/app/saved-notebooks# 


(emission) root@31c91ddf8794:/usr/src/app/saved-notebooks# PYTHONPATH=.. python bin/generate_plots.py energy_calculations.ipynb default
/usr/src/app/saved-notebooks/bin/generate_plots.py:30: SyntaxWarning: "is not" with a literal. Did you mean "!="?
  if r.status_code is not 200:
About to download config from https://raw.githubusercontent.com/e-mission/nrel-openpath-deploy-configs/main/configs/cortezebikes.nrel-op.json
Successfully downloaded config with version 1 for Cortez 55+ eBike Program and data collection URL https://cortezebikes-openpath.nrel.gov/api/
label_options is unavailable for the dynamic_config in cortezebikes
Running at 2024-10-11T20:21:21.321738+00:00 with args Namespace(plot_notebook='energy_calculations.ipynb', program='default', date=None) for range (<Arrow [2023-06-01T00:00:00+00:00]>, <Arrow [2024-10-01T00:00:00+00:00]>)
Running at 2024-10-11T20:21:21.361600+00:00 with params [Parameter('year', int), Parameter('month', int), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:21:47.155261+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:21:55.382353+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:04.384330+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:11.270368+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:19.210182+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:25.217748+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=11), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:28.570007+00:00 with params [Parameter('year', int, value=2023), Parameter('month', int, value=12), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:31.937802+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=1), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:34.772511+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=2), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:37.531869+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=3), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:40.265116+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=4), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:43.069850+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=5), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:46.007959+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=6), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:48.782695+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=7), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:51.506007+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=8), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:54.223285+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=9), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
Running at 2024-10-11T20:22:56.942991+00:00 with params [Parameter('year', int, value=2024), Parameter('month', int, value=10), Parameter('program', str, value='default'), Parameter('study_type', str, value='program'), Parameter('mode_of_interest', str, value='e-bike'), Parameter('include_test_users', bool, value=False), Parameter('is_debug_mode', bool, value=False), Parameter('dynamic_labels', dict, value={}), Parameter('use_imperial', bool, value=True)]
(emission) root@31c91ddf8794:/usr/src/app/saved-notebooks# 

Results:

Charts for cortezebike:
Screenshot 2024-10-11 at 1 31 41 PM
Screenshot 2024-10-11 at 1 33 57 PM

Note: Everything looks good

…book_inferred_data(,..,add_footprint). Use base_mode AIR to filter out the trip without air mode, instead of internal label
@iantei
Copy link
Contributor Author

iantei commented Oct 15, 2024

Fix for the following:

#140

Use base_mode to filter out the AIR related trips.

Results:

Before (Using internal label) After (Using base mode)
internal label base_mode

Looks identical

@iantei iantei marked this pull request as ready for review October 15, 2024 18:49
Copy link
Contributor

@JGreenlee JGreenlee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a couple questions/suggestions below!

Comment on lines +507 to +509
def extract_footprint(footprint_dict, footprint_key):
if footprint_key in footprint_dict.keys():
return footprint_dict[footprint_key]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good refactor

viz_scripts/generic_metrics.ipynb Outdated Show resolved Hide resolved
viz_scripts/scaffolding.py Outdated Show resolved Hide resolved
@@ -254,7 +257,7 @@ async def map_trip_data(expanded_trip_df, study_type, dynamic_labels):

return expanded_trip_df

async def load_viz_notebook_inferred_data(year, month, program, study_type, dynamic_labels, include_test_users=False):
async def load_viz_notebook_inferred_data(year, month, program, study_type, dynamic_labels, include_test_users=False, add_footprint=False):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For my understanding, is this cleanup from something that was meant to be added in a previous PR?

I saw that load_viz_notebook_data already had the add_footprint parameter, but load_viz_notebook_inferred_data was missing it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, this is not a cleanup which was meant to be added in the previous PR. We were using add_footprint parameter to compute footprint calculations in labeled trips in energy_calculations notebook. Earlier, we were not using add_footprint parameter for load_viz_notebook_inferred_data.

We need to pass add_footprint parameter as True to enable extraction of base_mode. Therefore, this change is required, such that we have base_mode as column added in the expanded_ct_inferred data frame. This enables us for filtering AIR as the base_mode, instead of using mode_confirm_w_other = 'air'.

@@ -244,7 +246,7 @@
" plot_and_text_stacked_bar_chart(expanded_ct, lambda df: (df.groupby(\"mode_confirm_w_other\").agg({distance_col: 'count'}).sort_values(by=distance_col, ascending=False)), \n",
" \"Labeled by user\\n\"+stacked_bar_quality_text_labeled, ax[0], text_results[0], colors_mode, debug_df, values_to_translations)\n",
" plot_and_text_stacked_bar_chart(expanded_ct_inferred, lambda df: (df.groupby(\"mode_confirm_w_other\").agg({distance_col: 'count'}).sort_values(by=distance_col, ascending=False)), \n",
" \"Labeled and Inferred by OpenPATH\\n\"+stacked_bar_quality_text_inferred, ax[1], text_results[1], colors_mode, debug_df_inferred, values_to_translations)\n",
" \"Inferred from prior labels\\n\"+stacked_bar_quality_text_inferred, ax[1], text_results[1], colors_mode, debug_df_inferred, values_to_translations)\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good text change. More concise & clearer

iantei and others added 2 commits October 18, 2024 16:02
Co-authored-by: Jack Greenlee <JackAGreenlee@gmail.com>
Co-authored-by: Jack Greenlee <JackAGreenlee@gmail.com>
Copy link
Contributor

@JGreenlee JGreenlee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Ready for review by Shankari
Development

Successfully merging this pull request may close these issues.

2 participants