Support additional configuration options for aggregate data #408
Labels
enhancement
New feature or request
Comments
shankari
added a commit
to e-mission/e-mission-phone
that referenced
this issue
Apr 20, 2020
The switch to WKWebView forced the use of CORS. There is no workaround. https://ionicframework.com/docs/v3/wkwebview/#cors We didn't encounter this in covid-19 repo since we made all calls through the built-in native plugin that automatically adds the user token and sends out the call. In emission, though, we need make aggregate calls that should not have a user token. We had been using XHR for this (through the angular `$http` server) and all those calls broke. While we could make the appropriate change on the server side, that would not be consistent with our long term goal (e-mission/e-mission-docs#408, e-mission/e-mission-docs#288) So for now, I use the alternative documented here https://ionicframework.com/docs/v3/wkwebview/#i-cant-implement-cors Concretely: - add the native plugin - add a new method `CommHelper.getAggregateData` that wraps it in a promise - change all instances of `$http.post` -> `CommHelper.getAggregateData` - use the configured connectUrl instead of hardcoding - remove the hardcoded URL from index.html Bonus fix: - Dealt with error in the heatmap if there was no data and thus, no bounds by checking to see if the bounds were valid Testing done: Navigated through the app screens until all the aggregate calls were invoked (see below). No errors in the console. ``` [Log] getting aggregate data without user authentication from http://localhost:8080/result/metrics/timestamp with arguments {"freq":"D","start_time":1586131200,"end_time":1587254400,"metric_list":["duration","median_speed","count","distance"],"is_return_aggregate":true} (cordova.js, line 1540) [Log] getting aggregate data without user authentication from http://localhost:8080/result/heatmap/incidents/timestamp with arguments {"start_time":1587187062,"end_time":1587359861,"sel_region":null} (cordova.js, line 1540) [Log] getting aggregate data without user authentication from http://localhost:8080/result/heatmap/pop.route/local_date with arguments {"modes":null,"from_local_date":{"year":2020,"month":4,"day":17,"hour":22},"to_local_date":{"year":2020,"month":4,"day":18,"hour":22},"sel_region":null} (cordova.js, line 1540) [Log] getting aggregate data without user authentication from http://localhost:8080/result/heatmap/incidents/local_date with arguments {"modes":null,"from_local_date":{"year":2020,"month":4,"day":17,"hour":22},"to_local_date":{"year":2020,"month":4,"day":18,"hour":22},"sel_region":null} (cordova.js, line 1540) ```
shankari
added a commit
to shankari/e-mission-phone
that referenced
this issue
Mar 17, 2021
This fixes e-mission/e-mission-docs#408 and is a partial fix for e-mission/e-mission-docs#628 Fix is fairly straightforward: - introduce a `aggregate_call_auth` config option, similar to the server - if is it set to `no_auth`, use the `cordova.plugin.http.sendRequest` as before - if it is set to `user_only`, call `window.cordova.plugins.BEMServerComm.pushGetJSON` similar to the existing user calls `window.cordova.plugins.BEMServerComm.pushGetJSON` returns the data from the response instead of the response directly. So change the `no_auth` option to also return the response data, and change all calling functions to access `response.foo` instead of `response.data.foo` Bonus fix: `finally` breaks on android 27 emulator, use `then().catch()` instead. Testing done: - with a server configured for `no_auth` - client = `no_auth` works - with a server configured for `user_only` - client = `no_auth` fails ``` 2021-03-16 16:58:42.465 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "ERROR:Error loading aggregate data, averages not available{"status":403,"url":"http://10.0.2.2:8080/result/metrics/timestamp","headers":{"date":"Tue, 16 Mar 2021 23:59:25 GMT","content-length":"761","server":"Cheroot/8.4.2","x-android-selected-protocol":"http/1.1","x-android-response-source":"NETWORK 403","x-android-received-millis":"1615939122220","x-android-sent-millis":"1615939122206","content-type":"text/html; charset=UTF-8"},"error":"\n <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n <html>\n <head>\n <title>Error: 403 Forbidden</title>\n <style type=\"text/css\">\n html {background-color: #eee; font-family: sans-serif;}\n body {background-color: #fff; border: 1px solid #ddd;\n padding: 15px; margin: 15px;}\n pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n </style>\n </head>\n <body>\n <h1>Error: 403 Forbidden</h1>\n <p>Sorry, the requested URL <tt>'http://10.0.2.2:8080/result/metrics/timestamp'</tt>\n caused an error:</p>\n <pre>aggregations only available to users</pre>\n </body>\n </html>\n"}", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145) ``` - client = `user_only` succeeds ``` 2021-03-16 18:52:47,214:DEBUG:123145648730112:START POST /result/metrics/timestamp 2021-03-16 18:52:47,214:DEBUG:123145648730112:Aggregate call, checking user_only policy 2021-03-16 18:52:47,214:DEBUG:123145648730112:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'> 2021-03-16 18:52:47,215:DEBUG:123145648730112:Using the skip method to verify id token REPLACEMEkVVdF9rT of length 17 2021-03-16 18:52:47,216:DEBUG:123145648730112:retUUID = cf8ccb7b-84d7-40e4-a726-7691e614b042 2021-03-16 18:52:47,223:DEBUG:123145648730112:END POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.009236335754394531 ```
jf87
pushed a commit
to jf87/e-mission-server
that referenced
this issue
Jun 21, 2021
This fixes e-mission/e-mission-docs#408 It is also a partial fix for e-mission/e-mission-docs#628 We support 3 basic policies: - `no_auth`: full public access (backwards compatible behavior) - `user_only`: access only to existing users (new functionality, consistent with e-mission/e-mission-docs#408) - `never`: disable completely Other sophisticated access control for certain users only is out of the scope at this time Testing done: - set the policy to `no_auth` - aggregate call works ``` 2021-03-16 16:25:32,859:DEBUG:123145663979520:START POST /result/metrics/timestamp 2021-03-16 16:25:32,859:DEBUG:123145663979520:Aggregate call, checking {aggregate_call_support} policy 2021-03-16 16:25:32,859:DEBUG:123145663979520:metric_list = ['duration', 'median_speed', 'count', 'distance'] 2021-03-16 16:25:32,859:DEBUG:123145663979520:['duration -> <function get_duration at 0x7ffe61347cb0>', 'median_speed -> <function get_median_speed at 0x7ffe61347d40>', 'count -> <function get_count at 0x7ffe61347b90>', 'distance -> <function get_distance at 0x7ffe61347c20>'] 2021-03-16 16:25:32,859:DEBUG:123145663979520:for user None, returning timeseries <emission.storage.timeseries.aggregate_timeseries.AggregateTimeSeries object at 0x7ffe8052d790> 2021-03-16 16:25:32,867:DEBUG:123145663979520:END POST /result/metrics/timestamp 0.008590936660766602 ``` - user call works ``` 2021-03-16 16:25:32,866:DEBUG:123145669234688:START POST /result/metrics/timestamp 2021-03-16 16:25:32,867:DEBUG:123145669234688:User specific call, returning UUID 2021-03-16 16:25:32,868:DEBUG:123145669234688:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'> 2021-03-16 16:25:32,868:DEBUG:123145669234688:Using the skip method to verify id token REPLACEMEkVVdF9rT of length 17 2021-03-16 16:25:32,870:DEBUG:123145669234688:retUUID = cf8ccb7b-84d7-40e4-a726-7691e614b042 2021-03-16 16:25:32,876:DEBUG:123145669234688:END POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.009974002838134766 ``` - switch the policy to `user_only` - user call works ``` 2021-03-16 16:25:32,866:DEBUG:123145669234688:START POST /result/metrics/timestamp 2021-03-16 16:25:32,867:DEBUG:123145669234688:User specific call, returning UUID 2021-03-16 16:25:32,868:DEBUG:123145669234688:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'> 2021-03-16 16:25:32,868:DEBUG:123145669234688:Using the skip method to verify id token REPLACEMEkVVdF9rT of length 17 2021-03-16 16:25:32,870:DEBUG:123145669234688:retUUID = cf8ccb7b-84d7-40e4-a726-7691e614b042 2021-03-16 16:25:32,876:DEBUG:123145669234688:END POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.009974002838134766 ``` - aggregate call fails ``` 2021-03-16 16:59:25,517:DEBUG:123145504403456:START POST /result/metrics/timestamp 2021-03-16 16:59:25,517:DEBUG:123145504403456:Aggregate call, checking user_only policy 2021-03-16 16:59:25,518:DEBUG:123145504403456:END POST /result/metrics/timestamp 0.00035881996154785156 ``` with error ``` 2021-03-16 16:58:42.465 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "ERROR:Error loading aggregate data, averages not available{"status":403,"url":"http://10.0.2.2:8080/result/metrics/timestamp","headers":{"date":"Tue, 16 Mar 2021 23:59:25 GMT","content-length":"761","server":"Cheroot/8.4.2","x-android-selected-protocol":"http/1.1","x-android-response-source":"NETWORK 403","x-android-received-millis":"1615939122220","x-android-sent-millis":"1615939122206","content-type":"text/html; charset=UTF-8"},"error":"\n <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n <html>\n <head>\n <title>Error: 403 Forbidden</title>\n <style type=\"text/css\">\n html {background-color: #eee; font-family: sans-serif;}\n body {background-color: #fff; border: 1px solid #ddd;\n padding: 15px; margin: 15px;}\n pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n </style>\n </head>\n <body>\n <h1>Error: 403 Forbidden</h1>\n <p>Sorry, the requested URL <tt>&e-mission#39;http://10.0.2.2:8080/result/metrics/timestamp'</tt>\n caused an error:</p>\n <pre>aggregations only available to users</pre>\n </body>\n </html>\n"}", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145) ``` - switch the policy to `never`, fails with error ``` 2021-03-16 17:13:20.422 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "Error loading aggregate data, averages not available{"status":404,"url":"http://10.0.2.2:8080/result/metrics/timestamp","headers":{"date":"Wed, 17 Mar 2021 00:14:03 GMT","content-length":"754","server":"Cheroot/8.4.2","x-android-selected-protocol":"http/1.1","x-android-response-source":"NETWORK 404","x-android-received-millis":"1615940000171","x-android-sent-millis":"1615940000159","content-type":"text/html; charset=UTF-8"},"error":"\n <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n <html>\n <head>\n <title>Error: 404 Not Found</title>\n <style type=\"text/css\">\n html {background-color: #eee; font-family: sans-serif;}\n body {background-color: #fff; border: 1px solid #ddd;\n padding: 15px; margin: 15px;}\n pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n </style>\n </head>\n <body>\n <h1>Error: 404 Not Found</h1>\n <p>Sorry, the requested URL <tt>&e-mission#39;http://10.0.2.2:8080/result/metrics/timestamp'</tt>\n caused an error:</p>\n <pre>Aggregate calls not supported</pre>\n </body>\n </html>\n"}", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145) ``` - switch the policy to an invalid valid, fails with error ``` 2021-03-16 17:14:25.561 23394-23394/edu.berkeley.eecs.emission.devapp I/chromium: [INFO:CONSOLE(145)] "ERROR:Error loading aggregate data, averages not available{"status":500,"url":"http://10.0.2.2:8080/result/metrics/timestamp","headers":{"date":"Wed, 17 Mar 2021 00:15:08 GMT","content-length":"1550","server":"Cheroot/8.4.2","x-android-selected-protocol":"http/1.1","x-android-response-source":"NETWORK 500","x-android-received-millis":"1615940065310","x-android-sent-millis":"1615940065297","content-type":"text/html; charset=UTF-8"},"error":"\n <!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n <html>\n <head>\n <title>Error: 500 Internal Server Error</title>\n <style type=\"text/css\">\n html {background-color: #eee; font-family: sans-serif;}\n body {background-color: #fff; border: 1px solid #ddd;\n padding: 15px; margin: 15px;}\n pre {background-color: #eee; border: 1px solid #ddd; padding: 5px;}\n </style>\n </head>\n <body>\n <h1>Error: 500 Internal Server Error</h1>\n <p>Sorry, the requested URL <tt>&e-mission#39;http://10.0.2.2:8080/result/metrics/timestamp'</tt>\n caused an error:</p>\n <pre>Internal Server Error</pre>\n <h2>Exception:</h2>\n <pre>KeyError(&e-mission#39;foobar&e-mission#39;)</pre>\n <h2>Traceback:</h2>\n <pre>Traceback (most recent call last):\n File "/Users/kshankar/e-mission/e-mission-server/emission/net/api/bottle.py", line 997, in _handle\n out = route.call(**args)\n File "/Users/kshankar/e-mission/e-mission-server/emission/net/api/bottle.py", line 1998, in wrapper\n rv = callback(*a, **ka)\n File "emission/net/api/cfc_webapp.py", line 466, in summarize_metrics\n user_uuid = get_user_or_aggregate_auth(request)\n File "emission/net/api/cfc_webapp.py", line 621, in get_user_or_aggregate_auth\n return aggregate_call_map[aggregate_call_support](request)\nKeyError: &e-mission#39;foobar&e-mission#39;\n</pre>\n </body>\n </html>\n"}", source: http://localhost/_app_file_/data/user/0/edu.berkeley.eecs.emission.devapp/files/phonegapdevapp/www/index.html (145) ``` - changed the phone code to send a user token for aggregate calls as well, worked ``` 2021-03-16 18:52:47,214:DEBUG:123145648730112:START POST /result/metrics/timestamp 2021-03-16 18:52:47,214:DEBUG:123145648730112:Aggregate call, checking user_only policy 2021-03-16 18:52:47,214:DEBUG:123145648730112:methodName = skip, returning <class 'emission.net.auth.skip.SkipMethod'> 2021-03-16 18:52:47,215:DEBUG:123145648730112:Using the skip method to verify id token REPLACEMEkVVdF9rT of length 17 2021-03-16 18:52:47,216:DEBUG:123145648730112:retUUID = cf8ccb7b-84d7-40e4-a726-7691e614b042 2021-03-16 18:52:47,223:DEBUG:123145648730112:END POST /result/metrics/timestamp cf8ccb7b-84d7-40e4-a726-7691e614b042 0.009236335754394531 ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The current assumptions for aggregate data access are pretty simple:
In fact, in many cases, we distinguish between user-specific and aggregated calls to the same data (e.g. heatmap/metrics) by seeing whether or not a token is passed in.
We also use two different code paths in the app for the two accesses:
$httpwhich does not insert a tokenWe may want to offer more control over the aggregate data - e.g. only offer aggregate data to other registered users. We may also want to unify the code paths to avoid being subject to CSP changes. Note that this will make it harder to connect to server A as a user but retrieve aggregate data from server B.
More discussion in closed issue #288
The text was updated successfully, but these errors were encountered: