-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Common trip system building #647
Comments
to give some additional context: e-mission is currently written in angular1 also called angularJS on the internet. It is also a reactive framework in which you make changes to So concretely, if I want to display a username, I can have
then if the javascript has |
If you have built the native version of the app (the second set of instructions on the README) you should be able to open this app in your IDE. and on your phone if you like |
If you have built the UI-only change version of the app (first set of instructions in the README), you need to install the e-mission-devapp in an emulator and then connect to the live-reloading dev server. Please submit a PR to clarify this in the README. |
The previous attempts at creating a graph of common trips are generally in The original goal was to create a graph of people's regular patterns, similar to We called it a tour model because that is apparently what such models are called in the travel behavior literature. |
The data representation (I think) https://github.com/e-mission/e-mission-server/blob/master/emission/analysis/modelling/tour_model/tour_model.py which is a classic graph data structure. Not sure if that is the correct data structure we need to use now. This has some wrapper objects: And this is how the matrix was created: Note that this is not incremental, and we will almost certainly need incremental creation in order to avoid overwhelming the server over a two year data collection period. |
On the phone, this data was retrieved and displayed in the UI shown above. The related UI code is at: we just read it from the local cache, we saved to the cache in the |
Ideas:
Critical from energy standpoint is mode, replaced_mode or distance
As few opportunities for open response as possible; reduce the other category as much as possible.
suggestion: both But would be then show these to the user for confirmation as well? Ask for a week's worth of data every few months. This might be good not just for these mixed clusters but for all trips in general. Go back to primary data collection mode every few weeks. When we are in "secondary data collection mode" will we still ask for confirmation of automatically assigned labels? Yes, to some extent until we gain confidence in the algorithm assessment. |
for analysis alone, we want to assign labels based on the clusters that @corinne-hcr has built so far and evaluate accuracy. This does not appear to be a very heavy lift. As part of system integration, however, we need to use those clusters as a model. In other words, we need to:
Store the clustersWe need to come up with a data structure to store the clusters. This might be as simple as a list of lists e.g.
Update the clustersWhen should we update this model? We will create it after an initial intensive "primary data collection period" but then we will go to "secondary data collection" for a few months in which we ask for confirmation for trips but not full trip labeling. Do we assume that people will confirm and correct the majority of their trips during the secondary data collection period?
This might be a question to ask @andyduvall to weigh in on. If we rebuild the model every week, we may want to consider incremental updates in which we only add new trips to existing clusters, but that might be hard to do. We may want to wait on that and treat it as a future performance enhancement. Apply the modelAs we get new unlabeled trips, we need to access the stored cluster model, and assign labels according to the existing clusters. This will likely involve a more complicated data structure because just storing the trip ids may not be enough to figure out which cluster a new trip should fall into. We may need to store some cluster level attributes like distance/duration cutoffs so we can efficiently determine how to match the trip. This does need to be incremental as in we cannot recluster every time we label a new trip. We must use stored clusters. This is because we get ~ 5-10 trips a day per person and the cluster algorithm can be space and time intensive. I turned off the tour model earlier because it was too slow (need to get some numbers). Note also that @corinne-hcr cannot run two versions of the pipeline side by side while restructuring on her laptop. And rebuilding the model every trip will result in rebuilding multiple time a day for each user. Note also that there is a latency issue with taking too long - if the user looks at the diary and we haven't run the inference yet, it will look like there were no labels and the algorithm is "not working". |
System design of labeling trips:
|
Note that we may want to use different clusterings from different folds, generate labels for each clustering and determine the confidence of non-mixed clusters (not mixed |
Current "label screen" javascript is here In particular, note that Here's where we embed that data structure in the UI: notably |
@GabrielKS, can you put in your high level design into this issue? @PatGendre, who is deploying the platform in La Rochelle is interested. @asiripanich may be too. As deployers, as opposed to end-users, they are more likely to give feedback on the configuration options 😄 |
@PatGendre @asiripanich @shankari I've attached the second draft of my trip label inference system UI proposal in PDF and Word form; I've pasted the executive summary below. UI Draft 2.pdf Behind the scenes, a server-side trip inference algorithm will produce a data structure that comprises, for each trip, a list of label sets with probabilities. This allows a client-side algorithm to refine the inference as the user manually confirms or corrects labels; it will also aid in analysis when the user has not confirmed or corrected labels. Below is an example of what this data structure might look like. To integrate the inference system into the phone app user interface, we will make three main changes. First, we display unfilled labels in red, inferred labels in yellow, and manually filled-out or verified labels in green. Second, we add a To Label view on the Label screen that displays only trips that users are expected to manually provide input on, according to the criteria below. Third, we change when we notify users as detailed below. We will also make some smaller UI changes to improve usability, including adding a confirm button, a feature to label many of the same type of trip at once, and a map of trips. Below is a screenshot of progress so far, showing the red/yellow/green color scheme and the confirm button. Expectations and notification behavior will be highly configurable to support many use cases. The basic idea of this configuration is that for each label category, comprising red labels and yellow labels with varying degrees of certainty, trip administrators will be able to select an expectation setting and a notification setting. The user could be expected to label every trip in a given category, none of them, or some random sample in between. The user could be notified after every trip in a given category, at the end of the day, or less frequently. Study administrators will also be able to configure a “primary mode,” in which user input expectations are high, and a “secondary mode,” in which less is demanded of the user; the app can automatically cycle between primary and secondary modes according to a configurable schedule. Please see the full proposal for a complete configuration example and many more details. Please post or send me any feedback you may have! |
@jf87 not sure if you are planning to use labels |
@GabrielKS thanks a lot for the explanations :-) Here a few questions and thoughts :
|
@PatGendre Thanks for the thoughts; I'll do my best to respond given my limited knowledge of the e-mission system outside of the areas I'm directly working with:
|
@PatGendre to summarize: the current feature works on trip-level labels. We currently have trip-level user labels, but even after we switch to section-level user input for the mode, we will still want to have trip-level purpose labels. So this is not dependent on merging the A-Mission code. If/when we do have section-level user inputs, we can expand this functionality to the section level as well. |
@shankari @GabrielKS thanks for your detailed answers, it is pretty clear now :-) and definitely a very promising feature! |
Tracking initial staging deployment at: |
@PatGendre @jf87 @asiripanich Initial deployment complete, in beta testing. Picking tuning parameters is being tracked at: Most recent set of UI changes, which can give you a sense of how the feature works, is being tracked at: |
@GabrielKS I am writing down all the potential issues for next week here so (a) I don't forget them and (b) we can prioritize them. Please let me know if there's anything else in your notes that I am missing. UI + expectations:
Trip matching and modeling:
High-level end to end feature:
|
Additional things I have:More urgent:
Less urgent:
Fun quotes:
|
I wonder if this is the same as #658 |
Another usability question: What happens if you click the green checkmark and not all of them are auto-labeled? Some of them are red. I don't want to click it and mess something up. |
The confirm button iterates through all label types and, for each, populates the user input with the client-computed inference iff
So the explanation I've been giving to users, "the confirm button turns yellow labels green" is exactly correct; red labels are simply ignored. However, I've just realized that the confidence threshold the client has been using is not the one in the config file — the config file is on the server side and I never wrote the code to send it over — it's been using a placeholder value of 0.5. This explains why we've been seeing so many trips with mixed yellow and red labels! I will fix that ASAP. (This affects both what is displayed and what is confirmable as the logic only happens once, so at least it's consistent.)
|
I know, and I told the user verbally. But wanted to record their initial response for you to think about UX improvements. |
Ah, okay. Didn't get that that was the user talking. |
@GabrielKS Another issue that came up with at least two users today: they expected to see the auto-labeling in the diary and when it didn't show up, they thought that the auto-labeling didn't work. One of the users went to the diary after they saw that there were no trips in "To Label", even at the end of the week 😦 This sounds deceptively easy, but is actually going to require a fair amount of rewriting, because the diary retrieves data using a completely different API call than the label screen. I had thought about merging the diary and label screens, but was hoping to not tackle that just yet. Maybe we can do a minimal rewrite in which we retrieve the confirmed trips in addition to the user labels as currently retrieved. Then we can read the inferred labels from there. |
@GabrielKS Jeanne also brought up displaying only recent trips to users. Unfortunately, instead of only showing trips from the last n days, she wants to start with a clean slate on 16th Aug and then display all trips. Given that requirement, I think that maybe the easiest option is to just hardcode that into the client instead of putting it into the server since it doesn't seem super general. We would not cherry-pick that change into master, obviously. |
Filed #662 to track the UI issues that came up today. |
Filed #663 to track the "confidence too high" issue from e-mission/e-mission-eval-private-data#28 (comment) |
Closing this for now. We can track any pending problems in separate issues. |
This tracks the tasks required to actually close the loop on user interaction with labels.
The text was updated successfully, but these errors were encountered: