802 unique ids for anon users

[misaugstad]

Resolves #802
Partially resolves #783

Summary

Each anonymous user is now automatically assigned a unique that is treated exactly the same as registered users. They are essentially given a randomized username, email, and password that is overwritten if they ever decide to sign up. The user experience should not change at all for anonymous users, except that their contribution is saved until they are logged out by closing their browser, etc., and when an anonymous user signs up, their contribution is converted to the account they just created. But the rest of the experience is the same, in that the navbar still has the same look as it used to have for anonymous users, etc.

This update is incompatible with the dataset we collected for DC. Anonymous users were just implied by IP address when collecting the DC data, and I removed all the code that handles anonymous users in that way, and all the code that searches through the database for ip addresses to count ip-address-based contributions, cluster their labels, etc. So if you run the current code on the DC data, it will look like there have been no anonymous users at all. The implementation I've done here is meant to have anonymous user contributions recorded in a more robust way when collecting data in future cities.

Testing

• Visit the website in incognito/private mode; it should look exactly the same as it used to for anonymous users
• Check that the "User" entry in the role table has been updated to be called "Registered", and that a new entry with the role "Anonymous" has been added to the table.
• Check that a new user has been added to the user table and that they have the new "Anonymous" role. You can do this by logging in as an admin (in your non private/incognito window), and check the User tab of the admin page. If you sort by registration date, you should see a user that has just registered who has the Anonymous role. The user should have a randomly generated username and email address.
• In the webpage_activity table, you should also see a new entry with the activity "AnonAutoSignUp"
• As the anonymous user, complete the tutorial and do some auditing
  • The tool should look the same as it normally does for an anonymous user (you are still asked to sign up, there is a sign in button at the top, etc)
  • Go to the anon user's profile page (the admin version, because the anon user shouldn't see the option to go to a user profile page), and you should see the contribution there
• Try signing up (do this once using the navbar, once by going to the /signUp endpoint, and once when prompted while auditing; this means that you will have to do the sign-up, verify it worked, then close the browser and reopen in incognito/private to auto-generate a new anonymous user to try and sign up again). Verify that the following happens:
  • In the webpage_activity table, "SignUp" and then "SignIn" should be logged for that user.
  • In the user table, the user_id remains the same, but the username and email is updated to what you used to sign up (i.e., the old anon user entry is modified, instead of a new user being created).
  • The entry for this user in the user_role table is updated from the "Anonymous" role to the "Registered" role.
  • The registered user can view their user profile page, and it should show their contributions made before they signed up.
  • In the User tab of the admin page, the role, username, and email should be updated appropriately. The date registered value should also be updated to the time when you just signed up.
• Test that an anon user is created if you visit any webpage part of the website: You can try closing the private browser window and reopening it, then visit the website for the first time through different URLs. So try going right to localhost:9000/faq, and you should be able to check that a new anonymous user has been created. Try the same thing by going directly to /audit, /results, etc. Anything you can find to test it out. I tried to do the auto sign up for any URL that goes to an actual webpage (i.e., I excluded the straight up API endpoints that just return some JSON).
• Check that the queries for the admin page are still working correctly (note that the only anon user contributions will now be the ones you just made while testing). Check the following things:
  • On the Overview tab, the counts of users/audits/distance for anonymous users should include only the contributions you made while testsing.
  • On the Users tab, there is no longer two separate tables (one for registered and one for anon users), there is now only one (where anon users are included, just with their role shown as "Anonymous").
  • On the Analytics tab, the graphs that differentiate by users' roles (time spent auditing, missions completed, labels per user) should have only the data you just contributed while testing.
• Check that clustering still works correctly:
  • The user_clustering_session table used to have an is_anonymous column, and two optional columns: ip_address and user_id. Check that now is_anonymous and ip_address are no longer columns in the table, while user_id should be a required field now (i.e., not nullable).
  • Go to the /clustering page and run single and multi-user clustering to make sure that is still runs correctly. I think that after updating some Python library I am now getting a message about an is_copy function being deprecated, which is fine for now.
  • When the clustering finishes running, check the counts of rows in the various tables from when you tested the clustering last time. You should see that there are likely less than half as many entries in the user_clustering_session table as last time. This is because we are no longer looking at users who audited anonymously in the old way (where we only identify them by ip address). So this is only looking at registered users, and the new anon user accounts you created while testing.

Other small changes and how to test them

1. I removed unused imports from a ton of files (which is partially why there are 56 files changed for this PR). As long as the code compiles, and you are able to use the tool normally without it crashing, this is works fine 👍
2. Finishing the tutorial is no longer counted as a mission in the chart that counts the number of missions completed per user. You can test this by looking at the mean number of missions completed, as seen on the admin page analytics tab, and check that the mean missions completed is slightly lower for registered users is slightly lower on this branch than on the develop branch.
3. Removed quite a bit of code that was not being used anywhere, mostly code that was related to anonymous users. No need to test this, because it was not being called from anywhere :)
4. Removed the /student endpoint, which was used when the tool was beign tested on a group of students doing a summer course in 2015 (or maybe 2016)? I guess you can test by trying to go to the /student endpoint and seeing that there is nothing there 👍
5. Mostly in this commit I changed the indentation and such for a bunch of raw SQL queries to make them more readable. None of those should have actual changes to the structure of the query, so to test you can just copy any queries that were modified in that commit and paste them into Valentina Studio and see if the queries run. For queries that take parameters in the code, you replace the '?'s with appropriate values; for example, the query in the function selectAuditTaskInteractionsOfAUser takes a region_id and a user_id as parameters. This testing is mostly just to make sure that I didn't accidentally delete a comma or something that would cause the query not to run.

[aileenzeng]

• I wasn't able to get to the "You've been contributing a lot!" pop-up, so I wasn't able to test out signing up that way.

• The overview tab isn't including anonymous users that haven't completed an audit task. (EX: I have 2 anonymous users that haven't completed an audit task and 1 that has, so the display is saying 1 (1), rather than 3 (1)).

• Total Time Spent Auditing, Labels per User and Missions Completed By Users are blank on the admin/analytics page. I'm connected to sidewalk (not sidewalkturk) so maybe that's causing some problems?
  

• I'm using a bigger database dump, so I can't compare counts to what I got earlier:

Single User Clustering

Table	Rows
user_attribute	116226
user_attribute_label	143940
global_attribute	0
global_attribute_user_attribute	0

Multi User Clustering

Table	Rows
user_attribute	116226
user_attribute_label	143940
global_attribute	91321
global_attribute_user_attribute	116226

Single AND Multi User Clustering
(webpage didn't update - still says "Running...")

Table	Rows
user_attribute	116226
user_attribute_label	143940
global_attribute	91321
global_attribute_user_attribute	116226

[misaugstad]

Thanks for the testing!! I will take a look at these issues!

I'm connected to sidewalk (not sidewalkturk) so maybe that's causing some problems?

You should always use the normal sidewalk database, except for the one situation where you are classifying labels from the mturk study 😁

[misaugstad]

The overview tab isn't including anonymous users that haven't completed an audit task. (EX: I have 2 anonymous users that haven't completed an audit task and 1 that has, so the display is saying 1 (1), rather than 3 (1)).

This seems to be working correctly for me, so I just want to check two things. If they were anon user accounts where you then signed up with them, they would now be considered registered users. Also they needed to have started an audit task (so they would have needed to complete the tutorial and start the first task). Do the two anon user accounts you were talking about fall into either of these categories?

I wasn't able to get to the "You've been contributing a lot!" pop-up, so I wasn't able to test out signing up that way.

Okay I'm also not getting that anymore, I'll look into it.

Total Time Spent Auditing, Labels per User and Missions Completed By Users are blank on the admin/analytics page.

Yep, thank you! I found the problems and am working on them now 👍

I'm using a bigger database dump, so I can't compare counts to what I got earlier:

That's fine, as long as it seems to be working correctly, that should be good. I didn't make significant changes to the clustering, except to delete the old anon user code 👍

[aileenzeng]

This seems to be working correctly for me, so I just want to check two things. If they were anon user accounts where you then signed up with them, they would now be considered registered users. Also they needed to have started an audit task (so they would have needed to complete the tutorial and start the first task). Do the two anon user accounts you were talking about fall into either of these categories?

Aaah I see - in that case, it's working perfectly for me! The accounts that I had made earlier only visited the results/faq page (and didn't create accounts later on).

[misaugstad]

@aileenzeng prompting anon users to sign up should be fixed, and the charts on the admin page should be fixed as well. The anon user parts of those charts may look wrong because there are only a couple users. The only way to really test it is to make a few more anon user accounts (where they never sign up) with varying number of labels placed and missions completed. Then the charts should look normal.

[aileenzeng]

Hmm I can't seem to load the audit page:

I'm also getting some 404s on the admin analytics page:

[misaugstad]

Hmm I'm not having these issues... Can you just double check a few things:

1. Did you remember to run grunt?
2. Are you using the sidewalk database (you should be using sidewalk, not sidewalturk)
3. Did you run git pull?
4. If you did those things, could you empty the cache for the site and reload (in Chrome, you can do this by right-clicking on the refresh button and selecting "empty cache and hard reload")?

[aileenzeng]

Yes for the first three -- I'll try the fourth one!

[aileenzeng]

Got it working -- everything looks great!

One small issue: when I make an account through the audit screen, the window doesn't close. (A similar issue is logged in #1294)

[misaugstad]

Got it working -- everything looks great!

Yayyyy thank you! And so glad that it is all working now!! 😁

One small issue: when I make an account through the audit screen, the window doesn't close. (A similar issue is logged in #1294)

Yep, that's the same issue. I updated the title for that issue to reflect that it covers both signing in and signing up 😁

[misaugstad]

Merging into develop now!!