Implement PhET-iO for data collection

[brettfiedler]

PhET-iO was implemented by JG in the early prototypes of Quad. Collecting relevant needs and information here for future refinement according to research needs.

Talking to MK, he reported this to me in Slack (when talking about Ratio and Proportion, but the same applies here):

If we don't want to record to metacog for PhET-iO research, we can also use localFile recording, this just stores events in the browser for you to download from the console when done recording. You can try it out here:
3:48
https://phet-dev.colorado.edu/html/ratio-and-proportion/1.2.0-dev.10/phet-io/wrappers/record/?localFile
3:49
Its existence is poorly documented here! https://github.com/phetsims/phet-io-wrappers/blob/361a103ffac5d317d28bd9077fe8019cad72ffc1/common/js/Recorder.js#L10

[brettfiedler]

Tagging relevant Ratio and Proportion links:
phetsims/ratio-and-proportion#142
https://github.com/phetsims/special-ops/issues/202 (private)

[brettfiedler]

To bring up when we start in on PhET-iO development:

1. A way to synchronize PhET-iO data collection with other data collection methods e.g., screen capture, video camera

• Maybe a button in the Preferences menu that begins the recording process? Perhaps a sound that begins with that button audible on other collection sources?

3. Where to store the data? Locally? Server?

[jessegreenberg]

My understanding is that this is top priority for me this week.

[brettfiedler]

Copied over and edited from meeting notes:

Minimum functionality for data collection

Synchronization

• Precise Start/Stop to data collection (@zepumph added in Create a Clapperboard for synchronizing recording scenery-phet#739)
  • Don’t want to fully start data collection with press of button, but we can spit out the state at that point. Ignoring stuff before is “asking for trouble”.
  • Button in Pref menu w/ large visual indicator (visible from camera several feet away) and audible, unique sound (Let’s say 100ms duration for both to start)

Storage needs

• Metacog cloud storage with 4 digit unique identifier that can be pulled from the cloud server and parsed using a script similar to what @zepumph made in Brainstorming movement and sim state data output ratio-and-proportion#142 (comment).

Minimum variable tracking for data collection in Quadrilateral:

1. Time (global)
2. Sim model values (states and positions)

• Quad:
  • Vertex positions
  • angleToleranceInterval (current value) [some possible values: -inf, inf, 0.01, 0.05, 0.1, etc]
  • sideABsideCDParallelSideChecker.isParallelProperty: boolean, sideBCsideDAParallelSideChecker.isParallelProperty: boolean
  • Current shape
  • parallelSideChecker parallel state
  • Initialized constants and query parameter values

3. Input cluster (arrays corresponding to each input if available)
   • Drag Listeners for mouse,
   • Keyboard listeners,
   • [NOT FOR QUAD] mediaPipe interacted with (mediaPipeInteractedWithProperty) + direct mediaPipe landmarks
   • OpenCV connected property (Am I being interacted with: (is the sim updating? I think) does it have enough markers to update)

• Tangible connected property (If BLE/wired serial is connected: TRUE or FALSE)

@jessegreenberg, feel free to edit and add any details I missed!

[jessegreenberg]

Thanks @BLFiedler thats great! There were some quad specific things that were listed yesterday that I no longer saw in the document. I think they were parallelogram state and vertex positions. Was there anything else?

[brettfiedler]

I believe they just got rearranged and should be below Sim model values above. But that does remind me of something. Do we have any property that states how many sets of parallel lines we have?

[brettfiedler]

Mentioning parallel efforts in phetsims/ratio-and-proportion#468 for linking purposes.

[jessegreenberg]

Got it, thanks! I moved all state info under that checkbox.

Do we have any property that states how many sets of parallel lines we have?

We currently have a set of Property that would look like this in the data stream
sideABsideCDParallelSideChecker.isParallelProperty: boolean
sideBCsideDAParallelSideChecker.isParallelProperty: boolean

Would that work? We also have a parallelSidePairs array whose length is the number of pairs of parallel sides. That would not show up in state currently but could with some instrumentation. But I would prefer using ParallelSideChecker.

[brettfiedler]

Yeah, I think those would work!

[jessegreenberg]

Initialized constants and query parameter values

@BLFiedler can you please comment on which constants and query parameter values you would like? I scanned QuadrilateralQueryParameters and I am not sure if any are useful. All the tolerance intervals have a dynamic behavior(change over time) now and will show up in state in Properties. The rest are for debugging. Other constants do not change so we know their values without PhET-iO. But let me know if it would be helpful to have some appear in state.

[brettfiedler]

Sure - I think the idea behind this is if things end up changing over the course of development, there would be a log during the particular study on that specific version of any of the constants that may change later on. Also, having the information with the data prevents data loss in the future if for some reason versioning for the sims was lost or just flat out prevents someone from associating the data with the wrong version (I'm thinking about some of the confusions that may or may not have happened during the Voicing survey). But, you're also right that any adjusted variables associated with query parameters will be in the URL of the thing we send :)

Somewhere I think it would be good just to have the sim tell us (with no bias or possible typos from a researcher) the default/initialized values of these:
angleToleranceInterval
deviceAngleToleranceInterval
toleranceIntervalScaleFactor
shapeAngleToleranceInterval
deviceShapeAngleToleranceInterval
deviceShapeLengthToleranceInterval
lengthToleranceIntervalScaleFactor
shapeLengthToleranceInterval

[jessegreenberg]

OK thanks, wasn't sure the best way to instrument static fields for PhET-iO but I think I found something that works.

[jessegreenberg]

It would be nice for the Vector2 data to be broken out into two columns for x and y instead of one per Vector2.

This is done.

[jessegreenberg]

Sound was adjusted in phetsims/scenery-phet#739 and confirmed as OK by @BLFiedler.

@BLFiedler what is remaining for this issue? Is there any other data or anything else related to iO instrumentation needed?

[jessegreenberg]

Currently the parsing into CSV has data as null until it is first changed by the user. It would be helpful to fill in those null values with the defaults from the sim.

[jessegreenberg]

I got #97 (comment) for most Properties, for example now vertex positions are filled in with the initial values from the phetioEngine.phetioElementAddedEmitter.

[brettfiedler]

@jessegreenberg if you can upload all of the files generated on metacog on 6/11/2022, then we can close this issue until further studies. If the files aren't too large, you can just e-mail them to me and I'll put them in the appropriate data folder.

This backend collection is likely to be used in quad interviews this summer and fall - but we'll bring up a new issue with any changes needed.

[jessegreenberg]

Hey @BLFiedler, would this data still be useful at this point? I can't remember how to collect exactly, but could relearn. Sorry I let this slip.

[brettfiedler]

No worries, we did not find time to work on this element specifically. However, it would be nice to have for some testing on my end when I do find some time, as well as archival purposes. I believe it was fairly quick and maybe MK can get you up to speed quickly?

[jessegreenberg]

OK thanks, Ill take a look and connect with MK.

[zepumph]

I believe that @jessegreenberg has a good script for parsing the logs into data that is helpful for quad analysis, so is all you need help on getting the data from our data storage provider?

[jessegreenberg]

@zepumph yes please! I forgot where to go and how to access... :(

[zepumph]

I slacked you some data, let me know if you have any questions. Also can you please slack me the exact link used for the recording? I can't find it and need to update the phet-io documentation with it for the history books.

[jessegreenberg]

OK, after all this time...

I parsed the data provided by @zepumph and emailed the output (and raw data) to @BLFiedler.

I sent a slack message to @zepumph with the version number of the build we used for the recording.

What's next @BLFiedler?

[brettfiedler]

Thanks! All good here. Will let everyone know what becomes of the data as I find some off time to work on it!