Brainwide Trial Data

[CodyCBakerPhD]

If we scan all trials tables in the entire IBL dataset via

all_sessions_all_trials_keys = dict()
all_sessions_all_trials_all_shapes = defaultdict(dict)
for session in all_sessions:
    if one.list_datasets(eid=session["id"], filename={'object': ['trial?']}):
        trials = one.load_object(id=session["id"], obj="trials", collection="alf")
        all_sessions_all_trials_keys.update({session["id"]: trials.keys()})
        all_sessions_all_trials_all_shapes[session["id"]].update({trial_key: value.shape for trial_key, value in trials.items()})
    else:
        all_sessions_all_trials_keys.update({session["id"]: None})

unique_trials_keys_to_sessions = defaultdict(list)
for session, trial_keys in all_sessions_all_trials_keys.items():
    if trial_keys and set(trial_keys) not in [  # Sometimes the columns are out of order
        set(existing_trial_key) for existing_trial_key in unique_trials_keys_to_sessions.keys()
    ]:
        unique_trials_keys_to_sessions[tuple(trial_keys)].append(session)

we find that there are about 11 different configurations, and sometimes common fields for certain configurations have uneven shapes for their values (the ONE system outputs a stderr notification when this happens)

While it might in theory be possible to define a global trials interface, it would seem better practice implement this more modularly for each experiment

If you limit the above snippet to just Brainwide sessions, they all share a single common configuration, minus the goCueTrigger_times they asked us not to include

For the Brainwide trials data we have the custom columns (with descriptions used from https://docs.google.com/document/d/1OqIqqakPakHXRAwceYLwFY9gOrm8_P62XIfCTnHwstg/edit#)

Trials:
  stimOff_times: |
Time in seconds, relative to the session start, of the stimulus offset, as recorded by an external photodiode.
  goCue_times: |
The start time of the go cue tone. This is the time the sound is actually played, that is, the command sent through soundcard sync was fed back into Bpod.  The go cue tone is a 100ms 5 kHz sine wave and occurs approximately at the time of stimulus onset.
  response_times: |
The time at which a response was recorded.  This marks the end of the closed loop state in Bpod and occurs when either 60 seconds have elapsed since the go cue, or the rotary encoder reaches a position equivalent to the stimulus on the screen reaching + or - 35º azimuth.
  choice: |
The response type registered for each trial where -1 corresponds to turning the wheel CCW, +1 turning CW, and 0 being a timeout (‘no-go’) where the wheel wasn’t moved to threshold within the 60 second time window.
  stimOn_times: |
The time at which the visual stimulus appears on the screen, as detected by the photodiode which is placed over the sync square that flips colour each time the screen is redrawn.
  contrastLeft: |
The contrast of the stimulus that appears on the left side of the screen (-35º azimuth).  When there is a non-zero contrast on the right, contrastLeft == 0, when there is no contrast on either side (a ‘catch’ trial), contrastLeft == NaN.
  contrastRight: |
The contrast of the stimulus that appears on the right side of the screen (35º azimuth).  When there is a non-zero contrast on the left, contrastRight == 0, when there is no contrast on either side (a ‘catch’ trial), contrastRight == NaN.
  probabilityLeft: |
Probability that the stimulus will be on the left-hand side for the current trial. The probability of right is 1 minus this.  For repeat trials (trainingChoiceWorld) the probability is N(bias,4), where bias is calculated using responses from the last 10 trials.  During biased blocks (biasedChoiceWorld) the probability may be 0.5, 0.8, or 0.2.
  feedback_times: |
The time of feedback delivery. For correct trials this is the time of the valve TTL trigger command, for incorrect trials this is the time of the white noise output trigger.
  feedbackType: |
Whether the feedback was positive (+1) or negative (-1).  Positive feedback indicates a correct response rewarded with sugar water.  Negative feedback indicates a trial timeout or incorrect response followed by a white noise burst.
  rewardVolume: |
The volume of reward delivered on each trial.  On trials where feedbackType == -1, rewardVolume == 0.  The reward volume is typically within the range of 1.5 to 3 and should not change within a session.
  firstMovement_times: |
The time of the first detected movement of the trial with a sufficiently large amplitude.  To be counted, the movement must occur between go cue and before feedback time.  The movement onset is sometimes just before the cue (occurring in the gap between quiescence end and cue start, or during the quiescence period but sub-threshold).  The movement is sufficiently large if it is greater than or equal to .1 radians.

the shapes for each of these fields for each of the sessions in the Brainwide experiments have been verified to be consistent.

[CodyCBakerPhD]

I assume we will want to map the column names to NWB Best Practice convention

First proposal for this is

  stimOff_times -> stim_off_time
  goCue_times -> go_cue_time
  response_times: -> response_time
  choice -> choice
  stimOn_times -> stim_on_time
  contrastLeft -> contrast_left
  contrastRight -> contrast_right
  probabilityLeft -> probability_left
  feedback_times -> feedback_time
  feedbackType -> feedback_type
  rewardVolume -> reward_volume
  firstMovement_times -> first_movement_time

[CodyCBakerPhD]

Another thing to discuss would be that what gets returned by the trials query in their API is an unordered dictionary, and sometimes the keys are out of a standard order

In NWB, the columns are defined in an ordered fashion, and we should fix the columns to a standard order for consistency between all the NWB files for this experiment

First proposal for order of columns (independent of name mapping proposal above)

  choice
  feedbackType
  rewardVolume
  contrastLeft
  contrastRight
  probabilityLeft
  feedback_times
  response_times
  stimOff_times
  stimOn_times
  goCue_times
  firstMovement_times

the thinking here is to cluster the timing columns from the others and within that cluster similar prefixes