Stroop task using HSSM - parameters do not agree with those from HDDM

[moxtoby]

I am in the process of transitioning to HSSM from HDDM, learning to use correct HSSM syntax so I can possibly use HSSM to analyse the data in my current ongoing research. In my research I am utilising the Stroop task. I have been able to use HDDM to get the values of v (both congruent and incongruent conditions), a, and t. However, when I try to run the data with HSSM, I cannot seem to get similar values of v and a as those from HDDM.

I first started to run each participant individually (I have opened an issue about not being able to run HSSM for multiple participants #324).
My HDDM code to run the Stroop task is as follows:
m=hddm.HDDMRegressor(mydata,"v~C(cond,Treatment('Con'))",group_only_regressors=False,p_outlier=.05)
m.sample(10000, burn=1000)

This gives me the 2 drift rates (one each for the congruent and incongruent conditions) and the boundary separation for each participant.

In HSSM, the code I used to try to get the output for one participant is as follows:
L_stroop_model = hssm.HSSM(data=df_t, model = 'ddm', hierarchical = False, include=[{"name":"v", "prior":{"name":"Uniform", "lower":-8.0, "upper": 8.0}, "formula":"v~ 0+(1|cond)", },{"name":"a", "prior":{"name":"Uniform", "lower":0.2, "upper": 5.0},},{"name":"t", "prior":{"name":"Uniform", "lower":0.01, "upper":1.0},},],)
L_stroop_model.sample()

However, the output doesn't always converge (RHat around 1.5), and I sometimes have to run multiple iterations to make sure it converges. The estimated parameters are not close enough to those from HDDM to give me confidence. Am I using the correct HSSM syntax for the Stroop task? (I have updated the data file from HDDM to HSSM to include responses of -1 and 1. Reaction time is recorded in seconds.)

For information, I then tried to input data from all participants together in one data file, using the subj_idx column to identify the participants. The code from HDDM remains the same, and the output produces the 2 drift rates (v) for each condition, the a and t parameters, separately for each participant.

In HSSM, to run all participants together, I used the following code:
JLM_stroop_model = hssm.HSSM(data=df, model = 'ddm', hierarchical = True, include=[{"name":"v", "prior":{"name":"Uniform", "lower":-8.0, "upper": 8.0}, "formula":"v~ 0+(1|cond) + (1|subj_idx)", },{"name":"a", "prior":{"name":"Uniform", "lower":0.2, "upper": 5.0}, "formula":"a~ 0+(1|subj_idx)",},{"name":"t", "prior":{"name":"Uniform", "lower":0.01, "upper":1.0}, "formula":"t~ 0+(1|subj_idx)",},],)
JLM_stroop_model.sample()

The output from the above HSSM code indicates non-convergence (RHat much above 1.1) and the parameters are not at all similar to those from HDDM. I think I must be doing something wrong on the HSSM syntax. Can you help?

The HSSM version I am using is 0.1.5. The HDDM version is 0.8.0

For information, the v, a and t values for HDDM vs HSSM are as follows (when entering data one participant at a time):
For the 3 test participants:
HDDM v (con) : 2.79, 2.17, 2.82
HDDM v (inc) : 2.12, 1.86, 1.97
HDDM a : 1.57, 2.11, 1.37
HDDM t : 0.279, 0.311, 0.291

For the 3 test participants:
HSSM v (con) : 2.48, 1.65, 3.14
HSSM v (inc) : 1.32, 2.01, 2.89
HSSM a : 0.936, 1.1, 0.866
HSSM t : 0.371, 0.361, 0.406

I am hoping to transition to HSSM as I am finding the Gelman-Rubin convergence test not working in HDDM (another issue opened but I am not sure pymc is supported anymore pymc-devs/pymc2#205). I hope you can help.

For my study, I will be running the Stroop test, Stop Signal test (just the go-trials will use HDDM/HSSM), and Go/No-go test. I have raised this query for the Stroop test, and a separate query for the Stop Signal go-trials.

Many thanks.

[AlexanderFengler]

There are a few things that might have an impact on the results here.
One of them is the overall prior specification in the defaults for HSSM, which we are currently working on (the corresponding PR is open and we are very short of releasing the next version of HSSM).

However, as a first pass, could you try to run your HSSM model with the argument loglik_kind = "blackbox"?

This falls back on exactly the likelihood computation that was used in HDDM, so it would help zero in on the source of the discrepancies.

Best,
Alex

[moxtoby]

Thank you Alex. I will give this a go over the weekend and report back. Many thanks again.

Best wishes,
Michelle

[moxtoby]

Hello Alex,

With my Stroop data, if I enter the task data individually per participant, adding the loglik_kind = "blackbox" option seems to bring down the RHat in 2 of the 3 participants.

Conversely, for the SST Go trials, adding the loglik_kind = "blackbox" option raised the RHat parameter, the opposite of what happened with the Stroop data.

Should the "blackbox" option give inconsistent results across the two tasks? The only difference is the addition of "formula":"v~ 0+(1|cond)" in the Stroop HSSM command.

I have attached a spreadsheet with the values in HDDM vs HSSM (with and without "blackbox" option).

I am not sure at this point I feel comfortable with HSSM as the parameters estimated do not seem to be close enough to those from HDDM to give me comfort. If you can suggest anything else to try in HSSM to bring down RHat and make the output more consistent with those from HDDM, please let me know. Many thanks for your help.

Best wishes,
Michelle

Compare HDDM HSSM.xlsx

[AlexanderFengler]

Thank you for performing the contrast.
We will look into it.

Sidenote: If you use the blackbox version, you should actually run more samples (as in HDDM), so some of those rhat's will probably come down still.

It would be illustrative to see the actual chains, but in general you are right, the mean parameter estimates should align.

[moxtoby]

Thank you Alex.
I have now run HSSM but using 20000 samples and 2000 burn-in, and the results are a lot closer to those from HDDM. The RHat values are almost all 1.1 or less. For the one participant's SST Go data that did not have RHat under 1.1, I re-ran HSSM with 25000 samples and RHat has decreased further (now 1.13 or lower).
I have attached the an updated spreadsheet for your reference.

I am now more confident in using HSSM in Stroop and SST Go trials. I also would like to run HSSM with the Go/No-go trial. I will raise another query if I run into problems with it.

Finally, do you recommend running each participant one at a time in HSSM? When I put all the participants' data in one data file (using subj_idx column to identify each one), the values I get from HSSM does not make sense, and the RHat is very large. I have raised this query in thread #324. I hope you can help me with this as well. Thank you so much again.

Best wishes,
Michelle
Compare HDDM HSSM 2.xlsx

[moxtoby]

Hi Alex,

I have just commented on an outstanding thread on HSSM and Go/No-go task data (#173). As the thread is not recent (July 2023), should I open up a new thread for my query? Thank you.

Best wishes,
Michelle

[moxtoby]

Hi Alex, I have just managed to use HSSM successfully to estimate parameters which agrees with HDDM. (I forgot that z was set to 0.5 in HDDM in Stroop and SST Go trials). Thank you so much for your help. This issue can now be closed. I will open a new issue regarding Go/No-go data entry into HSSM.

Best wishes,
Michelle