brain.screenshot(time_viewer=True) crash

[crsegerie]

I was running the mne-bids-pipeline (so with mne-23dev cloned from GitHub in backend), but a bug appeared just after the creation of the reports of the first subject. After investigation, it turns out that the bug comes from MNE, from the option time_viewer=True in the screenshot method. By setting it to false, the image plots nicely.

I tried to make a reproducible example with sample data, but this does not reproduce the issue (Else). So I've also included here the (IF True) my stc file and my subjects file which isolate the code on my computer.

from pathlib import Path
import matplotlib
import matplotlib.pyplot as plt
import mne
from mne.datasets import sample
matplotlib.use('Qt5Agg')

if True:
    stc_fname = Path("mne_data/ds003392/derivatives",
                     'mne-bids-pipeline/sub-01/meg/',
                     'sub-01_task-localizer_coherent+dSPM+hemi-lh.stc')

    subjects_dir = Path("mne_data/ds003392/derivatives/freesurfer",
                        'subjects')
    subject = 'sub-01'
    stc = mne.read_source_estimate(stc_fname, subject=subject)
else:
    data_path = sample.data_path()
    fname = data_path + '/MEG/sample/sample_audvis-meg'

    stc = mne.read_source_estimate(fname)
    subject = "sample"
    subjects_dir = Path("C:/Users/charb/mne_data/MNE-sample-data/subjects")
brain = stc.plot(subject=subject, subjects_dir=subjects_dir,
                 initial_time=0.13, views=['lat'], hemi='split')
brain.toggle_interface()
brain._renderer.plotter.reset_camera()
brain._renderer.plotter.subplot(0, 0)
brain._renderer.plotter.reset_camera()
figs, ax = plt.subplots(figsize=(15, 10))

# BUG : Set time_viewer to False to remove the Bug
image = brain.screenshot(time_viewer=True)
ax.imshow(image)
ax.axis('off')
plt.show()

Expected results

An interactive plot.

Actual results

When I run the code, the figure appears one second and then crashes

(mne_dev) C:\Users\charb\Desktop\parietal>C:/Users/charb/anaconda3/envs/mne_dev/python.exe c:/Users/charb/Desktop/parietal/test.py
Using pyvista 3d backend.

Using control points [13.01218636 16.52222281 48.36637508]
Traceback (most recent call last):
  File "c:\Users\charb\Desktop\parietal\test.py", line 51, in <module>
    image = brain.screenshot(time_viewer=True)
  File "C:\Users\charb\anaconda3\envs\mne_dev\lib\site-packages\mne\viz\_brain\_brain.py", line 2631, in screenshot
    trace_img = np.reshape(
  File "<__array_function__ internals>", line 5, in reshape
  File "C:\Users\charb\anaconda3\envs\mne_dev\lib\site-packages\numpy\core\fromnumeric.py", line 299, in reshape
    return _wrapfunc(a, 'reshape', newshape, order=order)
  File "C:\Users\charb\anaconda3\envs\mne_dev\lib\site-packages\numpy\core\fromnumeric.py", line 58, in _wrapfunc
    return bound(*args, **kwds)
ValueError: cannot reshape array of size 1082952 into shape (1015,4)

(mne_dev) C:\Users\charb\Desktop\parietal>    

Additional information

I'm working on Windows.

(mne_dev) C:\Users\charb\Desktop\parietal\mne-bids-pipeline>python
Python 3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 04:59:43) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import mne; mne.sys_info()
Platform:      Windows-10-10.0.19041-SP0
Python:        3.9.2 | packaged by conda-forge | (default, Feb 21 2021, 04:59:43) [MSC v.1916 64 bit (AMD64)]
Executable:    C:\Users\charb\anaconda3\envs\mne_dev\python.exe
CPU:           Intel64 Family 6 Model 158 Stepping 10, GenuineIntel: 8 cores
Memory:        7.9 GB

mne:           0.23.dev0
numpy:         1.20.2 {blas=NO_ATLAS_INFO, lapack=lapack}
scipy:         1.6.2
matplotlib:    3.4.1 {backend=Qt5Agg}

sklearn:       0.24.1
numba:         0.53.1
nibabel:       3.2.1
nilearn:       0.7.1
dipy:          1.4.0
cupy:          Not found
pandas:        1.2.4
mayavi:        4.7.2
pyvista:       0.29.1 {pyvistaqt=0.3.0, OpenGL 4.5.0 - Build 26.20.100.6911 via Intel(R) UHD Graphics 630}
vtk:           9.0.1
PyQt5:         5.12.3
>>>

[welcome]

Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️

[larsoner]

I'm guessing this is some sort of off-by-one error as:

>>> 1082952 // 4 / 1014
267.0

@GuillaumeFavelier can you look?

[GuillaumeFavelier]

I added it in the list, I'll fix it ASAP.

[hoechenberger]

Thanks @GuillaumeFavelier!

I don't want to be pushy, but if you could kindly move this up on your priorities list this would be most appreciated :) I'm working with @crsegerie and unfortunately this bug is currently a blocker in his project.

Let me know if there's any way I can help you track this down! Happy to have a call too.

🤗

[crsegerie]

Yep. It would be most appreciated! And props to @hoechenberger who first isolated the bug

[GuillaumeFavelier]

I'm trying to reproduce so I downloaded sub-01_task-localizer_coherent+dSPM+hemi-lh.stc and the sub-01 folder but I obtain the following when I run the script:

/tmp/bug.py in <module>
     10     subjects_dir = 'subjects'
     11     subject = 'sub-01'
---> 12     stc = mne.read_source_estimate(stc_fname)
     13 else:
     14     data_path = sample.data_path()

~/source/mne-python/mne/source_estimate.py in read_source_estimate(fname, subject)
    301             fname += '-stc'
    302         elif any(stc_exist) or any(w_exist):
--> 303             raise IOError("Hemisphere missing for %r" % fname_arg)
    304         else:
    305             raise IOError("SourceEstimate File(s) not found for: %r"

OSError: Hemisphere missing for 'sub-01_task-localizer_coherent+dSPM+hemi-lh.stc'

[GuillaumeFavelier]

Also, as a way to mitigate since it's a blocker, does it work if you change the size parameter in stc.plot before calling screenshot()?

[hoechenberger]

OSError: Hemisphere missing for 'sub-01_task-localizer_coherent+dSPM+hemi-lh.stc'

@crsegerie Can you share the STC for the right hemisphere too? Both are needed here

[crsegerie]

Yes @GuillaumeFavelier, if I add size=(500, 500) in the stc.plot, the bug disappears !

[GuillaumeFavelier]

the bug disappears !

I like your enthusiasm :) For me, it's still there in a corner case of stc.plot 😅

[crsegerie]

Here is the right hemisphere : https://drive.google.com/file/d/1y7puZ7RPMGV6uB6tN_Pu-MgamuaAFvlb/view?usp=sharing

(In the script only left hemisphere is imported ? Weird, but ok...)

[hoechenberger]

(In the script only left hemisphere is imported ? Weird, but ok...)

Yeah, MNE tries to automatically find the other hemisphere. It's a bit odd, yes…

[GuillaumeFavelier]

Thanks for sharing. I still cannot reproduce locally but I'll try to simulate this scenario anyway and come up with a fix eventually.

About the origin of the issue itself, it's not clear so I can only guess here. Reasons why the trace plot would not have the correct size could be either unexpected DPI configuration or PyQt5 inconsistencies on Windows for example.

[hoechenberger]

@GuillaumeFavelier yes I also suspected some kind of hi-DPI interaction

@crsegerie can you check if you have any of those settings for high-resolution displays activated?

https://windowsreport.com/fix-hidpi-issues-windows-10/

[crsegerie]

I checked my graphic settings, but did not find any anomalies.

[crsegerie]

if it is more convenient for you, we can arrange a meeting if you want to explore my debugger on VS Code?

[hoechenberger]

@GuillaumeFavelier So the default size is 800; why would changing that to 500 change anything? What's the rationale? (It seems to work so this is great – but I have no idea why that works?)

[larsoner]

PyQt5 inconsistencies on Windows for example.

Probably this, or something similar. My guess is that there is a situation where the sizes reported are inconsistent, or we don't use them correctly, and that whether or not this situation occurs depends on the size itself and/or interactions with the window (resizes, etc.). So trying different size options will tell us if some work and others don't, and then we could start looking at various MNE / Qt / matplotlib / vtk variables so see what each layer "thinks" the size should be, and thus figure out who's getting it wrong and why.

[hoechenberger]

Thanks @larsoner!

I just realized that @crsegerie switched from size=800 (default) to size=(500, 500), i.e., a tuple. Maybe that's also part of the solution…

[larsoner]

IIRC an int just gets turned into (size, size) internally so size=800 should give identical results to (800, 800)

[crsegerie]

Ok. I've tested (500, 500), 500, (800, 800), and 800 and it's not a matter of tuple not tuple... Only the 500 are working

[hoechenberger]

Ok. I've tested (500, 500), 500, (800, 800), and 800 and it's not a matter of tuple not tuple... Only the 500 are working

You mean 500 and (500, 500) are working, yes?

[crsegerie]

Only the size matter, not the tuple

[crsegerie]

OK. 500 working, 600 working, 799 working, 800 not working, 801 not working

[larsoner]

Can you see if you can reproduce with a simple example using sample data? For example:

import os
import os.path as op
import mne
from mne.datasets import sample

data_path = sample.data_path()
sample_dir = op.join(data_path, 'MEG', 'sample')
subjects_dir = op.join(data_path, 'subjects')
fname_stc = op.join(sample_dir, 'sample_audvis-meg')
stc = mne.read_source_estimate(fname_stc, subject='sample')
brain = stc.plot(subjects_dir=subjects_dir, size=500, inital_time=0.1)
img = brain.screenshot(time_viewer=True)
print(img.shape)

For me I get:

(669, 500, 3)

Can you tell me if this works for you at 500 and 800? And if one or both of them succeed, what the print statement prints?

Also, what is your screen resolution? And can you tell me if this prints something other than 1.:

>>> python -c "from mne.conftest import pixel_ratio; print(pixel_ratio.__wrapped__())"
1.0

[crsegerie]

There is a small typo inital_time > initial_time.

For me the code works for both 500 and 800.

For 500 I obtain : (667, 500, 3)
For 800 I obtain : (1067, 800, 3)

My screen resolution : 1920x1080.

When I run the tests, I get 1.

[crsegerie]

So the only difference is my 667 and your 669