Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improved sequence unrolling performance #79

Open
wants to merge 25 commits into
base: main
Choose a base branch
from
Open

Improved sequence unrolling performance #79

wants to merge 25 commits into from

Conversation

Toreil
Copy link
Collaborator

@Toreil Toreil commented Feb 24, 2025

Rewrite of the way the sequence is unrolled with a substantial improvement in performance. Also appears to have addressed an issue where. depending on resolution/field of view, the entire sequence would be executed instantaneously.

LUMC-LowFieldMRI and others added 13 commits December 10, 2024 12:30
…o be tested in imaging to ensure there's no unexpected changes.
Fixes deblanking IO signal not being transmitted
Fixes improper handling of rf ring down time
Shims were added to the gradient waveform but also set as offsets on the card level, meaning shim levels were double what they should be. Shims are no longer added to the gradient waveform and now use the card offsets exclusively. This has the added benefit of hugely increasing sequence unrolling speed.
Removes unnecessary usage of SimpleNamespace for information about ADC events, just returns the ADC gate signal and reference waveform now.

Also addressed some linting issues
Fixed gradient limit calculations. Now correctly throw error if gradient waveform exceeds limits and a seperate error if the combined waveform and shim offset exceeds limits.

Also fixed linting errors
Address linting errors
Copy link

github-actions bot commented Feb 24, 2025

Coverage

Coverage Report
FileStmtsMissCoverMissing
src/console/interfaces
   acquisition_data.py1202480%127–132, 141–142, 148–149, 152–153, 157–161, 176–177, 188–193, 199, 245
   acquisition_parameter.py63494%77, 92, 115, 143
src/console/pulseq_interpreter
   sequence_provider.py2564483%89, 91, 93, 95–97, 113, 143, 148, 151–153, 185–188, 216–219, 233–234, 274, 298, 322, 331–334, 342–344, 451, 457, 462–466, 492, 522–526, 546, 616–617
src/console/spcm_control
   abstract_device.py63630%2–119
   acquisition_control.py1601600%3–373
   rx_device.py2002000%2–474
   tx_device.py2122120%2–509
src/console/spcm_control/spcm
   pyspcm.py1431430%3–276
   tools.py52520%3–122
src/console/utilities
   json_encoder.py8362%22–24
   load_config.py27270%2–119
   plot.py17170%2–26
src/console/utilities/sequences/calibration
   fid_tx_adjust.py18194%54
   se_tx_adjust.py25292%56–57
src/console/utilities/sequences/spectrometry
   fid.py15150%2–61
   se_spectrum.py24292%42, 65
src/console/utilities/sequences/tse
   tse_3d.py2438266%109–110, 112–113, 115–116, 123–124, 126–128, 130–133, 136–147, 152–155, 162–173, 190, 212–219, 225, 254, 317–334, 341–342, 407, 410–422, 440, 442, 531–541
TOTAL4891105179% 

Tests Skipped Failures Errors Time
209 0 💤 0 ❌ 0 🔥 9.806s ⏱️

Copy link
Owner

@schote schote left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the PR @Toreil! I added some comments to the code, have you been able to verify the sequence calculation updates? Maybe a direct comparison of the unrolled sequence waveforms from the old implementation vs. the new implementation is a good test.

@schote
Copy link
Owner

schote commented Feb 24, 2025

Were you able to find out what caused the problem you are describing?

@Toreil
Copy link
Collaborator Author

Toreil commented Feb 25, 2025

Were you able to find out what caused the problem you are describing?

Not specifically. I saw the same thing happen when I did the bit shift for the acquisition gate signals in the wrong direction which essentially caused the gate to be high continuously, so whatever the bug was in the old code it must've been causing that gate signal to be high.

The other time I saw incorrect acquisition gating was with the fix I pushed previously which was when trying to replicate the control signals on the extra GPIO cards when those wernt installed. It would cause the TX card (process?) to crash but it would have jitter on the output of the GPIO pins on the card so the Rx card would just randomly acquire a few samples here and there. Very different behaviour, so I think it was related to the gate signal, not something else.

@Toreil Toreil requested a review from schote February 25, 2025 10:46
@Toreil
Copy link
Collaborator Author

Toreil commented Feb 26, 2025

Thank you for the PR @Toreil! I added some comments to the code, have you been able to verify the sequence calculation updates? Maybe a direct comparison of the unrolled sequence waveforms from the old implementation vs. the new implementation is a good test.

Yeah I did a direct comparison both on the bench and with imaging experiments and performance was identical. As far as I can tell merge is ready to go (I don't understand the failed tests, I ran them locally and there was no problem). I also tested seperately on Python 3.12 and confirm it's all working so the dependencies can be updated to reflect that.

@schote
Copy link
Owner

schote commented Feb 26, 2025

Thank you for the PR @Toreil! I added some comments to the code, have you been able to verify the sequence calculation updates? Maybe a direct comparison of the unrolled sequence waveforms from the old implementation vs. the new implementation is a good test.

Yeah I did a direct comparison both on the bench and with imaging experiments and performance was identical. As far as I can tell merge is ready to go (I don't understand the failed tests, I ran them locally and there was no problem). I also tested seperately on Python 3.12 and confirm it's all working so the dependencies can be updated to reflect that.

That is great, but I think it would be good to have an objective measure. For example. by unrolling a reference sequence with the current and the updated version of the sequence provider and reporting the sum of absolute difference between the resulting arrays (should be 0). I know this causes some extra work, but we have had some bugs in the past because we did not check changes carefully. I will have a look at the pytest error.
Regarding the static tests, mypy throws some errors. You should be able to see them by running mypy src/ in the project root. The CI which executes the test always uses a fresh install of the package. It could be, that there is a version missmatch. Let me know, if you need further support with this.

The phase of RF pulses is set relative to time since the first RF pulse of the sequence. This commit fixes an issue where that time was not being considered properly.
@Toreil
Copy link
Collaborator Author

Toreil commented Mar 4, 2025

That is great, but I think it would be good to have an objective measure. For example. by unrolling a reference sequence with the current and the updated version of the sequence provider and reporting the sum of absolute difference between the resulting arrays (should be 0). I know this causes some extra work, but we have had some bugs in the past because we did not check changes carefully. I will have a look at the pytest error. Regarding the static tests, mypy throws some errors. You should be able to see them by running mypy src/ in the project root. The CI which executes the test always uses a fresh install of the package. It could be, that there is a version missmatch. Let me know, if you need further support with this.

I ran a set of comparisons using the example sequences in the examples folder since they cover all basic functions of Pulseq. There are/were small differences but they're all attributable to rounding of the number of samples (which for shaped RF pulses has an outsized visual effect due to using resample which depends on the number of samples). The fast-unroll uses round to cast to int, whereas the main branch casts to int which always rounds down. Typically this manifests as a 20 us delay being 400 samples on the fast-unroll and 399 on the main branch of the sequence unroll for instance. I propose that since the small changes are due to round being used and round gives a more accurate number of samples the small differences are not detrimental (perhaps even an improvement) and this branch is ready to merge.

@schote
Copy link
Owner

schote commented Mar 5, 2025

That is great, but I think it would be good to have an objective measure. For example. by unrolling a reference sequence with the current and the updated version of the sequence provider and reporting the sum of absolute difference between the resulting arrays (should be 0). I know this causes some extra work, but we have had some bugs in the past because we did not check changes carefully. I will have a look at the pytest error. Regarding the static tests, mypy throws some errors. You should be able to see them by running mypy src/ in the project root. The CI which executes the test always uses a fresh install of the package. It could be, that there is a version missmatch. Let me know, if you need further support with this.

I ran a set of comparisons using the example sequences in the examples folder since they cover all basic functions of Pulseq. There are/were small differences but they're all attributable to rounding of the number of samples (which for shaped RF pulses has an outsized visual effect due to using resample which depends on the number of samples). The fast-unroll uses round to cast to int, whereas the main branch casts to int which always rounds down. Typically this manifests as a 20 us delay being 400 samples on the fast-unroll and 399 on the main branch of the sequence unroll for instance. I propose that since the small changes are due to round being used and round gives a more accurate number of samples the small differences are not detrimental (perhaps even an improvement) and this branch is ready to merge.

Could you share some of these results then? Maybe just post some magnitude/phase images of the different states here for documentation, that would be great!
I'll have a look at the remaining errors in pytest and the static tests.

@berksilemek
Copy link
Collaborator

berksilemek commented Mar 5, 2025

Hi all,

Do not have the full story here. The conversation triggered me: if int or rounding errors causes some phase issues that I have not been able to solve at issue #66 , which was apparent at 3T experiments as well. Have you tested if Decimal works (worse calculation performance compared to round or int but maybe the most stable for imaging) ? I implemented in Rx device as follows:

# Calculate gate duration
gate_length = Decimal(str(timestamp_1)) - Decimal(str(timestamp_0))

# Calculate the number of adc gate sample points (per channel)
gate_sample = int(round(gate_length * (Decimal(str(self.sample_rate)) * Decimal("1e6"))))   

This fixes small rounding errors for delay and phase instabilities as mentioned above.

@schote
Copy link
Owner

schote commented Mar 6, 2025

Hi all,

Do not have the full story here. The conversation triggered me: if int or rounding errors causes some phase issues that I have not been able to solve at issue #66 , which was apparent at 3T experiments as well. Have you tested if Decimal works (worse calculation performance compared to round or int but maybe the most stable for imaging) ? I implemented in Rx device as follows:

# Calculate gate duration
gate_length = Decimal(str(timestamp_1)) - Decimal(str(timestamp_0))

# Calculate the number of adc gate sample points (per channel)
gate_sample = int(round(gate_length * (Decimal(str(self.sample_rate)) * Decimal("1e6"))))   

This fixes small rounding errors for delay and phase instabilities as mentioned above.

Hi Berk,

thanks for your input on this, I think this is more about convention when unrolling the sequence.
Instead of truncating the calculated number of sample points by int(...), the use of round yields an extra sample point for some cases. However, because the waveforms are interpolated anyway, this does not matter. This is about a one data point on a 50 ns sampling grid.

Thank you for the readme update! Please use a seperate PR (even for such small changes) in the future.

@Toreil
Copy link
Collaborator Author

Toreil commented Mar 6, 2025

Normal - vs Fast-unroll

Comparison of gradient waveforms with the latest fix showing (essentially) identical waveforms, small difference are caused by different rounding of gradient event durations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: No status
Development

Successfully merging this pull request may close these issues.

Memory usage increases substantially after unrolling the sequence, when starting the acquisition.
4 participants