Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arecord tampering causes DSP crashes #489

Closed
cujomalainey opened this issue Oct 12, 2018 · 17 comments
Closed

Arecord tampering causes DSP crashes #489

cujomalainey opened this issue Oct 12, 2018 · 17 comments
Assignees
Labels
bug Something isn't working as expected GLK Applies to Gemini Lake platform P1 Blocker bugs or important features verified QA/Bug reporter verifies solution worked.
Milestone

Comments

@cujomalainey
Copy link
Contributor

cujomalainey commented Oct 12, 2018

I am able to cause the DSP to crash pretty easily with a simple script.
while arecord -D hw:0,99 -f S32_LE -d 1; do :; done

Then by tampering with loop by using C-z (to suspend) or C-c (to interrupt) you can crash the dsp pretty easily.

Example of DSP panic by interrupting loop

[  103.538428] sof-audio sof-audio: error : DSP panic!
[  103.538479] sof-audio sof-audio: status: fw entered - code 00000005
[  103.538645] sof-audio sof-audio: error: can't enter idle
[  103.538648] sof-audio sof-audio: error: trace point 00004000
[  103.538652] sof-audio sof-audio: error: DSP Firmware Oops
[  103.538655] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[  103.538658] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[  103.538661] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[  103.538665] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[  103.538668] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[  103.538671] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[  103.538674] sof-audio sof-audio: stack dump from 0xbe07fec0
[  103.538678] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[  103.538681] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[  103.538684] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe00361c 0x00000000
[  103.538687] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe05b074 0xbe05b050
[  103.538690] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[  103.538693] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[  103.538697] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[  103.538700] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[  103.538703] sof-audio sof-audio: error: waking up any trace sleepers

a run where I simply suspend the stream (no resume)


[   82.612077] sof-audio sof-audio: firmware boot complete
[   90.108473] sof-audio sof-audio: error : DSP panic!
[   90.108538] sof-audio sof-audio: status: fw entered - code 00000005
[   90.108731] sof-audio sof-audio: error: can't enter idle
[   90.108734] sof-audio sof-audio: error: trace point 00004000
[   90.108737] sof-audio sof-audio: error: DSP Firmware Oops
[   90.108740] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[   90.108744] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[   90.108747] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[   90.108751] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[   90.108754] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[   90.108757] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[   90.108760] sof-audio sof-audio: stack dump from 0xbe07fec0
[   90.108764] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[   90.108767] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[   90.108770] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe003600 0x00000000
[   90.108773] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe05b074 0xbe05b050
[   90.108776] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[   90.108780] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[   90.108783] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[   90.108786] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[   90.108789] sof-audio sof-audio: error: waking up any trace sleepers

Another run where I suspend then resume arecord (note, did not check if crash happened before resume)

[1]+  Stopped                 arecord -D hw:0,99 -f S32_LE -d 1
localhost ~ # fg
arecord -D hw:0,99 -f S32_LE -d 1
overrun!!! (at least 1473.533 ms long)
arecord: pcm_read:2143: read error: Connection timed out

the kernel output

[   24.127078] sof-audio sof-audio: firmware boot complete
[   27.873506] sof-audio sof-audio: error : DSP panic!
[   27.873538] sof-audio sof-audio: status: fw entered - code 00000005
[   27.873701] sof-audio sof-audio: error: can't enter idle
[   27.873704] sof-audio sof-audio: error: trace point 00004000
[   27.873708] sof-audio sof-audio: error: DSP Firmware Oops
[   27.873711] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[   27.873715] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[   27.873718] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[   27.873721] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[   27.873725] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[   27.873728] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[   27.873731] sof-audio sof-audio: stack dump from 0xbe07fec0
[   27.873735] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[   27.873738] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[   27.873741] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe003600 0x00000000
[   27.873745] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe05b074 0xbe05b050
[   27.873748] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[   27.873751] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[   27.873754] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[   27.873758] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[   27.873761] sof-audio sof-audio: error: waking up any trace sleepers
[   28.847341] sof-audio sof-audio: error: ipc timed out for 0x60040000 size 0xc
[   28.847369] sof-audio sof-audio: status: fw entered - code 00000005
[   28.847508] sof-audio sof-audio: error: can't enter idle
[   28.847514] sof-audio sof-audio: error: trace point 00004000
[   28.847520] sof-audio sof-audio: error: DSP Firmware Oops
[   28.847526] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[   28.847533] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[   28.847559] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[   28.847562] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[   28.847566] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[   28.847569] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[   28.847573] sof-audio sof-audio: stack dump from 0xbe07fec0
[   28.847576] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[   28.847579] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[   28.847583] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe003600 0x00000000
[   28.847586] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe05b074 0xbe05b050
[   28.847589] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[   28.847593] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[   28.847596] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[   28.847600] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[   28.847603] sof-audio sof-audio: error: waking up any trace sleepers
[   28.847616]  DMIC01: ASoC: trigger FE failed -110
[   29.151353] sof-audio sof-audio: error: ipc timed out for 0x60050000 size 0xc
[   29.151381] sof-audio sof-audio: status: fw entered - code 00000005
[   29.151523] sof-audio sof-audio: error: can't enter idle
[   29.151529] sof-audio sof-audio: error: trace point 00004000
[   29.151535] sof-audio sof-audio: error: DSP Firmware Oops
[   29.151541] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[   29.151547] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[   29.151554] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[   29.151560] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[   29.151566] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[   29.151587] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[   29.151590] sof-audio sof-audio: stack dump from 0xbe07fec0
[   29.151594] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[   29.151597] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[   29.151600] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe003600 0x00000000
[   29.151603] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe05b074 0xbe05b050
[   29.151606] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[   29.151610] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[   29.151613] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[   29.151616] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[   29.151619] sof-audio sof-audio: error: waking up any trace sleepers
[   29.151632]  DMIC01: ASoC: trigger FE failed -110

These results are all from V0.5 drop, will continue testing as I move around the tree

@ranj063
Copy link
Collaborator

ranj063 commented Oct 12, 2018

@singalsu is this related to the crash that you are seeing after 46 iterations of dmic capture ?

@xiulipan
Copy link
Contributor

@cujomalainey
Seems duplicate to #489
And like some DMIC issues

@cujomalainey
Copy link
Contributor Author

@xiulipan you linked to this issue, do you mean #487? I think thats very possible

@xiulipan
Copy link
Contributor

@cujomalainey
Yes, #487.
I think this panic will only happen with DMIC record.
And it is a software panic report by sa_agent.

@lgirdwood
Copy link
Member

@singalsu Looks like another vector that may help you identifying root cause. @ZhendanYang can you try this test with SSP only to confirm it's DMIC issue.

@cujomalainey
Copy link
Contributor Author

cujomalainey commented Oct 16, 2018

@lgirdwood I don't thinks its DMIC specific, I was able to produce something similar by running a similar script and tampering with it but instead using speaker-test. Here is the DSP crash from that.

[  341.973324] sof-audio sof-audio: error : DSP panic!
[  341.973356] sof-audio sof-audio: status: fw entered - code 00000005
[  341.973581] sof-audio sof-audio: error: can't enter idle
[  341.973583] sof-audio sof-audio: error: trace point 00004000
[  341.973587] sof-audio sof-audio: error: DSP Firmware Oops
[  341.973590] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[  341.973593] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[  341.973596] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[  341.973599] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[  341.973602] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[  341.973605] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[  341.973607] sof-audio sof-audio: stack dump from 0xbe07fec0
[  341.973611] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[  341.973614] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[  341.973617] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe00361c 0x00dbb993
[  341.973620] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe059d74 0xbe059d50
[  341.973623] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[  341.973626] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[  341.973629] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[  341.973631] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[  341.973634] sof-audio sof-audio: error: waking up any trace sleepers
[  343.712330] sof-audio sof-audio: error: ipc timed out for 0x60040000 size 0xc
[  343.712358] sof-audio sof-audio: status: fw entered - code 00000005
[  343.712523] sof-audio sof-audio: error: can't enter idle
[  343.712526] sof-audio sof-audio: error: trace point 00004000
[  343.712529] sof-audio sof-audio: error: DSP Firmware Oops
[  343.712532] sof-audio sof-audio: error: Exception Cause: IllegalInstructionCause, Illegal instruction
[  343.712535] sof-audio sof-audio: EXCCAUSE 0x00000000 EXCVADDR 0x00000000 PS       0x00060725 SAR     0x00000005
[  343.712538] sof-audio sof-audio: EPC1     0xbe0230b9 EPC2     0xbe010bee EPC3     0xbe010bee EPC4    0x00000000
[  343.712541] sof-audio sof-audio: EPC5     0xbe01cbe5 EPC6     0x00000000 EPC7     0x00000000 DEPC    0x00000000
[  343.712544] sof-audio sof-audio: EPS2     0x00060720 EPS3     0x00060720 EPS4     0x00000000 EPS5    0x00060120
[  343.712547] sof-audio sof-audio: EPS6     0x00000000 EPS7     0x00000000 INTENABL 0x00010198 INTERRU 0x00000222
[  343.712549] sof-audio sof-audio: stack dump from 0xbe07fec0
[  343.712552] sof-audio sof-audio: 0xbe07fec0: 0xbe015142 0xbe07ff50 0xbe04c858 0xbe04d6a4
[  343.712555] sof-audio sof-audio: 0xbe07fec4: 0xfe0228b4 0xbe07ff10 0xbe0410d0 0x00000001
[  343.712558] sof-audio sof-audio: 0xbe07fec8: 0x00000000 0x00004b00 0xbe00361c 0x00dbb993
[  343.712561] sof-audio sof-audio: 0xbe07fecc: 0xfe0228b4 0xbe07ff20 0xbe059d74 0xbe059d50
[  343.712564] sof-audio sof-audio: 0xbe07fed0: 0x7e010bee 0xbe07ff40 0x00030082 0x00000005
[  343.712567] sof-audio sof-audio: 0xbe07fed4: 0x00001604 0xbe07ff40 0x00030082 0x00000005
[  343.712570] sof-audio sof-audio: 0xbe07fed8: 0x00000001 0x00000016 0x016e6263 0xbe05d718
[  343.712573] sof-audio sof-audio: 0xbe07fedc: 0xbe010b11 0xbe07ffa0 0x00000000 0x00000001
[  343.712576] sof-audio sof-audio: error: waking up any trace sleepers
[  343.712582]  Speakers: ASoC: trigger FE failed -110

@lgirdwood
Copy link
Member

@cujomalainey ok, must be something in the pipeline or DMA related.
@ZhendanYang can you try modifying the script and use speaker-test and check this applies to both HDA and SSP endpoints. This will tell us if it's pipeline or DMA specific (since both use different DMA)

@lgirdwood lgirdwood added bug Something isn't working as expected P1 Blocker bugs or important features br-master GLK Applies to Gemini Lake platform labels Oct 17, 2018
@ZhendanYang
Copy link

On up2, tests are blocked by #443 . Please @keqiaozhang if it's a GLK issue.

@mmaka1
Copy link

mmaka1 commented Oct 18, 2018

@cujomalainey @lgirdwood I am able to repro DSP crash easily with the arecord loop, however I get 'out of memory', not the original 'can't enter idle'. With the #495 applied, there is no repro on my board. No other issues like the 'idle' one appeared.
Are you still observing this with 079558d?

@cujomalainey
Copy link
Contributor Author

I cannot reproduce this currently at master. I'm not seeing any errors in the trace buffer. These is the only errors I'm seeing in the kernel logs.

[  322.234723] sof-audio sof-audio: XRUN for unknown stream, msg_id 0
and
[  362.033758] sof-audio sof-audio: error: DSP trace buffer overflow 16 bytes. Total messages 479

@lgirdwood
Copy link
Member

@cujomalainey A PR went in today from @mmaka1 that fixed a leak in the DMA.
@ZhendanYang can you validate at your end too.

@mengdonglin
Copy link
Collaborator

@keqiaozhang can you verify this on Yorp? I think Zhendan's validation on UP2 can still be blocked by #443

@keqiaozhang
Copy link
Collaborator

@cujomalainey @lgirdwood
I cannot reproduce this issue on my Yorp with latest sof-master. @cujomalainey please try with latest code base.

  1. sof-master: 459ccd2
  2. kernel-cros-sof-v4.14-rebase: 3ad0e4ddebf
  3. soft-master: a1dfcb7

@cujomalainey
Copy link
Contributor Author

@keqiaozhang I checked out the same location and ran it 10 times with tampering and it has passed every time.

@mengdonglin mengdonglin added the verified QA/Bug reporter verifies solution worked. label Oct 24, 2018
@lgirdwood lgirdwood added this to the 1.3 milestone Apr 1, 2019
@lgirdwood
Copy link
Member

lgirdwood commented Apr 1, 2019

@stevyan @mengdonglin shouldn't this be re-opened if it's listed as related with a PM bug now ?

@keqiaozhang
Copy link
Collaborator

@lgirdwood
I didn't get why this issue should be re-opened.
But I re-tested this issue on GLK and I didn't observe this issue.
Test recipe:
sof/tplg: tag/v1.3-rc1
kernel: tag/cros-sof-v4.14-rebase-v5-032819

@lgirdwood
Copy link
Member

@keqiaozhang thanks, I think I copy pasted from the daily test but it looks like kernel bug with same number as this. Sorry about this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working as expected GLK Applies to Gemini Lake platform P1 Blocker bugs or important features verified QA/Bug reporter verifies solution worked.
Projects
None yet
Development

No branches or pull requests

9 participants