Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

graph_color_d2_symmetric_double_int_int_TestExecSpace seg faults in cuda/10.1 + Volta nightly test on kokkos-dev-2 #634

Closed
ndellingwood opened this issue Mar 4, 2020 · 5 comments
Closed

graph_color_d2_symmetric_double_int_int_TestExecSpace seg faults in cuda/10.1 + Volta nightly test on kokkos-dev-2 #634

ndellingwood opened this issue Mar 4, 2020 · 5 comments
Assignees
@brian-kelley
Labels
bug fixed-in-develop bug InDevelop

Comments

@ndellingwood
Copy link
Contributor

The following test began seg faulting in the nightly cuda/10.1 test on kokkos-dev-2 (Volta70 arch), this did not occur in the UVM build:

02:44:31 3: [ RUN      ] cuda.graph_graph_color_d2_symmetric_double_int_int_TestExecSpace
02:44:31 3/8 Test #3: graph_cuda .......................***Exception: SegFault  3.52 sec

@brian-kelley I'm guessing this is related to the changes from PR #623?

Reproducer:

./kokkos-kernels/scripts/cm_test_all_sandia cuda/10.1 --build-list=Cuda_OpenMP
@ndellingwood
Copy link
Contributor Author

Adding link to failing nightly build output:

@brian-kelley
Copy link
Contributor

@ndellingwood I was able to replicate a crash. It was actually in resolveConflictsSerial(), which runs on host, but I think cuda tends to generate more conflicts.

@brian-kelley
Copy link
Contributor

This bug happens randomly when a raw malloc'd array happens to contain all zeros. The fix was just to initialize it.

brian-kelley added a commit to brian-kelley/kokkos-kernels that referenced this issue Mar 4, 2020
@brian-kelley
Fixed D2 resolveConflictsSerial() memory safety (kokkos#634)
1c1d91f 
Buffer "forbidden" was read in this loop:
while(forbidden[c] == v) c++;

but forbidden was not initialized. If it happened to be filled with 0,
then when coloring the first vertex this loop would read past the end
of the array (causing random crashes, see kokkos#634)
This was referenced Mar 4, 2020
Closed
Merged
brian-kelley added a commit that referenced this issue Mar 4, 2020
@brian-kelley
Merge pull request #638 from brian-kelley/Fix634
0424574 
Fixed D2 resolveConflictsSerial() memory safety (#634)
@brian-kelley
Copy link
Contributor

@ndellingwood How do I log into jenkins? I want to keep an eye on things like this.

@ndellingwood
Copy link
Contributor Author

If you aren't able to sign in when clicking the link above you'll need to request through WebCARS, I'll send you an email.

@crtrott crtrott closed this as completed Apr 14, 2020
@kokkos-devops-admin kokkos-devops-admin mentioned this issue Jun 21, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brian-kelley brian-kelley

Labels
bug fixed-in-develop bug InDevelop
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@brian-kelley @crtrott @ndellingwood