Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

>=raft-0.18.2: "test_uv_truncate_snapshot.c:42: assertion failed: status == result->status (23 == 0)" #35

Closed
ganto opened this issue Dec 21, 2023 · 16 comments
Closed

>=raft-0.18.2: "test_uv_truncate_snapshot.c:42: assertion failed: status == result->status (23 == 0)" #35

ganto opened this issue Dec 21, 2023 · 16 comments
Labels
bug Something isn't working help wanted Extra attention is needed

Comments

@ganto
Copy link
Owner

ganto commented Dec 21, 2023 •
edited
Loading

Attempting to build the RPM of >=raft-0.18.2 (from cowsql/raft) via mock on Fedora 38 fails with:

FAIL: test/unit/uv                                                                                                                                                                                                                                                                                                            
==================                                                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                                                              
Running test suite with seed 0x7fe5bd45...                                                                                                                                                                                                                                                                                    
UvWriterClose/aio                                                                                                                                                                                                                                                                                                             
  dir-fs=btrfs                                              [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=ext4                                               [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=xfs                                                [ SKIP  ]                                                                                                                                                                                                                                                         
UvWriterClose/threadpool                                                                                                                                                                                                                                                                                                      
  dir-fs=tmpfs                                              [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=zfs                                                [ SKIP  ]                                                                                                                                                                                                                                                         
UvWriterSubmit/noResources                                                                                                                                                                                                                                                                                                    
  dir-fs=tmpfs                                              [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=zfs                                                [ SKIP  ]                                                                                                                                                                                                                                                         
UvWriterSubmit/concurrentSame                                                                                                                                                                                                                                                                                                 
  dir-fs=tmpfs                                              [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=ext4                                               [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=btrfs                                              [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=xfs                                                [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=zfs                                                [ SKIP  ]                                                                                                                                                                                                                                                         
UvWriterSubmit/concurrent                                                                                                                                                                                                                                                                                                     
  dir-fs=tmpfs                                              [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=ext4                                               [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=btrfs                                              [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=xfs                                                [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=zfs                                                [ SKIP  ]                                                                                                                                                                                                                                                         
UvWriterSubmit/beyondEOF                                                                                                                                                                                                                                                                                                      
  dir-fs=tmpfs                                              [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=ext4                                               [ SKIP  ]                                                                                                                                                                                                                                                         
  dir-fs=btrfs                                              [ SKIP  ]
  dir-fs=xfs                                                [ SKIP  ]
  dir-fs=zfs                                                [ SKIP  ]
UvWriterSubmit/vecTwice                                     
  dir-fs=tmpfs                                              [ SKIP  ]
  dir-fs=ext4                                               [ SKIP  ]
  dir-fs=btrfs                                              [ SKIP  ]
  dir-fs=xfs                                                [ SKIP  ]
  dir-fs=zfs                                                [ SKIP  ]
UvWriterSubmit/vec                                          
  dir-fs=tmpfs                                              [ SKIP  ]
  dir-fs=ext4                                               [ SKIP  ]
  dir-fs=btrfs                                              [ SKIP  ]
  dir-fs=xfs                                                [ SKIP  ]
  dir-fs=zfs                                                [ SKIP  ]
UvWriterSubmit/twice                                        
  dir-fs=tmpfs                                              [ SKIP  ]
  dir-fs=ext4                                               [ SKIP  ]
  dir-fs=btrfs                                              [ SKIP  ]
  dir-fs=xfs                                                [ SKIP  ]
  dir-fs=zfs                                                [ SKIP  ]
UvWriterSubmit/two                                          
  dir-fs=tmpfs                                              [ SKIP  ]
  dir-fs=ext4                                               [ SKIP  ]
  dir-fs=btrfs                                              [ SKIP  ]
  dir-fs=xfs                                                [ SKIP  ]
  dir-fs=zfs                                                [ SKIP  ]
UvWriterSubmit/one                                          
  dir-fs=tmpfs                                              [ SKIP  ]
  dir-fs=ext4                                               [ SKIP  ]
  dir-fs=btrfs                                              [ SKIP  ]
  dir-fs=xfs                                                [ SKIP  ]
  dir-fs=zfs                                                [ SKIP  ]
UvWriterInit/noResources                                    [ ERROR ]
Error: test/unit/test_uv_writer.c:256: assertion failed: _rv == RAFT_TOOMANY (0 == 23)
Error: child killed by signal 6 (Aborted)
UvOsJoin/dirAndFilenameMax                                  [ OK    ] [ 0.00003054 / 0.00003026 CPU ]
UvOsJoin/dirAndFilenameTooLong                              [ OK    ] [ 0.00001283 / 0.00001264 CPU ]
UvOsJoin/filenameTooLong                                    [ OK    ] [ 0.00000978 / 0.00000937 CPU ]
UvOsJoin/dirTooLong                                         [ OK    ] [ 0.00001116 / 0.00001086 CPU ]
UvOsJoin/basic                                              [ OK    ] [ 0.00000747 / 0.00000724 CPU ]
UvFsRenameFile/same                                         [ OK    ] [ 0.04239300 / 0.00061866 CPU ]
UvFsRenameFile/rename                                       [ OK    ] [ 0.03020970 / 0.00062390 CPU ]
UvFsMakeFile/exists                                         [ OK    ] [ 0.05205354 / 0.00108483 CPU ]
UvFsMakeFile/notExists                                      [ OK    ] [ 0.04082608 / 0.00059668 CPU ]
UvFsProbeCapabilities/noResources                           [ SKIP  ]
UvFsProbeCapabilities/noSpace                               [ SKIP  ]
UvFsProbeCapabilities/noAccess                              [ OK    ] [ 0.00003693 / 0.00003672 CPU ]
UvFsProbeCapabilities/aio                                   
  dir-fs=btrfs                                              [ SKIP  ]
  dir-fs=ext4                                               [ SKIP  ]
  dir-fs=xfs                                                [ SKIP  ]
UvFsProbeCapabilities/zfsDirectIO                           [ SKIP  ]
UvFsProbeCapabilities/tmpfs                                 [ SKIP  ]
UvFsAllocateFile/noSpace                                    
  dir-fs=tmpfs                                              [ SKIP  ]
UvFsAllocateFile/fileAlreadyExists                          [ OK    ] [ 0.00003526 / 0.00003513 CPU ]
UvFsAllocateFile/dirNoExists                                [ OK    ] [ 0.00002953 / 0.00002947 CPU ]
UvFsAllocateFile/success                                    [ OK    ] [ 0.00005135 / 0.00005133 CPU ]
UvFsOpenFileForReading/noExists                             [ OK    ] [ 0.00001578 / 0.00001562 CPU ]
UvFsSyncDir/noExists                                        [ OK    ] [ 0.00003273 / 0.00003256 CPU ]
UvFsCheckDir/notWritable                                    [ OK    ] [ 0.00002304 / 0.00002278 CPU ]
UvFsCheckDir/notDir                                         [ OK    ] [ 0.00003940 / 0.00003929 CPU ]
UvFsCheckDir/notDirPrefix                                   [ OK    ] [ 0.00002816 / 0.00002796 CPU ]
UvFsCheckDir/permissionDenied                               [ OK    ] [ 0.00003916 / 0.00003906 CPU ]
UvFsCheckDir/doesNotExist                                   [ OK    ] [ 0.00001378 / 0.00001368 CPU ]
UvFsCheckDir/exists                                         [ OK    ] [ 0.00001266 / 0.00001241 CPU ]
21 of 22 (95%) tests successful, 55 (71%) test skipped.
FAIL test/unit/uv (exit status: 1)

FAIL: test/integration/uv
=========================

Running test suite with seed 0x21ca1d33...
snapshot_truncate/snapshotThenTruncate                      [ ERROR ]
     src/uv_append.c:598 - enqueue 4 entries
    src/uv_prepare.c:166 - create open segment open-1
     src/uv_append.c:630 - set uv->append_next_index 5
     src/uv_append.c:598 - enqueue 4 entries
     src/uv_append.c:630 - set uv->append_next_index 9
     src/uv_append.c:598 - enqueue 4 entries
     src/uv_append.c:630 - set uv->append_next_index 13
   src/uv_snapshot.c:622 - put snapshot at 6, keeping 8192
     src/uv_append.c:815 - UvBarrier uv->append_next_index:13
   src/uv_truncate.c:173 - uv truncate 8
     src/uv_append.c:815 - UvBarrier uv->append_next_index:8
    src/uv_prepare.c:232 - completed creation of open-1
    src/uv_prepare.c:166 - create open segment open-2
   src/uv_finalize.c: 39 - finalize open-1 into 0000000000000001-0000000000000004
    src/uv_prepare.c:232 - completed creation of open-2
     src/uv_append.c:462 - prepare segment ready failed (23)
     src/uv_append.c:111 - rollback uv->append_next_index was:8
     src/uv_append.c:114 - rollback uv->append_next_index now:4
     src/uv_append.c:111 - rollback uv->append_next_index was:4
     src/uv_append.c:114 - rollback uv->append_next_index now:0
Error: test/integration/test_uv_truncate_snapshot.c:42: assertion failed: status == result->status (23 == 0)
Error: child killed by signal 6 (Aborted)
@ganto ganto added bug Something isn't working help wanted Extra attention is needed labels Dec 21, 2023
@stgraber
Copy link

@freeekanayaka

@ganto
Copy link
Owner Author

ganto commented Dec 21, 2023

With Fedora 38 I can reproduce this issue with every build (only tried x86_64 so far). However with Fedora 39 and Rawhide I cannot reproduce it and the test suite passes. They all use the same libuv-1:1.47.0-3.

@freeekanayaka
Copy link

freeekanayaka commented Dec 21, 2023 •
edited
Loading

Attempting to build the RPM of >=raft-0.18.2 (from cowsql/raft) via mock on Fedora 38 fails with:

...
UvWriterInit/noResources                                    [ ERROR ]
Error: test/unit/test_uv_writer.c:256: assertion failed: _rv == RAFT_TOOMANY (0 == 23)
Error: child killed by signal 6 (Aborted)

This one is probably spurious. The test suite assumes that the host system is not running workloads that could interfer with the test, for example processes that use the AIO kernel subsystem.

Is the system running something like mysql, postgresql or other similar service that might make use of asynchronous I/O?

@freeekanayaka
Copy link 

snapshot_truncate/snapshotThenTruncate                      [ ERROR ]
     src/uv_append.c:598 - enqueue 4 entries
    src/uv_prepare.c:166 - create open segment open-1
     src/uv_append.c:630 - set uv->append_next_index 5
     src/uv_append.c:598 - enqueue 4 entries
     src/uv_append.c:630 - set uv->append_next_index 9
     src/uv_append.c:598 - enqueue 4 entries
     src/uv_append.c:630 - set uv->append_next_index 13
   src/uv_snapshot.c:622 - put snapshot at 6, keeping 8192
     src/uv_append.c:815 - UvBarrier uv->append_next_index:13
   src/uv_truncate.c:173 - uv truncate 8
     src/uv_append.c:815 - UvBarrier uv->append_next_index:8
    src/uv_prepare.c:232 - completed creation of open-1
    src/uv_prepare.c:166 - create open segment open-2
   src/uv_finalize.c: 39 - finalize open-1 into 0000000000000001-0000000000000004
    src/uv_prepare.c:232 - completed creation of open-2
     src/uv_append.c:462 - prepare segment ready failed (23)
     src/uv_append.c:111 - rollback uv->append_next_index was:8
     src/uv_append.c:114 - rollback uv->append_next_index now:4
     src/uv_append.c:111 - rollback uv->append_next_index was:4
     src/uv_append.c:114 - rollback uv->append_next_index now:0
Error: test/integration/test_uv_truncate_snapshot.c:42: assertion failed: status == result->status (23 == 0)
Error: child killed by signal 6 (Aborted)

This one might be a similar reason as the other one, error code 23 is RAFT_TOOMANY which gets returned if some resources where exhausted, perhaps because some other process is interferring or taking them.

Are these tests being run in parallel? (e.g. with make -j<N>)

@freeekanayaka
Copy link

If there's no way for you to run these tests in a fully isolated environment, one thing I could do is to disable this kind of environment-sensitive tests by default, and only run them if some special environment variable is set (which we would do in our own CI).

@ganto
Copy link
Owner Author

ganto commented Dec 21, 2023 •
edited
Loading

I'm test building this on my workstation via mock which creates a separate build root. Although I'm not running any databases there is a lot running on the same kernel for sure.

Let me submit it to COPR and see if it works better there. But as far as I understand the Fedora build infrastructure is on a shared kernel using mock too.

Are these tests being run in parallel? (e.g. with make -j<N>)

Yes, it's run with -j6.

@freeekanayaka
Copy link

I'm test building this on my workstation via mock which creates a separate build root. Although I'm not running any databases there is a lot running on the same kernel for sure.

Let me submit it to COPR and see if it works better there. But as far as I understand the Fedora build infrastructure is on a shared kernel using mock too.

I see. I think I'll disable those tests by default then.

Are these tests being run in parallel? (e.g. with make -j<N>)

Yes, it's run with -j6.

Ok, please try to run that without -j6.

@ganto
Copy link
Owner Author

ganto commented Dec 21, 2023 •
edited
Loading

Ok, the COPR environment seems to be more to the liking of the test suite 😃

It built successfully for all architectures and Fedora releases: https://copr.fedorainfracloud.org/coprs/ganto/lxc4/build/6781882/ (c875d31)

@freeekanayaka
Copy link

Ok, the COPR environment seems to be more to the liking of the test suite 😃

It built successfully for all architectures and Fedora releases: https://copr.fedorainfracloud.org/coprs/ganto/lxc4/build/6781882/ (c875d31)

Great :) In that case, I'd rather not put any workaround in place in the upstream code, at least for now. What do you think?

@ganto
Copy link
Owner Author

ganto commented Dec 21, 2023

Yes, I agree. I'll also try to build it on Koji (the official Fedora build system) later to check if anything pops up there. I'll let you know.

@u5r0
Copy link

u5r0 commented Dec 22, 2023

@ganto This looks related to a raft conflict I got while dnf update

image

@ganto
Copy link
Owner Author

ganto commented Dec 22, 2023

This looks related to a raft conflict I got while dnf update

That was expected as I haven't rebuilt dqlite against the new raft version. Should be fixed now.

@ganto
Copy link
Owner Author

ganto commented Dec 30, 2023

@freeekanayaka: Unfortunately the official Fedora build service Koji seems to be a bit more sensitive:

  • My first build attempt (task ID 111051096) failed for aarch64 with the known errors:
UvWriterInit/noResources                                    [ ERROR ]
Error: test/unit/test_uv_writer.c:256: assertion failed: _rv == RAFT_TOOMANY (0 == 23)
Error: child killed by signal 6 (Aborted)
[...]
21 of 22 (95%) tests successful, 55 (71%) test skipped.
FAIL test/unit/uv (exit status: 1)
FAIL: test/integration/uv
=========================
Running test suite with seed 0x586be6bb...
snapshot_truncate/snapshotThenTruncate                      [ ERROR ]
     src/uv_append.c:598 - enqueue 4 entries
    src/uv_prepare.c:166 - create open segment open-1
     src/uv_append.c:630 - set uv->append_next_index 5
     src/uv_append.c:598 - enqueue 4 entries
     src/uv_append.c:630 - set uv->append_next_index 9
     src/uv_append.c:598 - enqueue 4 entries
     src/uv_append.c:630 - set uv->append_next_index 13
   src/uv_snapshot.c:622 - put snapshot at 6, keeping 8192
     src/uv_append.c:815 - UvBarrier uv->append_next_index:13
   src/uv_truncate.c:173 - uv truncate 8
     src/uv_append.c:815 - UvBarrier uv->append_next_index:8
    src/uv_prepare.c:232 - completed creation of open-1
    src/uv_prepare.c:166 - create open segment open-2
    src/uv_prepare.c:232 - completed creation of open-2
    src/uv_prepare.c:166 - create open segment open-3
   src/uv_finalize.c: 39 - finalize open-1 into 0000000000000001-0000000000000004
    src/uv_prepare.c:232 - completed creation of open-3
    src/uv_prepare.c:166 - create open segment open-4
   src/uv_finalize.c: 78 - uv finalize after work segment 0xaaaaf517d3b0 cb status:0
   src/uv_finalize.c: 93 - unblock barrier or close
            src/uv.c:189 - uv maybe fire close cb
    src/uv_prepare.c:232 - completed creation of open-4
    src/uv_prepare.c:166 - create open segment open-5
   src/uv_finalize.c: 39 - finalize open-2 into 0000000000000005-0000000000000008
    src/uv_prepare.c:232 - completed creation of open-5
   src/uv_finalize.c: 78 - uv finalize after work segment 0xaaaaf517d640 cb status:0
   src/uv_finalize.c: 93 - unblock barrier or close
            src/uv.c:189 - uv maybe fire close cb
   src/uv_finalize.c: 39 - finalize open-3 into 0000000000000009-0000000000000012
   src/uv_finalize.c: 78 - uv finalize after work segment 0xaaaaf517b950 cb status:0
   src/uv_finalize.c: 93 - unblock barrier or close
            src/uv.c:189 - uv maybe fire close cb
   src/uv_snapshot.c:505 - snapshot write start
   src/uv_snapshot.c:508 - snapshot write end 0
       src/uv_list.c: 92 - segment 0000000000000001-0000000000000004
       src/uv_list.c: 92 - segment 0000000000000005-0000000000000008
       src/uv_list.c: 92 - segment 0000000000000009-0000000000000012
       src/uv_list.c: 92 - segment open-4
       src/uv_list.c: 92 - segment open-5
       src/uv_list.c: 97 - ignore snapshot-1-6-3844702642
       src/uv_list.c: 82 - snapshot snapshot-1-6-3844702642.meta
   src/uv_truncate.c:125 - uv truncate barrier cb
   src/uv_truncate.c:144 - set truncate work
     src/uv_append.c:896 - UvUnblock triggered barrier request callback.
   src/uv_truncate.c: 26 - uv truncate work cb
       src/uv_list.c: 92 - segment 0000000000000001-0000000000000004
       src/uv_list.c: 92 - segment 0000000000000005-0000000000000008
       src/uv_list.c: 92 - segment 0000000000000009-0000000000000012
       src/uv_list.c: 92 - segment open-4
       src/uv_list.c: 92 - segment open-5
       src/uv_list.c: 97 - ignore snapshot-1-6-3844702642
       src/uv_list.c: 82 - snapshot snapshot-1-6-3844702642.meta
    src/uv_segment.c:1087 - truncate 5-8 at 8
   src/uv_truncate.c: 93 - uv truncate work cb ok
   src/uv_truncate.c:110 - uv truncate after work cb status:0
   src/uv_truncate.c:115 - clear truncate work
     src/uv_append.c:901 - UvUnblock queue empty
     src/uv_append.c:930 - uv barrier close
            src/uv.c:189 - uv maybe fire close cb
            src/uv.c:189 - uv maybe fire close cb
            src/uv.c:189 - uv maybe fire close cb
            src/uv.c:189 - uv maybe fire close cb
            src/uv.c:189 - uv maybe fire close cb
Error: test/integration/test_uv_truncate_snapshot.c:244: assertion failed: _ret == 0 (18 == 0)
Error: child killed by signal 6 (Aborted)
  • And the second attempt even failed for i686, x86_64 and aarch64 (task ID 111057385)
UvWriterInit/noResources                                    [ ERROR ]
Error: test/unit/test_uv_writer.c:256: assertion failed: _rv == RAFT_TOOMANY (0 == 23)
Error: child killed by signal 6 (Aborted)
[...]
21 of 22 (95%) tests successful, 55 (71%) test skipped.
FAIL test/unit/uv (exit status: 1)
FAIL: test/integration/uv
=========================
Running test suite with seed 0xe735ce3e...
snapshot_truncate/snapshotThenTruncate                      [ OK    ] [ 0.06217178 / 0.00694586 CPU ]
truncate/closingMultiple                                    [ ERROR ]
Error: test/integration/test_uv_truncate.c:99: assertion failed: rv_ == 0 (18 == 0)
Error: child killed by signal 6 (Aborted)

A way to only selectively run these tests would be helpful in my case.

@freeekanayaka
Copy link

@ganto I had prepared cowsql/raft#109 which I could sure merge.

However, I'd like to first rule out the possibility that these failures are caused by those two tests being run in parallel. Please can you confirm that this build is not running these tests with make -j<N> check?

If it is running them with make -j<N> check, please could you drop the -j? It's ok to build the source code in parallel (make -j<N>), but it's not ok to run the tests in parallel (make -J<N> check).

If it's not possible or easy for you to drop the -j from the tests, we can try to merge the PR above and it should indeed help, although I'm not 100% sure that it is enough to solve all possible problems related to running tests in parallel (it;s probably enough, but I'd need to double check).

@ganto
Copy link
Owner Author

ganto commented Dec 30, 2023

but it's not ok to run the tests in parallel (make -J check)

Ok. After the recent success I completely forgot about that. Will need to dig into the RPM macros to see how this works.

@ganto
Copy link
Owner Author

ganto commented Dec 30, 2023

With serial make check it was successful (task ID 2338658). Sorry for the false alert and thanks for your support.

@ganto ganto closed this as completed Dec 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working help wanted Extra attention is needed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@freeekanayaka @ganto @stgraber @u5r0