Build on MacOS #173

rabits · 2021-02-20T06:43:39Z

Patch to support the MacOS for go-dqlite, basically a copy of abandoned PR #119 with some cleaning and MacOS-specific changes.

Now it's ready for review - I tested it successfully with macos->macos and macos->linux connection and regular DB workload.
Tests are in the go-dqlite PR here: canonical/go-dqlite#132

The most important change that fixed the freeze is moving r->leader_state.change = req; prior to call of the clientChangeConfiguration function (because the macos IO implementation is not async and executes directly, so quite sure even on linux it could fail in case the async executor will decide to run IO operation right after the call).

The other patches in series:

codecov-io · 2021-02-24T04:00:51Z

Codecov Report

Merging #173 (5f6ae82) into master (fefc557) will increase coverage by 0.05%.
The diff coverage is 69.23%.

@@            Coverage Diff             @@
##           master     canonical/raft#173      +/-   ##
==========================================
+ Coverage   86.27%   86.32%   +0.05%     
==========================================
  Files          47       47              
  Lines        7532     7531       -1     
  Branches     2000     2003       +3     
==========================================
+ Hits         6498     6501       +3     
+ Misses        928      923       -5     
- Partials      106      107       +1

Impacted Files	Coverage Δ
src/byte.c	`98.57% <ø> (ø)`
src/fixture.c	`92.99% <ø> (ø)`
src/heap.c	`100.00% <ø> (ø)`
src/uv.c	`86.45% <0.00%> (-0.51%)`	⬇️
src/uv_os.c	`81.55% <ø> (ø)`
src/uv_snapshot.c	`75.97% <ø> (ø)`
src/uv_fs.c	`74.87% <60.00%> (-0.07%)`	⬇️
src/raft.c	`82.50% <100.00%> (-1.46%)`	⬇️
src/uv_writer.c	`81.13% <100.00%> (ø)`
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fefc557...5f6ae82. Read the comment docs.

example/server.c

src/client.c

src/heap.c

src/uv.c

src/uv_fs.c

src/uv_os.h

rabits · 2021-04-22T04:17:02Z

Hi @stgraber could you please check this change as well?

stgraber · 2021-04-22T04:47:37Z

I think I'd feel a lot more confident about this (and easier to review/revert) if it was split into a bunch of commits for the various type of changes done in there.

rabits · 2021-04-22T05:10:13Z

@stgraber you mean half-working ones? I checked the changelist again and almost any line is needed to be sure that the project could be built or work at all... Maybe you have some clues on how to split it, because it's quite none in my head right now...

pycaster · 2021-05-07T15:47:26Z

@rabits Thank you for the PR. I am looking forward for this to land on master. However, I got your changes to build on MAC with this patch to raft, master of dqlite but seeing this error when I ran the dqlite-demo example,

dqlite-demo --api 127.0.0.1:8001 --db 127.0.0.1:9001
Assertion failed: (amount == (int)page_size), function vfsWalWrite, file src/vfs.c, line 1079.
[1]    79045 abort      dqlite-demo --api 127.0.0.1:8001 --db 127.0.0.1:9001

Any idea what's going here?

rabits · 2021-05-07T17:40:59Z

Hi @pycaster , yeah, still working on the patches - but with the latest releases. So still need to prepare CI and retest them again.

Regarding the error you see - looks like some assert failed, hard to tell right now, but let's check when CI part will be ready.

codecov-commenter · 2021-05-10T06:51:41Z

Codecov Report

Merging #173 (23bade6) into master (b71e303) will decrease coverage by 0.15%.
The diff coverage is 80.43%.

@@            Coverage Diff             @@
##           master     canonical/raft#173      +/-   ##
==========================================
- Coverage   86.79%   86.63%   -0.16%     
==========================================
  Files          47       47              
  Lines        7602     7648      +46     
  Branches     2030     2046      +16     
==========================================
+ Hits         6598     6626      +28     
- Misses        900      918      +18     
  Partials      104      104

Impacted Files	Coverage Δ
src/byte.c	`98.57% <ø> (ø)`
src/heap.c	`100.00% <ø> (ø)`
src/uv_metadata.c	`91.66% <0.00%> (ø)`
src/uv_segment.c	`90.09% <ø> (ø)`
src/uv_snapshot.c	`76.62% <ø> (+0.64%)`	⬆️
src/uv_os.c	`80.58% <25.00%> (-0.98%)`	⬇️
src/client.c	`78.12% <66.66%> (-0.71%)`	⬇️
src/uv_fs.c	`75.05% <85.71%> (-0.58%)`	⬇️
src/raft.c	`88.75% <100.00%> (+1.25%)`	⬆️
src/uv.c	`85.71% <100.00%> (-1.36%)`	⬇️
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b71e303...23bade6. Read the comment docs.

rabits · 2021-05-11T06:56:45Z

@MathieuBordere , I tested osx build but looks like it's disabled for now - do you know a way to make it through?

MathieuBordere · 2021-05-11T07:45:20Z

@rabits Looks like our Travis account needs funding :-) should be functional again soon.

rabits · 2021-05-11T17:46:19Z

Oh, sorry - hopefully it's not me who eaten the budget... Is osx pricey on travis?

rabits · 2021-05-17T05:00:22Z

Hi @MathieuBordere , could you please check the CI log - I'm a little bit confused why the test could fail:

send/changeToUnconnectedAddress                             [ ERROR ]
Error: test/integration/test_uv_send.c:281: uv_run: condition not met in 20 iterations
Error: child killed by signal 6

Also not sure if the other tests are fixed correctly, so am I going in the right direction?

MathieuBordere · 2021-05-17T14:10:53Z

Hi @MathieuBordere , could you please check the CI log - I'm a little bit confused why the test could fail:
send/changeToUnconnectedAddress                             [ ERROR ]
Error: test/integration/test_uv_send.c:281: uv_run: condition not met in 20 iterations
Error: child killed by signal 6
Also not sure if the other tests are fixed correctly, so am I going in the right direction?

It's not obvious why it would fail, I propose that you try to run the test locally and debug it.

The steps in the failing test are:

Try to send a message to a server at a certain address
Try to send a message to the same server at a different, disconnected address, hitting the code path

raft/src/uv_send.c

Line 426 in 34b3ae0

if (strcmp((*client)->address, address) != 0) {

but the connection will fail.
Try to send a message to the same server at the original, connected address (this is where it fails for you). My guess is that

raft/src/uv_send.c

Line 487 in 34b3ae0

rv = uvGetClient(uv, message->server_id, message->server_address, &client);

has a non-0 return value.

penberg · 2021-11-11T18:53:13Z

@MathieuBordere I can reproduce the problem locally.

It does not look like uvGetClient() is the culprit (it can't return non-zero unless memory allocation fails AFAICT).

Can you walk me through how a new connection is supposed to appear in the test and who should flush the pending messages?

I hacked tracing.h as follows:

diff --git a/src/tracing.h b/src/tracing.h
index aeaddb2..9b0d9e2 100644
--- a/src/tracing.h
+++ b/src/tracing.h
@@ -16,6 +16,8 @@ extern struct raft_tracer StderrTracer;
 /* Emit a debug message with the given tracer. */
 #define Tracef(TRACER, ...)                                              \
     do {                                                                 \
+fprintf(stderr, __VA_ARGS__); \
+fprintf(stderr, "\n"); \
         if (TRACER != NULL && TRACER->emit != NULL && TRACER->enabled) { \
             static char _msg[1024];                                      \
             snprintf(_msg, sizeof _msg, __VA_ARGS__);                    \

and see the following:

penberg@vonneumann raft % make test/integration/uv && ./test/integration/uv send/changeToUnconnectedAddress
make: `test/integration/uv' is up to date.
Running test suite with seed 0x8ff66663...
send/changeToUnconnectedAddress                             [ ERROR ]
no connection available -> enqueue message
connect attempt completed -> status unknown error
send pending messages
connection available -> write message
no connection available -> enqueue message
no connection available -> enqueue message
connect attempt completed -> status no connection to remote server available
timer expired -> attempt to reconnect
connect attempt completed -> status no connection to remote server available
timer expired -> attempt to reconnect
connect attempt completed -> status no connection to remote server available
Error: test/integration/test_uv_send.c:284: uv_run: condition not met in 5 iterations
Error: child killed by signal 6
0 of 1 (0%) tests successful, 0 (0%) test skipped.

MathieuBordere · 2021-11-15T22:47:04Z

Thanks for this, will have a look later this week!

calvin2021y · 2022-07-30T14:11:12Z

a lot developer use MacOS, please consider fix this.

rabits · 2023-04-22T16:55:48Z

Oh, just recalled that this PR is still here. Sorry folks, I used dqlite in my project before, but figured out some other solution and quite sure will not going to continue work on this PR - gets too complicated to support this. Don't really mind if someone wants to continue the work)

This was referenced Feb 20, 2021

Support compile with clang on MacOS canonical/dqlite#285

Merged

Build on MacOS with TLS support canonical/go-dqlite#132

Merged

rabits marked this pull request as draft February 20, 2021 07:14

rabits force-pushed the macos_build branch from 3b7db7d to 5f6ae82 Compare February 24, 2021 03:57

rabits mentioned this pull request Mar 20, 2021

Cross platform support canonical/dqlite#593

Open

rabits force-pushed the macos_build branch from 5f6ae82 to 6ddabdb Compare March 28, 2021 19:52

rabits marked this pull request as ready for review March 28, 2021 19:52

rabits mentioned this pull request Mar 28, 2021

Fixed the segfault in case raft uses direct IO backend canonical/dqlite#288

Merged

MathieuBordere reviewed Mar 30, 2021

View reviewed changes

example/server.c Outdated Show resolved Hide resolved

MathieuBordere reviewed Mar 30, 2021

View reviewed changes

src/client.c Show resolved Hide resolved

MathieuBordere reviewed Mar 30, 2021

View reviewed changes

src/heap.c Show resolved Hide resolved

MathieuBordere reviewed Mar 30, 2021

View reviewed changes

src/heap.c Show resolved Hide resolved

src/heap.c Outdated Show resolved Hide resolved

src/uv.c Outdated Show resolved Hide resolved

src/uv_fs.c Outdated Show resolved Hide resolved

src/uv_os.h Outdated Show resolved Hide resolved

rabits force-pushed the macos_build branch 2 times, most recently from 55297d0 to 2d208a0 Compare April 2, 2021 02:01

MathieuBordere requested a review from stgraber April 2, 2021 06:52

rabits mentioned this pull request Apr 11, 2021

Fixed a couple of issues with unix sockets on MacOS canonical/dqlite#290

Merged

rabits force-pushed the macos_build branch 4 times, most recently from 119f949 to 23bade6 Compare May 10, 2021 06:48

rabits force-pushed the macos_build branch 3 times, most recently from e99ef7d to 1ffd87d Compare May 10, 2021 08:15

rabits force-pushed the macos_build branch from 1ffd87d to fabc324 Compare May 11, 2021 06:10

Build on MacOS

218928d

rabits force-pushed the macos_build branch from fabc324 to 4bfa80a Compare May 11, 2021 06:15

rabits force-pushed the macos_build branch 2 times, most recently from 35f1979 to da01c58 Compare May 16, 2021 22:33

Initial CI for MacOS

79a980f

rabits force-pushed the macos_build branch from da01c58 to 79a980f Compare May 17, 2021 04:30

freeekanayaka mentioned this pull request Apr 20, 2023

Implement server-side role management canonical/dqlite#480

Merged

rabits closed this Apr 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Build on MacOS #173

Build on MacOS #173

rabits commented Feb 20, 2021 •

edited

Loading

codecov-io commented Feb 24, 2021

rabits commented Apr 22, 2021

stgraber commented Apr 22, 2021

rabits commented Apr 22, 2021

pycaster commented May 7, 2021

rabits commented May 7, 2021

codecov-commenter commented May 10, 2021 •

edited

Loading

rabits commented May 11, 2021

MathieuBordere commented May 11, 2021 •

edited

Loading

rabits commented May 11, 2021

rabits commented May 17, 2021

MathieuBordere commented May 17, 2021 •

edited

Loading

penberg commented Nov 11, 2021 •

edited

Loading

MathieuBordere commented Nov 15, 2021

calvin2021y commented Jul 30, 2022

rabits commented Apr 22, 2023

Build on MacOS #173

Build on MacOS #173

Conversation

rabits commented Feb 20, 2021 • edited Loading

codecov-io commented Feb 24, 2021

Codecov Report

rabits commented Apr 22, 2021

stgraber commented Apr 22, 2021

rabits commented Apr 22, 2021

pycaster commented May 7, 2021

rabits commented May 7, 2021

codecov-commenter commented May 10, 2021 • edited Loading

Codecov Report

rabits commented May 11, 2021

MathieuBordere commented May 11, 2021 • edited Loading

rabits commented May 11, 2021

rabits commented May 17, 2021

MathieuBordere commented May 17, 2021 • edited Loading

penberg commented Nov 11, 2021 • edited Loading

MathieuBordere commented Nov 15, 2021

calvin2021y commented Jul 30, 2022

rabits commented Apr 22, 2023

rabits commented Feb 20, 2021 •

edited

Loading

codecov-commenter commented May 10, 2021 •

edited

Loading

MathieuBordere commented May 11, 2021 •

edited

Loading

MathieuBordere commented May 17, 2021 •

edited

Loading

penberg commented Nov 11, 2021 •

edited

Loading