Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Epoch System #459

Merged
merged 63 commits into from
Sep 18, 2023
Merged
Show file tree
Hide file tree
Changes from 59 commits
Commits
Show all changes
63 commits
Select commit Hold shift + click to select a range
d57d1f6
disk: add epoch handling to `u3_disk_init`
matthew-levan Apr 12, 2023
4be7ec3
u3: add dir list to `u3_dire`
matthew-levan Apr 13, 2023
89e9198
disk: add `u3_disk_last_epoc` and `u3_disk_migrate`
matthew-levan Apr 13, 2023
796de55
u3: make `u3e_backup` copy snapshot to arbitrary path
matthew-levan Apr 14, 2023
6d00b08
disk: add `epoc` functions
matthew-levan Apr 14, 2023
488adc3
cli: add `roll` subcommand for epoch rollovers
matthew-levan Apr 17, 2023
f0289f9
cli: make `chop` work with epochs
matthew-levan Apr 17, 2023
c54d6c0
disk: migrate existing piers to v1 format (epochs)
matthew-levan Apr 18, 2023
f8834e0
disk: validate current snapshot before migration
matthew-levan Apr 26, 2023
b93508e
lmdb: factor siz_i calculation
matthew-levan Apr 28, 2023
bb1f786
disk: create new epoch when vere version is different
matthew-levan Apr 28, 2023
5b432b6
cli: make `chop` leave only the latest two epochs
matthew-levan May 1, 2023
2e9da02
disk: create a new epoch when version change detected
matthew-levan May 1, 2023
1b952d0
c3: add `c3_link`
matthew-levan May 16, 2023
b0da9be
disk: make migration idempotent (crash safe)
matthew-levan May 18, 2023
6c38d90
Merge branch 'develop' into i/313/epoch
matthew-levan May 25, 2023
3b1ee9a
cli: fix `chop` bug
matthew-levan May 25, 2023
b8b52c2
disk: move format version to `version.h`
matthew-levan May 26, 2023
6582f14
disk: use `PRIc3_d` instead of `PRIu64`
matthew-levan May 30, 2023
b35b413
disk: various epoch improvements
matthew-levan May 31, 2023
ba1a3af
disk: function style fixes
matthew-levan May 31, 2023
9b45ad2
u3: remove snapshot backup in `u3e_save` and unused `u3m_backup`
matthew-levan Jun 1, 2023
fb762b2
disk: add event check to `u3_disk_epoc_good`
matthew-levan Jun 1, 2023
83b5737
cleanup whitespace
barter-simsum Jun 1, 2023
6dc1bd6
data.mdb -> lock.mdb
barter-simsum Jun 1, 2023
a0d239b
disk: add event check in `u3_disk_epoc_good`
matthew-levan Jun 1, 2023
d6fcb98
disk: fix duplicate code
matthew-levan Jun 2, 2023
1886844
remove redundant dut_o in u3_disk_migrate
barter-simsum Jun 2, 2023
d59b0af
disk: make `u3_disk_epoc_vere` safer
matthew-levan Jun 5, 2023
c50906d
disk: cleanup `#include` directives
matthew-levan Jun 5, 2023
6118566
lmdb: restore `siz_i` parameter in `u3_lmdb_init`
matthew-levan Jun 5, 2023
d5aff59
disk: define `siz_i` as a global, static constant
matthew-levan Jun 6, 2023
c045119
Merge branch 'develop' into i/313/epoch
matthew-levan Jun 6, 2023
cc2557c
disk: pare down `u3_disk_epoc_good`
matthew-levan Jun 9, 2023
8703f11
disk: remove `u3_disk_epoc_good` and add argument to `u3_disk_epoc_init`
matthew-levan Jun 12, 2023
fdb2d86
Merge branch 'jb/moar-replay' into msl/replay-with-epochs
matthew-levan Jun 13, 2023
52f76bc
wip
matthew-levan Jun 15, 2023
0236e18
Merge branch 'develop' into msl/replay-with-epochs
matthew-levan Jun 19, 2023
20a3af0
disk: set `epo_u`
matthew-levan Jun 20, 2023
e5723ad
disk: `urbit play` with epochs
matthew-levan Jun 21, 2023
c9a624d
events: better path naming
matthew-levan Jun 21, 2023
981893c
Merge branch 'develop' into msl/replay-with-epochs
matthew-levan Jun 21, 2023
01ecd2e
disk: fix merge error
matthew-levan Jun 22, 2023
4336422
events: remove unnecessary/erroneous `mkdir -p` implementation
matthew-levan Jun 22, 2023
ac3dd76
trim trailing whitespace
matthew-levan Jun 22, 2023
6c6ac0c
events: `mkdir -p` for `_ce_image_open` (needed for boot)
matthew-levan Jun 23, 2023
64c5454
events: prevent `0B` snapshot files from being created during `play`
matthew-levan Jun 23, 2023
b6541b7
events: remove `mkdir -p` functionality from `_ce_image_open`
matthew-levan Jun 23, 2023
299c6ed
disk: create `.urb/chk` on init; simplify `log_u->dun_d` setting
matthew-levan Jun 23, 2023
8a3df1d
play: full replay when only epoch 0 exists
matthew-levan Jun 27, 2023
fa9cddd
events/disk/pier: implement `u3_disk_vere_diff`
matthew-levan Jun 28, 2023
4d35ba2
pier: move `.urb/chk` creation into `u3m_pier`
matthew-levan Jun 28, 2023
8668d73
cli: subcommands auto-migrate disk format
matthew-levan Jun 30, 2023
5e3f253
cli: fix `play` boot
matthew-levan Jul 2, 2023
1140c4e
play: support full replay of unmigrated piers without valid snapshot
matthew-levan Jul 5, 2023
fcd9bcc
always replay synchronously on restart
matthew-levan Jul 6, 2023
f229c4d
deprecated async replay codepath
matthew-levan Jul 6, 2023
0eaba9a
Merge branch 'develop' into msl/replay-with-epochs
matthew-levan Jul 12, 2023
3c8c0b2
disk: fix migrate scenarios
matthew-levan Jul 12, 2023
1556567
disk/events: simplify code
matthew-levan Jul 18, 2023
7593d40
Merge branch 'develop' into msl/replay-with-epochs
matthew-levan Jul 25, 2023
f9a2d31
disk: add "migration starting" printf
matthew-levan Jul 26, 2023
ea6f278
disk: move `u3C.wag_w` assignment before boot call
matthew-levan Aug 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
matthew-levan marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -1,3 +1,7 @@
# IDEs.
/.vscode
/.idea

# Bazel.
/.user.bazelrc
/bazel-*
Expand Down
2 changes: 2 additions & 0 deletions pkg/c3/defs.h
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,8 @@
mkdir(a, b);})
# define c3_rmdir(a) ({ \
rmdir(a);})
# define c3_link(a, b) ({ \
link(a, b);})
# define c3_unlink(a) ({ \
unlink(a);})
# define c3_fopen(a, b) ({ \
Expand Down
100 changes: 56 additions & 44 deletions pkg/noun/events.c
Original file line number Diff line number Diff line change
Expand Up @@ -398,23 +398,14 @@ _ce_ephemeral_open(c3_i* eph_i)
/* _ce_image_open(): open or create image.
*/
static c3_o
_ce_image_open(u3e_image* img_u)
_ce_image_open(u3e_image* img_u, c3_c* ful_c)
{
c3_i mod_i = O_RDWR | O_CREAT;
c3_c ful_c[8193];

snprintf(ful_c, 8192, "%s", u3P.dir_c);
c3_mkdir(ful_c, 0700);

snprintf(ful_c, 8192, "%s/.urb", u3P.dir_c);
c3_mkdir(ful_c, 0700);

snprintf(ful_c, 8192, "%s/.urb/chk", u3P.dir_c);
c3_mkdir(ful_c, 0700);

snprintf(ful_c, 8192, "%s/.urb/chk/%s.bin", u3P.dir_c, img_u->nam_c);
if ( -1 == (img_u->fid_i = c3_open(ful_c, mod_i, 0666)) ) {
fprintf(stderr, "loom: c3_open %s: %s\r\n", ful_c, strerror(errno));
c3_c pax_c[8192];
snprintf(pax_c, 8192, "%s/%s.bin", ful_c, img_u->nam_c);
if ( -1 == (img_u->fid_i = c3_open(pax_c, mod_i, 0666)) ) {
fprintf(stderr, "loom: c3_open %s: %s\r\n", pax_c, strerror(errno));
return c3n;
}
else if ( c3n == _ce_image_stat(img_u, &img_u->pgs_w) ) {
Expand Down Expand Up @@ -1345,54 +1336,75 @@ _ce_image_copy(u3e_image* fom_u, u3e_image* tou_u)
return c3y;
}

/* u3e_backup(): copy snapshot to .urb/bhk (if it doesn't exist yet).
/* u3e_backup(): copy snapshot from [pux_c] to [pax_c],
* overwriting optionally. note that image files must
* be named "north" and "south".
*/
c3_o
u3e_backup(c3_o ovw_o)
u3e_backup(c3_c* pux_c, c3_c* pax_c, c3_o ovw_o)
{
u3e_image nop_u = { .nam_c = "north", .pgs_w = 0 };
u3e_image sop_u = { .nam_c = "south", .pgs_w = 0 };
c3_i mod_i = O_RDWR | O_CREAT; // XX O_TRUNC ?
c3_c ful_c[8193];
// source image files from [pux_c]
u3e_image nux_u = { .nam_c = "north" };
u3e_image sux_u = { .nam_c = "south" };

// destination image files to [pax_c]
u3e_image nax_u = { .nam_c = "north" };
u3e_image sax_u = { .nam_c = "south" };

matthew-levan marked this conversation as resolved.
Show resolved Hide resolved
c3_i mod_i = O_RDWR | O_CREAT;

snprintf(ful_c, 8192, "%s/.urb/bhk", u3P.dir_c);
if ( !pux_c || !pax_c ) {
fprintf(stderr, "loom: image backup: bad path\r\n");
return c3n;
}

if ( (c3n == ovw_o) && c3_mkdir(ful_c, 0700) ) {
if ( (c3n == ovw_o) && c3_mkdir(pax_c, 0700) ) {
if ( EEXIST != errno ) {
fprintf(stderr, "loom: image backup: %s\r\n", strerror(errno));
}
return c3n;
}

snprintf(ful_c, 8192, "%s/.urb/bhk/%s.bin", u3P.dir_c, nop_u.nam_c);

if ( -1 == (nop_u.fid_i = c3_open(ful_c, mod_i, 0666)) ) {
fprintf(stderr, "loom: c3_open %s: %s\r\n", ful_c, strerror(errno));
// open source image files if they exist
//
c3_c nux_c[8193];
snprintf(nux_c, 8192, "%s/%s.bin", pux_c, nux_u.nam_c);
if ( (0 != access(nux_c, F_OK)) || (c3n == _ce_image_open(&nux_u, pux_c)) ) {
fprintf(stderr, "loom: couldn't open north image at %s\r\n", pux_c);
return c3n;
}
c3_c sux_c[8193];
snprintf(sux_c, 8192, "%s/%s.bin", pux_c, sux_u.nam_c);
if ( (0 != access(sux_c, F_OK)) || (c3n == _ce_image_open(&sux_u, pux_c)) ) {
fprintf(stderr, "loom: couldn't open south image at %s\r\n", pux_c);
return c3n;
}

snprintf(ful_c, 8192, "%s/.urb/bhk/%s.bin", u3P.dir_c, sop_u.nam_c);

if ( -1 == (sop_u.fid_i = c3_open(ful_c, mod_i, 0666)) ) {
fprintf(stderr, "loom: c3_open %s: %s\r\n", ful_c, strerror(errno));
// open destination image files
c3_c nax_c[8193];
snprintf(nax_c, 8192, "%s/%s.bin", pax_c, nax_u.nam_c);
if ( -1 == (nax_u.fid_i = c3_open(nax_c, mod_i, 0666)) ) {
fprintf(stderr, "loom: c3_open %s: %s\r\n", nax_c, strerror(errno));
return c3n;
}
c3_c sax_c[8193];
snprintf(sax_c, 8192, "%s/%s.bin", pax_c, sax_u.nam_c);
if ( -1 == (sax_u.fid_i = c3_open(sax_c, mod_i, 0666)) ) {
fprintf(stderr, "loom: c3_open %s: %s\r\n", sax_c, strerror(errno));
return c3n;
}

if ( (c3n == _ce_image_copy(&u3P.nor_u, &nop_u))
|| (c3n == _ce_image_copy(&u3P.sou_u, &sop_u)) )
if ( (c3n == _ce_image_copy(&nux_u, &nax_u))
|| (c3n == _ce_image_copy(&sux_u, &sax_u)) )
{

c3_unlink(ful_c);
snprintf(ful_c, 8192, "%s/.urb/bhk/%s.bin", u3P.dir_c, nop_u.nam_c);
c3_unlink(ful_c);
snprintf(ful_c, 8192, "%s/.urb/bhk", u3P.dir_c);
c3_rmdir(ful_c);
c3_unlink(nax_c);
c3_unlink(sax_c);
fprintf(stderr, "loom: image backup failed\r\n");
return c3n;
}

close(nop_u.fid_i);
close(sop_u.fid_i);
close(nax_u.fid_i);
close(sax_u.fid_i);
fprintf(stderr, "loom: image backup complete\r\n");
return c3y;
}
Expand Down Expand Up @@ -1502,8 +1514,6 @@ u3e_save(u3_post low_p, u3_post hig_p)
}

u3e_toss(low_p, hig_p);

u3e_backup(c3n);
}

/* _ce_toss_pages(): discard ephemeral pages.
Expand Down Expand Up @@ -1573,8 +1583,10 @@ u3e_live(c3_o nuu_o, c3_c* dir_c)

// Open image files.
//
if ( (c3n == _ce_image_open(&u3P.nor_u)) ||
(c3n == _ce_image_open(&u3P.sou_u)) )
c3_c chk_c[8193];
snprintf(chk_c, 8193, "%s/.urb/chk", u3P.dir_c);
if ( (c3n == _ce_image_open(&u3P.nor_u, chk_c)) ||
(c3n == _ce_image_open(&u3P.sou_u, chk_c)) )
{
fprintf(stderr, "boot: image failed\r\n");
exit(1);
Expand Down
7 changes: 4 additions & 3 deletions pkg/noun/events.h
Original file line number Diff line number Diff line change
Expand Up @@ -74,10 +74,11 @@

/** Functions.
**/
/* u3e_backup(): copy the snapshot from chk to bhk.
/* u3e_backup(): copy the snapshot from [pux_c] to [pax_c],
* overwriting optional.
*/
c3_o
u3e_backup(c3_o ovw_o);
c3_o
u3e_backup(c3_c* pux_c, c3_c* pax_c, c3_o ovw_o);

/* u3e_fault(): handle a memory fault.
*/
Expand Down
46 changes: 37 additions & 9 deletions pkg/noun/manage.c
Original file line number Diff line number Diff line change
Expand Up @@ -1762,14 +1762,6 @@ _cm_limits(void)
# endif
}

/* u3m_backup(): copy snapshot to .urb/bhk (if it doesn't exist yet).
*/
c3_o
u3m_backup(c3_o ovw_o)
{
return u3e_backup(ovw_o);
}

/* u3m_fault(): handle a memory event with libsigsegv protocol.
*/
c3_i
Expand Down Expand Up @@ -2090,6 +2082,42 @@ u3m_stop()
u3je_secp_stop();
}

/* u3m_pier(): make a pier.
*/
c3_c*
u3m_pier(c3_c* dir_c)
{
c3_c ful_c[8193];

u3C.dir_c = dir_c;

snprintf(ful_c, 8192, "%s", dir_c);
if ( c3_mkdir(ful_c, 0700) ) {
if ( EEXIST != errno ) {
fprintf(stderr, "loom: pier create: %s\r\n", strerror(errno));
exit(1);
}
}

snprintf(ful_c, 8192, "%s/.urb", dir_c);
if ( c3_mkdir(ful_c, 0700) ) {
if ( EEXIST != errno ) {
fprintf(stderr, "loom: .urb create: %s\r\n", strerror(errno));
exit(1);
}
}

snprintf(ful_c, 8192, "%s/.urb/chk", dir_c);
if ( c3_mkdir(ful_c, 0700) ) {
if ( EEXIST != errno ) {
fprintf(stderr, "loom: .urb/chk create: %s\r\n", strerror(errno));
exit(1);
}
}

return strdup(dir_c);
}

/* u3m_boot(): start the u3 system. return next event, starting from 1.
*/
c3_d
Expand All @@ -2105,7 +2133,7 @@ u3m_boot(c3_c* dir_c, size_t len_i)

/* Activate the storage system.
*/
nuu_o = u3e_live(c3n, dir_c);
nuu_o = u3e_live(c3n, u3m_pier(dir_c));

/* Activate tracing.
*/
Expand Down
5 changes: 5 additions & 0 deletions pkg/noun/manage.h
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,11 @@
c3_d
u3m_boot(c3_c* dir_c, size_t len_i);

/* u3m_pier(): make a pier.
*/
c3_c*
u3m_pier(c3_c* dir_c);

/* u3m_boot_lite(): start without checkpointing.
*/
c3_d
Expand Down
6 changes: 6 additions & 0 deletions pkg/noun/version.h
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,10 @@ typedef c3_w u3e_version;
#define U3E_VER1 1
#define U3E_VERLAT U3E_VER1

/* DISK FORMAT
*/

#define U3D_VER1 1
#define U3D_VERLAT U3L_VER1

#endif /* ifndef U3_VERSION_H */
Loading