status?

[benoitc]

What is the current status of leveled? Can it start to be used in small prods? Is there any release planned soon?

[martinsumner]

Benoit, it is close but not quite there yet.

The positives:

• 100% test coverage. including testing of difficult scenarios (simulating file corruption etc);
• More than 100 days of volume testing with no issues (mainly as a Riak backend);
• Passed some initial property-based testing (focused on preserving data through startup/shutdown scenarios).

There are a few things still needed before I can say release 1.0 is ready:

• More property-based testing (and @russelldb is promising to test it again soon);
• A thorough code-review (much of the code has been read by me, and me alone so far);
• One intermittent test failure (which I'm working on at the moment).

Currently, the only target I have for releasing it is to be ready for Riak 3.0, and Riak 3.0 is not likely to be out until Autumn 2018 at the earliest. However, if there was broader interest in using it outside of Riak I could potentially do the necessary work to declare 1.0 for June 2018.

Let me know if you'd like me to push forward more aggressively to make it release ready. Are you thinking of comparing leveled against your work on rocksdb?

[benoitc]

@martinsumner I discussed with some people about having a backend in erlang for the storage of barrel instead of using a nif like my rocksdb binding. leveled is interresting on that purpose :) I will try and report but any tagged release would be cool of course.

[ghost]

Would love to introduce leveled to the Elixir community, any moves to improve compatibility would be much appreciated. Happy to help with any resources I can bring to the table.

[ghost]

Including writing an Elixir compatibility layer/wrapper.

[martinsumner]

Are there obvious barriers to Elixir compatibility at the moment - anything you want me to look at?

[ghost]

I think I did a pull request a while back but haven't been near it since. I'll create a poc project first to bootstrap.

…

On Sun, 29 Apr 2018, 13:19 Martin Sumner, ***@***.***> wrote: Are there obvious barriers to Elixir compatibility at the moment - anything you want me to look at? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#131 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Aego9h5JUZLiXMIfADBNjutilpVwEPMLks5tta_BgaJpZM4TrfW-> .

[martinsumner]

Just a note that It is tested up to OTP19 only at the moment. I've not done anything yet to tackle OTP20.

[martinsumner]

As an aside on the OTP20 thing. It is compatible with OTP20, other than the deprecation warnings over gen_fsm. Interestingly, the ct tests run about 30% faster in OTP20, so there does appear to be some real advantages with moving forward with releases.

The two FSMs are leveled_cdb and leveled_sst (basically the two modules that actually write to disk). I may simply convert these to gen_server to get round the deprecation of the FSM.

[martinsumner]

OTP21 works in that it will pass all eunit and ct tests.

However dialyzer isn't happy. There are two places where the function output spec is either an atom or some other Erlang object type - and in bot these places, dialyzer acts as if only the atom can be the result:

https://github.com/martinsumner/leveled/blob/master/src/leveled_codec.erl#L649-L672
https://github.com/martinsumner/leveled/blob/master/src/leveled_pmanifest.erl#L326-L339

Dialyzer is happy with this in OTP 20, but in OTP 21:

MartinWork:leveled martinsumner$ rebar3 dialyzer
===> Verifying dependencies...
===> Compiling leveled
===> Dialyzer starting, this may take a while...
===> Updating plt...
===> Resolving files...
===> Updating base plt...
===> Resolving files...
===> Checking 187 files in "/Users/martinsumner/.cache/rebar3/rebar3_21.0_plt"...
===> Copying "/Users/martinsumner/.cache/rebar3/rebar3_21.0_plt" to "/Users/martinsumner/dbroot/leveled/_build/default/rebar3_21.0_plt"...
===> Checking 187 files in "/Users/martinsumner/dbroot/leveled/_build/default/rebar3_21.0_plt"...
===> Doing success typing analysis...
===> Resolving files...
===> Analyzing 18 files with "/Users/martinsumner/dbroot/leveled/_build/default/rebar3_21.0_plt"...

src/leveled_codec.erl
 627: The pattern {LMD1, TTL} can never match the type 'no_index'
 918: Function parsedate_test/0 has no local return
 926: The created fun has no local return
 926: The pattern 'true' can never match the type 'false'
 935: Function check_pd/2 will never be called
 948: The variable __V can never match since previous clauses completely covered the type 'no_index'

src/leveled_penciller.erl
 800: The call leveled_pmanifest:key_lookup(any(), 0, 'all') breaks the contract (manifest(), integer(), leveled_codec:ledger_key()) -> 'false' | manifest_entry()
 832: The variable FP can never match since previous clauses completely covered the type 'false'
1188: The pattern <State, 'true'> can never match the type <#state{persisted_sqn::integer(),ledger_sqn::integer(),root_path::string(),clerk::'undefined' | pid(),levelzero_pending::boolean(),levelzero_constructor::'undefined' | pid(),levelzero_cache::[any()],levelzero_size::integer(),levelzero_maxcachesize::'undefined' | integer(),levelzero_cointoss::boolean(),is_snapshot::boolean(),snapshot_fully_loaded::boolean(),source_penciller::'undefined' | pid(),levelzero_astree::'undefined' | [any()],work_ongoing::boolean(),work_backlog::boolean(),timings::'no_timing' | #pcl_timings{sample_count::integer(),foundmem_time::integer(),found0_time::integer(),found1_time::integer(),found2_time::integer(),found3_time::integer(),foundlower_time::integer(),missed_time::integer(),foundmem_count::integer(),found0_count::integer(),found1_count::integer(),found2_count::integer(),found3_count::integer(),foundlower_count::integer(),missed_count::integer()},timings_countdown::integer(),compression_method::'lz4' | 'native'},'false'>
1218: Function timed_fetch_mem/6 has no local return
1259: The variable FP can never match since previous clauses completely covered the type 'false'
1301: The variable _ can never match since previous clauses completely covered the type 'not_present'
1589: Function update_statetimings/2 will never be called
1602: Function log_timings/1 will never be called
1641: The pattern {_, 0} can never match the type {'not_found' | 'tuple' | {[75,...],[86,...]},'basement' | 2 | 3}
1645: The pattern {_, 1} can never match the type {'not_found' | 'tuple' | {[75,...],[86,...]},'basement' | 2 | 3}

src/leveled_pmanifest.erl
1004: Function keylookup_manifest_test/0 has no local return
1012: The variable __V can never match since previous clauses completely covered the type 'false'
1013: The variable __V can never match since previous clauses completely covered the type 'false'
1014: The variable __V can never match since previous clauses completely covered the type 'false'
1015: The variable __V can never match since previous clauses completely covered the type 'false'
1016: The variable __V can never match since previous clauses completely covered the type 'false'
1018: The created fun has no local return
1018: The pattern "pid_z2" can never match the type 'false'
1050: Function ext_keylookup_manifest_test/0 has no local return
1125: The created fun has no local return
1125: The pattern "pid_y3" can never match the type 'false'
===> Warnings written to /Users/martinsumner/dbroot/leveled/_build/default/21.0.dialyzer_warnings
===> Warnings occurred running dialyzer: 27

Most interesting is this line in leveled_codec - 926: The pattern 'true' can never match the type 'false'
As the test for this line passes (i.e. the assertion that it is true passes), so it doesn't seem to make sense that dialyzer claims this can only be false.

[martinsumner]

My fault, not dialyzer (as always).

OTP 21 fixes a dialyzer bug where use of '|' could mask issues. The non-atom part of the '|' is wrong in both cases:

The first element in the tuple emitted by parse_date is a list() not a binary(). The output of key_lookup if not 'false' is a manifest owner, not a manifest entry.

[martinsumner]

This was thrown up in OTP 21 - due to this fix erlang/otp#1722

[martinsumner]

Just a note, that leveled is now being subject to extensive property-based testing. No doubt this will throw up some issues, as property-based testing always does - but once these are resolved we should be ready to tag 1.0

[martinsumner]

The first round of property-based testing is complete.

This came back clean, but a number of bugs have been thrown up by further testing and code analysis in parallel - in particular related to handling data corruption and loss of data from disk being handled alongside multiple stop/starts of the store. These detected bugs have now been resolved.

I've tagged a release candidate v0.9.0, with a target said for a 1.0 release at the end of October.

[benoitc]

@martinsumner maybe you can put a release on hex.pm also.

[benoitc]

bump ^^ :)

[martinsumner]

Sorry. I hear you. I have tasks to tag this and get it fully released as part of Riak Release 2.9.

Sorry, I know I've promised this before. It is on my list!

[martinsumner]

https://hex.pm/packages/leveled

[benoitc]

closing the issue. Sorry for the delay to take care of it i've been side tracked :( Thanks for all the work done :) I will let you know when the project i'm working with will be online :)