Skip to content

Commit

Permalink
[#9] Implement configurable and dynamic pick/compute shard and node f…
Browse files Browse the repository at this point in the history
…unctions.
  • Loading branch information
cabol committed Jun 1, 2016
1 parent 534948b commit 1386ae0
Show file tree
Hide file tree
Showing 9 changed files with 540 additions and 358 deletions.
56 changes: 35 additions & 21 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,37 +43,51 @@ Once into the Erlang console:
```erlang
% let's create a table, such as you would create it with ETS, with 4 shards
> shards:new(mytab1, [{n_shards, 4}]).
{mytab1,{shards_local,set,4}}
{mytab1,{4,#Fun<shards_local.pick_shard.3>,set}}
```

Exactly as ETS, `shards:new/2` function receives 2 arguments, the name of the table and
the options. With `shards` there are additional options:

* `{n_shards, pos_integer()}`: defines the number of local shards in which the table will
be split.
* `{n_shards, pos_integer()}`: allows to set the desired number of shards. By default,
the number of shards is calculated from the total online schedulers.

* `{scope, l | g}`: defines `shards` scope, in other words, if sharding will be applied
locally (`l`) or global/distributed (`g`).
locally (`l`) or global/distributed (`g`) – default is `l`.

* `{pick_shard_fun, pick_shard_fun()}`: Function to pick the **shard** on which the `key`
will be handled locally – used by `shards_local`. See the spec [HERE](https://github.com/cabol/shards/blob/master/src/shards_local.erl#L145).

* `{pick_node_fun, pick_node_fun()}`: Function to pick the **node** on which the `key`
will be handled globally/distributed – used by `shards_dist`. See the spec [HERE](https://github.com/cabol/shards/blob/master/src/shards_local.erl#L150).

Also the `shards:new/2` function returns a tuple of two elements:
> **NOTE:** By default `shards` uses a built-in functions to pick the **shard** (local scope)
and the **node** (distributed scope) on which the key will be handled. BUT you can overwrite
them and set your own functions, they are totally configurable by table, so you can have
different tables with different pick-functions each.

Besides, the `shards:new/2` function returns a tuple of two elements:

```erlang
{mytab1, {shards_local, set, 4}}
^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^
1st 2nd
{mytab1, {4, #Fun<shards_local.pick_shard.3>, set}}
^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1st 2nd
```

The first element is the name of the created table; `mytab1`. And the second one is the
[State](./src/shards_local.erl#L147): `{shards_local, set, 4}`. We'll talk about
the **State** later, and see how it can be used.
[State](./src/shards_local.erl#L159): `{4, #Fun<shards_local.pick_shard.3>, set}`.
We'll talk about the **State** later, and see how it can be used.

> **NOTE:** For more information about `shards:new/2` go [HERE](./src/shards_local.erl#L796).
Let's continue:

```erlang
% create another one with default number of shards, which is the total of online
% schedulers; in my case is 8 (4 cores, 2 threads each).
% This value is calculated calling: erlang:system_info(schedulers_online)
> shards:new(mytab2, []).
{mytab2,{shards_local,set,8}}
> shards:new(mytab2, []).
{mytab2,{8,#Fun<shards_local.pick_shard.3>,set}}

% now open the observer so you can see what happened
> observer:start().
Expand Down Expand Up @@ -155,7 +169,7 @@ The module `shards` is a wrapper on top of two main modules:
the distributed part later.

When you use `shards` on top of `shards_local`, a call to the control ETS table owned by `shards_owner_sup`
must be done, in order to recover the [State](./src/shards_local.erl#L147), mentioned previously.
must be done, in order to recover the [State](./src/shards_local.erl#L159), mentioned previously.
Most of the `shards_local` functions receives the **State** as parameter, so it must be fetched before
to call it. You can check how `shards` module is implemented [HERE](./src/shards.erl).

Expand All @@ -168,17 +182,17 @@ or wherever you want. E.g.:
```erlang
% take a look at the 2nd element of the returned tuple, that is the state
> shards:new(mytab, [{n_shards, 4}]).
{mytab, {shards_local, set, 4}}
^^^^^^^^^^^^^^^^^^^^^^
{mytab,{4,#Fun<shards_local.pick_shard.3>,set}}
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

% State: {shards_local, set, 4}
% 1st element is the module called by shards
% 2nd element is the type of ETS table
% 3rd element is the number of shards.
% State: {4, #Fun<shards_local.pick_shard.3>, set}
% 1st element is the number of shards
% 2nd element is the pick function to select the shard
% 3rd element is the type of ETS table

% you can also get the state at any time you want
> shards:state(mytab).
{shards_local, set, 4}
{4,#Fun<shards_local.pick_shard.3>,set}
```

Most of the cases this is not necessary, `shards` wrapper is more than enough, it adds only a
Expand Down Expand Up @@ -217,7 +231,7 @@ $ erl -sname c@localhost -pa _build/default/lib/*/ebin -s shards
% when a tables is created with {scope, g}, the module shards_dist is used
% internally by shards
> shards:new(mytab, [{n_shards, 4}, {scope, g}]).
{mytab,{shards_dist,set,4}}
{mytab,{4,#Fun<shards_local.pick_shard.3>,set}}
```

**3.** Setup the `shards` cluster.
Expand Down
4 changes: 2 additions & 2 deletions c_src/jumping_hash.c
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jumping_hash (unsigned long long key, unsigned int num_buckets) {
}

static ERL_NIF_TERM
calculate (ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) {
compute (ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) {
unsigned long key;
unsigned int num_buckets;

Expand All @@ -30,7 +30,7 @@ calculate (ErlNifEnv *env, int argc, const ERL_NIF_TERM argv[]) {
}

static ErlNifFunc nif_funcs[] = {
{"calculate", 2, calculate}
{"compute", 2, compute}
};

ERL_NIF_INIT (jumping_hash, nif_funcs, NULL, NULL, NULL, NULL);
4 changes: 2 additions & 2 deletions src/hash/jumping_hash.erl
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
-module(jumping_hash).

-export([calculate/2]).
-export([compute/2]).
-on_load(init/0).

-define(APPNAME, shards).
Expand All @@ -11,7 +11,7 @@
%%%===================================================================

%% @hidden
calculate(_, _) ->
compute(_, _) ->
not_loaded(?LINE).

%% @private
Expand Down
Loading

0 comments on commit 1386ae0

Please sign in to comment.