DAOS-4763 dfs: New API for anchor split to allow parallelizing dfs_readdir #2652

mchaarawi · 2020-05-07T14:29:16Z

To optimize find performance, we need to be able to execute a parallel
readdir. To do that we need to be able to split an anchor among multiple
clients.

This PR adds 2 new API calls each to DAOS obj and DFS:

daos/dfs_obj_anchor_split() to suggest a splitting of anchors by DAOS (internally based on number of shards), but also let user select a desired split factor. The latter is not supported yet.
daos/dfs_obj_anchor_set() to set 1 anchor based on an index from the num of anchors returned from dfs_obj_anchor_split().

Signed-off-by: Mohamad Chaarawi mohamad.chaarawi@intel.com

To optimize find performance, we need to be able to execute a parallel readdir. To do that we need to be able to split an anchor among multiple clients. For this, we will split it depending on the number of shards a directory is shared over. We need a new API for the anchor split based on the shard number, and a new API for DFS to return the number of shards of a directory or file. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

johannlombardi · 2020-05-09T14:56:56Z

src/client/array/dc_array.c

+	*num_shards = layout->ol_nr;
+	daos_obj_layout_free(layout);
+	return 0;
+}


I am not clear on how this is intended to be used for an array TBH. The number of shards in the future might not be stable too (with GIGA+), so i am not sure whether this is a good idea to expose it to the end users.

yea i was debating this myself.. I'll remove that.

johannlombardi · 2020-05-09T15:10:01Z

src/include/daos_obj.h

+/**
+ * Set an anchor for enumeration for one shard only.
+ *
+ * \param[in]	oh	Object open handle.


There is no object handle in the API below.

johannlombardi · 2020-05-09T15:19:04Z

src/include/daos_obj.h

+ * \param[in/out]
+ *		anchor	Anchor modified for 1 shard only
+ *
+ * \return		0		Success and consistent


it should always succeed.. i don't know where i copied that from

johannlombardi · 2020-05-09T15:27:45Z

src/include/daos_obj.h

+ * \return		0		Success and consistent
+ */
+int
+daos_obj_anchor_split(uint32_t shard, daos_anchor_t *anchor);


What I had in mind was more something like this:

/** * Generate an array of anchors that split the dkey address space for parallel enumeration. * * \param[in] oh Object open handle. * \param[in/out] * nr Size of anchors array. If 0 is passed, return the optimal size back. * Number of split anchors returned. * \param[in/out] * anchors Array of split anchors * * \return ... */ int daos_obj_anchor_split(daos_handle_t oh, uint32_t *nr, daos_anchor_t *anchors);

hmm, but how would you distribute the array of anchors to everyone?
The way it works today is that one guy get the optimal number of anchors (which is number of shards today), and inserts works into the queue for each shard. Others pick that up and generate an anchor using this function for a particular shard. otherwise I will probably need to modify this patch more to allow an anchor to span a range but not all shards, which isn't supported today.

johannlombardi · 2020-05-09T15:32:17Z

src/object/cli_obj.c

+dc_obj_anchor_split(uint32_t shard, daos_anchor_t *anchor) {
+	daos_anchor_set_zero(anchor);
+	dc_obj_shard2anchor(anchor, shard);
+	daos_anchor_set_flags(anchor, DIOF_TO_SPEC_SHARD);


Does it return DAOS_ANCHOR_TYPE_EOF when we reach the end of the shard?

yes, but it required that fix i made above on line 2156

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

daosbuild1

Style warning(s) for job https://build.hpdd.intel.com/job/daos-stack/job/daos/job/PR-2652/3/
Please review https://wiki.hpdd.intel.com/display/DC/Coding+Rules

daosbuild1 · 2020-05-12T22:11:05Z

src/include/daos_fs.h

+/**
+ * Set an anchor with an index based on split done with dfs_obj_anchor_split.
+ * The anchor passed will be re-intialized and set to start and finish iteration
+ * based on the specified index. 


(style) trailing whitespace

daosbuild1 · 2020-05-12T22:11:06Z

src/include/daos_fs.h

+ */
+int
+dfs_obj_anchor_set(dfs_obj_t *obj, uint32_t index, daos_anchor_t *anchor);
+


(style) trailing whitespace

daosbuild1 · 2020-05-12T22:11:06Z

src/client/dfs/dfs.c

+	/** TBD - support more than per shard iteration */
+	if (*nr !=0 && *nr != layout->ol_nr) {
+		D_ERROR("For now, num anchors should be the same as what is"
+			"reported as optimal\n");


(style) break quoted strings at a space character

daosbuild1 · 2020-05-12T22:11:06Z

src/client/dfs/dfs.c

+	if (*nr !=0 && *nr != layout->ol_nr) {
+		D_ERROR("For now, num anchors should be the same as what is"
+			"reported as optimal\n");
+		D_GOTO(out, rc = ENOSYS);


(style) ENOSYS means 'invalid syscall nr' and nothing else

daosbuild1 · 2020-05-12T22:11:06Z

src/client/dfs/dfs.c

+		return rc;
+
+	/** TBD - support more than per shard iteration */
+	if (*nr !=0 && *nr != layout->ol_nr) {


(style) spaces required around that '!=' (ctx:WxV)

daosbuild1 · 2020-05-12T22:11:13Z

Test stage checkpatch completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-2652/3/execution/node/56/log

daosbuild1 · 2020-05-12T23:27:38Z

Test stage Functional_Hardware_Small completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-2652/3/execution/node/671/log

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Updated patch

done for regular KV object. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

daosbuild1 · 2020-05-18T16:35:48Z

Test stage Functional_Hardware_Small completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-2652/5/execution/node/685/log

daosbuild1 · 2020-05-18T17:05:08Z

Test stage Functional_Hardware_Large completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-2652/5/execution/node/826/log

daosbuild1 · 2020-05-18T17:47:28Z

Test stage Functional_Hardware_Medium completed with status FAILURE. https://build.hpdd.intel.com//job/daos-stack/job/daos/view/change-requests/job/PR-2652/5/execution/node/814/log

mchaarawi · 2020-05-19T19:54:11Z

@wangdi1 i think at least one of the rebuild test failures might be due to my change to fix iteration to stop when the SPEC flag is added to the anchor.
Could you please advise?

wangdi1 · 2020-05-19T21:15:43Z

@wangdi1 i think at least one of the rebuild test failures might be due to my change to fix iteration to stop when the SPEC flag is added to the anchor.
Could you please advise?

@mchaarawi I checked the result for #6, it looks like
The failure in rebuild0-10 caused by DAOS-4623
The failure in rebuild26 caused by DAOS-4626,

I do not see other rebuild failures. or you mean other failures?

mchaarawi · 2020-05-19T21:18:18Z

@wangdi1 i think at least one of the rebuild test failures might be due to my change to fix iteration to stop when the SPEC flag is added to the anchor.
Could you please advise?

@mchaarawi I checked the result for #6, it looks like
The failure in rebuild0-10 caused by DAOS-4623
The failure in rebuild26 caused by DAOS-4626,

I do not see other rebuild failures. or you mean other failures?

well in here:
https://build.hpdd.intel.com/blue/organizations/jenkins/daos-stack%2Fdaos/detail/PR-2652/6/tests

I see 1 new failures and three existing. i thought the 1 new is caused by my patch.
if not, and it is known, then please never-mind this.

wangdi1 · 2020-05-19T22:32:37Z

@wangdi1 i think at least one of the rebuild test failures might be due to my change to fix iteration to stop when the SPEC flag is added to the anchor.
Could you please advise?
@mchaarawi I checked the result for #6, it looks like
The failure in rebuild0-10 caused by DAOS-4623
The failure in rebuild26 caused by DAOS-4626,
I do not see other rebuild failures. or you mean other failures?

well in here:
https://build.hpdd.intel.com/blue/organizations/jenkins/daos-stack%2Fdaos/detail/PR-2652/6/tests

I see 1 new failures and three existing. i thought the 1 new is caused by my patch.
if not, and it is known, then please never-mind this.

oh, that is 4623. Hmm, not sure why this is marked as new.

mchaarawi force-pushed the mschaara/split_anchor branch from 7995d93 to a0b33e5 Compare May 7, 2020 20:15

mchaarawi changed the title ~~WIP - split anchor for parallel readdir~~ DAOS-4763 dfs + object: Add new API for anchor split and DFS get_shard. May 7, 2020

mchaarawi requested a review from johannlombardi May 8, 2020 14:47

johannlombardi reviewed May 9, 2020

View reviewed changes

Merge branch 'master' into mschaara/split_anchor

3e5817d

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

mchaarawi changed the title ~~DAOS-4763 dfs + object: Add new API for anchor split and DFS get_shard.~~ DAOS-4763 dfs: Add new API for anchor split to allow parallelizing dfs_readdir May 12, 2020

mchaarawi changed the title ~~DAOS-4763 dfs: Add new API for anchor split to allow parallelizing dfs_readdir~~ DAOS-4763 dfs: New API for anchor split to allow parallelizing dfs_readdir May 12, 2020

daosbuild1 previously requested changes May 12, 2020

View reviewed changes

fix checkpatch errors

c50bf20

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

mchaarawi requested a review from johannlombardi May 18, 2020 13:36

add object level API instead of only DFS API to allow the split to be

d218328

done for regular KV object. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

johannlombardi approved these changes May 19, 2020

View reviewed changes

mchaarawi requested review from wangdi1 and gnailzenh May 19, 2020 13:55

Merge branch 'master' into mschaara/split_anchor

90642f9

johannlombardi merged commit ea8e4f0 into master May 21, 2020

johannlombardi deleted the mschaara/split_anchor branch May 21, 2020 06:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DAOS-4763 dfs: New API for anchor split to allow parallelizing dfs_readdir #2652

DAOS-4763 dfs: New API for anchor split to allow parallelizing dfs_readdir #2652

mchaarawi commented May 7, 2020 •

edited

Loading

johannlombardi May 9, 2020

mchaarawi May 9, 2020

johannlombardi May 9, 2020

johannlombardi May 9, 2020

mchaarawi May 9, 2020

johannlombardi May 9, 2020

mchaarawi May 9, 2020 •

edited

Loading

johannlombardi May 9, 2020

mchaarawi May 9, 2020

daosbuild1 left a comment

daosbuild1 May 12, 2020

daosbuild1 May 12, 2020

daosbuild1 May 12, 2020

daosbuild1 May 12, 2020

daosbuild1 May 12, 2020

daosbuild1 commented May 12, 2020

daosbuild1 commented May 12, 2020

daosbuild1 commented May 18, 2020

daosbuild1 commented May 18, 2020

daosbuild1 commented May 18, 2020

mchaarawi commented May 19, 2020

wangdi1 commented May 19, 2020

mchaarawi commented May 19, 2020

wangdi1 commented May 19, 2020

DAOS-4763 dfs: New API for anchor split to allow parallelizing dfs_readdir #2652

DAOS-4763 dfs: New API for anchor split to allow parallelizing dfs_readdir #2652

Conversation

mchaarawi commented May 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mchaarawi May 9, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daosbuild1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daosbuild1 commented May 12, 2020

daosbuild1 commented May 12, 2020

daosbuild1 commented May 18, 2020

daosbuild1 commented May 18, 2020

daosbuild1 commented May 18, 2020

mchaarawi commented May 19, 2020

wangdi1 commented May 19, 2020

mchaarawi commented May 19, 2020

wangdi1 commented May 19, 2020

mchaarawi commented May 7, 2020 •

edited

Loading

mchaarawi May 9, 2020 •

edited

Loading