Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DAOS-6999 object: add shard id to daos_shard_tgt #4959

Merged
merged 3 commits into from
Mar 17, 2021
Merged

DAOS-6999 object: add shard id to daos_shard_tgt #4959

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
069423b
Merge branch 'master' into daos_6999
Nasf-Fan Mar 9, 2021
2335d74
DAOS-6999 object: add shard id to daos_shard_tgt
Mar 11, 2021
675f0e4
DAOS-6999 cpd: distinguish shard index and identifier
Nasf-Fan Mar 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/include/daos/object.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -206,6 +206,7 @@ struct daos_obj_layout {
struct daos_shard_tgt {
uint32_t st_rank; /* rank of the shard */
uint32_t st_shard; /* shard index */
uint32_t st_shard_id; /* shard id */
liuxuezhao marked this conversation as resolved.
Show resolved Hide resolved
uint32_t st_tgt_id; /* target id */
uint16_t st_tgt_idx; /* target xstream index */
/* target idx for EC obj, only used for client */
Expand Down
3 changes: 2 additions & 1 deletion src/object/cli_obj.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
#define CLI_OBJ_IO_PARMS 8
#define NIL_BITMAP (NULL)

#define OBJ_TGT_INLINE_NR (16)
#define OBJ_TGT_INLINE_NR (12)
struct obj_req_tgts {
/* to save memory allocation if #targets <= OBJ_TGT_INLINE_NR */
struct daos_shard_tgt ort_tgts_inline[OBJ_TGT_INLINE_NR];
Expand Down Expand Up @@ -942,6 +942,7 @@ obj_shard_tgts_query(struct dc_object *obj, uint32_t map_ver, uint32_t shard,

shard_tgt->st_rank = obj_shard->do_target_rank;
shard_tgt->st_shard = shard;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, did not check carefully last time, seems original st_shard is already the shard_id?
assume the different is that they possibly different when grp_nr > 1. For example for an obj_class with grp_nr = 2, grp_size = 3.
Then "shard index" can be [0, 2], but "shard id" can be [0, 5] right?
So seems the original st_shard is "shard id" already?

and in obj_shard_open() "oid.id_shard = obj_shard->do_shard;" seems incorrect, because obj_shard->do_shard is "shard index" but need to be "shard id".

It really confuse to with both the two things, it would be great to only have one (the "shard id"), do we really need the "shard index"?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assume the different is that they possibly different when grp_nr > 1. For example for an obj_class with grp_nr = 2, grp_size = 3.
Then "shard index" can be [0, 2], but "shard id" can be [0, 5] right?
So seems the original st_shard is "shard id" already?

oh, this is for shard extending case (during reintegration/extending). Usually, shard_id is the offset of the shard in layout. (not within the group). But during reintegration, some extra shards (old stale shards) might be added to the layout, then this is not true anymore, so we have to separate them as st_shard(offset) and st_shard_id(real id). Otherwise server forward will use wrong shard to forward, thus causing corruption.

hmm, do_shard is the shard_id, which comes from po_shard. or I miss sth?.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"do_shard is the shard_id, which comes from po_shard" you are right, thanks.

shard_tgt->st_shard_id = obj_shard->do_id.id_shard;
shard_tgt->st_tgt_idx = obj_shard->do_target_idx;
rc = obj_shard2tgtid(obj, shard, map_ver, &shard_tgt->st_tgt_id);
obj_shard_close(obj_shard);
Expand Down
2 changes: 1 addition & 1 deletion src/object/obj_rpc.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -718,7 +718,7 @@ crt_proc_struct_daos_cpd_sub_req(crt_proc_t proc,
* be used as part of the vos object cache index.
*
* It is not important what the id_shard is, that
* is packed via daos_cpd_req_idx::dcri_shard_idx.
* is packed via daos_cpd_req_idx::dcri_shard_id.
*/
rc = crt_proc_daos_unit_oid_t(proc, &oid);
}
Expand Down
8 changes: 6 additions & 2 deletions src/object/obj_rpc.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@
* These are for daos_rpc::dr_opc and DAOS_RPC_OPCODE(opc, ...) rather than
* crt_req_create(..., opc, ...). See daos_rpc.h.
*/
#define DAOS_OBJ_VERSION 2
#define DAOS_OBJ_VERSION 3
/* LIST of internal RPCS in form of:
* OPCODE, flags, FMT, handler, corpc_hdlr and name
*/
Expand Down Expand Up @@ -467,13 +467,17 @@ struct daos_cpd_sub_req {
*/
struct daos_cpd_req_idx {
/* Shard index of the object for the sub request on this DAOS target. */
uint32_t dcri_shard_idx;
uint32_t dcri_shard_off;
/* Shard identifier of the object for the sub request. */
uint32_t dcri_shard_id;
/* The index (relative to the first sub request for its transaction)
* of sub-request in the 'oci_sub_reqs' array. For parsing convenience,
* DCSO_READ requests firstly, then modification ones. The update and
* punch are sorted as their original executed order.
*/
uint32_t dcri_req_idx;
/* 64-bits alignment. */
uint32_t dcri_padding;
};

/**
Expand Down
4 changes: 3 additions & 1 deletion src/object/obj_tx.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1246,8 +1246,10 @@ dc_tx_classify_common(struct dc_tx *tx, struct daos_cpd_sub_req *dcsr,

dcri = &dtrg->dtrg_req_idx[dtrg->dtrg_read_cnt +
dtrg->dtrg_write_cnt];
dcri->dcri_shard_idx = shard->do_shard;
dcri->dcri_shard_off = idx;
dcri->dcri_shard_id = shard->do_shard;
dcri->dcri_req_idx = req_idx;
dcri->dcri_padding = 0;

if (read)
dtrg->dtrg_read_cnt++;
Expand Down
4 changes: 2 additions & 2 deletions src/object/srv_obj.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3576,7 +3576,7 @@ ds_obj_dtx_handle_one(crt_rpc_t *rpc, struct daos_cpd_sub_head *dcsh,
D_GOTO(out, rc = -DER_PROTO);
}

dcsr->dcsr_oid.id_shard = dcri[i].dcri_shard_idx;
dcsr->dcsr_oid.id_shard = dcri[i].dcri_shard_id;
rc = vos_fetch_begin(ioc->ioc_vos_coh, dcsr->dcsr_oid,
dcsh->dcsh_epoch.oe_value,
&dcsr->dcsr_dkey, dcsr->dcsr_nr,
Expand All @@ -3600,7 +3600,7 @@ ds_obj_dtx_handle_one(crt_rpc_t *rpc, struct daos_cpd_sub_head *dcsh,
/* P2: vos_update_begin. */
for (i = 0; i < dcde->dcde_write_cnt; i++) {
dcsr = &dcsrs[dcri[i].dcri_req_idx];
dcsr->dcsr_oid.id_shard = dcri[i].dcri_shard_idx;
dcsr->dcsr_oid.id_shard = dcri[i].dcri_shard_id;

if (dcsr->dcsr_opc != DCSO_UPDATE)
continue;
Expand Down
8 changes: 5 additions & 3 deletions src/object/srv_obj_remote.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,7 +133,7 @@ ds_obj_remote_update(struct dtx_leader_handle *dlh, void *data, int idx,
orw->orw_iod_array.oia_oiod_nr = orw->orw_iod_array.oia_iod_nr;
orw->orw_iod_array.oia_offs = tgt_oiod->oto_offs;
}
orw->orw_oid.id_shard = shard_tgt->st_shard;
orw->orw_oid.id_shard = shard_tgt->st_shard_id;
uuid_copy(orw->orw_co_hdl, orw_parent->orw_co_hdl);
uuid_copy(orw->orw_co_uuid, orw_parent->orw_co_uuid);
orw->orw_shard_tgts.ca_count = orw_parent->orw_shard_tgts.ca_count;
Expand Down Expand Up @@ -359,7 +359,7 @@ ds_obj_cpd_clone_reqs(struct dtx_leader_handle *dlh, struct daos_shard_tgt *tgt,

oiod = obj_ec_tgt_oiod_get(split->osr_tgt_oiods,
dcsr_parent[idx].dcsr_ec_tgt_nr,
dcri_parent->dcri_shard_idx -
dcri_parent->dcri_shard_off -
dcu_parent->dcu_start_shard);
D_ASSERT(oiod != NULL);

Expand All @@ -370,8 +370,10 @@ ds_obj_cpd_clone_reqs(struct dtx_leader_handle *dlh, struct daos_shard_tgt *tgt,
}
}

dcde->dcde_reqs[i].dcri_shard_idx = dcri_parent->dcri_shard_idx;
dcde->dcde_reqs[i].dcri_shard_off = dcri_parent->dcri_shard_off;
dcde->dcde_reqs[i].dcri_shard_id = dcri_parent->dcri_shard_id;
dcde->dcde_reqs[i].dcri_req_idx = i;
dcde->dcde_reqs[i].dcri_padding = dcri_parent->dcri_padding;
}

out:
Expand Down