Skip to content
This repository has been archived by the owner on May 3, 2024. It is now read-only.

CORTX-33586: ctgdump is crashing after new btree merge #1974

Merged
merged 5 commits into from
Jul 15, 2022

Conversation

kanchan-chaudhari
Copy link
Contributor

Issue:

  • ctg_open() is responsible for allocating tree pointer and
    opening the btree. This was missing before accessing the
    catalogue tree in the ctgdump().

Fix:

  • Added ctg_open() call before accessing tree pointer.
  • Added m0_btree_close() to close tree in ctg_fini().

Signed-off-by: Kanchan Chaudhari kanchan.chaudhari@seagate.com

Problem Statement

  • Problem statement

Design

  • For Bug, Describe the fix here.
  • For Feature, Post the link for design

Coding

Checklist for Author

  • Coding conventions are followed and code is consistent

Testing

Checklist for Author

  • Unit and System Tests are added
  • Test Cases cover Happy Path, Non-Happy Path and Scalability
  • Testing was performed with RPM

Impact Analysis

Checklist for Author/Reviewer/GateKeeper

  • Interface change (if any) are documented
  • Side effects on other features (deployment/upgrade)
  • Dependencies on other component(s)

Review Checklist

Checklist for Author

  • JIRA number/GitHub Issue added to PR
  • PR is self reviewed
  • Jira and state/status is updated and JIRA is updated with PR link
  • Check if the description is clear and explained

Documentation

Checklist for Author

  • Changes done to WIKI / Confluence page / Quick Start Guide

Issue:
- ctg_open() is responsible for allocating tree pointer and
  opening the btree. This was missing before accessing the
  catalogue tree in the ctgdump().

Fix:
- Added ctg_open() call before accessing tree pointer.
- Added m0_btree_close() to close tree in ctg_fini().

Signed-off-by: Kanchan Chaudhari <kanchan.chaudhari@seagate.com>
@kanchan-chaudhari
Copy link
Contributor Author

Tested patch with s3ios and 1N deployment with Custom Build #7171, IOs are working fine.

Copy link
Contributor

@madhavemuri madhavemuri left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@huanghua78
Copy link

Tested patch with s3ios and 1N deployment with Custom Build #7171, IOs are working fine.

This utility is an offline tool.
So, it does not impact the normal deployment and normal I/O operations.

The following test must be run and verified:
cas/st/ctgdump_tesh.sh

It has not been added to ST list. Maybe it's time to add it into the list now.

@huanghua78
Copy link

huanghua78 commented Jul 12, 2022

Please apply the following patch, and include this patch into your PR.
This is to fix a wrong modification to this script. (see commit 58066d6, #1644)
And then run the following test:
sudo sh cas/st/ctgdump_tesh.sh test

diff --git a/cas/st/ctgdump_tesh.sh b/cas/st/ctgdump_tesh.sh
index dfe59c44a..89ad3fb70 100755
--- a/cas/st/ctgdump_tesh.sh
+++ b/cas/st/ctgdump_tesh.sh
@@ -56,7 +56,7 @@ kvs_create_n_insert()

        echo "$M0_SRC_DIR/utils/m0kv" "${MOTR_PARAM}" index create "$DIX_FID"

-       "$M0_SRC_DIR/utils/m0kv" "${MOTR_PARAM}"                                     \
+       "$M0_SRC_DIR/utils/m0kv" ${MOTR_PARAM}                                     \
                                index create "$DIX_FID"                            \
                         || {
                rc=$?
@@ -64,7 +64,7 @@ kvs_create_n_insert()
        }

        for ((j=0; j<$num_of_kv; j++)); do
-               "$M0_SRC_DIR/utils/m0kv" "${MOTR_PARAM}"                                       \
+               "$M0_SRC_DIR/utils/m0kv" ${MOTR_PARAM}                                       \
                        index put    "$DIX_FID" "$key_prefix-$j" "$val_prefix-$j"
        done
        return $rc
@@ -95,7 +95,7 @@ ctgdump_comp()
                               -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 262144 -K 100663296 \
                               -k 262144 -c ${dir}/confd/conf.xc -e ${cas_ep} -f ${proc_fid} str ${dix_fid} "
                echo "running: $dump_cmd $dump_cmd_args"
-               $dump_cmd "$dump_cmd_args" | sort > "$SANDBOX_DIR/ios$i.dump"
+               $dump_cmd $dump_cmd_args | sort > "$SANDBOX_DIR/ios$i.dump"
        done

        # Compare each dump file

You will see something like this:

bash-4.4$ sudo sh cas/st/ctgdump_tesh.sh test
[sudo] password for 520428:
vm.max_map_count = 30000000
motr_service_start: (N,K,S,P)=(2,1,1,4) nr_ios=4 multiple_pools=0
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00368365 s, 285 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00321724 s, 326 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00187618 s, 559 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00264693 s, 396 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00281914 s, 372 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00251679 s, 417 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00324389 s, 323 MB/s
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0068233 s, 154 MB/s

[104:
  {0x74| ((^t|1:0), 1, (11, 22), ^o|2:9, ^v|1:20, 1,
          [1: "4 2 1 1"],
          [1: ^n|1:2],
          [1: ^S|1:6],
          [3: ^o|1:9, ^o|20:1, ^o|2:9],
          [1: ^p|1:0], [0])},
  {0x70| ((^p|1:0), [3: ^o|1:9, ^o|20:1, ^o|2:9])},
  {0x6e| ((^n|1:2), 16000, 2, 3, 2, [8: ^r|1:100, ^r|1:0, ^r|1:1, ^r|1:2, ^r|1:3, ^r|1:4, ^r|1:5, ^r|1:6])},

  {0x72| ((^r|1:100), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:1", [1: ^s|1:101])},
  {0x72| ((^r|1:0), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:900", [8: ^s|1:0, ^s|11:0, ^s|6:0, ^s|7:0,                     ^s|3:0, ^s|13:0, ^s|15:0, ^s|16:0])},
  {0x72| ((^r|1:1), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:901", [8: ^s|1:1, ^s|11:1, ^s|6:1, ^s|7:1,                     ^s|3:1, ^s|13:1, ^s|15:1, ^s|16:1])},
  {0x72| ((^r|1:2), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:902", [8: ^s|1:2, ^s|11:2, ^s|6:2, ^s|7:2,                     ^s|3:2, ^s|13:2, ^s|15:2, ^s|16:2])},
  {0x72| ((^r|1:3), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:903", [8: ^s|1:3, ^s|11:3, ^s|6:3, ^s|7:3,                     ^s|3:3, ^s|13:3, ^s|15:3, ^s|16:3])},
 {0x72| ((^r|1:4), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:800", [3: ^s|2:0, ^s|12:0, ^s|3:4])},
 {0x72| ((^r|1:5), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:34:1",
                          [3: ^s|1:6, ^s|1:7, ^s|3:6])},
 {0x72| ((^r|1:6), [1:3], 0, 0, 0, 0, "10.230.240.203@tcp:12345:33:100",
                          [2: ^s|8:0, ^s|3:7])},
  {0x73| ((^s|8:0), @M0_CST_CONFD, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x73| ((^s|1:6), @M0_CST_HA, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:7), @M0_CST_FIS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
  {0x73| ((^s|1:101), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:1"], [0], [0])},




 {0x73| ((^s|2:0), @M0_CST_MDS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|12:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},
  {0x73| ((^s|3:4), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:800"], [0], [0])},

  {0x73| ((^s|1:0), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [1: ^d|1:1])},
  {0x73| ((^s|11:0), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|6:0), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|7:0), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|3:0), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
 {0x73| ((^s|13:0), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:900"], [0], [1: ^d|20:0])},
  {0x73| ((^s|15:0), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|16:0), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:900"], [0], [0])},
  {0x73| ((^s|1:1), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [1: ^d|1:2])},
  {0x73| ((^s|11:1), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|6:1), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|7:1), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|3:1), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
 {0x73| ((^s|13:1), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:901"], [0], [1: ^d|20:1])},
  {0x73| ((^s|15:1), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|16:1), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:901"], [0], [0])},
  {0x73| ((^s|1:2), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [1: ^d|1:3])},
  {0x73| ((^s|11:2), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|6:2), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|7:2), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|3:2), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
 {0x73| ((^s|13:2), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:902"], [0], [1: ^d|20:2])},
  {0x73| ((^s|15:2), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|16:2), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:902"], [0], [0])},
  {0x73| ((^s|1:3), @M0_CST_IOS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [1: ^d|1:4])},
  {0x73| ((^s|11:3), @M0_CST_ADDB2, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|6:3), @M0_CST_SNS_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|7:3), @M0_CST_SNS_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|3:3), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
 {0x73| ((^s|13:3), @M0_CST_CAS, [1: "10.230.240.203@tcp:12345:33:903"], [0], [1: ^d|20:3])},
  {0x73| ((^s|15:3), @M0_CST_DIX_REP, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},
  {0x73| ((^s|16:3), @M0_CST_DIX_REB, [1: "10.230.240.203@tcp:12345:33:903"], [0], [0])},

 {0x73| ((^s|3:6), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:34:1"], [0], [0])},
 {0x73| ((^s|3:7), @M0_CST_RMS, [1: "10.230.240.203@tcp:12345:33:100"], [0], [0])},
  {0x64| ((^d|1:1), 0, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop1")},
 {0x6b| ((^k|1:1), ^d|1:1, [1: ^v|1:10])},
 {0x6a| ((^j|1:1), ^k|1:1, [0])},
 {0x64| ((^d|1:2), 1, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop2")},
 {0x6b| ((^k|1:2), ^d|1:2, [1: ^v|1:10])},
 {0x6a| ((^j|1:2), ^k|1:2, [0])},
 {0x64| ((^d|1:3), 2, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop3")},
 {0x6b| ((^k|1:3), ^d|1:3, [1: ^v|1:10])},
 {0x6a| ((^j|1:3), ^k|1:3, [0])},
 {0x64| ((^d|1:4), 3, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop4")},
 {0x6b| ((^k|1:4), ^d|1:4, [1: ^v|1:10])},
 {0x6a| ((^j|1:4), ^k|1:4, [0])},
  {0x53| ((^S|1:6), [1: ^a|1:6], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x61| ((^a|1:6), [1: ^e|1:7], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x65| ((^e|1:7), ^n|1:2, [1: ^c|1:8], [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x63| ((^c|1:8), [8: ^k|1:1, ^k|1:2, ^k|1:3, ^k|1:4, ^k|20:0, ^k|20:1, ^k|20:2, ^k|20:3],
                            [3: ^v|1:10, ^v|1:20, ^v|2:10])},
  {0x6f| ((^o|1:9), 0, [3: ^v|1:10, ^v|0x40000000000001:11, ^v|0x40000000000001:12])},
  {0x76| ((^v|1:10), {0| (2, 1,
                                            1, 4,
                                            [5: 0, 0, 0, 0, 1],
                                            [1: ^j|1:5])})},
  {0x76| ((^v|0x40000000000001:11), {1| (0, ^v|1:10, [5: 0, 0, 0, 0, 1])})},
  {0x76| ((^v|0x40000000000001:12), {1| (1, ^v|1:10, [5: 0, 0, 0, 0, 2])})},
  {0x6a| ((^j|1:5), ^S|1:6, [1: ^j|1:6])},
  {0x6a| ((^j|1:6), ^a|1:6, [1: ^j|1:7])},
  {0x6a| ((^j|1:7), ^e|1:7, [1: ^j|1:8])},
  {0x6a| ((^j|1:8), ^c|1:8, [4: ^j|1:1, ^j|1:2, ^j|1:3, ^j|1:4])} ,
{0x6f| ((^o|2:9), 0, [1: ^v|2:10])},
{0x76| ((^v|2:10), {0| (4, 0, 0, 4, [5: 0, 0, 0, 0, 1], [1: ^j|2:5])})},
{0x6a| ((^j|2:5), ^S|1:6, [1: ^j|2:6])},
{0x6a| ((^j|2:6), ^a|1:6, [1: ^j|2:7])},
{0x6a| ((^j|2:7), ^e|1:7, [1: ^j|2:8])},
{0x6a| ((^j|2:8), ^c|1:8, [4: ^j|2:1, ^j|2:2, ^j|2:3, ^j|2:4])},
{0x6a| ((^j|2:1), ^k|1:1, [0])},
 {0x6a| ((^j|2:2), ^k|1:2, [0])},
 {0x6a| ((^j|2:3), ^k|1:3, [0])},
 {0x6a| ((^j|2:4), ^k|1:4, [0])}  ,
{0x64| ((^d|20:0), 4, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop25")},
{0x6b| ((^k|20:0), ^d|20:0, [1: ^v|1:20])},
{0x6a| ((^j|20:100), ^k|20:0, [0])},
{0x64| ((^d|20:1), 5, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop26")},
{0x6b| ((^k|20:1), ^d|20:1, [1: ^v|1:20])},
{0x6a| ((^j|20:101), ^k|20:1, [0])},
{0x64| ((^d|20:2), 6, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop27")},
{0x6b| ((^k|20:2), ^d|20:2, [1: ^v|1:20])},
{0x6a| ((^j|20:102), ^k|20:2, [0])},
{0x64| ((^d|20:3), 7, 4, 1, 4096, 596000000000, 3, 4, "/dev/loop28")},
{0x6b| ((^k|20:3), ^d|20:3, [1: ^v|1:20])},
{0x6a| ((^j|20:103), ^k|20:3, [0])},
{0x6f| ((^o|20:1), 0, [1: ^v|1:20])},
{0x76| ((^v|1:20), {0| (1, 3, 0,
                                                    4,
                                                    [5: 0, 0, 0, 0, 3],
                                                    [1: ^j|20:1])})},
{0x6a| ((^j|20:1), ^S|1:6, [1: ^j|20:2])},
{0x6a| ((^j|20:2), ^a|1:6, [1: ^j|20:3])},
{0x6a| ((^j|20:3), ^e|1:7, [1: ^j|20:4])},
{0x6a| ((^j|20:4), ^c|1:8, [4:  ^j|20:100, ^j|20:101, ^j|20:102, ^j|20:103])}]
cd /var/motr/systest-2528905/confd && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs                          -w 4 -m 65536               -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T linux -e libfab:10.230.240.203@tcp:12345:35:1                      -c /var/motr/systest-2528905/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-2528905/confd && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -f '<0x7200000000000001:6>' -T linux -e libfab:10.230.240.203@tcp:12345:33:100 -c /var/motr/systest-2528905/confd/conf.xc |& tee -a m0d.log
cd /var/motr/systest-2528905/ha && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:35:1 -c /var/motr/systest-2528905/confd/conf.xc |& tee -a m0mkfs.log
motr[2529091]:  f800  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/systest-2528905/mds1 && exec                       /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad                         -D db -S stobs -A linuxstob:addb-stobs                            -w 4 -m 65536                           -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:800                  -c /var/motr/systest-2528905/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-2528905/ios1 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:900 -f '<0x7200000000000001:0>' -c /var/motr/systest-2528905/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-2528905/ios2 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:901 -f '<0x7200000000000001:1>' -c /var/motr/systest-2528905/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-2528905/ios3 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:902 -f '<0x7200000000000001:2>' -c /var/motr/systest-2528905/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-2528905/ios4 && exec /home/520428/work/cortx-motr/utils/mkfs/m0mkfs -F -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:35:903 -f '<0x7200000000000001:3>' -c /var/motr/systest-2528905/confd/conf.xc |& tee -a m0mkfs.log
cd /var/motr/systest-2528905/ha && exec /home/520428/work/cortx-motr/motr/m0d -D db -S stobs -A linuxstob:addb-stobs                         -w 4 -m 65536                           -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -T ad -e libfab:10.230.240.203@tcp:12345:34:1                     -c /var/motr/systest-2528905/confd/conf.xc -f '<0x7200000000000001:5>'  -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
motr[2529578]:  aaa0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr HA agent started.
cd /var/motr/systest-2528905/mds1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:800 -f '<0x7200000000000001:4>' -H 10.230.240.203@tcp:12345:34:1 -c /var/motr/systest-2528905/confd/conf.xc |& tee -a m0d.log
motr[2529652]:  3020  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
cd /var/motr/systest-2528905/ios1 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:900 -f '<0x7200000000000001:0>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/systest-2528905/ios2 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:901 -f '<0x7200000000000001:1>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/systest-2528905/ios3 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:902 -f '<0x7200000000000001:2>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
cd /var/motr/systest-2528905/ios4 && exec /home/520428/work/cortx-motr/motr/m0d -T ad -d disks.conf -D db -S stobs -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 307200 -K 100663296 -k 307200 -e libfab:10.230.240.203@tcp:12345:33:903 -f '<0x7200000000000001:3>' -H 10.230.240.203@tcp:12345:34:1 |& tee -a m0d.log
Motr confd started.
Motr mdservices started.
motr[2529739]:  6430  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[2529733]:  c4d0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[2529736]:  cdd0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
motr[2529726]:  cb00  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
Started
lt-m0d: systemd notifications not allowed

Press CTRL+C to quit.
Motr ioservices started.
/home/520428/work/cortx-motr/utils/m0kv -l 10.230.240.203@tcp:12345:33:1000  -h 10.230.240.203@tcp:12345:34:1 -p 0x7000000000000001:0               -f 0x7200000000000001:64 -s  index create 7800000000012345:12345
motr[2530019]:  3ef0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
create done, rc: 0
Done, rc:  0
motr[2530064]:  2050  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530109]:  9af0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530154]:  9080  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530199]:  4c80  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530244]:  feb0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530289]:  c560  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530334]:  3660  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530379]:  9b90  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530424]:   120  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530469]:  6e30  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530514]:  5cd0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530559]:  bb40  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530604]:   c20  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530649]:  c810  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530694]:  b590  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
motr[2530739]:  38a0  ERROR  [conf/helpers.c:552:m0_conf_process2service_get]  <! rc=-2
operation rc: 0
put done, rc: 0
Done, rc:  0
stopping /home/520428/work/cortx-motr/motr/m0d processes...
=== pids of services: 2529091 2529578 2529652 2529726 2529733 2529736 2529739 ===
Shutting down services one by one. mdservice is the last.
----- 2529091 stopping--------lt-m0d: got signal 1
motr[2529091]:  ea00  FATAL  [lib/assert.c:50:m0_panic]  panic: fatal signal delivered at unknown() (unknown:0)  [git: 2.0.0-853-7-gc079391c9] /var/motr/systest-2528905/confd/m0trace.2529091.2022-07-12-18:20:05
Motr panic: fatal signal delivered at unknown() unknown:0 (errno: 4) (last failed: none) [git: 2.0.0-853-7-gc079391c9] pid: 2529091  /var/motr/systest-2528905/confd/m0trace.2529091.2022-07-12-18:20:05
Motr panic reason: signo: 11
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_backtrace+0x20)[0x7f8d3e1c0c70]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_arch_panic+0xdf)[0x7f8d3e1c0e1f]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_panic+0x127)[0x7f8d3e1aedc7]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x3d1e68)[0x7f8d3e1c0e68]
/lib64/libpthread.so.0(+0x12b30)[0x7f8d3d85fb30]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_rpc_conn_is_snd+0x0)[0x7f8d3e23b7e0]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_rpc_post+0x51)[0x7f8d3e24a321]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_rm_outgoing_send+0xe8)[0x7f8d3e22bbd8]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_rm_request_out+0x1fd)[0x7f8d3e22bf1d]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x44385f)[0x7f8d3e23285f]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x443f5c)[0x7f8d3e232f5c]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x444134)[0x7f8d3e233134]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_rm_owner_windup+0x68)[0x7f8d3e2331b8]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x447589)[0x7f8d3e236589]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x447676)[0x7f8d3e236676]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_reqh_service_stop+0x8e)[0x7f8d3e229cee]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(cs_service_fini+0x7b)[0x7f8d3e1ce8bb]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(+0x3dfae0)[0x7f8d3e1ceae0]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_module_fini+0x70)[0x7f8d3e201a50]
/home/520428/work/cortx-motr/motr/.libs/libmotr.so.2(m0_cs_fini+0x2d)[0x7f8d3e1d2f4d]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x4011cc]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7f8d3c0814a3]
/home/520428/work/cortx-motr/motr/.libs/lt-m0d[0x40145e]
----- 2529091 stopped --------
----- 2529578 stopping--------lt-m0d: got signal 1
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
motr[2529578]:  b810   WARN  [ha/link.c:1514:ha_link_outgoing_fom_tick]  rlk_rc=-110 endpoint=10.230.240.203@tcp:12345:33:1000
----- 2529578 stopped --------
----- 2529652 stopping--------lt-m0d: got signal 1
motr[2529652]:  6810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x2067ff0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=98 delivered=98 next=98 assign=124)
motr[2529652]:  6810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=51 hl=0x2067ff0 ep=10.230.240.203@tcp:12345:34:1
motr[2529652]:  6a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 2529652 stopped --------
----- 2529726 stopping--------lt-m0d: got signal 1
motr[2529726]:  b810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0xd61fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=104)
motr[2529726]:  b810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0xd61fb0 ep=10.230.240.203@tcp:12345:34:1
motr[2529726]:  ba70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 2529726 stopped --------
----- 2529733 stopping--------lt-m0d: got signal 1
motr[2529733]:  8810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x2107fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=118)
motr[2529733]:  8810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x2107fb0 ep=10.230.240.203@tcp:12345:34:1
motr[2529733]:  8a70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 2529733 stopped --------
----- 2529736 stopping--------lt-m0d: got signal 1
motr[2529736]:  f810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x1347fb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=132)
motr[2529736]:  f810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x1347fb0 ep=10.230.240.203@tcp:12345:34:1
motr[2529736]:  fa70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 2529736 stopped --------
----- 2529739 stopping--------lt-m0d: got signal 1
motr[2529739]:  e810   WARN  [ha/link.c:1285:ha_link_outgoing_fop_replied]  rc=-110 nr=1 hl=0x26eefb0 ep=10.230.240.203@tcp:12345:34:1 lq_tags=(confirmed=64 delivered=64 next=64 assign=146)
motr[2529739]:  e810   WARN  [ha/link.c:1289:ha_link_outgoing_fop_replied]  old_rc=0 old_nr=32 hl=0x26eefb0 ep=10.230.240.203@tcp:12345:34:1
motr[2529739]:  ea70   WARN  [ha/entrypoint.c:563:ha_entrypoint_client_fom_tick]  rlk_rc=-110
----- 2529739 stopped --------
Motr services stopped.
{key: iter-äää-0}, {val: something1_anotherstring2*YETanotherstring3-0}
{key: iter-äää-1}, {val: something1_anotherstring2*YETanotherstring3-1}
{key: iter-äää-2}, {val: something1_anotherstring2*YETanotherstring3-2}
{key: iter-äää-3}, {val: something1_anotherstring2*YETanotherstring3-3}
{key: iter-äää-4}, {val: something1_anotherstring2*YETanotherstring3-4}
{key: iter-äää-5}, {val: something1_anotherstring2*YETanotherstring3-5}
{key: iter-äää-6}, {val: something1_anotherstring2*YETanotherstring3-6}
{key: iter-äää-7}, {val: something1_anotherstring2*YETanotherstring3-7}
{key: iter-äää-8}, {val: something1_anotherstring2*YETanotherstring3-8}
{key: iter-äää-9}, {val: something1_anotherstring2*YETanotherstring3-9}
{key: iter-äää-10}, {val: something1_anotherstring2*YETanotherstring3-10}
{key: iter-äää-11}, {val: something1_anotherstring2*YETanotherstring3-11}
{key: iter-äää-12}, {val: something1_anotherstring2*YETanotherstring3-12}
{key: iter-äää-13}, {val: something1_anotherstring2*YETanotherstring3-13}
{key: iter-äää-14}, {val: something1_anotherstring2*YETanotherstring3-14}
{key: iter-äää-15}, {val: something1_anotherstring2*YETanotherstring3-15}
running: /home/520428/work/cortx-motr/cas/m0ctgdump -T ad -d /var/motr/systest-2528905/ios1/disks.conf -D /var/motr/systest-2528905/ios1/db -S /var/motr/systest-2528905/ios1/stobs        -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 262144 -K 100663296                          -k 262144 -c /var/motr/systest-2528905/confd/conf.xc -e libfab:10.230.240.203@tcp:12345:33:900 -f 0x7200000000000001:0 str 7800000000012345:12345
running: /home/520428/work/cortx-motr/cas/m0ctgdump -T ad -d /var/motr/systest-2528905/ios2/disks.conf -D /var/motr/systest-2528905/ios2/db -S /var/motr/systest-2528905/ios2/stobs        -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 262144 -K 100663296                          -k 262144 -c /var/motr/systest-2528905/confd/conf.xc -e libfab:10.230.240.203@tcp:12345:33:900 -f 0x7200000000000001:0 str 7800000000012345:12345
running: /home/520428/work/cortx-motr/cas/m0ctgdump -T ad -d /var/motr/systest-2528905/ios3/disks.conf -D /var/motr/systest-2528905/ios3/db -S /var/motr/systest-2528905/ios3/stobs        -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 262144 -K 100663296                          -k 262144 -c /var/motr/systest-2528905/confd/conf.xc -e libfab:10.230.240.203@tcp:12345:33:900 -f 0x7200000000000001:0 str 7800000000012345:12345
running: /home/520428/work/cortx-motr/cas/m0ctgdump -T ad -d /var/motr/systest-2528905/ios4/disks.conf -D /var/motr/systest-2528905/ios4/db -S /var/motr/systest-2528905/ios4/stobs        -A linuxstob:addb-stobs -w 4 -m 65536 -q 16 -N 100663296 -C 262144 -K 100663296                          -k 262144 -c /var/motr/systest-2528905/confd/conf.xc -e libfab:10.230.240.203@tcp:12345:33:900 -f 0x7200000000000001:0 str 7800000000012345:12345
All CTG dumps match!
ctgdump: test status: SUCCESS
bash-4.4$
bash-4.4$
bash-4.4$
bash-4.4$

(Please ignore the panic there. It will be fixed in #1976)

madhavemuri and others added 2 commits July 13, 2022 16:35
@kanchan-chaudhari
Copy link
Contributor Author

In reference to #1974 (comment), we will take this task as a part of separate ticket. CC: @huanghua78

Signed-off-by: Kanchan Chaudhari <kanchan.chaudhari@seagate.com>
Copy link

@huanghua78 huanghua78 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK

@cortx-admin
Copy link

Jenkins CI Result : Motr#1475

Motr Test Summary

Test ResultCountInfo
❌Failed22
📁

01motr-single-node/43m0crate
01motr-single-node/52motr-singlenode-sanity
01motr-single-node/01net
01motr-single-node/03console
01motr-single-node/02rpcping
02motr-single-node/07m0d-fatal
02motr-single-node/67fdmi-plugin-multi-filters
02motr-single-node/53clusterusage-alert
02motr-single-node/41motr-conf-update
03motr-single-node/61sns-repair-motr-1n-1f
03motr-single-node/08spiel-multi-confd
03motr-single-node/62sns-repair-motr-mf
03motr-single-node/63sns-repair-motr-1k-1f
03motr-single-node/60sns-repair-motr-1f
03motr-single-node/66sns-repair-motr-abort-quiesce
03motr-single-node/24motr-dix-repair-lookup-insert-spiel
03motr-single-node/68sns-repair-motr-shutdown
03motr-single-node/64sns-repair-motr-ios-fail
03motr-single-node/24motr-dix-repair-lookup-insert-m0repair
03motr-single-node/04sss
03motr-single-node/65sns-repair-motr-abort
01motr-single-node/00userspace-tests

🏁Skipped15
📁

01motr-single-node/28sys-kvs
01motr-single-node/35m0singlenode
01motr-single-node/04initscripts
01motr-single-node/37protocol
02motr-single-node/51kem
02motr-single-node/20rpc-session-cancel
02motr-single-node/10pver-assign
02motr-single-node/21fsync-single-node
02motr-single-node/13dgmode-io
02motr-single-node/14poolmach
02motr-single-node/11m0t1fs
02motr-single-node/26motr-user-kernel-tests
02motr-single-node/08spiel
03motr-single-node/06conf
03motr-single-node/36spare-reservation

✔️Passed3
📁

01motr-single-node/05confgen
01motr-single-node/06hagen
01motr-single-node/01kernel-tests

Total40🔗

CppCheck Summary

   Cppcheck: No new warnings found 👍

@kanchan-chaudhari
Copy link
Contributor Author

@rkothiya
Copy link
Contributor

Since the results of ci/cd job is not automatically updated here, I am pasting the results. Below are known failures :
image

@rkothiya rkothiya merged commit 65bcdcb into Seagate:main Jul 15, 2022
@kanchan-chaudhari kanchan-chaudhari deleted the CORTX-33586 branch July 15, 2022 13:09
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants