CORTX-33789: Hare Config stage fail with Motr Clients #2146

pavankrishnat · 2022-08-02T13:47:30Z

CORTX-33789: Hare Config stage fail with Motr Clients
Problem:
If num_client_inst is specified a non-zero value, deployment is failing.
Issue is with search key. For data nodes we used 'io':'motr', it returns the
list of machine IDs which includes both data node and motr client nodes,
because of which deployment was failing.

Solution:
Separated search key for data node and motr client node.

Signed-off-by: pavankrishnat pavan.k.thunuguntla@seagate.com

vaibhavparatwar · 2022-08-03T03:24:03Z

retest this please

vaibhavparatwar · 2022-08-03T13:50:08Z

@mssawant @d-nayak @Shreya-18 please review this PR. Premerge failure is because of some infra issue.

mssawant · 2022-08-03T16:45:11Z

@pavankrishnat can you please elaborate the problem and solution in the commit message?

vaibhavparatwar · 2022-08-04T03:26:04Z

@pavankrishnat request you to add appropriate commit message

provisioning/miniprov/hare_mp/store.py

supriyachavan4398 · 2022-08-04T05:56:05Z

With below image:

cortx-docker.colo.seagate.com/seagate/cortx-rgw:2.0.0-7433-custom-ci
cortx-docker.colo.seagate.com/seagate/cortx-data:2.0.0-7433-custom-ci
cortx-docker.colo.seagate.com/seagate/cortx-all:2.0.0-7433-custom-ci
cortx-docker.colo.seagate.com/seagate/cortx-control:2.0.0-7433-custom-ci

Deployed 6N cluster without motr client successfully at https://eos-jenkins.colo.seagate.com/job/Cortx-Automation/job/RGW/job/setup-cortx-rgw-cluster/10407/console

[root@ssc-vm-g4-rhev4-0710 ~]# kubectl get pods
NAME                             READY   STATUS    RESTARTS        AGE
cortx-consul-client-2pgsr        1/1     Running   0               6m24s
cortx-consul-client-2sc5f        1/1     Running   0               6m24s
cortx-consul-client-6q5ch        1/1     Running   0               6m24s
cortx-consul-client-k2p76        1/1     Running   0               6m24s
cortx-consul-client-vmmlw        1/1     Running   0               6m24s
cortx-consul-client-z2b2s        1/1     Running   0               6m24s
cortx-consul-server-0            1/1     Running   0               6m24s
cortx-consul-server-1            1/1     Running   0               6m24s
cortx-consul-server-2            1/1     Running   0               6m24s
cortx-control-776fd76b88-7jmlr   1/1     Running   0               6m24s
cortx-data-g0-0                  3/3     Running   0               6m24s
cortx-data-g0-1                  3/3     Running   0               6m23s
cortx-data-g0-2                  3/3     Running   0               6m22s
cortx-data-g0-3                  3/3     Running   0               6m21s
cortx-data-g0-4                  3/3     Running   0               6m20s
cortx-data-g0-5                  3/3     Running   0               6m19s
cortx-data-g1-0                  3/3     Running   0               6m24s
cortx-data-g1-1                  3/3     Running   0               6m23s
cortx-data-g1-2                  3/3     Running   0               6m21s
cortx-data-g1-3                  3/3     Running   0               6m21s
cortx-data-g1-4                  3/3     Running   0               6m20s
cortx-data-g1-5                  3/3     Running   0               6m19s
cortx-ha-6b9984c66c-s7cwc        3/3     Running   3 (4m16s ago)   6m24s
cortx-kafka-0                    1/1     Running   0               6m23s
cortx-kafka-1                    1/1     Running   0               6m22s
cortx-kafka-2                    1/1     Running   0               6m22s
cortx-server-0                   2/2     Running   0               6m24s
cortx-server-1                   2/2     Running   0               6m24s
cortx-server-2                   2/2     Running   0               6m24s
cortx-server-3                   2/2     Running   0               6m23s
cortx-server-4                   2/2     Running   0               6m23s
cortx-server-5                   2/2     Running   0               6m22s
cortx-zookeeper-0                1/1     Running   0               6m24s
cortx-zookeeper-1                1/1     Running   0               6m24s
cortx-zookeeper-2                1/1     Running   0               6m24s

Deployed 6N cluster with motr client successfully at https://eos-jenkins.colo.seagate.com/job/Cortx-kubernetes/job/setup-cortx-cluster-solution-input/657/console

    motr:
      num_client_inst: 1
      extra_configuration: ""

[root@ssc-vm-g4-rhev4-0710 k8_cortx_cloud]# kubectl get pods
NAME                             READY   STATUS    RESTARTS   AGE
cortx-client-0                   2/2     Running   0          4m53s
cortx-client-1                   2/2     Running   0          4m53s
cortx-client-2                   2/2     Running   0          4m53s
cortx-client-3                   2/2     Running   0          4m53s
cortx-client-4                   2/2     Running   0          4m53s
cortx-client-5                   2/2     Running   0          4m53s
cortx-consul-client-56lnd        1/1     Running   0          4m53s
cortx-consul-client-6ctcj        1/1     Running   0          4m53s
cortx-consul-client-bfj5q        1/1     Running   0          4m53s
cortx-consul-client-hjvwp        1/1     Running   0          4m53s
cortx-consul-client-mcddk        1/1     Running   0          4m53s
cortx-consul-client-p8l9s        1/1     Running   0          4m53s
cortx-consul-server-0            1/1     Running   0          4m53s
cortx-consul-server-1            1/1     Running   0          4m53s
cortx-consul-server-2            1/1     Running   0          4m53s
cortx-control-776fd76b88-pgss6   1/1     Running   0          4m53s
cortx-data-g0-0                  3/3     Running   0          4m53s
cortx-data-g0-1                  3/3     Running   0          4m52s
cortx-data-g0-2                  3/3     Running   0          4m51s
cortx-data-g0-3                  3/3     Running   0          4m50s
cortx-data-g0-4                  3/3     Running   0          4m49s
cortx-data-g0-5                  3/3     Running   0          4m48s
cortx-data-g1-0                  3/3     Running   0          4m51s
cortx-data-g1-1                  3/3     Running   0          4m50s
cortx-data-g1-2                  3/3     Running   0          4m49s
cortx-data-g1-3                  3/3     Running   0          4m49s
cortx-data-g1-4                  3/3     Running   0          4m48s
cortx-data-g1-5                  3/3     Running   0          4m47s
cortx-ha-6b9984c66c-z7qbc        3/3     Running   0          4m53s
cortx-kafka-0                    1/1     Running   0          4m52s
cortx-kafka-1                    1/1     Running   0          4m51s
cortx-kafka-2                    1/1     Running   0          4m51s
cortx-server-0                   2/2     Running   0          4m53s
cortx-server-1                   2/2     Running   0          4m53s
cortx-server-2                   2/2     Running   0          4m53s
cortx-server-3                   2/2     Running   0          4m53s
cortx-server-4                   2/2     Running   0          4m52s
cortx-server-5                   2/2     Running   0          4m51s
cortx-zookeeper-0                1/1     Running   0          4m53s
cortx-zookeeper-1                1/1     Running   0          4m53s
cortx-zookeeper-2                1/1     Running   0          4m53s

[root@ssc-vm-g4-rhev4-0710 k8_cortx_cloud]# kubectl exec -it cortx-client-0 -c cortx-hax -- /bin/bash
[root@cortx-client-0 /]# hctl status
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x0 'storage-set-1__sns'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'Profile_the_pool': 'storage-set-1__sns' 'storage-set-1__dix' None
Services:
    cortx-data-g0-0.cortx-data-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:cortx-data-g0-0.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0x1          inet:tcp:cortx-data-g0-0.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0x2          inet:tcp:cortx-data-g0-0.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-data-g0-1.cortx-data-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x3          inet:tcp:cortx-data-g0-1.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0x4          inet:tcp:cortx-data-g0-1.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0x5          inet:tcp:cortx-data-g0-1.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-server-4.cortx-server-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x2c         inet:tcp:cortx-server-4.cortx-server-headless.cortx.svc.cluster.local@22001
    [started]  rgw_s3              0x7200000000000001:0x2d         inet:tcp:cortx-server-4.cortx-server-headless.cortx.svc.cluster.local@22501
    cortx-server-5.cortx-server-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x2e         inet:tcp:cortx-server-5.cortx-server-headless.cortx.svc.cluster.local@22001
    [started]  rgw_s3              0x7200000000000001:0x2f         inet:tcp:cortx-server-5.cortx-server-headless.cortx.svc.cluster.local@22501
    cortx-client-0.cortx-client-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x30         inet:tcp:cortx-client-0.cortx-client-headless.cortx.svc.cluster.local@22001
    [unknown]  motr_client         0x7200000000000001:0x31         inet:tcp:cortx-client-0.cortx-client-headless.cortx.svc.cluster.local@21501
    cortx-client-1.cortx-client-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x32         inet:tcp:cortx-client-1.cortx-client-headless.cortx.svc.cluster.local@22001
    [unknown]  motr_client         0x7200000000000001:0x33         inet:tcp:cortx-client-1.cortx-client-headless.cortx.svc.cluster.local@21501
    cortx-client-2.cortx-client-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x34         inet:tcp:cortx-client-2.cortx-client-headless.cortx.svc.cluster.local@22001
    [unknown]  motr_client         0x7200000000000001:0x35         inet:tcp:cortx-client-2.cortx-client-headless.cortx.svc.cluster.local@21501
    cortx-client-3.cortx-client-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x36         inet:tcp:cortx-client-3.cortx-client-headless.cortx.svc.cluster.local@22001
    [unknown]  motr_client         0x7200000000000001:0x37         inet:tcp:cortx-client-3.cortx-client-headless.cortx.svc.cluster.local@21501
    cortx-client-4.cortx-client-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x38         inet:tcp:cortx-client-4.cortx-client-headless.cortx.svc.cluster.local@22001
    [unknown]  motr_client         0x7200000000000001:0x39         inet:tcp:cortx-client-4.cortx-client-headless.cortx.svc.cluster.local@21501
    cortx-client-5.cortx-client-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x3a         inet:tcp:cortx-client-5.cortx-client-headless.cortx.svc.cluster.local@22001
    [unknown]  motr_client         0x7200000000000001:0x3b         inet:tcp:cortx-client-5.cortx-client-headless.cortx.svc.cluster.local@21501
    cortx-data-g0-2.cortx-data-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x6          inet:tcp:cortx-data-g0-2.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0x7          inet:tcp:cortx-data-g0-2.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0x8          inet:tcp:cortx-data-g0-2.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-data-g0-3.cortx-data-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x9          inet:tcp:cortx-data-g0-3.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:cortx-data-g0-3.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0xb          inet:tcp:cortx-data-g0-3.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-data-g0-4.cortx-data-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0xc          inet:tcp:cortx-data-g0-4.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0xd          inet:tcp:cortx-data-g0-4.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0xe          inet:tcp:cortx-data-g0-4.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-data-g0-5.cortx-data-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0xf          inet:tcp:cortx-data-g0-5.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0x10         inet:tcp:cortx-data-g0-5.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0x11         inet:tcp:cortx-data-g0-5.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-data-g1-0.cortx-data-headless.cortx.svc.cluster.local  (RC)
    [started]  hax                 0x7200000000000001:0x12         inet:tcp:cortx-data-g1-0.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0x13         inet:tcp:cortx-data-g1-0.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0x14         inet:tcp:cortx-data-g1-0.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-data-g1-1.cortx-data-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x15         inet:tcp:cortx-data-g1-1.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0x16         inet:tcp:cortx-data-g1-1.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0x17         inet:tcp:cortx-data-g1-1.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-data-g1-2.cortx-data-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x18         inet:tcp:cortx-data-g1-2.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0x19         inet:tcp:cortx-data-g1-2.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0x1a         inet:tcp:cortx-data-g1-2.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-data-g1-3.cortx-data-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x1b         inet:tcp:cortx-data-g1-3.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0x1c         inet:tcp:cortx-data-g1-3.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0x1d         inet:tcp:cortx-data-g1-3.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-data-g1-4.cortx-data-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x1e         inet:tcp:cortx-data-g1-4.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0x1f         inet:tcp:cortx-data-g1-4.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0x20         inet:tcp:cortx-data-g1-4.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-data-g1-5.cortx-data-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x21         inet:tcp:cortx-data-g1-5.cortx-data-headless.cortx.svc.cluster.local@22001
    [started]  ioservice           0x7200000000000001:0x22         inet:tcp:cortx-data-g1-5.cortx-data-headless.cortx.svc.cluster.local@21001
    [started]  confd               0x7200000000000001:0x23         inet:tcp:cortx-data-g1-5.cortx-data-headless.cortx.svc.cluster.local@21002
    cortx-server-0.cortx-server-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x24         inet:tcp:cortx-server-0.cortx-server-headless.cortx.svc.cluster.local@22001
    [started]  rgw_s3              0x7200000000000001:0x25         inet:tcp:cortx-server-0.cortx-server-headless.cortx.svc.cluster.local@22501
    cortx-server-1.cortx-server-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x26         inet:tcp:cortx-server-1.cortx-server-headless.cortx.svc.cluster.local@22001
    [started]  rgw_s3              0x7200000000000001:0x27         inet:tcp:cortx-server-1.cortx-server-headless.cortx.svc.cluster.local@22501
    cortx-server-2.cortx-server-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x28         inet:tcp:cortx-server-2.cortx-server-headless.cortx.svc.cluster.local@22001
    [started]  rgw_s3              0x7200000000000001:0x29         inet:tcp:cortx-server-2.cortx-server-headless.cortx.svc.cluster.local@22501
    cortx-server-3.cortx-server-headless.cortx.svc.cluster.local
    [started]  hax                 0x7200000000000001:0x2a         inet:tcp:cortx-server-3.cortx-server-headless.cortx.svc.cluster.local@22001
    [started]  rgw_s3              0x7200000000000001:0x2b         inet:tcp:cortx-server-3.cortx-server-headless.cortx.svc.cluster.local@22501
[root@cortx-client-0 /]#

Within data pods, able to run hare mini provisioning stages with the consul source type

/opt/seagate/cortx/utils/bin/utils_setup post_install --config yaml:///etc/cortx/cluster.conf --services all && /opt/seagate/cortx/motr/bin/motr_setup post_install --config yaml:///etc/cortx/cluster.conf --services io && /opt/seagate/cortx/hare/bin/hare_setup post_install --config consul://cortx-consul-server:8500/conf --services all && /opt/seagate/cortx/utils/bin/utils_setup prepare --config yaml:///etc/cortx/cluster.conf --services all && /opt/seagate/cortx/motr/bin/motr_setup prepare --config yaml:///etc/cortx/cluster.conf --services io && /opt/seagate/cortx/hare/bin/hare_setup prepare --config consul://cortx-consul-server:8500/conf --services all && /opt/seagate/cortx/utils/bin/utils_setup config --config yaml:///etc/cortx/cluster.conf --services all && /opt/seagate/cortx/motr/bin/motr_setup config --config yaml:///etc/cortx/cluster.conf --services io && /opt/seagate/cortx/hare/bin/hare_setup config --config consul://cortx-consul-server:8500/conf --services all
/opt/seagate/cortx/utils/bin/utils_setup init --config yaml:///etc/cortx/cluster.conf --services all && /opt/seagate/cortx/motr/bin/motr_setup init --config yaml:///etc/cortx/cluster.conf --services io && /opt/seagate/cortx/hare/bin/hare_setup init --config consul://cortx-consul-server:8500/conf --services all

It's generates hare config files successfully

[root@cortx-data-g0-5 7346289b7c972ecff9112565e95c9076]# date
Thu Aug  4 06:25:14 UTC 2022
[root@cortx-data-g0-5 7346289b7c972ecff9112565e95c9076]# ls -l
total 49772
-rw-r--r-- 1 root root    22071 Aug  4 06:24 cluster.yaml
-rw-r--r-- 1 root root   130670 Aug  4 06:24 confd.dhall
-rw-r--r-- 1 root root    64367 Aug  4 06:24 confd.xc
drwxr-xr-x 5 root root     4096 Aug  4 05:49 consul
-rw-r--r-- 1 root root     4217 Aug  4 06:24 consul-agents.json
drwxr-xr-x 2 root root     4096 Aug  4 06:24 consul-client-conf
-rw-r--r-- 1 root root     1193 Aug  4 06:24 consul-env
-rw-r--r-- 1 root root   180370 Aug  4 06:24 consul-kv.json
drwxr-xr-x 2 root root     4096 Aug  4 05:50 consul-server-conf
-rwx------ 1 root root 16842752 Aug  4 06:24 m0trace.1319.2022-08-04-06:24:39
-rwx------ 1 root root 16842752 Aug  4 05:50 m0trace.569.2022-08-04-05:50:27
-rwx------ 1 root root 16842752 Aug  4 06:21 m0trace.82.2022-08-04-05:50:49
-rw-r--r-- 1 root root       59 Aug  4 06:24 node-name
drwxr-xr-x 5 root root     4096 Aug  4 05:50 sysconfig

cc @pavankrishnat, @vaibhavparatwar, @mssawant

mssawant

Please add a comment mentioning presently, both client and data pods are returned by the conf store search query. But data pods are filtered out while creating node in _create_node().

mssawant · 2022-08-04T17:00:01Z

Tried 3N bootstrap using hctl --mkfs bootstrap,

[root@ssc-vm-g4-rhev4-0554 ~]# hctl bootstrap --mkfs /root/multinode.yaml
2022-08-04 08:10:17: Generating cluster configuration... OK
2022-08-04 08:10:22: Starting Consul server on this node............ OK
2022-08-04 08:10:33: Importing configuration into the KV store... OK
2022-08-04 08:10:35: Starting Consul on other nodes...Consul ready on all nodes
2022-08-04 08:10:37: Updating Consul configuration from the KV store... OK
2022-08-04 08:10:41: Waiting for the RC Leader to get elected...... OK
2022-08-04 08:10:44: Starting Motr (phase1, mkfs)... OK
2022-08-04 08:10:52: Starting Motr (phase1, m0d)... OK
2022-08-04 08:10:55: Starting Motr (phase2, mkfs)... OK
2022-08-04 08:11:07: Starting Motr (phase2, m0d)... OK
2022-08-04 08:11:14: Checking health of services... OK
[root@ssc-vm-g4-rhev4-0554 ~]#
[root@ssc-vm-g4-rhev4-0554 ~]# hctl status -d
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x0 'sns-pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'default': 'sns-pool' None None
Services:
    ssc-vm-g4-rhev4-0554.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:10.230.240.235@22001
    [started]  confd               0x7200000000000001:0x1          inet:tcp:10.230.240.235@21002
    [started]  ioservice           0x7200000000000001:0x2          inet:tcp:10.230.240.235@21003
    [started]  ioservice           0x7200000000000001:0x3          inet:tcp:10.230.240.235@21004
    ssc-vm-rhev4-2612.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x4          inet:tcp:10.230.241.94@22001
    [started]  confd               0x7200000000000001:0x5          inet:tcp:10.230.241.94@21002
    [started]  ioservice           0x7200000000000001:0x6          inet:tcp:10.230.241.94@21003
    [started]  ioservice           0x7200000000000001:0x7          inet:tcp:10.230.241.94@21004
    ssc-vm-rhev4-2613.colo.seagate.com
    [started]  hax                 0x7200000000000001:0x8          inet:tcp:10.230.248.118@22001
    [started]  confd               0x7200000000000001:0x9          inet:tcp:10.230.248.118@21002
    [started]  ioservice           0x7200000000000001:0xa          inet:tcp:10.230.248.118@21003
Devices:
    ssc-vm-g4-rhev4-0554.colo.seagate.com
    [online]  /dev/sdb
    [online]  /dev/sdc
    [online]  /dev/null
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/null
    ssc-vm-rhev4-2612.colo.seagate.com
    [online]  /dev/sdb
    [online]  /dev/sdc
    [online]  /dev/null
    [online]  /dev/sdd
    [online]  /dev/sde
    [online]  /dev/null
    ssc-vm-rhev4-2613.colo.seagate.com
    [online]  /dev/sdb
    [online]  /dev/sdc
    [online]  /dev/null
[root@ssc-vm-g4-rhev4-0554 ~]#

pavankrishnat · 2022-08-05T04:14:19Z

@mssawant, Manually bootstrapped a Single Node setup with current PR branch which worked.

[root@ssc-vm-rhev4-2906 ~]# hctl status
Bytecount:
    critical : 0
    damaged : 0
    degraded : 0
    healthy : 0
Data pool:
    # fid name
    0x6f00000000000001:0x0 'the pool'
Profile:
    # fid name: pool(s)
    0x7000000000000001:0x0 'default': 'the pool' None None
Services:
    ssc-vm-rhev4-2906.colo.seagate.com  (RC)
    [started]  hax                 0x7200000000000001:0x0          inet:tcp:10.230.240.238@22001
    [started]  confd               0x7200000000000001:0x1          inet:tcp:10.230.240.238@21002
    [started]  ioservice           0x7200000000000001:0x2          inet:tcp:10.230.240.238@21003
    [unknown]  m0_client_other     0x7200000000000001:0x3          inet:tcp:10.230.240.238@22501
    [unknown]  m0_client_other     0x7200000000000001:0x4          inet:tcp:10.230.240.238@22502
[root@ssc-vm-rhev4-2906 ~]#
[root@ssc-vm-rhev4-2906 ~]# rpm -qa | grep cortx
cortx-motr-devel-2.0.0-1_git3f098b1_3.10.0_1160.el7.x86_64
cortx-hare-2.0.0-1_git18e0722.el7.x86_64
cortx-motr-2.0.0-1_git3f098b1_3.10.0_1160.el7.x86_64
cortx-hare-debuginfo-2.0.0-1_git18e0722.el7.x86_64
cortx-motr-debuginfo-2.0.0-1_git3f098b1_3.10.0_1160.el7.x86_64
cortx-py-utils-2.0.0-2_bccd313.noarch
[root@ssc-vm-rhev4-2906 ~]#

Problem: If num_client_inst is specified a non-zero value, deployment is failing. Issue is with search key. For data nodes we used 'io':'motr', it returns the list of machine IDs which includes both data node and motr client nodes, because of which deployment was failing. Solution: Separated search key for data node and motr client node. Signed-off-by: pavankrishnat <pavan.k.thunuguntla@seagate.com>

cla-bot bot added the cla-signed label Aug 2, 2022

auto-assign bot assigned d-nayak, mssawant, Shreya-18 and vaibhavparatwar Aug 2, 2022

pavankrishnat requested review from d-nayak, mssawant, Shreya-18 and vaibhavparatwar August 2, 2022 13:48

pavankrishnat force-pushed the CORTX33789 branch from a786fb1 to 27a08fa Compare August 3, 2022 12:34

pavankrishnat force-pushed the CORTX33789 branch from 38b5881 to 64950ba Compare August 3, 2022 18:46

pavankrishnat force-pushed the CORTX33789 branch from 64950ba to 18e0722 Compare August 4, 2022 04:01

vaibhavparatwar approved these changes Aug 4, 2022

View reviewed changes

mssawant reviewed Aug 4, 2022

View reviewed changes

provisioning/miniprov/hare_mp/store.py Show resolved Hide resolved

mssawant approved these changes Aug 4, 2022

View reviewed changes

pavankrishnat force-pushed the CORTX33789 branch from cc0a89b to f1d4643 Compare August 5, 2022 04:32

pavankrishnat force-pushed the CORTX33789 branch from a9b592a to addefdb Compare August 5, 2022 04:34

d-nayak approved these changes Aug 5, 2022

View reviewed changes

d-nayak merged commit 4458b54 into Seagate:main Aug 5, 2022

pavankrishnat deleted the CORTX33789 branch August 22, 2022 10:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CORTX-33789: Hare Config stage fail with Motr Clients #2146

CORTX-33789: Hare Config stage fail with Motr Clients #2146

pavankrishnat commented Aug 2, 2022 •

edited

Loading

vaibhavparatwar commented Aug 3, 2022

vaibhavparatwar commented Aug 3, 2022

mssawant commented Aug 3, 2022

vaibhavparatwar commented Aug 4, 2022

supriyachavan4398 commented Aug 4, 2022 •

edited

Loading

mssawant left a comment

mssawant commented Aug 4, 2022

pavankrishnat commented Aug 5, 2022

CORTX-33789: Hare Config stage fail with Motr Clients #2146

CORTX-33789: Hare Config stage fail with Motr Clients #2146

Conversation

pavankrishnat commented Aug 2, 2022 • edited Loading

vaibhavparatwar commented Aug 3, 2022

vaibhavparatwar commented Aug 3, 2022

mssawant commented Aug 3, 2022

vaibhavparatwar commented Aug 4, 2022

supriyachavan4398 commented Aug 4, 2022 • edited Loading

mssawant left a comment

Choose a reason for hiding this comment

mssawant commented Aug 4, 2022

pavankrishnat commented Aug 5, 2022

pavankrishnat commented Aug 2, 2022 •

edited

Loading

supriyachavan4398 commented Aug 4, 2022 •

edited

Loading