gluster-block create failure #187

salamani · 2019-03-29T06:25:41Z

I have followed the readme and built glusterblock v0.3 from source.
Following services are running: glusterd, rpcbind & tcmu-runner
$ gluster-block create block-test/sample-block ha 1 <my_host> 1GiB -d
failed to configure on z-ub18-4 configure failed
RESULT:FAIL

Any extra configuration is need for it?

pkalever · 2019-03-29T07:41:18Z

@salamani Hi!

Currently gluster-block doesn't work with hostnames, can you please give a try passing IP address ?

like:
$ gluster-block create block-test/sample-block 192.168.0.5 1GiB --json-pretty

salamani · 2019-03-29T08:26:05Z

$ gluster-block create block-test/sample-block 9.12.13.234 1GiB --json-pretty
{
"RESULT":"FAIL",
"errCode":255,
"errMsg":"failed to configure on 9.47.78.234 configure failed"
}

pkalever · 2019-03-29T08:42:52Z

@salamani thanks for trying this out.

Can you provide below details:

change GB_LOG_LEVEL to TRACE in /etc/sysconfig/gluster-blockd and restart gluster-blockd, now run
$ gluster-block create block-test/sample-block 9.12.13.234 1GiB --json-pretty

Above step will help collect the debug logs.

Now please attach below details:

logs from /var/log/gluster-block
describe versions of various packages
targetcli, python-rtslib, python-configshell and tcmu-runner
mention few details about your environment, OS version, Kernel version ...

itisravi · 2019-03-29T09:48:39Z

gluster-block-logs.tar.gz

I'm hitting the same issue as well. This is on fedora 26 with source install of latest master branches of glusterfs and gluster-block.

[root@f26 gluster-block]# targetcli --version
/usr/bin/targetcli version 2.1.fb46
[root@f26 gluster-block]# tcmu-runner --version
tcmu-runner 1.1.3
[root@f26 gluster-block]# python --version
Python 2.7.15
[root@f26 gluster-block]# uname -a
Linux f26 4.16.11-100.fc26.x86_64 #1 SMP Tue May 22 20:02:12 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

salamani · 2019-03-29T10:15:02Z

$gluster-block create block-test/sample-block 9.12.13.234 1GiB --json-pretty
{
"RESULT":"FAIL",
"errCode":255,
"errMsg":"failed to configure on 9.12.13.234 configure failed"
}

Please find below logs from folder: /var/log/gluster-block
gluster-blockd.log
gluster-block-gfapi.log
gluster-block-cli.log

$targetcli --version
/usr/bin/targetcli version 2.1.fb43
$tcmu-runner --version
tcmu-runner 1.4.1
$python --version
Python 2.7.15rc1
$uname -a
Linux 4.4.0-109-generic #132-Ubuntu SMP Tue Jan 9 19:52:39 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

pkalever · 2019-03-29T13:57:59Z

@itisravi can you please try using some other IP address on host, like 192.168.x.x or 10.70.x.x ?

[2019-03-29 09:43:06.182952] ERROR: glfs_open(sample-block) failed[No such file or directory] [at glfs-operations.c+701 :]

also I see related log:
[2019-03-29 09:43:05.920969] ERROR: backend creation failed for: block-test/sample-block [at block_svc_routines.c+4053 :]

So its like:
gluster-block creates that file called sample-block in /block-store directory of gluster volume block-test
when tcmu-runner tries to access it ,glfs_open is failing

HTH

itisravi · 2019-03-29T14:02:45Z

I had created the gluster volume with 127.0.0.2 IP which @pkalever found was the cause of the issue. After creating the block-test volume with 192.x.x.x IP, it worked for me:

root@f26 ~]# gluster  v info

Volume Name: block-test
Type: Replicate
Volume ID: 5452dcdd-aa07-48d1-9704-1d5c68792688
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 192.168.122.137:/bricks/brick1
Brick2: 192.168.122.137:/bricks/brick2
Brick3: 192.168.122.137:/bricks/brick3
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

[root@f26 ~]# gluster-block create block-test/sample-block  192.168.122.137 1GiB --json-pretty
{
  "IQN":"iqn.2016-12.org.gluster-block:9e1e9d6b-dc6b-4b0d-8bff-7a7239dbfc25",
  "PORTAL(S)":[
    "192.168.122.137:3260"
  ],
  "RESULT":"SUCCESS"
}

pkalever · 2019-03-29T16:52:35Z

@salamani From the logs you have shared above, I have noticed this:

[2019-03-29 10:02:48.657425] ERROR: glfs_open(sample-block) failed[No such file or directory] [at glfs-operations.c+379 :]

and

[2019-03-29 10:02:48.395668] ERROR: backend creation failed for: block-test/sample-block [at block_svc_routines.c+1963 :]

Which are very alike to what @itisravi 's setup has.

Can you retry using the primary IP of the node for creating block volume ?

HTH

salamani · 2019-03-31T10:21:01Z

I am still facing same issue
$ gluster v info
Volume Name: block-test
Type: Distribute
Volume ID: 8e573d92-b89f-4bc0-820a-9cf37b767f23
Status: Started
Snapshot Count: 0
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 9.12.13.234:/tmp/block
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
cluster.brick-multiplex: off

$ gluster-block create block-test/sample-block 9.12.13.234 1GiB --json-pretty
{
"RESULT":"FAIL",
"errCode":255,
"errMsg":"failed to configure on 9.12.13.234 configure failed"
}

gluster-blockd.log
gluster-block-gfapi.log
gluster-block-cli.log

pkalever · 2019-04-01T05:38:32Z

@salamani but you seem to be using the same IP address as before '9.12.13.234'

Can you check/paste
$ ip route
$ ip addr

Commands to see the primary/Other IP's associated with the node ?

Thanks!

salamani · 2019-04-01T06:27:40Z

I have tried on new machine 9.47.78.234, logs and issue is exactly same as with 9.12.13.234 before.
$gluster-block create block-test/sample-block 9.47.78.234 1GiB --json-pretty
{
"RESULT":"FAIL",
"errCode":255,
"errMsg":"failed to configure on 9.47.78.234 configure failed"
}

$ gluster v info
Volume Name: block-test
Type: Distribute
Volume ID: 07c917ac-4ba8-43de-bdab-b8d2ae0ddaf8
Status: Started
Snapshot Count: 0
Number of Bricks: 1
Transport-type: tcp
Bricks:
Brick1: 9.47.78.234:/tmp/block
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
cluster.brick-multiplex: off

$ip route
default via 9.47.79.254 dev enc1000 onlink
9.47.64.0/20 dev enc1000 proto kernel scope link src 9.47.78.234
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1

$ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: enc1000: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 02:be:ef:0e:18:7c brd ff:ff:ff:ff:ff:ff
inet 9.47.78.234/20 brd 9.47.79.255 scope global enc1000
valid_lft forever preferred_lft forever
inet6 fd55:faaf:e1ab:3ed:79d0:e86d:4388:94e6/64 scope global temporary dynamic
valid_lft 538661sec preferred_lft 19713sec
inet6 fd55:faaf:e1ab:3ed:d5b:98ab:3f42:c0f6/64 scope global temporary deprecated dynamic
valid_lft 452812sec preferred_lft 0sec
inet6 fd55:faaf:e1ab:3ed:7d9d:5334:9822:36f7/64 scope global temporary deprecated dynamic
valid_lft 366963sec preferred_lft 0sec
inet6 fd55:faaf:e1ab:3ed:7ccb:427e:317d:5979/64 scope global temporary deprecated dynamic
valid_lft 281113sec preferred_lft 0sec
inet6 fd55:faaf:e1ab:3ed:e9a6:13c7:94cb:488a/64 scope global temporary deprecated dynamic
valid_lft 195263sec preferred_lft 0sec
inet6 fd55:faaf:e1ab:3ed:987a:816a:44bf:9da8/64 scope global temporary deprecated dynamic
valid_lft 109414sec preferred_lft 0sec
inet6 fd55:faaf:e1ab:3ed:f8b1:cb7c:523:7cad/64 scope global temporary deprecated dynamic
valid_lft 23563sec preferred_lft 0sec
inet6 fd55:faaf:e1ab:3ed:be:efff:fe0e:187c/64 scope global dynamic mngtmpaddr
valid_lft 2591969sec preferred_lft 604769sec
inet6 fe80::be:efff:fe0e:187c/64 scope link
valid_lft forever preferred_lft forever
4: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:02:7f:92:42 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:2ff:fe7f:9242/64 scope link
valid_lft forever preferred_lft forever
444: vetheaacd39@if443: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 9e:45:4a:58:65:7f brd ff:ff:ff:ff:ff:ff link-netnsid 3
inet6 fe80::9c45:4aff:fe58:657f/64 scope link
valid_lft forever preferred_lft forever

pkalever · 2019-04-01T06:43:02Z

Can you try it with below command:

$ gluster-block create block-test/sample-block-1 172.17.0.1 1GiB --json-pretty

Thanks!

salamani · 2019-04-01T07:56:59Z

I have observed the same issue!

$gluster-block create block-test/sample-block-1 172.17.0.1 1GiB --json-pretty
{
"RESULT":"FAIL",
"errCode":255,
"errMsg":"failed to configure on 172.17.0.1 configure failed"
}

gluster-block-cli.log
gluster-block-configshell.log
gluster-blockd.log
gluster-block-gfapi.log

pkalever · 2019-04-01T10:16:50Z

@salamani Strange!

From the gfapi.log I see:

[2019-04-01 07:47:21.828066] W [dict.c:761:dict_ref] (-->/usr/local/lib/glusterfs/5.3/xlator/performance/quick-read.so(+0x7f46) [0x3ff96107f46] -->/usr/local/lib/glusterfs/5.3/xlator/performance/io-cache.so(+0xc14c) [0x3ff9618c14c] -->/usr/local/lib/libglusterfs.so.0(dict_ref+0x76) [0x3ffa599cffe] ) 0-dict: dict is NULL [Invalid argument]
[2019-04-01 07:47:21.834779] ERROR: glfs_open(sample-block-1) failed[No such file or directory] [at glfs-operations.c+379 :<blockGetMetaInfo>]

From path '/usr/local/lib/libglusterfs.so.0' It looks like you have glusterfs installations from source and .deb package (coz gluster-block look for this) ? Can you uninstall the source install binaries/libraries and just install from .dep package only ?

and the error:
ERROR: glfs_open(sample-block-1) failed[No such file or directory] [at glfs-operations.c+379 :]

means, gluster-block cannot access the backend block hosting volume.

You could simply download this file and have a test to see if the issue is from gfapi side.

Also do you see anything wrong in tcmu-runner.log/tcmu-runner-gfapi.log ? can you copy them here ?

Otherwise, Frankly speaking we never tested gluster-block on ubuntu (but, was tested by users on fedora/centos and rhel), I can try reproduce this issue when I have some time this week on ubuntu, can you share the details about the setup (OS image version) and how you installed stuff and setup gluster block hosting volume etc ...

Thanks!

salamani · 2019-04-02T04:20:18Z

I built glusterfs from source and created the service to run.
$./tests/gfapi-test block-test 9.47.78.234
Test works as expected!
tcmu-runner.log

pkalever · 2019-04-02T11:05:03Z

@salamani sounds good.

Could you share your environment details requested above. We can try reproducing this and get back to you.

Thanks!

salamani · 2019-04-04T13:26:55Z

I have installed glusterfs 5.3 from source, tcmu-runner from source.
dependencies installed : targetcli-fb python python-dev make automake autoconf libtool flex bison pkg-config libssl-dev libxml2-dev python-dev libaio-dev libibverbs-dev librdmacm-dev libreadline-dev liblvm2-dev libglib2.0-dev liburcu-dev libcmocka-dev libsqlite3-dev libacl1-dev pkg-config libuuid1 libjson-c-dev

$glusterd --version
glusterfs 5.3
$ targetcli --version
/usr/bin/targetcli version 2.1.fb43
$tcmu-runner --version
tcmu-runner 1.4.0
$ python --version
Python 2.7.15rc1
$ uname -a
Linux ub28 4.15.0-20-generic #21-Ubuntu SMP Tue Apr 24 06:16:15 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

cmd_history.log
gluster-block-cli.log
gluster-blockd.log
gluster-block-gfapi.log
tcmu-runner.log

salamani · 2019-04-08T04:16:15Z

@pkalever any updates on this?

salamani · 2019-04-18T06:25:59Z

any updates here

lxbsz · 2019-04-18T07:06:58Z

@salamani
It seems it is the same with #209 from your logs.

Could you paste the following metadata ?

$ mount..... /mnt // to check whether the block meta file is exist ? If exist then
$ cat /mnt/block-meta/block-volume

lxbsz · 2019-04-18T07:28:28Z

@salamani
It seems it is the same with #209 from your logs.

Could you paste the following metadata ?

$ mount..... /mnt // to check whether the block meta file is exist ? If exist then
$ cat /mnt/block-meta/block-volume

Sorry, please ignore this.

pkalever · 2019-04-18T11:25:45Z

@salamani Although we never supported gluster-block on ubuntu, may be its time, we should update the README guide to help others about how to install gluster-block and get it running on Ubuntu. I had seen the pain today :-(

Will update this issue later with the steps I followed to install gluster-block on ubuntu 18.04.02, which might at-least help easily point as a reference.

I have started from here:

root@block:~# uname -a
Linux block 4.15.0-47-generic #50-Ubuntu SMP Wed Mar 13 10:44:52 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
root@block:~# cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
root@block:~#

root@block:~/gluster-block# gluster-block create sample/block 192.168.124.208 1MiB
failed to configure on 192.168.124.208 configure failed
RESULT:FAIL

Checking logs:

[2019-04-18 10:55:54.675240] INFO: create cli request, volume=sample blockname=block mpath=1 blockhosts=192.168.124.208 authmode=0 size=1048576, rbsize=0 [at block_svc_routines.c+3873 :<block_create_cli_1_svc_st>]
[2019-04-18 10:55:54.767901] INFO: create request, volume=sample volserver=192.168.124.208 blockname=block blockhosts=192.168.124.208 filename=21bdc99e-f9f6-4881-a47a-47afc9181b6c authmode=0 passwd= size=1048576 [at block_svc_routines.c+4294 :<block_create_common>]
[2019-04-18 10:55:54.950296] ERROR: backend creation failed for: sample/block [at block_svc_routines.c+4056 :<blockValidateCommandOutput>]
[2019-04-18 10:55:54.950335] INFO: command exit code, -1 [at block_svc_routines.c+4473 :<block_create_common>]
[2019-04-18 10:55:54.970340] ERROR: failed in remote create for block block on host 192.168.124.208 volume sample [at block_svc_routines.c+972 :<glusterBlockCreateRemote>]
[2019-04-18 10:55:54.970462] WARNING: glusterBlockCreateRemoteAsync: return -1 failed in remote async create for block block on volume sample with hosts 192.168.124.208 [at block_svc_routines.c+4002 :<block_create_cli_1_svc_st>]
[2019-04-18 10:55:54.971672] WARNING: No Spare nodes to create (block): rollingback creation of target on volume sample with given hosts 192.168.124.208 [at block_svc_routines.c+3066 :<glusterBlockAuditRequest>]
[2019-04-18 10:55:54.990168] INFO: delete request, blockname=block filename=21bdc99e-f9f6-4881-a47a-47afc9181b6c [at block_svc_routines.c+4729 :<block_delete_1_svc_st>

I have enabled DEBUG logs and noticed that the issue is from targetcli/rtslib being too old versions than what gluster-block expects,

Please do this:

# apt-get remove targetcli-fb python-rtslib-fb python3-configshell-fb python3-rtslib-fb
# apt-get install python-setuptools python-gobject
# mkdir /etc/target
# git clone https://github.com/open-iscsi/targetcli-fb
# cd targetcli-fb
# ./setup.py install
# cd - 
# git clone https://github.com/open-iscsi/rtslib-fb
# cd rtslib-fb/
# ./setup.py install
# cd -
# git clone https://github.com/open-iscsi/configshell-fb
# cd configshell-fb/
# ./setup.py install
# cd -

root@block:~# systemctl restart gluster-blockd

root@block:~# gluster-block create sample/block 192.168.124.208 1GiB --json-pretty
{
  "IQN":"iqn.2016-12.org.gluster-block:01dc0c3c-4597-42b1-9037-444563d477fa",
  "PORTAL(S)":[
    "192.168.124.208:3260"
  ],
  "RESULT":"SUCCESS"
}
root@block:~# gluster-block delete sample/block --json-pretty
{
  "SUCCESSFUL ON":[
    "192.168.124.208"
  ],
  "RESULT":"SUCCESS"
}
root@block:~#

Note:
With the recent changes to master gluster-blockd won't start with old targetcli version, indicating the msg in the logs, like:

[2019-04-18 10:39:06.254292] INFO: Distro ID=ubuntu. Current kernel version: '4.15.0-47-generic'. [at gluster-blockd.c+410 :<gbMinKernelVersionCheck>]
[2019-04-18 10:39:06.257415] INFO: starting with tcmu-runner version - 1.4.0 [at gluster-blockd.c+500 :<gbDependenciesVersionCheck>]
[2019-04-18 10:39:06.373240] ERROR: current targetcli version is 2.1.fb43, gluster-block need atleast - 2.1.fb46 [at gluster-blockd.c+507 :<gbDependenciesVersionCheck>]
[2019-04-18 10:39:06.373266] ERROR: Please install the recommended dependency version and try again. [at gluster-blockd.c+517 :<gbDependenciesVersionCheck>]

Also, you should compile, with an additional flag with recent changes to master to disable TIRPC on ubuntu machines:

# ./configure --with-systemddir=/usr/lib/systemd/system --enable-tirpc=no

HTH.

pkalever · 2019-04-18T12:46:56Z

Created a new issue for reference guide: #215

pkalever · 2019-04-18T12:49:19Z

@salamani feel free to review and share your experience/thoughts on #215

pkalever · 2020-05-15T08:15:18Z

I hope there is nothing remaining to be taken care, closing this now.

pkalever mentioned this issue Apr 2, 2019

tests: add simple gfapi access test #193

Merged

pkalever closed this as completed May 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gluster-block create failure #187

gluster-block create failure #187

salamani commented Mar 29, 2019 •

edited

Loading

pkalever commented Mar 29, 2019 •

edited

Loading

salamani commented Mar 29, 2019 •

edited

Loading

pkalever commented Mar 29, 2019 •

edited

Loading

itisravi commented Mar 29, 2019 •

edited

Loading

salamani commented Mar 29, 2019 •

edited

Loading

pkalever commented Mar 29, 2019

itisravi commented Mar 29, 2019

pkalever commented Mar 29, 2019

salamani commented Mar 31, 2019

pkalever commented Apr 1, 2019

salamani commented Apr 1, 2019

pkalever commented Apr 1, 2019

salamani commented Apr 1, 2019 •

edited

Loading

pkalever commented Apr 1, 2019 •

edited

Loading

salamani commented Apr 2, 2019 •

edited

Loading

pkalever commented Apr 2, 2019

salamani commented Apr 4, 2019 •

edited

Loading

salamani commented Apr 8, 2019

salamani commented Apr 18, 2019

lxbsz commented Apr 18, 2019 •

edited

Loading

lxbsz commented Apr 18, 2019

pkalever commented Apr 18, 2019

pkalever commented Apr 18, 2019

pkalever commented Apr 18, 2019

pkalever commented May 15, 2020

gluster-block create failure #187

gluster-block create failure #187

Comments

salamani commented Mar 29, 2019 • edited Loading

pkalever commented Mar 29, 2019 • edited Loading

salamani commented Mar 29, 2019 • edited Loading

pkalever commented Mar 29, 2019 • edited Loading

itisravi commented Mar 29, 2019 • edited Loading

salamani commented Mar 29, 2019 • edited Loading

pkalever commented Mar 29, 2019

itisravi commented Mar 29, 2019

pkalever commented Mar 29, 2019

salamani commented Mar 31, 2019

pkalever commented Apr 1, 2019

salamani commented Apr 1, 2019

pkalever commented Apr 1, 2019

salamani commented Apr 1, 2019 • edited Loading

pkalever commented Apr 1, 2019 • edited Loading

salamani commented Apr 2, 2019 • edited Loading

pkalever commented Apr 2, 2019

salamani commented Apr 4, 2019 • edited Loading

salamani commented Apr 8, 2019

salamani commented Apr 18, 2019

lxbsz commented Apr 18, 2019 • edited Loading

lxbsz commented Apr 18, 2019

pkalever commented Apr 18, 2019

pkalever commented Apr 18, 2019

pkalever commented Apr 18, 2019

pkalever commented May 15, 2020

salamani commented Mar 29, 2019 •

edited

Loading

pkalever commented Mar 29, 2019 •

edited

Loading

salamani commented Mar 29, 2019 •

edited

Loading

pkalever commented Mar 29, 2019 •

edited

Loading

itisravi commented Mar 29, 2019 •

edited

Loading

salamani commented Mar 29, 2019 •

edited

Loading

salamani commented Apr 1, 2019 •

edited

Loading

pkalever commented Apr 1, 2019 •

edited

Loading

salamani commented Apr 2, 2019 •

edited

Loading

salamani commented Apr 4, 2019 •

edited

Loading

lxbsz commented Apr 18, 2019 •

edited

Loading