Docker, overlay2, XFS

[qrpike]

Installation details
Scylla version (or git commit hash): 2.0.1 Docker Image
Cluster size: 5
OS (RHEL/CentOS/Ubuntu/AWS AMI): CoreOS Stable + Kubernetes

Platform (physical/VM/cloud instance type/docker):
Bare Metal, CoreOS pxe boot, kubernetes with docker 1.12.9.

Hardware: sockets=2 cores=16 hyperthreading=32 memory=192gb
Disks: (SSD/HDD, count): hardware RAID0 2x 480GB SSDs.

I have several boxes with no raid, and SSD1=ext4 for system, and SSD2=xfs for scylla directory mounted into the docker container.

However I tried to convert one to RAID0 xfs the entire disk, with overlay2 instead of overlay storage driver and I get the following error when trying to start without dev=1

2017-11-29T07:55:29.25786859Z running: (['/usr/lib/scylla/scylla_dev_mode_setup', '--developer-mode', '0'],)
2017-11-29T07:55:29.263598678Z running: (['/usr/lib/scylla/scylla_io_setup'],)
2017-11-29T07:55:29.614283438Z Unexpected end of /proc/mounts line `overlay / overlay rw,context="system_u:object_r:svirt_lxc_file_t:s0:c200,c725",relatime,lowerdir=l/XOJCS3DNECMTAKOUW5WDRPZDBG:l/46VBVJ3ZPSLZW7UUMXBSVDZFYZ:l/23IYUSRHOZEIPRTPTVUMFVUH45:l/O3RDJWBS73EOFEABS4N6BQOUUJ:l/G6FPCSVDONNYOZ54OUNEECGIDY:l/7DKBRO6NFYJRU77D3VRBCEWLAK:l/WJAHYI4EL4ADOMZSJJNLCPED6P:l/4C244OA4LQFPLTNTKXHDSFY4GQ:l/MYBYWCANH6MXGAWP7FXWIFGSFI:l/GLEICF6PK4UWUC6LXXRIO6GJNS:l/DR6PYYW66EHHQWDO6OO6TVCNED:l/DQ5VXNXEWL5Z6FG2LGUBFCAUYB:l/LSRORZLHJQSUJ2N3WIP3ICS3TI:l/K5BZRRK7RSOVTWZZQXO7HTCNXG:l/B3M222'
2017-11-29T07:55:29.671799048Z terminate called after throwing an instance of 'std::system_error'
2017-11-29T07:55:29.671841733Z   what():  open: No such file or directory
2017-11-29T07:55:29.902811474Z ERROR:root:/var/lib/scylla/data did not pass validation tests, it may not be on XFS and/or has limited disk space.
2017-11-29T07:55:29.902884122Z This is a non-supported setup, and performance is expected to be very bad.
2017-11-29T07:55:29.902899492Z For better performance, placing your data on XFS-formatted directories is required.
2017-11-29T07:55:29.902913871Z To override this error, enable developer mode as follow:
2017-11-29T07:55:29.902926494Z sudo /usr/lib/scylla/scylla_dev_mode_setup --developer-mode 1
2017-11-29T07:55:29.921158473Z failed!
2017-11-29T07:55:29.921208116Z Traceback (most recent call last):
2017-11-29T07:55:29.921224546Z   File "/docker-entrypoint.py", line 15, in <module>
2017-11-29T07:55:29.921238993Z     setup.io()
2017-11-29T07:55:29.921252371Z   File "/scyllasetup.py", line 33, in io
2017-11-29T07:55:29.921265051Z     self._run(['/usr/lib/scylla/scylla_io_setup'])
2017-11-29T07:55:29.921277797Z   File "/scyllasetup.py", line 22, in _run
2017-11-29T07:55:29.921290776Z     subprocess.check_call(*args, **kwargs)
2017-11-29T07:55:29.921303125Z   File "/usr/lib64/python3.4/subprocess.py", line 558, in check_call
2017-11-29T07:55:29.921315914Z     raise CalledProcessError(retcode, cmd)
2017-11-29T07:55:29.921328781Z subprocess.CalledProcessError: Command '['/usr/lib/scylla/scylla_io_setup']' returned non-zero exit status 1

I'm guessing its a difference between overlay and overlay2 maybe? Not sure - but really don't want to run the nodes in dev mode.

Thanks,

[qrpike]

I solved this for the meantime with replacing the scylla_io_setup file and changed these lines to hardcode the evaluation directory:

        else:
            if os.environ.has_key("SCYLLA_CONF"):
                conf_dir = os.environ["SCYLLA_CONF"]
            else:
                conf_dir = "/etc/scylla"
            cfg = yaml.load(open(os.path.join(conf_dir, "scylla.yaml")))
            data_dir = "/var/lib/scylla"
            iotune_args = []
            if cpudata.cpuset():
                iotune_args += [ "--cpuset", ",".join(map(str, cpudata.cpuset())) ]
            elif cpudata.smp():
                iotune_args += [ "--smp", cpudata.smp() ]

[slivne]

@penberg - can you provide your input on this

[avikivity]

I think it's a bug in hwloc. It tried to parse /proc/mounts and fails.

[eshizhan]

same problem with 2.0.0 docker image.

[shivakar]

Yes. The bug is in hwloc. open-mpi/hwloc#142
It has been fixed in hwloc 1.11.3 about 2 years ago. Unfortunately, CentOS 7 still has hwloc 1.11.2.
So this won't be fixed until hwloc is updated in CentOS 7.

[avikivity]

Well, we can package a newer hwloc, or someone with a Red Hat subscription can ask for a fix.

[avikivity]

Actually, looks like RHEL 7.5 packages 1.11.8, so the problem is solved. We just have to wait until CentOS 7.5 is released.

[avikivity]

CentOS 7.5 was released some years ago, closing.