overlay2 is not posix compliant so rpm is failing

[robnagler]

overlay2 is the wrong choice for Docker container storage even though Docker clearly states:

overlay2 is the preferred storage driver, for all currently supported Linux distributions, and requires no extra configuration.

However, we are seeing errors like this:

rpmdbNextIterator: skipping h#     680 blob size(6044): BAD, 8 + 16 * il(74) + dl(4852)
failed to replace old database with new database!

Even with a clean rebuild, it fails, because overlay2 does not support rename for directories, and Docker notes this as a limitation:

Limitations on OverlayFS compatibility:
rename(2): OverlayFS does not fully support the rename(2) system call. Your application needs to detect its failure and fall back to a “copy and unlink” strategy.

Of course, the one thing that's important in Unix (and I'm guessing POSIX) is directory rename be atomic.

Sirepo build is stuck at the moment so I'm working on this.

[robnagler]

This workaround works so the container doesn't corrupt rpm db, and therefore doesn't need to be rebuilt, which avoids directory rename issue.

There are many tickets associated with this problem. I'd rather avoid reformatting our disks to support device-mapper in non-loopback mode so this fix avoids the immediate issue. The fundamental issue is that overlay2 is not POSIX-compliant, but the issues and documentation are conflicting (see below). We definitely create our XFS file systems with ftype=1.

The correct solution is to install yum/dnf-ovl-plugin which does the touch /var/lib/rpm/* but checks to see if / is overlayfs. See radiasoft/download#65

• For new installations, I would also recommend giving the overlay2 storage driver a try; a lot of fixes have gone into current versions of the RHEL / CentOS kernels to improve overlayfs support (and it's therefore now used by default by docker).
• This is now a kernel version dependent issue. I think it would be wise to document and close this issue. There is nothing we can do about it for kernels which do not have the patch to support this.
  Overall this is not an issue within rpm but a - well known - limitation of the overlayfs. Closing.
  

[robnagler]

See my comment on RH#1646543.