Cache causes corrupt data when filesystem is full

[lunaneff]

I'm writing a Nix package similar to ttf-ms-win11-auto on the AUR, but because Nix sandboxes package builds and so you can't mount filesystems, I'm actually building the package in a small VM (using nixpkgs' runInLinuxVM). By default, this VM has 512MB of RAM, and /tmp (where I'm putting the cache) is a tmpfs of size 256MB. This isn't enough, and if you attempt to build it using these parameters, the build fails while extracting the fonts from install.wim because httpdirfs returns corrupt data:

build logs

building '/nix/store/xhcjrczgl04fd2a1zlqsz3z60klijpfh-windows-fonts-win11-22H2.drv'...
cSeaBIOS (version rel-1.16.2-0-gea1b7a073390-prebuilt.qemu.org)
iPXE (http://ipxe.org) 00:06.0 CA00 PCI2.10 PnP PMM+1EFD0E50+1EF30E50 CA00
Booting from ROM...

Probing EDD (edd=off to disable)... ocloading kernel modules...

[    0.252681] Invalid ELF header magic: != ELF

[    0.255857] Invalid ELF header magic: != ELF

[    0.260607] Invalid ELF header magic: != ELF

[    0.263970] Invalid ELF header magic: != ELF

[    0.266798] Invalid ELF header magic: != ELF

[    0.355418] Invalid ELF header magic: != ELF

[    0.360386] Invalid ELF header magic: != ELF

[    0.364367] Invalid ELF header magic: != ELF

[    0.367771] Invalid ELF header magic: != ELF

[    0.371557] Invalid ELF header magic: != ELF

[    0.377234] Invalid ELF header magic: != ELF

[    0.386895] Invalid ELF header magic: != ELF

[    0.390398] Invalid ELF header magic: != ELF

[    0.394368] Invalid ELF header magic: != ELF

[    0.402659] Invalid ELF header magic: != ELF

[    0.409957] Invalid ELF header magic: != ELF

[    0.482844] Invalid ELF header magic: != ELF

[    0.490862] Invalid ELF header magic: != ELF

[    0.497495] Invalid ELF header magic: != ELF

[    0.505586] Invalid ELF header magic: != ELF

[    0.512287] Invalid ELF header magic: != ELF

[    0.523335] Invalid ELF header magic: != ELF

mounting Nix store...

mounting host's temporary directory...

starting stage 2 (/nix/store/2yias079qax7ca569naz07m4h495pp93-vm-run-stage2)

setting up network...

mounting httpdirfs...

print_version: HTTPDirFS version 1.2.3

print_version: libcurl SSL engine: OpenSSL/3.0.9

LinkTable_print: --------------------------------------------

LinkTable_print:  LinkTable 0x1f47240 for https://software-static.download.prss.microsoft.com/dbazure/988969d5-f34g-4e03-ac9d-1f9786c66751/22621.525.220925-0207.ni_release_svc_refresh_CLIENTENTERPRISEEVAL_OEMRET_x64FRE_en-us.iso

LinkTable_print: --------------------------------------------

LinkTable_print: 0 H 0  https://software-static.download.prss.microsoft.com/dbazure/988969d5-f34g-4e03-ac9d-1f9786c66751/22621.525.220925-0207.ni_release_svc_refresh_CLIENTENTERPRISEEVAL_OEMRET_x64FRE_en-us.iso

LinkTable_print: 1 F 5088602112 22621.525.220925-0207.ni_release_svc_refresh_CLIENTENTERPRISEEVAL_OEMRET_x64FRE_en-us.iso https://software-static.download.prss.microsoft.com/dbazure/988969d5-f34g-4e03-ac9d-1f9786c66751/22621.525.220925-0207.ni_release_svc_refresh_CLIENTENTERPRISEEVAL_OEMRET_x64FRE_en-us.iso

LinkTable_print: --------------------------------------------

LinkTable_print:  Invalid link count: 0

LinkTable_print: --------------------------------------------

none / tmpfs rw,relatime 0 0

devtmpfs /dev devtmpfs rw,relatime,size=238068k,nr_inodes=59517,mode=755 0 0

none /dev/shm tmpfs rw,relatime 0 0

none /dev/pts devpts rw,relatime,mode=600,ptmxmode=000 0 0

store /nix/store 9p rw,dirsync,relatime,loose,access=client,trans=virtio 0 0

none /tmp tmpfs rw,relatime 0 0

none /run tmpfs rw,relatime,mode=755 0 0

xchg /tmp/xchg 9p rw,sync,dirsync,relatime,access=client,trans=virtio 0 0

none /proc proc rw,relatime 0 0

none /sys sysfs rw,relatime 0 0

httpdirfs /tmp/tmp.cQ6ajg5Y7U fuse.httpdirfs rw,nosuid,nodev,relatime,user_id=0,group_id=0 0 0

mounting iso...

mount: /tmp/tmp.JWM9Bvd9Cf: WARNING: source write-protected, mounted read-only.

extracting fonts...
7-Zip [64] 17.05 : Copyright (c) 1999-2021 Igor Pavlov : 2017-08-28

p7zip Version 17.05 (locale=C,Utf16=off,HugeFiles=on,64 bits,1 CPU x64)
Scanning the drive for archives:

1 file, 4374753355 bytes (4173 MiB)
Extracting archive: /tmp/tmp.JWM9Bvd9Cf/sources/install.wim
WARNINGS:

Some files have incorrect reference count
--

Path = /tmp/tmp.JWM9Bvd9Cf/sources/install.wim

Type = wim

WARNING = Some files have incorrect reference count

Physical Size = 4374753355

Size = 8946659063

Packed Size = 4365912645

Method = LZX:15

Cluster Size = 32768

Created = 2022-09-25 02:47:56

Modified = 2022-09-25 02:58:14

Comment = 4374750599227581015071645059301772888108530x01D8D0890x378BD5F10x01D8D08A0xA7E4EBC409Microsoft Windows Operating SystemEnterpriseEvalClient010.0.22621.1;2016-01-01T00:00:00ZIMAGE_STATE_GENERALIZE_RESEAL_TO_OOBEWinNTTerminal Serveren-USen-US100226215250ni_releaseWINDOWSWindows 11 Enterprise EvaluationWindows 11 Enterprise EvaluationEnterpriseEvalWindows 11 Enterprise EvaluationWindows 11 Enterprise Evaluation

Version = 1.13

Multivolume = -

Volume = 1

Volumes = 1

Images = 1
ERROR: Data Error : Windows/Fonts/msyh.ttc

ERROR: Data Error : Windows/Fonts/YuGothL.ttc

ERROR: Data Error : Windows/Fonts/YuGothB.ttc

ERROR: Data Error : Windows/Fonts/YuGothM.ttc

ERROR: Data Error : Windows/Fonts/YuGothR.ttc

ERROR: Data Error : Windows/Fonts/simsun.ttc

ERROR: Data Error : Windows/Fonts/simsunb.ttf

ERROR: Data Error : Windows/Fonts/mingliub.ttc
Sub items Errors: 8
Archives with Errors: 1
Warnings: 1
Sub items Errors: 8

[   32.026711] reboot: Power down

After increasing RAM remounting /tmp with a larger size, the build succeeds. Without cache it also succeeds, but the build takes ~2 minutes instead of ~30 seconds.

For my specific usecase I can just increase the size of the filesystem, but I'm sure it'd save someone else a headache if you could make httpdirfs print a warning and/or delete old cached data when running out of space.

[fangfufu]

What is your underlying filesystem for the cache?

[lunaneff]

It was a tmpfs as I said in the original report

[fangfufu]

Does tmpfs allow sparse allocation?

[lunaneff]

I don't know, how could I check?

[LuShuailing]

Hey, I have the same problem using httpdirfs as a data transfer channel like mount nfs, httpdirfs seems to keep taking up space during the transfer. When I use the -o direct_io option, the transfer completes but showing corrupted data on my computer. How can I only fetch data from httpdirfs without letting httpdirfs save it? Thanks.

[fangfufu]

@LuShuailing , turn off the cache with --cache?