Writing a huge number of files in a directory (ext4 fs with 1024-byte inodes)

[KaneRoot]

Bug Report

Hello. I made an application that intensively uses the filesystem and I noticed a crash when writing a lot of files in a directory of an ext4 filesystem. I tried several configurations, including playing with other filesystems (BTRFS and TMPFS) and it seems to crash only on ext4 with 1024-byte inodes (seems to work fine with 4096-byte blocks, but I actually need to lower that for my application).

Here is an application I made to reproduce the bug:

# Create files in #{dbdir}/data/ and store the last entry number in #{dbdir}/last-entry.
dbdir = "db"
Dir.mkdir_p "#{dbdir}/data"

# Number of the last entry.
last_entry_filename = "#{dbdir}/last-entry"

# Where to start? Where to end?
current_entry_number = ((File.read last_entry_filename).to_i rescue 0) + 1
limit = ARGV[0].to_i rescue 1_000_000 # Bug happens around 300k files created.

while current_entry_number <= limit
	filename = "#{dbdir}/data/file-#{current_entry_number}"

	File.write filename, "hello #{current_entry_number}"

	# Write the last entry that has successfully been added.
	File.write last_entry_filename, "#{current_entry_number}"

	STDOUT.write "\rfile #{current_entry_number}/#{limit}".to_slice

	current_entry_number += 1
end

puts
puts "done!"

The bug:

/tmp/ext4 $ burning-some-inodes 5000000
file 308956/5000000Unhandled exception: Error opening file with mode 'w': 'db/data/file-308957': No space left on device (File::Error)
  from /home/user/tmp/crystal/src/crystal/system/unix/file.cr:12:7 in '??'
  from /home/user/tmp/crystal/src/file.cr:741:12 in 'write'
  from /home/user/tmp/crystal/src/string.cr:5667:5 in '__crystal_main'
  from /home/user/tmp/crystal/src/crystal/main.cr:129:5 in 'main'
  from /home/user/tmp/crystal/src/crystal/system/unix/main.cr:10:3 in 'main'
  from /lib/x86_64-linux-gnu/libc.so.6 in '??'
  from /lib/x86_64-linux-gnu/libc.so.6 in '__libc_start_main'
  from /home/user/burning-some-inodes in '_start'
  from ???

I still have plenty of space and free inodes.

$ df -h .
Filesystem      Size  Used Avail Use% Mounted on
/dev/loop1000   2,5G  312M  2,0G  14% /tmp/ext4
$ df -i .
Filesystem       Inodes  IUsed   IFree IUse% Mounted on
/dev/loop1000  10002432 308971 9693461    4% /tmp/ext4

Thus, I can still actually write new files when I do that in my terminal!
Furthermore, if I move the db directory (created by the provided code) but keep it in the same partition and restart my application, new files are created up to the same number of files in the directory (in this example: 308957).
However, if I change something rather trivial such as what is being printed in the terminal or the filenames, I get different results.

I'm on Ubuntu 24.10 with the crystal version provided by snap:

Crystal 1.15.0 [7b9e2ef80] (2025-01-09)
LLVM: 18.1.6
Default target: x86_64-unknown-linux-gnu

But I also tried it with master (same bug at the same file number):

Crystal 1.16.0-dev [204b6a9] (2025-02-01)
LLVM: 19.1.1
Default target: x86_64-pc-linux-gnu

In case you want to reproduce the bug without creating a partition on a real disk, here is a makefile to make a fake ext4 partition from a generated empty file in /tmp that is then mounted on /tmp/ext4:

EXT4_DIR ?= /tmp/ext4
EXT4_IMG ?= /tmp/ext4.img
EXT4_SIZ ?= 5000
EXT4_SECTOR_SIZE ?= 1024
EXT4_INODE_NUMBER ?= 10000000
#EXT4_SECTOR_SIZE ?= 4096
#EXT4_INODE_NUMBER ?= 1500000
EXT4_INODE_SIZE ?= 256
EXT4_LOOP ?= /dev/loop1000

ext4: ext4-mkdir ext4-create ext4-loop ext4-format ext4-mount
ext4-mkdir:; [ ! -d $(EXT4_DIR) ] && mkdir $(EXT4_DIR) 2>/dev/null || :
ext4-create:
	[ ! -f $(EXT4_IMG) ] && dd if=/dev/zero of=$(EXT4_IMG) bs=1M count=$(EXT4_SIZ) || :
ext4-loop:; losetup --sector-size $(EXT4_SECTOR_SIZE) $(EXT4_LOOP) $(EXT4_IMG)
ext4-format:; mkfs.ext4 -b $(EXT4_SECTOR_SIZE) -I $(EXT4_INODE_SIZE) -N $(EXT4_INODE_NUMBER) -L EXT4DATA $(EXT4_LOOP)
ext4-mount:; mount $(EXT4_LOOP) $(EXT4_DIR)

unext4: ext4-umount ext4-unloop ext4-rm
ext4-unloop:; [ -b $(EXT4_LOOP) ] && losetup -d $(EXT4_LOOP) || :
ext4-umount:; umount $(EXT4_DIR) || :
ext4-rm:; rm $(EXT4_IMG) || :

Thanks and have a fun time with this one! 😆

[ysbaddaden]

You're reaching an errno, so that's a system error. Isn't it reproducible in another language?

The inodes declare the total number of files for the filesystem. Maybe the parameters to mkfs.ext4 are limiting the number of files per directory?

[KaneRoot]

Thanks for your quick answer.

You're reaching an errno, so that's a system error. Isn't it reproducible in another language?

I can still write files so this shouldn't be related to the filesystem. I could write the same application in C if you want but since I can write files from my terminal I don't see the point.

The inodes declare the total number of files for the filesystem. Maybe the parameters to mkfs.ext4 are limiting the number of files per directory?

I can still write files in the same directory. Moreover, as I said in the bug report, when I change what is printed on the terminal the number of files I can write in the directory changes, thus I guess the bug should be memory-related.

Also, you can see the parameters I actually used to create the partition. From what I understand from wikipedia (https://en.wikipedia.org/wiki/Ext4) ext4 now enables billions of files in a single directory so we are far from it. And again, even with ext4 I managed to write millions of files in a single directory (with 4-kiB blocks).