Rewrite batch mode to use temp directory (#136)

[gammazero]

Continued from #136
Credit to @requilence for most of this PR

Problem

When uploading large files through UnixFS or other IPLD structures, the current batch implementation keeps everything in memory and writes blocks individually. If the application crashes mid-process, partial blocks remain in FlatFS, making garbage collection extremely difficult since there's no way to identify which blocks belong to incomplete operations.

Solution

This PR rewrites the batch mode to use a temporary directory approach:

1. Atomic batch operations: All blocks in a batch are written to a temporary directory first, then renamed to their final locations only on commit. If the process crashes, no partial data remains in the main datastore - the temp directory is cleaned on restart.
2. Memory efficiency: Instead of keeping all data in memory, blocks are written directly to temp files, enabling large file uploads without high RAM usage.
3. Read operations in batch: Added BatchReader interface implementing Get/Has/GetSize methods. This is essential for IPLD structures which need to verify block links during construction. Without batch reads, you cannot build complex IPLD structures that reference blocks added earlier in the same batch. This follows standard database transaction semantics where reads within a transaction see uncommitted writes.

Implementation Details

• Each batch creates a unique temp directory under .temp/
• Files are written without sharding in temp
• On commit, files are atomically renamed to their sharded destinations
• On discard or crash, temp files are cleaned up (on the next startup). This code was already existing
• Thread-safe with mutex protection
• Batch can be reused after commit/discard (because this behavior of Batch was before my changes)

[gammazero]

Triage: Open PR in kubo and test on kubo staging.

[lidel]

Lgtm, as long we address two things:

1. Updated godoc (bd468ee) to reflect the latest version, but unsure about one thing – see inline.
2. IIUC two concurrent goroutines calling os.Create() and Write() on the same file can error or corrupt data – see inline.

[lidel]

Bit worried about this on Windows: i suspect concurrent async writes (multiple ipfs add) may cause file handle issues, things like golang/go#34681 – flatfs uses https://github.com/alexbrainman/goissue34681 for a reason, we likely need similar wrapper on windows here (old code used tempFileOnce from util_windows.go).

@gammazero too late on my end to write code, but we probably need to create createFile equivalent that does something similar (pseudocode):

util_unix.go:

  func createFile(name string) (*os.File, error) {
  	return os.Create(name)
  }

util_windows.go:

  func createFile(name string) (*os.File, error) {
  	return goissue34681.OpenFile(name, os.O_RDWR|os.O_CREATE|os.O_TRUNC, 0666)
  }

Then finally here:

  file, err := createFile(tempFile)

Windows concurrency is janky, unsure if we need a retry wrapper, maybe can go without it, but in case, it could be something like:

util_windows.go:

  func createFile(name string) (*os.File, error) {
  	var file *os.File
  	var err error
  	for i := 0; i < RetryAttempts; i++ {
  		file, err = createFileOnce(name)
  		if err == nil || !isTooManyFDError(err) {
  			break
  		}
  		time.Sleep(time.Duration(i+1) * RetryDelay)
  	}
  	return file, err
  }

[gammazero]

Done, without retry wrapper.

[hsanjuan]

Triage:

• Should test windows well. Perhaps stress-test with a test in this repo. Potential problem with file with duplicate blocks writing in parallel etc.