Inaccurate depth count after readError on "current" file

[kev1n80]

It seems like DiskQueue does not accurately update depth when a read error occurs (in handleReadError()) or in the scenario where the machine finishes reading a file, deletes a file, but crashes before writing to the MetaData file.

[mreiferson]

Yes, those edge cases are possible. What we do is perform checks when you reach the tail of the queue:

go-diskqueue/diskqueue.go

Lines 521 to 536 in 99f6193

	// we've reached the end of the diskqueue
	// if depth isn't 0 something went wrong
	if depth != 0 {
	if depth < 0 {
	d.logf(ERROR,
	"DISKQUEUE(%s) negative depth at tail (%d), metadata corruption, resetting 0...",
	d.name, depth)
	} else if depth > 0 {
	d.logf(ERROR,
	"DISKQUEUE(%s) positive depth at tail (%d), data loss, resetting 0...",
	d.name, depth)
	}
	// force set depth 0
	d.depth = 0
	d.needSync = true
	}

[kev1n80]

Hello, I'm not sure if I should make a new issue or respond here. I found that in the case where the user is reading all of the messages in the queue and the last file is corrupted, the depth never becomes 0. Within the if clause in line 579, the readFileNum == the writeFileNum and writePos is set to 0 as is the readPos later on in the function. After the readPos is set to 0, shouldn't checkTailCorruption be called or depth be set to 0 if readFileNum == writeFileNum?

go-diskqueue/diskqueue.go

Lines 579 to 605 in 99f6193

	if d.readFileNum == d.writeFileNum {
	// if you can't properly read from the current write file it's safe to
	// assume that something is fucked and we should skip the current file too
	if d.writeFile != nil {
	d.writeFile.Close()
	d.writeFile = nil
	}
	d.writeFileNum++
	d.writePos = 0
	}
	badFn := d.fileName(d.readFileNum)
	badRenameFn := badFn + ".bad"
	d.logf(WARN,
	"DISKQUEUE(%s) jump to next file and saving bad file as %s",
	d.name, badRenameFn)
	err := os.Rename(badFn, badRenameFn)
	if err != nil {
	d.logf(ERROR,
	"DISKQUEUE(%s) failed to rename bad diskqueue file %s to %s",
	d.name, badFn, badRenameFn)
	}
	d.readFileNum++
	d.readPos = 0

[mreiferson]

Yea, you're right, we should call d.checkTailCorruption(d.depth) at the end of handleReadError(), want to open a PR?

[kev1n80]

Yes I would love to open a PR and add the change: call d.checkTailCorruption(d.depth) at the end of handleReadError(). Thank you!