Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

maxBytesPerFile read check off-by-one #15

Closed
wants to merge 3 commits into from

Conversation

xkeyideal
Copy link

No description provided.

@xkeyideal
Copy link
Author

#14

@xkeyideal
Copy link
Author

@ploxiln

Please see the newest pr code.

use curReadFileSize instead of maxBytesPerFile for move to next file.

TODO: each data file should embed the maxBytesPerFile as the first 8 bytes (at creation time) ensuring that the value can change without affecting runtime be solved.

diskqueue.go Outdated
@@ -625,7 +644,16 @@ func (d *diskQueue) ioLoop() {
count = 0
}

if (d.readFileNum < d.writeFileNum) || (d.readPos < d.writePos) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this expression did not need to change (and the original is more concise)

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the change just for my debug, and forget change back.
I will change it back, and submit another pr.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for another PR: you can do a git interactive-rebase, and force-push this branch, to make any change to this PR

// the value can change without affecting runtime
if d.nextReadPos > d.maxBytesPerFile {
// only readFileNum less than writeFileNum need move next file
if d.readFileNum < d.writeFileNum && d.nextReadPos >= d.curReadFileSize {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if d.readFileNum < d.writeFileNum needed to be added here, I'll think on it a bit...

Adding a test or two would help :) In particular, one that fails with current master, and passes with this branch. Note that Travis-CI currently fails to post "commit status" for the "legacy travis-ci.org integration". I've updated a couple of other repos to use GitHub Actions instead, I'll get around to this one eventually. In the meantime, you can view test results at https://travis-ci.org/github/nsqio/go-diskqueue/pull_requests or run them locally.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when d.readFileNum == d.writeFileNum means reader and writer is same file, so when d.nextReadPos >= d.curReadFileSize the readFile can't be closed, the program should be waiting writeFile update d.curReadFileSize

And I think just pass now unit test will be ok.

@ploxiln
Copy link
Member

ploxiln commented Apr 15, 2020

The new curReadFileSize logic looks good to me, thanks!
It will take some more time to review and validate, sorry.

@xkeyideal
Copy link
Author

I use the nsqio/go-diskqueue code develop read-write separation program, for readonly and read by write, thanks for your code.

I use curReadFileSize to solve switch files avoid maxBytesPerFile cause file data error

@mreiferson
Copy link
Member

mreiferson commented Dec 29, 2020

The logic here looks good and addresses an outstanding TODO, I've cleaned things up in #23, so let's wrap this up there.

Thanks!

@mreiferson mreiferson closed this Dec 29, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants