Sync fork #11
Open
Sync fork #11
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…of messages we read in the readFile. Also, reserve the last 4 bytes at the end of the file for the number of messages in that file.
…ile now contains readMessages and writeMessages.
…omplete file and completely reading a file.
…writeMessages and readMessages in skipToNextRWFile.
…or disk space limit.
…t that these numbers are accurate.
…esolve the writeMessages off by 1 error.
This reverts commit 652b548.
…s, and resolve the writeMessages off by 1 error." This reverts commit 48c7a85.
… exactly with a message less than 8 bytes.
This reverts commit e9b23ac.
Track the Number of Messages
* Remove readBytes and update testing. * Add comment for readMsgSize. * Modify disk size limit features only if disk size limit feature is being used . * Reset write bytes during read error if read file is write file. * Track writeBytes and readMsgSize. * Update test. * Test writeBytes is accurate. * Make code more readable. * Revert back to using msgSize instead of readMsgSize in order to make minimal changes. * Update max bad file size to include numFileMsgsBytes.
* Remove readBytes and update testing. * Add comment for readMsgSize. * Modify disk size limit features only if disk size limit feature is being used . * Increase disk size limit in testing, and get MetaData file size. * Add comments to metaDataFileSize func for better readability. * Add logic to remove readFile if we are going to surpass the Disk Size Limit. * Abstract code from moveForward and create makeSpace function to be used when DiskQueue needs to make space. * Rename func to make it more readable. * Make space when the new writeMessage will surpass the disk size limit. Test that DiskQueue never surpasses the disk size limit. * Make space until there is enough space to write the message * Abstract code and make a reachFileSizeLimit flag. * Update testing to account for metadata file size. * Handle the deletion of .bad files if it exists. * Get oldest bad file info does not need to return an error. It either finds a bad file or does not. * Track size of bad files. * Decrease badBytes when we delete a bad file and move badBytes to check if we need to make disk space. * Remove badBytes from metadata and make it an atomic field. * Recalculate badBytes if it is a negative number. * Add comments and rename varaibles to improve readability. * Update metaData file size now that badBytes is removed from metaData file. * Prevent user from writing data that is bigger than disk size limit. * Make the code more readable by changing the order of checks in writeOne(). * Test the scenario when msgSize needs the deletion of several files. * Use regex when getting diskqueue files with .bad extension. * Update the tracking of .bad files to use regex. * Remove log messages. * Start writing a test to check if .bad files are deleted or accounted for correctly. * Test that DiskQueue deleted .bad files first in order to make Disk Space. * Update with to other branches - variable name change and adjust writeBytes, writeMsgs, and readMsgs in handReadError. * Adjust writeBytes as a result of the initial overestimation of the size of a bad file when we cannot get its accurate size. * Reset messages in handleReadError * Use MatchString as opposed to Match. * Throw error. * Instead of overestimating bad file size, underestime it. This ensures we will never go over disk limit. * Add a check to see if readFile is corrupted. * Remove DEBUG logs. * test .bad files. * In progress... corrupt file and then have readOne() deem it corrupted. * Make code more readable. * Ensure that diskqueue handles corrupted files correctly. * Remove badBytes and get the accurate writeBytes data every time a writeFile is deleted or a readFile turns into a bad file. * Abstract general function to walk through all of the files in the directory DiskQueue writes and reads in. * Abstract code to its own function. * Remove readMsgSize and make code more readable. * Make test code more readable. * Update go.mod * Make regexp constants and close metadatafile after getting its size. * Change writeBytes to totalDiskSpaceUsed and have it track writeBytes and badBytes for now. * Update panic messages. * Track the total disk size with one variable rather than several smaller ones. * Remove unnecessary comments. * Break up huge function into two smaller functions. * Ensure that global regExp are created on start. * Add testing of depth, add info logs, and test when disk size limit is too small (not enough space for meta data file). * Update name of dq objects to match function name. * Use ReadDir instead of WalkDir * Replace go v1.16 functions and objects with <v1.16 functions and objects. * Have metaData file size stay as 56 bytes and update testing. * Add the number of bytes that were removed when a file is removed. * Only get the size if the file was able to be removed successfully. * Add function to get filder size and update log when removing read file to make space. * Make the code more readable. * Extract chunk of code into a new function. * Improve readability of the code and update code from nsqio#29 issue 29 * totalDiskSpaceUsed does not change in handleReadError * Iterate through each file rather than relying on the totalDiskSpaceUsed variable. * Revert "Merge branch 'master' into DiskSizeLimit" This reverts commit 0c456b4, reversing changes made to a821855.
Updated the Build status, Go reference, and latest Github release
* trivial improvement of some checks * use file instead of bufio.Reader in readNumOfMessages from file end Signed-off-by: Leon Ziyang Zhang <leonzz@google.com>
Update \d\d\d\d\d\d to \d+
* Add PeekChan() based on nsqio@2cb4338 * Add peekchan test
Co-authored-by: Mikhail Faraponov <mfaraponov@cloudlinux.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Start the the sync process by merging master branch with this branch that contains nsqio:master updated code.