midx: use more structured data for expire #107
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
derrickstolee
force-pushed
the
midx-expire/ms
branch
from
96979d9 to
05c0e20
Compare
Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
derrickstolee
force-pushed
the
midx-expire/ms
branch
from
05c0e20 to
5e95f49
Compare
derrickstolee
added a commit
to microsoft/VFSForGit
that referenced
this pull request
Jan 7, 2019
… fix and trace2 v3 * [PR 107 in microsoft/git](microsoft/git#107) includes a fix for the way we permute the set of packs during a `git multi-pack-index expire` command. Specifically, this fixes an issue when we expire packs and add packs to the multi-pack-index at the same time (and the order of those packs). * [PR 100 in microsoft/git](microsoft/git#100) includes updates to trace2.
dscho
pushed a commit
that referenced
this pull request
Feb 27, 2019
This replaces #106. That PR was complicated and difficult to understand because we didn't use structured data, but instead relied on our simple arrays and overloaded that data. This is a bigger change, but results in code that is (hopefully) easier to understand. The new flow for writing a multi-pack-index is as follows: 1. Construct a list of `midx_info` structs that contain the details of the packs. This list starts with the packs in the existing midx, followed by the new packs to add. Keep track of the `orig_pack_int_id` for these packs. 2. Construct the list of object entries. The `pack_int_id` we use here corresponds to the `orig_pack_int_id` for the pack we are using. 3. Sort the packs by name. 4. If we have packs to drop, identify where they are in the list of packs. We can use the sorted nature of the list to know we will find them in the correct order. 5. Determine the `new_pack_int_id` for each `struct midx_info` by tracking how many are dropped by that point in the list. 6. Construct a new permutation array that maps from `orig_pack_int_id` to `new_pack_int_id`. If the pack is expired, then the value used here is invalid and will error if any object tries to use that value. 7. Count the length of the pack names we will write, and modify the length to be properly aligned if necessary. 8. Write the midx as usual, tracking that we have `packs.nr - drop_count` packs to write. 9. When writing the object offsets, use `packs.perm` to translate from the `orig_pack_int_id` to `new_pack_int_id`. While this PR is just one giant commit, I will peel parts across multiple commits for upstream. These will be interleaved with the commits already in `microsoft/git:master`.
dscho
pushed a commit
that referenced
this pull request
Mar 29, 2019
This replaces #106. That PR was complicated and difficult to understand because we didn't use structured data, but instead relied on our simple arrays and overloaded that data. This is a bigger change, but results in code that is (hopefully) easier to understand. The new flow for writing a multi-pack-index is as follows: 1. Construct a list of `midx_info` structs that contain the details of the packs. This list starts with the packs in the existing midx, followed by the new packs to add. Keep track of the `orig_pack_int_id` for these packs. 2. Construct the list of object entries. The `pack_int_id` we use here corresponds to the `orig_pack_int_id` for the pack we are using. 3. Sort the packs by name. 4. If we have packs to drop, identify where they are in the list of packs. We can use the sorted nature of the list to know we will find them in the correct order. 5. Determine the `new_pack_int_id` for each `struct midx_info` by tracking how many are dropped by that point in the list. 6. Construct a new permutation array that maps from `orig_pack_int_id` to `new_pack_int_id`. If the pack is expired, then the value used here is invalid and will error if any object tries to use that value. 7. Count the length of the pack names we will write, and modify the length to be properly aligned if necessary. 8. Write the midx as usual, tracking that we have `packs.nr - drop_count` packs to write. 9. When writing the object offsets, use `packs.perm` to translate from the `orig_pack_int_id` to `new_pack_int_id`. While this PR is just one giant commit, I will peel parts across multiple commits for upstream. These will be interleaved with the commits already in `microsoft/git:master`.
dscho
pushed a commit
that referenced
this pull request
May 25, 2019
This replaces #106. That PR was complicated and difficult to understand because we didn't use structured data, but instead relied on our simple arrays and overloaded that data. This is a bigger change, but results in code that is (hopefully) easier to understand. The new flow for writing a multi-pack-index is as follows: 1. Construct a list of `midx_info` structs that contain the details of the packs. This list starts with the packs in the existing midx, followed by the new packs to add. Keep track of the `orig_pack_int_id` for these packs. 2. Construct the list of object entries. The `pack_int_id` we use here corresponds to the `orig_pack_int_id` for the pack we are using. 3. Sort the packs by name. 4. If we have packs to drop, identify where they are in the list of packs. We can use the sorted nature of the list to know we will find them in the correct order. 5. Determine the `new_pack_int_id` for each `struct midx_info` by tracking how many are dropped by that point in the list. 6. Construct a new permutation array that maps from `orig_pack_int_id` to `new_pack_int_id`. If the pack is expired, then the value used here is invalid and will error if any object tries to use that value. 7. Count the length of the pack names we will write, and modify the length to be properly aligned if necessary. 8. Write the midx as usual, tracking that we have `packs.nr - drop_count` packs to write. 9. When writing the object offsets, use `packs.perm` to translate from the `orig_pack_int_id` to `new_pack_int_id`. While this PR is just one giant commit, I will peel parts across multiple commits for upstream. These will be interleaved with the commits already in `microsoft/git:master`.
dscho
pushed a commit
that referenced
this pull request
May 27, 2019
This replaces #106. That PR was complicated and difficult to understand because we didn't use structured data, but instead relied on our simple arrays and overloaded that data. This is a bigger change, but results in code that is (hopefully) easier to understand. The new flow for writing a multi-pack-index is as follows: 1. Construct a list of `midx_info` structs that contain the details of the packs. This list starts with the packs in the existing midx, followed by the new packs to add. Keep track of the `orig_pack_int_id` for these packs. 2. Construct the list of object entries. The `pack_int_id` we use here corresponds to the `orig_pack_int_id` for the pack we are using. 3. Sort the packs by name. 4. If we have packs to drop, identify where they are in the list of packs. We can use the sorted nature of the list to know we will find them in the correct order. 5. Determine the `new_pack_int_id` for each `struct midx_info` by tracking how many are dropped by that point in the list. 6. Construct a new permutation array that maps from `orig_pack_int_id` to `new_pack_int_id`. If the pack is expired, then the value used here is invalid and will error if any object tries to use that value. 7. Count the length of the pack names we will write, and modify the length to be properly aligned if necessary. 8. Write the midx as usual, tracking that we have `packs.nr - drop_count` packs to write. 9. When writing the object offsets, use `packs.perm` to translate from the `orig_pack_int_id` to `new_pack_int_id`. While this PR is just one giant commit, I will peel parts across multiple commits for upstream. These will be interleaved with the commits already in `microsoft/git:master`.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This replaces #106. That PR was complicated and difficult to understand because we didn't use structured data, but instead relied on our simple arrays and overloaded that data.
This is a bigger change, but results in code that is (hopefully) easier to understand. The new flow for writing a multi-pack-index is as follows:
Construct a list of
midx_infostructs that contain the details of the packs. This list starts with the packs in the existing midx, followed by the new packs to add. Keep track of theorig_pack_int_idfor these packs.Construct the list of object entries. The
pack_int_idwe use here corresponds to theorig_pack_int_idfor the pack we are using.Sort the packs by name.
If we have packs to drop, identify where they are in the list of packs. We can use the sorted nature of the list to know we will find them in the correct order.
Determine the
new_pack_int_idfor eachstruct midx_infoby tracking how many are dropped by that point in the list.Construct a new permutation array that maps from
orig_pack_int_idtonew_pack_int_id. If the pack is expired, then the value used here is invalid and will error if any object tries to use that value.Count the length of the pack names we will write, and modify the length to be properly aligned if necessary.
Write the midx as usual, tracking that we have
packs.nr - drop_countpacks to write.When writing the object offsets, use
packs.permto translate from theorig_pack_int_idtonew_pack_int_id.While this PR is just one giant commit, I will peel parts across multiple commits for upstream. These will be interleaved with the commits already in
microsoft/git:master.