TEST PR: speed up index load through parallelization #20

Test clean-up and corrections. * es/test-fixes: (26 commits) t5608: fix broken &&-chain t9119: fix broken &&-chains t9000-t9999: fix broken &&-chains t7000-t7999: fix broken &&-chains t6000-t6999: fix broken &&-chains t5000-t5999: fix broken &&-chains t4000-t4999: fix broken &&-chains t3030: fix broken &&-chains t3000-t3999: fix broken &&-chains t2000-t2999: fix broken &&-chains t1000-t1999: fix broken &&-chains t0000-t0999: fix broken &&-chains t9814: simplify convoluted check that command correctly errors out t9001: fix broken "invoke hook" test t7810: use test_expect_code() instead of hand-rolled comparison t7400: fix broken "submodule add/reconfigure --force" test t7201: drop pointless "exit 0" at end of subshell t6036: fix broken "merge fails but has appropriate contents" tests t5505: modernize and simplify hard-to-digest test t5406: use write_script() instead of birthing shell script manually ...

"git diff --color-moved" feature has further been tweaked. * sb/diff-color-move-more: diff.c: offer config option to control ws handling in move detection diff.c: add white space mode to move detection that allows indent changes diff.c: factor advance_or_nullify out of mark_color_as_moved diff.c: decouple white space treatment from move detection algorithm diff.c: add a blocks mode for moved code detection diff.c: adjust hash function signature to match hashmap expectation diff.c: do not pass diff options as keydata to hashmap t4015: avoid git as a pipe input xdiff/xdiffi.c: remove unneeded function declarations xdiff/xdiff.h: remove unused flags

"git checkout" and "git worktree add" learned to honor checkout.defaultRemote when auto-vivifying a local branch out of a remote tracking branch in a repository with multiple remotes that have tracking branches that share the same names. * ab/checkout-default-remote: checkout & worktree: introduce checkout.defaultRemote checkout: add advice for ambiguous "checkout <branch>" builtin/checkout.c: use "ret" variable for return checkout: pass the "num_matches" up to callers checkout.c: change "unique" member to "num_matches" checkout.c: introduce an *_INIT macro checkout.h: wrap the arguments to unique_tracking_name() checkout tests: index should be clean after dwim checkout

Code restructuring and a small fix to transport protocol v2 during fetching. * jt/fetch-pack-negotiator: fetch-pack: introduce negotiator API fetch-pack: move common check and marking together fetch-pack: make negotiation-related vars local fetch-pack: use ref adv. to prune "have" sent fetch-pack: directly end negotiation if ACK ready fetch-pack: clear marks before re-marking fetch-pack: split up everything_local()

Parsing of -L[<N>][,[<M>]] parameters "git blame" and "git log" take has been tweaked. * is/parsing-line-range: log: prevent error if line range ends past end of file blame: prevent error if range ends past end of file

lookup_commit_reference() and friends have been updated to find in-core object for a specific in-core repository instance. * sb/object-store-lookup: (32 commits) commit.c: allow lookup_commit_reference to handle arbitrary repositories commit.c: allow lookup_commit_reference_gently to handle arbitrary repositories tag.c: allow deref_tag to handle arbitrary repositories object.c: allow parse_object to handle arbitrary repositories object.c: allow parse_object_buffer to handle arbitrary repositories commit.c: allow get_cached_commit_buffer to handle arbitrary repositories commit.c: allow set_commit_buffer to handle arbitrary repositories commit.c: migrate the commit buffer to the parsed object store commit-slabs: remove realloc counter outside of slab struct commit.c: allow parse_commit_buffer to handle arbitrary repositories tag: allow parse_tag_buffer to handle arbitrary repositories tag: allow lookup_tag to handle arbitrary repositories commit: allow lookup_commit to handle arbitrary repositories tree: allow lookup_tree to handle arbitrary repositories blob: allow lookup_blob to handle arbitrary repositories object: allow lookup_object to handle arbitrary repositories object: allow object_as_type to handle arbitrary repositories tag: add repository argument to deref_tag tag: add repository argument to parse_tag_buffer tag: add repository argument to lookup_tag ...

Various glitches in the heuristics of merge-recursive strategy have been documented in new tests. * en/t6042-insane-merge-rename-testcases: t6042: add testcase covering long chains of rename conflicts t6042: add testcase covering rename/rename(2to1)/delete/delete conflict t6042: add testcase covering rename/add/delete conflict type

"git fetch" learned a new option "--negotiation-tip" to limit the set of commits it tells the other end as "have", to reduce wasted bandwidth and cycles, which would be helpful when the receiving repository has a lot of refs that have little to do with the history at the remote it is fetching from. * jt/fetch-nego-tip: fetch-pack: support negotiation tip whitelist

For a large tree, the index needs to hold many cache entries allocated on heap. These cache entries are now allocated out of a dedicated memory pool to amortize malloc(3) overhead. * jm/cache-entry-from-mem-pool: block alloc: add validations around cache_entry lifecyle block alloc: allocate cache entries from mem_pool mem-pool: fill out functionality mem-pool: add life cycle management functions mem-pool: only search head block for available space block alloc: add lifecycle APIs for cache_entry structs read-cache: teach make_cache_entry to take object_id read-cache: teach refresh_cache_entry to take istate

"git gc --auto" opens file descriptors for the packfiles before spawning "git repack/prune", which would upset Windows that does not want a process to work on a file that is open by another process. The issue has been worked around. * kg/gc-auto-windows-workaround: gc --auto: release pack files before auto packing

"git grep" learned the "--only-matching" option. * tb/grep-only-matching: grep.c: teach 'git grep --only-matching' grep.c: extract show_line_header()

"git rebase --rebase-merges" mode now handles octopus merges as well. * js/rebase-merge-octopus: rebase --rebase-merges: adjust man page for octopus support rebase --rebase-merges: add support for octopus merges merge: allow reading the merge commit message from a file

The recursive merge strategy did not properly ensure there was no change between HEAD and the index before performing its operation, which has been corrected. * en/dirty-merge-fixes: merge: fix misleading pre-merge check documentation merge-recursive: enforce rule that index matches head before merging t6044: add more testcases with staged changes before a merge is invoked merge-recursive: fix assumption that head tree being merged is HEAD merge-recursive: make sure when we say we abort that we actually abort t6044: add a testcase for index matching head, when head doesn't match HEAD t6044: verify that merges expected to abort actually abort index_has_changes(): avoid assuming operating on the_index read-cache.c: move index_has_changes() from merge.c

Tests to cover various conflicting cases have been added for merge-recursive. * en/t6036-merge-recursive-tests: t6036: add a failed conflict detection case: regular files, different modes t6036: add a failed conflict detection case with conflicting types t6036: add a failed conflict detection case with submodule add/add t6036: add a failed conflict detection case with submodule modify/modify t6036: add a failed conflict detection case with symlink add/add t6036: add a failed conflict detection case with symlink modify/modify

Tests to cover conflict cases that involve submodules have been added for merge-recursive. * en/t7405-recursive-submodule-conflicts: t7405: verify 'merge --abort' works after submodule/path conflicts t7405: add a directory/submodule conflict t7405: add a file/submodule conflict

"git rebase" started exporting GIT_DIR environment variable and exposing it to hook scripts when part of it got rewritten in C. Instead of matching the old scripted Porcelains' behaviour, compensate by also exporting GIT_WORK_TREE environment as well to lessen the damage. This can harm existing hooks that want to operate on different repository, but the current behaviour is already broken for them anyway. * bc/sequencer-export-work-tree-as-well: sequencer: pass absolute GIT_WORK_TREE to exec commands

"git send-email" when using in a batched mode that limits the number of messages sent in a single SMTP session lost the contents of the variable used to choose between tls/ssl, unable to send the second and later batches, which has been fixed. * jm/send-email-tls-auth-on-batch: send-email: fix tls AUTH when sending batch

Add a server-side knob to skip commits in exponential/fibbonacci stride in an attempt to cover wider swath of history with a smaller number of iterations, potentially accepting a larger packfile transfer, instead of going back one commit a time during common ancestor discovery during the "git fetch" transaction. * jt/fetch-negotiator-skipping: negotiator/skipping: skip commits during fetch

The lazy clone support had a few places where missing but promised objects were not correctly tolerated, which have been fixed. * jt/tags-to-promised-blobs-fix: tag: don't warn if target is missing but promised revision: tolerate promised targets of tags

Look for broken "&&" chains that are hidden in subshell, many of which have been found and corrected. * es/chain-lint-in-subshell: t/chainlint.sed: drop extra spaces from regex character class t/chainlint: add chainlint "specialized" test cases t/chainlint: add chainlint "complex" test cases t/chainlint: add chainlint "cuddled" test cases t/chainlint: add chainlint "loop" and "conditional" test cases t/chainlint: add chainlint "nested subshell" test cases t/chainlint: add chainlint "one-liner" test cases t/chainlint: add chainlint "whitespace" test cases t/chainlint: add chainlint "basic" test cases t/Makefile: add machinery to check correctness of chainlint.sed t/test-lib: teach --chain-lint to detect broken &&-chains in subshells

The singleton commit-graph in-core instance is made per in-core repository instance. * jt/commit-graph-per-object-store: commit-graph: add repo arg to graph readers commit-graph: store graph in struct object_store commit-graph: add free_commit_graph commit-graph: add missing forward declaration object-store: add missing include commit-graph: refactor preparing commit graph

Signed-off-by: Junio C Hamano <gitster@pobox.com>

The option help text for the force-with-lease option to "git push" reads like this: $ git push -h 2>&1 | grep -e force-with-lease --force-with-lease[=<refname>:<expect>] which comes from having N_("refname>:<expect") as the argument help text in the source code, with an aparent lack of "<" and ">" at both ends. It turns out that parse-options machinery takes the whole string and encloses it inside a pair of "<>", to make it easier for majority cases that uses a single token placeholder. The help string was written in a funnily unbalanced way knowing that the end result would balance out, by somebody who forgot the presence of PARSE_OPT_LITERAL_ARGHELP, which is the escape hatch mechanism designed to help such a case. We just should use the official escape hatch instead. Because ":<expect>" part can be omitted to ask Git to guess, it may be more correct to spell it as "<refname>[:<expect>]", but that is not the focus of this topic. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Helped-by: René Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Don't translate the argument specification for --chmod; "+x" and "-x" are the literal strings that the commands accept. Separate alternatives using a pipe character instead of a slash, for consistency. Use the flag PARSE_OPT_LITERAL_ARGHELP to prevent parseopt from adding a pair of angular brackets around the argument help string, as that would wrongly indicate that users need to replace the literal strings with some kind of value. Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Parseopt wraps arguments in a pair of angular brackets by default, signifying that the user needs to replace it with a value of the documented type. Remove the pairs from the option definitions to duplication and confusion. Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Wrap both placeholders in the argument help string in angular brackets to signal that users needs replace them with some actual value. Use the flag PARSE_OPT_LITERAL_ARGHELP to prevent parseopt from adding another pair. Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Wrap each part of the argument help string in angular brackets to show that users need to replace them with actual values. Do that explicitly to balance the pairs nicely in the code and avoid confusing casual readers. Add the flag PARSE_OPT_LITERAL_ARGHELP to keep parseopt from adding another pair. Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Wrap the placeholders in the option help string for -w in pairs of angular brackets to document that users need to replace them with actual numbers. Use the flag PARSE_OPT_LITERAL_ARGHELP to prevent parseopt from adding another pair. Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Parseopt wraps argument help strings in a pair of angular brackets by default, to tell users that they need to replace it with an actual value. This is useful in most cases, because most option arguments are indeed single values of a certain type. The option PARSE_OPT_LITERAL_ARGHELP needs to be used in option definitions with arguments that have multiple parts or are literal strings. Stop adding these angular brackets if special characters are present, as they indicate that we don't deal with a simple placeholder. This simplifies the code a bit and makes defining special options slightly easier. Remove the flag PARSE_OPT_LITERAL_ARGHELP in the cases where the new and more cautious handling suffices. Signed-off-by: Rene Scharfe <l.s.r@web.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

want_color_fd() is designed to work only with standard output and error file descriptors and stores information about each descriptor in an array. However, it doesn't verify that the passed-in descriptor lives within that set, which, with a buggy caller, could lead to access or assignment outside the array bounds. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Acked-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

This test description looks like it was written with the originally observed behavior ("causes segfault") rather than the desired and now current behavior ("does not cause segfault"). Fix it. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The color group in config.txt is actually sorted but changes in sb/blame-color broke this. Reorder color.blame.* and move blame.coloring back to the rest of blame.* (and reorder that group too while we're there) Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

After making a change to the documentation, it's easy to forget to check the rendered version to make sure it was formatted as you intended. And simply doing a diff between the two built versions is less trivial than you might hope: - diffing the roff or html output isn't particularly readable; what we really care about is what the end user will see - you have to tweak a few build variables to avoid spurious differences (e.g., version numbers, build times) Let's provide a script that builds and installs the manpages for two commits, renders the results using "man", and diffs the result. Since this is time-consuming, we'll also do our best to avoid repeated work, keeping intermediate results between runs. Some of this could probably be made a little less ugly if we built support into Documentation/Makefile. But by relying only on "make install-man" working, this script should work for generating a diff between any two versions, whether they include this script or not. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Git for Windows' original 4aa8b8c (Teach 'git pull' to handle --rebase=interactive, 2011-10-21) had support for the very convenient abbreviation git pull --rebase=i which was later lost when it was ported to the builtin `git pull`, and it was not introduced before the patch eventually made it into Git as f5eb87b (pull: allow interactive rebase with --rebase=interactive, 2016-01-13). However, it is *really* a useful short hand for the occasional rebasing pull on branches that do not usually want to be rebased. So let's reintroduce this convenience, at long last. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Tests for "git am --[no-]scissors" [1] work in the following way: 1. Create files with commit messages 2. Use these files to create expected commits 3. Generate eml file with patch from expected commits 4. Create commits using git am with these eml files 5. Compare these commits with expected The test for "git am --scissors" is supposed to take an e-mail with a scissors line and in-body "Subject:" header and demonstrate that the subject line from the e-mail itself is overridden by the in-body header and that only text below the scissors line is included in the commit message of the commit created by the invocation of "git am --scissors". However, the setup of the test incorrectly uses a commit without the scissors line and without the in-body header in the commit message, producing eml file not suitable for testing of "git am --scissors". This can be checked by intentionally breaking is_scissors_line function in mailinfo.c, for example, by changing string ">8", which is used by the test. With such change the test should fail, but does not. Fix broken test by generating eml file with scissors line and in-body header "Subject:". Since the two tests for --scissors and --no-scissors options are there to test cutting or keeping the commit message, update both tests to change the test file in the same way, which allows us to generate only one eml file to be passed to git am. To clarify the intention of the test, give files and tags more explicit names. [1]: introduced in bf72ac1 (t4150: tests for am --[no-]scissors, 2015-07-19) Signed-off-by: Andrei Rybak <rybak.a.v@gmail.com> Reviewed-by: Paul Tan <pyokagan@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The --exec option's implementation is not really well-prepared for --rebase-merges. Demonstrate this. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Commit 3ac68a9 (help: add --config to list all available config - 2018-05-26) makes generate-cmdlist.sh adds a new input source config.txt but it's not a Makefile dependency. Any changes in config.txt will not trigger command-list.h regeneration and the config list in this file becomes outdated. Correct the dependency. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

A fair number of tests need to check that the filesystem supports file names including "funny" characters, like newline, tab, and double-quote. Jonathan Nieder suggested that this be extracted into a lazy prereq in the top-level `test-lib.sh`. This patch effects that change. The FUNNYNAMES prereq now uniformly requires support for newlines, tabs, and double-quotes in filenames. This very slightly decreases the power of some tests, which might have run previously on a system that supports (e.g.) newlines and tabs but not double-quotes, but now will not. This seems to me like an acceptable tradeoff for consistency. One test (`t/t9902-completion.sh`) defined FUNNYNAMES to further require the separators \034 through \037, the test for which was implemented using the Bash-specific $'\034' syntax. I've elected to leave this one as is, renaming it to FUNNIERNAMES. After this patch, `git grep 'test_$set\|lazy$_prereq.*FUNNYNAMES'` has only one result. Signed-off-by: William Chargin <wchargin@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

From a security perspective, it seems that SHA-256, BLAKE2, SHA3-256, K12, and so on are all believed to have similar security properties. All are good options from a security point of view. SHA-256 has a number of advantages: * It has been around for a while, is widely used, and is supported by just about every single crypto library (OpenSSL, mbedTLS, CryptoNG, SecureTransport, etc). * When you compare against SHA1DC, most vectorized SHA-256 implementations are indeed faster, even without acceleration. * If we're doing signatures with OpenPGP (or even, I suppose, CMS), we're going to be using SHA-2, so it doesn't make sense to have our security depend on two separate algorithms when either one of them alone could break the security when we could just depend on one. So SHA-256 it is. Update the hash-function-transition design doc to say so. After this patch, there are no remaining instances of the string "NewHash", except for an unrelated use from 2008 as a variable name in t/t9700/test.pl. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Acked-by: Linus Torvalds <torvalds@linux-foundation.org> Acked-by: brian m. carlson <sandals@crustytoothpaste.net> Acked-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Acked-by: Dan Shumow <danshu@microsoft.com> Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

On Fedora-derived systems, the apache httpd package installs modules under /usr/lib{,64}/httpd/modules, depending on whether the system is 32- or 64-bit. A symlink from /etc/httpd/modules is created which points to the proper module path. Use it to support apache on Fedora, CentOS, and Red Hat systems. Written with assistance of Todd Zullinger <tmz@pobox.com> and Junio C Hamano <gitster@pobox.com>. Signed-off-by: Sebastian Kisela <skisela@redhat.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The generated apache2 config fails with apache >= 2.4. The error log states: AH00136: Server MUST relinquish startup privileges before accepting connections. Please ensure mod_unixd or other system security module is loaded. AH00016: Configuration Failed Fix this by loading the unixd module. This works with older httpd as well, so no IfVersion conditional is needed. (Tested with httpd-2.2.15 on CentOS-6.) Written with assistance of Todd Zullinger <tmz@pobox.com> Signed-off-by: Sebastian Kisela <skisela@redhat.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The words "save" and "safe" are both very wonderful words, each with their own set of meanings. Let's not confuse them with one another save on occasion of a pun. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

We should not interrupt. sentences in the middle. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

A test making use of test_must_fail was failing like this: fatal: ambiguous argument '|': unknown revision or path not in the working tree. when the intent was to verify that a specific string was not found in the output of the git diff command, i.e. that grep returned non-zero. Fix the test to do that. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

We can get rid of some quoted tabs and make a few tests slightly easier to read and edit by just asking for the names of the files modified, since that's all these tests were interested in anyway. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

When a git command is on the left side of a pipe, the pipe will swallow its exit status, preventing us from detecting failures in said commands. Restructure the tests to put the output in a temporary file to avoid this problem. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

test -e, test -s, etc. do not provide nice error messages when we hit test failures, so use the test_* helper functions from test-lib-functions.sh. Also, add test_path_exists() to test-lib-function.sh while at it, so that we don't need to worry whether submodule/.git is a file or a directory. It currently is a file with contents of the form gitdir: ../.git/modules/submodule but it could be changed in the future to be a directory; this test only really cares that it exists. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

A user reported a submodule issue regarding a section mix-up, but it could be boiled down to the following test case: $ git init test && cd test $ git config foo."Bar".key test $ git config foo."bar".key test $ tail -n 3 .git/config [foo "Bar"] key = test key = test Sub sections are case sensitive and we have a test for correctly reading them. However we do not have a test for writing out config correctly with case sensitive subsection names, which is why this went unnoticed in 6ae996f (git_config_set: make use of the config parser's event stream, 2018-04-09) Unfortunately we have to make a distinction between old style configuration that looks like [foo.Bar] key = test and the new quoted style as seen above. The old style is documented as case-agnostic, hence we need to keep 'strncasecmp'; although the resulting setting for the old style config differs from the configuration. That will be fixed in a follow up patch. Reported-by: JP Sugarbroad <jpsugar@google.com> Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The bug was noticed when writing the previous patch; a fix for this bug is not easy though: If we choose to ignore the case of the subsection (and revert most of the code of the previous patch, just keeping s/strncasecmp/strcmp/), then we'd introduce new sections using the new syntax, such that -------- [section.subsection] key = value1 -------- git config section.Subsection.key value2 would result in -------- [section.subsection] key = value1 [section.Subsection] key = value2 -------- which is even more confusing. A proper fix would replace the first occurrence of 'key'. As the syntax is deprecated, let's prefer to not spend time on fixing the behavior and just document it instead. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The following phrase could be interpreted multiple ways: "To pretend you have a file with mode and sha1 at path" In particular, I can think of two: 1. Pretend we have some new file, which happens to have a given mode and sha1 2. Pretend one of the files we are already tracking has a different mode and sha1 than what it really does I think people could easily assume either case while reading, but the example command provided doesn't actually handle the first case, which caused some minor frustration to at least one user. Modify the example command so that it correctly handles both cases, and re-order the wording in a way that makes it more likely folks will assume the first interpretation. I believe the new example shouldn't pose any obstacles to those wanting the second interpretation (at worst, they pass an unnecessary extra flag). Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Back when we removed `git apply --index-info` in 2007, we forgot to adjust the documentation for update-index that reads its output. Let's reorder the description of three formats to present the other two formats that are still generated by git commands before this format, and stop mentioning `git apply --index-info`. Signed-off-by: Junio C Hamano <gitster@pobox.com>

The colorization is controlled with the config setting "color.remote". Supported keywords are "error", "warning", "hint" and "success". They are highlighted if they appear at the start of the line, which is common in error messages, eg. ERROR: commit is missing Change-Id The Git push process itself prints lots of non-actionable messages (eg. bandwidth statistics, object counters for different phases of the process). This obscures actionable error messages that servers may send back. Highlighting keywords in the sideband draws more attention to those messages. The background for this change is that Gerrit does server-side processing to create or update code reviews, and actionable error messages (eg. missing Change-Id) must be communicated back to the user during the push. User research has shown that new users have trouble seeing these messages. The highlighting is done on the client rather than server side, so servers don't have to grow capabilities to understand terminal escape codes and terminal state. It also consistent with the current state where Git is control of the local display (eg. prefixing messages with "remote: "). The highlighting can be configured using color.remote.<KEYWORD> configuration settings. Since the keys are matched case insensitively, we match the keywords case insensitively too. Finally, this solution is backwards compatible: many servers already prefix their messages with "error", and they will benefit from this change without requiring a server update. By contrast, a server-side solution would likely require plumbing the TERM variable through the git protocol, so it would require changes to both server and client. Helped-by: Duy Nguyen <pclouds@gmail.com> Signed-off-by: Han-Wen Nienhuys <hanwen@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The idea of `--exec` is to append an `exec` call after each `pick`. Since the introduction of fixup!/squash! commits, this idea was extended to apply to "pick, possibly followed by a fixup/squash chain", i.e. an exec would not be inserted between a `pick` and any of its corresponding `fixup` or `squash` lines. The current implementation uses a dirty trick to achieve that: it assumes that there are only pick/fixup/squash commands, and then *inserts* the `exec` lines before any `pick` but the first, and appends a final one. With the todo lists generated by `git rebase --rebase-merges`, this simple implementation shows its problems: it produces the exact wrong thing when there are `label`, `reset` and `merge` commands. Let's change the implementation to do exactly what we want: look for `pick` lines, skip any fixup/squash chains, and then insert the `exec` line. Lather, rinse, repeat. Note: we take pains to insert *before* comment lines whenever possible, as empty commits are represented by commented-out pick lines (and we want to insert a preceding pick's exec line *before* such a line, not afterward). While at it, also add `exec` lines after `merge` commands, because they are similar in spirit to `pick` commands: they add new commits. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

A subsequent patch will teach repack to run pack-objects with some same and some different arguments if repacking of promisor objects is required. Refactor the setup of the pack-objects cmd so that setting up the arguments common to both is done in a function. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Currently, repack does not touch promisor packfiles at all, potentially causing the performance of repositories that have many such packfiles to drop. Therefore, repack all promisor objects if invoked with -a or -A. This is done by an additional invocation of pack-objects on all promisor objects individually given, which takes care of deduplication and allows the resulting packfiles to respect flags such as --max-pack-size. Signed-off-by: Jonathan Tan <jonathantanmy@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

When gpg-interface API unified support for signature verification codepaths for signed tags and signed commits in mid 2015 at around v2.6.0-rc0~114, we accidentally loosened the GPG signature verification. Before that change, signed commits were verified by looking for "G"ood signature from GPG, while ignoring the exit status of "gpg --verify" process, while signed tags were verified by simply passing the exit status of "gpg --verify" through. The unified code we currently have ignores the exit status of "gpg --verify" and returns successful verification when the signature matches an unexpired key regardless of the trust placed on the key (i.e. in addition to "G"ood ones, we accept "U"ntrusted ones). Make these commands signal failure with their exit status when underlying "gpg --verify" (or the custom command specified by "gpg.program" configuration variable) does so. This essentially changes their behaviour in a backward incompatible way to reject signatures that have been made with untrusted keys even if they correctly verify, as that is how "gpg --verify" behaves. Note that the code still overrides a zero exit status obtained from "gpg" (or gpg.program) if the output does not say the signature is good or computes correctly but made with untrusted keys, to catch a poorly written wrapper around "gpg" the user may give us. We could exclude "U"ntrusted support from this fallback code, but that would be making two backward incompatible changes in a single commit, so let's avoid that for now. A follow-up change could do so if desired. Helped-by: Vojtech Myslivec <vojtech.myslivec@nic.cz> Helped-by: brian m. carlson <sandals@crustytoothpaste.net> Helped-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The t5552 test script uses GIT_TRACE_PACKET to monitor what git-fetch sends and receives. However, because we're accessing a local repository, the child upload-pack also sends trace output to the same file. On Linux, this works out OK. We open the trace file with O_APPEND, so all writes are atomically positioned at the end of the file. No data can be overwritten or omitted. And since we prepare our small writes in a strbuf and write them with a single write(), we should see each line as an atomic unit. The order of lines between the two processes is undefined, but the test script greps only for "fetch>" or "fetch<" lines. So under Linux, the test results are deterministic. The test fails intermittently on Windows, however, reportedly even overwriting bits of the output file (i.e., O_APPEND does not seem to give us an atomic position+write). Since the test only cares about the trace output from fetch, we can just disable the output from upload-pack. That doesn't solve the greater question of O_APPEND/trace issues under Windows, but it easily fixes the flakiness from this test. Reported-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The problem solved by the code introduced in this commit goes like this: given two sets of items, and a cost matrix which says how much it "costs" to assign any given item of the first set to any given item of the second, assign all items (except when the sets have different size) in the cheapest way. We use the Jonker-Volgenant algorithm to solve the assignment problem to answer questions such as: given two different versions of a topic branch (or iterations of a patch series), what is the best pairing of commits/patches between the different versions? Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

This command does not do a whole lot so far, apart from showing a usage that is oddly similar to that of `git tbdiff`. And for a good reason: the next commits will turn `range-branch` into a full-blown replacement for `tbdiff`. At this point, we ignore tbdiff's color options, as they will all be implemented later using diff_options. Since f318d73 (generate-cmds.sh: export all commands to command-list.h, 2018-05-10), every new command *requires* a man page to build right away, so let's also add a blank man page, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

At this stage, `git range-diff` can determine corresponding commits of two related commit ranges. This makes use of the recently introduced implementation of the linear assignment algorithm. The core of this patch is a straight port of the ideas of tbdiff, the apparently dormant project at https://github.com/trast/tbdiff. The output does not at all match `tbdiff`'s output yet, as this patch really concentrates on getting the patch matching part right. Note: due to differences in the diff algorithm (`tbdiff` uses the Python module `difflib`, Git uses its xdiff fork), the cost matrix calculated by `range-diff` is different (but very similar) to the one calculated by `tbdiff`. Therefore, it is possible that they find different matching commits in corner cases (e.g. when a patch was split into two patches of roughly equal length). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

This patch lets `git range-diff` use the same order as tbdiff. The idea is simple: for left-to-right readers, it is natural to assume that the `git range-diff` is performed between an older vs a newer version of the branch. As such, the user is probably more interested in the question "where did this come from?" rather than "where did that one go?". To that end, we list the commits in the order of the second commit range ("the newer version"), inserting the unmatched commits of the first commit range as soon as all their predecessors have been shown. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Just like tbdiff, we now show the diff between matching patches. This is a "diff of two diffs", so it can be a bit daunting to read for the beginner. An alternative would be to display an interdiff, i.e. the hypothetical diff which is the result of first reverting the old diff and then applying the new diff. Especially when rebasing frequently, an interdiff is often not feasible, though: if the old diff cannot be applied in reverse (due to a moving upstream), an interdiff can simply not be inferred. This commit brings `range-diff` closer to feature parity with regard to tbdiff. To make `git range-diff` respect e.g. color.diff.* settings, we have to adjust git_branch_config() accordingly. Note: while we now parse diff options such as --color, the effect is not yet the same as in tbdiff, where also the commit pairs would be colored. This is left for a later commit. Note also: while tbdiff accepts the `--no-patches` option to suppress these diffs between patches, we prefer the `-s` (or `--no-patch`) option that is automatically supported via our use of diff_opt_parse(). And finally note: to support diff options, we have to call `parse_options()` such that it keeps unknown options, and then loop over those and let `diff_opt_parse()` handle them. After that loop, we have to call `parse_options()` again, to make sure that no unknown options are left. Helped-by: Thomas Gummerer <t.gummerer@gmail.com> Helped-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

When comparing commit messages, we need to keep in mind that they are indented by four spaces. That is, empty lines are no longer empty, but have "trailing whitespace". When displaying them in color, that results in those nagging red lines. Let's just right-trim the lines in the commit message, it's not like trailing white-space in the commit messages are important enough to care about in `git range-diff`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The main information in the `range-diff` view comes from the list of matching and non-matching commits, the diffs are additional information. Indenting them helps with the reading flow. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

When showing the diff between corresponding patches of the two branch versions, we have to make up a fake filename to run the diff machinery. That filename does not carry any meaningful information, hence tbdiff suppresses it. So we should, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

This not only uses "dashed stand-ins" for "pairs" where one side is missing (i.e. unmatched commits that are present only in one of the two commit ranges), but also adds onelines for the reader's pleasure. This change brings `git range-diff` yet another step closer to feature parity with tbdiff: it now shows the oneline, too, and indicates with `=` when the commits have identical diffs. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

We are comparing complete, formatted commit messages with patches. There are no function names here, so stop looking for them. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

These are essentially lifted from https://github.com/trast/tbdiff, with light touch-ups to account for the command now being named `git range-diff`. Apart from renaming `tbdiff` to `range-diff`, only one test case needed to be adjusted: 11 - 'changed message'. The underlying reason it had to be adjusted is that diff generation is sometimes ambiguous. In this case, a comment line and an empty line are added, but it is ambiguous whether they were added after the existing empty line, or whether an empty line and the comment line are added *before* the existing empty line. And apparently xdiff picks a different option here than Python's difflib. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Arguably the most important part of `git range-diff`'s output is the list of commits in the two branches, together with their relationships. For that reason, tbdiff introduced color-coding that is pretty intuitive, especially for unchanged patches (all dim yellow, like the first line in `git show`'s output) vs modified patches (old commit is red, new commit is green). Let's imitate that color scheme. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

This "color" simply reverts background and foreground. It will be used in the upcoming "dual color" mode of `git range-diff`, where we will reverse colors for the -/+ markers and the fragment headers of the "outer" diff. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

When diffing diffs, it can be quite daunting to figure out what the heck is going on, as there are nested +/- signs. Let's make this easier by adding a flag in diff_options that allows color-coding the outer diff sign with inverted colors, so that the preimage and postimage is colored like the diff it is. Of course, this really only makes sense when the preimage and postimage *are* diffs. So let's not expose this flag via a command-line option for now. This is a feature that was invented by git-tbdiff, and it will be used by `git range-diff` in the next commit, by offering it via a new option: `--dual-color`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

When showing what changed between old and new commits, we show a diff of the patches. This diff is a diff between diffs, therefore there are nested +/- signs, and it can be relatively hard to understand what is going on. With the --dual-color option, the preimage and the postimage are colored like the diffs they are, and the *outer* +/- sign is inverted for clarity. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

When displaying a diff of diffs, it is possible that there is an outer `+` before a context line. That happens when the context changed between old and new commit. When that context line starts with a tab (after the space that marks it as context line), our diff machinery spits out a white-space error (space before tab), but in this case, that is incorrect. Rather than adding a specific whitespace flag that specifically ignores the first space in the output (and might miss other problems with the white-space warnings), let's just skip handling white-space errors in dual color mode to begin with. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The bulk of this patch consists of a heavily butchered version of tbdiff's README written by Thomas Rast and Thomas Gummerer, lifted from https://github.com/trast/tbdiff. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Tab completion of `git range-diff` is very convenient, especially given that the revision arguments to specify the commit ranges to compare are typically more complex than, say, what is normally passed to `git log`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

As pointed out by Elijah Newren, tbdiff has this neat little alignment trick where it outputs the commit pairs with patch numbers that are padded to the maximal patch number's width: 1: cafedead = 1: acefade first patch [...] 314: beefeada < 314: facecab up to PI! Let's do the same in range-diff, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

After using this command extensively for the last two months, this developer came to the conclusion that even if the dual color mode still leaves a lot of room for confusion about what was actually changed, the non-dual color mode is substantially worse in that regard. Therefore, we really want to make the dual color mode the default. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

It *is* a confusing thing to look at a diff of diffs. All too easy is it to mix up whether the -/+ markers refer to the "inner" or the "outer" diff, i.e. whether a `+` indicates that a line was added by either the old or the new diff (or both), or whether the new diff does something different than the old diff. To make things easier to process for normal developers, we introduced the dual color mode which colors the lines according to the commit diff, i.e. lines that are added by a commit (whether old, new, or both) are colored in green. In non-dual color mode, the lines would be colored according to the outer diff: if the old commit added a line, it would be colored red (because that line addition is only present in the first commit range that was specified on the command-line, i.e. the "old" commit, but not in the second commit range, i.e. the "new" commit). However, this dual color mode is still not making things clear enough, as we are looking at two levels of diffs, and we still only pick a color according to *one* of them (the outer diff marker is colored differently, of course, but in particular with deep indentation, it is easy to lose track of that outer diff marker's background color). Therefore, let's add another dimension to the mix. Still use green/red/normal according to the commit diffs, but now also dim the lines that were only in the old commit, and use bold face for the lines that are only in the new commit. That way, it is much easier not to lose track of, say, when we are looking at a line that was added in the previous iteration of a patch series but the new iteration adds a slightly different version: the obsolete change will be dimmed, the current version of the patch will be bold. At least this developer has a much easier time reading the range-diffs that way. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The commit-graph files are binary files, so they should not be compared with 'test_cmp', because that might cause issues like crashing[1] or infinite loop[2] on Windows, where 'test_cmp' is a shell function to deal with random LF-CRLF conversions[3]. Use 'test_cmp_bin' instead. 1 - b93e6e3 (t5000, t5003: do not use test_cmp to compare binary files, 2014-06-04) 2 - f9f3851 (t9300: use test_cmp_bin instead of test_cmp to compare binary files, 2014-09-12) 3 - 4d715ac (Windows: a test_cmp that is agnostic to random LF <> CRLF conversions, 2013-10-26) Reviewed-by: Derrick Stolee <dstolee@microsoft.com> Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Two tests added in dade47c (commit-graph: add repo arg to graph readers, 2018-07-11) prepare the contents of 'expect' files by 'echo'ing the results of command substitutions. That's unncessary, avoid them by directly saving the output of the commands executed in those command substitutions. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Eliminate an unnecessary prompt to continue after failed merger, by not calling the prompt_after_failed_merge function when only one iteration remains. Uses positional parameters to count files in the list to make it easier to see if we have any more paths to process from within the loop. Signed-off-by: Nicholas Guriev <guriev-ns@ya.ru> Signed-off-by: Junio C Hamano <gitster@pobox.com>

chainlint.sed swallows top-level here-docs to avoid being fooled by content which might look like start-of-subshell. It likewise swallows here-docs in subshells to avoid marking content lines as breaking the &&-chain, and to avoid being fooled by content which might look like end-of-subshell, start-of-nested-subshell, or other specially-recognized constructs. At the time of implementation, it was believed that it was not possible to support arbitrary here-doc tag names since 'sed' provides no way to stash the opening tag name in a variable for later comparison against a line signaling end-of-here-doc. Consequently, tag names are hard-coded, with "EOF" being the only tag recognized at the top-level, and only "EOF", "EOT", and "INPUT_END" being recognized within subshells. Also, special care was taken to avoid being confused by here-docs nested within other here-docs. In practice, this limited number of hard-coded tag names has been "good enough" for the 13000+ existing Git test, despite many of those tests using tags other than the recognized ones, since the bodies of those here-docs do not contain content which would fool the linter. Nevertheless, the situation is not ideal since someone writing new tests, and choosing a name not in the "blessed" set could potentially trigger a false-positive. To address this shortcoming, upgrade chainlint.sed to handle arbitrary here-doc tag names, both at the top-level and within subshells. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

A here-doc tag can be quoted ('EOF') or escaped (\EOF) to suppress interpolation within the body. Although, chainlint recognizes escaped tags, it does not know about quoted tags. For completeness, teach it to recognize quoted tags, as well. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

For multi-line $(...) expressions nested within subshells, chainlint.sed only recognizes: x=$( echo foo && ... but it is not unlikely that test authors may also cuddle the command with the opening "$(", so support that style, as well: x=$(echo foo && ... The closing ")" is already correctly recognized when cuddled or not. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

After swallowing a here-doc, chainlint.sed assumes that no other processing needs to be done on the line aside from checking for &&-chain breakage; likewise, after folding a multi-line quoted string. However, it's conceivable (even if unlikely in practice) that both a here-doc and a multi-line quoted string might commence on the same line: cat <<\EOF && echo "foo bar" data EOF Support this case by sending the line (after swallowing and folding) through the normal processing sequence rather than jumping directly to the check for broken &&-chain. This change also allows other somewhat pathological cases to be handled, such as closing a subshell on the same line starting a here-doc: ( cat <<-\INPUT) data INPUT or, for instance, opening a multi-line $(...) expression on the same line starting a here-doc: x=$(cat <<-\END && data END echo "x") among others. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

chainlint.sed recognizes multi-line quoted strings within subshells: echo "abc def" >out && so it can avoid incorrectly classifying lines internal to the string as breaking the &&-chain. To identify the first line of a multi-line string, it checks if the line contains a single quote. However, this is fragile and can be easily fooled by a line containing multiple strings: echo "xyz" "abc def" >out && Make detection more robust by checking for an odd number of quotes rather than only a single one. (Escaped quotes are not handled, but support may be added later.) The original multi-line string recognizer rather cavalierly threw away all but the final quote, whereas the new one is careful to retain all quotes, so the "expected" output of a couple existing chainlint tests is updated to account for this new behavior. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

This extract from contrib/subtree/t7900 triggered a false positive due to three chainlint limitations: * recognizing only a "blessed" set of here-doc tag names in a subshell ("EOF", "EOT", "INPUT_END"), of which "TXT" is not a member * inability to recognize multi-line $(...) when the first statement of the body is cuddled with the opening "$(" * inability to recognize multiple constructs on a single line, such as opening a multi-line $(...) and starting a here-doc Now that all of these shortcomings have been addressed, turn this rather pathological bit of shell coding into a chainlint test case. Signed-off-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Correct a comment referring to the removal of just the branch to also refer to the tag. This should have been changed in my ca3065e ("fetch tests: add a tag to be deleted to the pruning tests", 2018-02-09) when the tag deletion was added, but I missed it at the time. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

It should be "is not an empty string" not "is not empty string". This fixes wording originally introduced in ab9b313 ("Documentation: multi-head fetch.", 2005-08-24). Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

These flags were split between cache.h and packfile.h, because some of the flags apply only to packs. However, they share a single numeric namespace, since both are respected for the packed variant. Let's make sure they're defined together so that nobody accidentally adds a new flag in one location that duplicates the other. While we're here, let's also put them in an enum (which helps debugger visibility) and use "(1<<n)" rather than counting powers of 2 manually. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

It's not wrong to pass our flags in an "unsigned", as we know it will be at least as large as the enum. However, using the enum in the declaration makes it more obvious where to find the list of flags. While we're here, let's also drop the "extern" noise-words from the declarations, per our modern coding style. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

We already mention the local/alternate behavior of these functions, but we can help clarify a few other behaviors: - there's no need to mention LOCAL_ONLY specifically, since we already reference the flags by type (and as we add more flags, we don't want to have to mention each) - clarify that reachability doesn't matter here; this is all accessible objects - what ordering/uniqueness guarantees we give - how pack-specific flags are handled for the loose case Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

We currently iterate over objects within a pack in .idx order, which uses the object hashes. That means that it is effectively random with respect to the location of the object within the pack. If you're going to access the actual object data, there are two reasons to move linearly through the pack itself: 1. It improves the locality of access in the packfile. In the cold-cache case, this may mean fewer disk seeks, or better usage of disk cache. 2. We store related deltas together in the packfile. Which means that the delta base cache can operate much more efficiently if we visit all of those related deltas in sequence, as the earlier items are likely to still be in the cache. Whereas if we visit the objects in random order, our cache entries are much more likely to have been evicted by unrelated deltas in the meantime. So in general, if you're going to access the object contents pack order is generally going to end up more efficient. But if you're simply generating a list of object names, or if you're going to end up sorting the result anyway, you're better off just using the .idx order, as finding the pack order means generating the in-memory pack-revindex. According to the numbers in 8b8dfd5 (pack-revindex: radix-sort the revindex, 2013-07-11), that takes about 200ms for linux.git, and 20ms for git.git (those numbers are a few years old but are still a good ballpark). That makes it a good optimization for some cases (we can save tens of seconds in git.git by having good locality of delta access, for a 20ms cost), but a bad one for others (e.g., right now "cat-file --batch-all-objects --batch-check="%(objectname)" is 170ms in git.git, so adding 20ms to that is noticeable). Hence this patch makes it an optional flag. You can't actually do any interesting timings yet, as it's not plumbed through to any user-facing tools like cat-file. That will come in a later patch. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The test for --batch-all-objects in t1006 covers a variety of object storage situations, but one thing it doesn't cover is that we avoid mentioning duplicate objects. We won't have any because running "git repack -ad" will have packed them all and deleted the loose ones. This does work (because we sort and de-dup the output list), but it's good to include it in our test. And doubly so for when we add an unordered mode which has to de-dup in a different way. Note that we cannot just re-create one of the objects, as Git will omit the write of an object that is already present. However, we can create a new pack with one of the objects, which forces the duplication. One alternative would be to just use "git repack -a" instead of "-ad". But then _every_ object would be duplicated as loose and packed, and we might miss a bug that omits packed objects (because we'd show their loose counterparts). Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

We're not really doing the batch-show operation in these callbacks, but just collecting the set of objects. That distinction will become more important in a future patch, so let's rename them now to avoid cluttering that diff. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

If you're going to access the contents of every object in a packfile, it's generally much more efficient to do so in pack order, rather than in hash order. That increases the locality of access within the packfile, which in turn is friendlier to the delta base cache, since the packfile puts related deltas next to each other. By contrast, hash order is effectively random, since the sha1 has no discernible relationship to the content. This patch introduces an "--unordered" option to cat-file which iterates over packs in pack-order under the hood. You can see the results when dumping all of the file content: $ time ./git cat-file --batch-all-objects --buffer --batch | wc -c 6883195596 real 0m44.491s user 0m42.902s sys 0m5.230s $ time ./git cat-file --unordered \ --batch-all-objects --buffer --batch | wc -c 6883195596 real 0m6.075s user 0m4.774s sys 0m3.548s Same output, different order, way faster. The same speed-up applies even if you end up accessing the object content in a different process, like: git cat-file --batch-all-objects --buffer --batch-check | grep blob | git cat-file --batch='%(objectname) %(rest)' | wc -c Adding "--unordered" to the first command drops the runtime in git.git from 24s to 3.5s. Side note: there are actually further speedups available for doing it all in-process now. Since we are outputting the object content during the actual pack iteration, we know where to find the object and could skip the extra lookup done by oid_object_info(). This patch stops short of that optimization since the underlying API isn't ready for us to make those sorts of direct requests. So if --unordered is so much better, why not make it the default? Two reasons: 1. We've promised in the documentation that --batch-all-objects outputs in hash order. Since cat-file is plumbing, people may be relying on that default, and we can't change it. 2. It's actually _slower_ for some cases. We have to compute the pack revindex to walk in pack order. And our de-duplication step uses an oidset, rather than a sort-and-dedup, which can end up being more expensive. If we're just accessing the type and size of each object, for example, like: git cat-file --batch-all-objects --buffer --batch-check my best-of-five warm cache timings go from 900ms to 1100ms using --unordered. Though it's possible in a cold-cache or under memory pressure that we could do better, since we'd have better locality within the packfile. And one final question: why is it "--unordered" and not "--pack-order"? The answer is again two-fold: 1. "pack order" isn't a well-defined thing across the whole set of objects. We're hitting loose objects, as well as objects in multiple packs, and the only ordering we're promising is _within_ a single pack. The rest is apparently random. 2. The point here is optimization. So we don't want to promise any particular ordering, but only to say that we will choose an ordering which is likely to be efficient for accessing the object content. That leaves the door open for further changes in the future without having to add another compatibility option. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

This code is only needed for diff-tree (since f0c6b2a ([PATCH] Optimize diff-tree -[CM] --stdin - 2005-05-27)). Let the caller do the preparation instead and avoid read_index() in diff.c code. read_index() should be avoided (in addition to the_index) because it uses get_index_file() underneath to get the path $GIT_DIR/index. This effectively pulls the_repository in and may become the only reason to pull a 'struct repository *' in diff.c. Let's keep the dependencies as few as possible and kick it back to diff-tree.c Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

This puts update_main_cache_tree() and write_cache_as_tree() in the same group of "index compat" functions that assume the_index implicitly, which should only be used within builtin/ or t/helper. sequencer.c is also updated to not use these functions. As of now, no files outside builtin/ use these functions anymore. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Make the attr API take an index_state instead of assuming the_index in attr code. All call sites are converted blindly to keep the patch simple and retain current behavior. Individual call sites may receive further updates to use the right index instead of the_index. There is one ugly temporary workaround added in attr.c that needs some more explanation. Commit c24f3ab (apply: file commited with CRLF should roundtrip diff and apply - 2017-08-19) forces one convert_to_git() call to NOT read the index at all. But what do you know, we read it anyway by falling back to the_index. When "istate" from convert_to_git is now propagated down to read_attr_from_array() we will hit segfault somewhere inside read_blob_data_from_index. The right way of dealing with this is to kill "use_index" variable and only follow "istate" but at this stage we are not ready for that: while most git_attr_set_direction() calls just passes the_index to be assigned to use_index, unpack-trees passes a different one which is used by entry.c code, which has no way to know what index to use if we delete use_index. So this has to be done later. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Make the convert API take an index_state instead of assuming the_index in convert.c. All external call sites are converted blindly to keep the patch simple and retain current behavior. Individual call sites may receive further updates to use the right index instead of the_index. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Make the match_patchspec API and friends take an index_state instead of assuming the_index in dir.c. All external call sites are converted blindly to keep the patch simple and retain current behavior. Individual call sites may receive further updates to use the right index instead of the_index. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

write_eolinfo() does take an istate as function argument and it should be used instead of the_index. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

This function mark_new_skip_worktree() has an argument named the_index which is also the name of a global variable. While they have different types (the global the_index is not a pointer) mistakes can easily happen and it's also confusing for readers. Rename the function argument to something other than the_index. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Prior to fba92be, this code implicitly (and incorrectly) assumes the_index when running the exclude machinery. fba92be helps show this problem clearer because unpack-trees operation is supposed to work on whatever index the caller specifies... not specifically the_index. Update the code to use "istate" argument that's originally from mark_new_skip_worktree(). From the call sites, both in unpack_trees(), you can see that this function works on two separate indexes: o->src_index and o->result. The second mark_new_skip_worktree() so far has incorecctly applied exclude rules on o->src_index instead of o->result. It's unclear what is the consequences of this, but it's definitely wrong. [1] fba92be (dir: convert is_excluded_from_list to take an index - 2017-05-05) Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Both functions that are updated in this commit are called by verify_absent(), which is part of the "unpack-trees" operation that is supposed to work on any index file specified by the caller. Thanks to Brandon [1] [2], an implicit dependency on the_index is exposed. This commit fixes it. In both functions, it makes sense to use src_index to check for exclusion because it's almost unchanged and should give us the same outcome as if running the exclude check before the unpack. It's "almost unchanged" because we do invalidate cache-tree and untracked cache in the source index. But this should not affect how exclude machinery uses the index: to see if a file is tracked, and to read a blob from the index instead of worktree if it's marked skip-worktree (i.e. it's not available in worktree) [1] a0bba65 (dir: convert is_excluded to take an index - 2017-05-05 [2] 2c1eb10 (dir: convert read_directory to take an index - 2017-05-05) Helped-by: Elijah Newren <newren@gmail.com> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

checkout-index.c needs update because if checkout->istate is NULL, ie_match_stat() will crash. Previously this is ie_match_stat(&the_index, ..) so it will not crash, but it is not technically correct either. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Since attr checking API now take the index, there's no need to set an index in advance with this call. Most call sites are straightforward because they either pass the_index or NULL (which defaults back to the_index previously). There's only one suspicious call site in unpack-trees.c where it sets a different index. This code in unpack-trees is about to check out entries from the new/temporary index after merging is done in it. The attributes will be used by entry.c code to do crlf conversion if needed. entry.c now respects struct checkout's istate field, and this field is correctly set in unpack-trees.c, there should be no regression from this change. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

With 'struct archive_args' gaining new repository pointer, we don't have to assume the_repository in the archive backends anymore. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

we're going to remove the dependency on the_index by moving 'struct index_state *' to somewhere inside struct apply_state. Let's make sure relevant functions have access to this struct now and reduce the diff noise when the actual conversion happens. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

We're moving away from the_index in this code. "struct index_state *" could be added to struct apply_state. But let's aim long term and put struct repository here instead so that we could even avoid more global states in the future. The index will be available via apply_state->repo->index. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Use apply_state->repo->index instead of the_index (in most cases, unless we need to use a temporary index in some functions). Let the callers (am and apply) tell us what to use, instead of always assuming to operate on the_index. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Side note, since we gain access to the right repository, we can stop rely on the_repository in this code as well. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The Windows CRT implements O_APPEND "manually": on write() calls, the file pointer is set to EOF before the data is written. Clearly, this is not atomic. And in fact, this is the root cause of failures observed in t5552-skipping-fetch-negotiator.sh and t5503-tagfollow.sh, where different processes write to the same trace file simultanously; it also occurred in t5400-send-pack.sh, but there it was worked around in 71406ed ("t5400: avoid concurrent writes into a trace file", 2017-05-18). Fortunately, Windows does support atomic O_APPEND semantics using the file access mode FILE_APPEND_DATA. Provide an implementation that does. This implementation is minimal in such a way that it only implements the open modes that are actually used in the Git code base. Emulation for other modes can be added as necessary later. To become aware of the necessity early, the unusal error ENOSYS is reported if an unsupported mode is encountered. Diagnosed-by: Johannes Schindelin <Johannes.Schindelin@gmx.de> Helped-by: Jeff Hostetler <git@jeffhostetler.com> Signed-off-by: Johannes Sixt <j6t@kdbg.org> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The test 'pack-objects to file can use bitmap' added in 645c432 (pack-objects: use reachability bitmap index when generating non-stdout pack, 2016-09-10) is silently buggy and doesn't check what it's supposed to. In 't5310-pack-bitmaps.sh', the 'list_packed_objects' helper function does what its name implies by running: git show-index <"$1" | cut -d' ' -f2 The test in question invokes this function like this: list_packed_objects <packa-$packasha1.idx >packa.objects && list_packed_objects <packb-$packbsha1.idx >packb.objects && test_cmp packa.objects packb.objects Note how these two callsites don't specify the name of the pack index file as the function's parameter, but redirect the function's standard input from it. This triggers an error message from the shell, as it has no filename to redirect from in the function, but this error is ignored, because it happens upstream of a pipe. Consequently, both invocations produce empty 'pack{a,b}.objects' files, and the subsequent 'test_cmp' happily finds those two empty files identical. Fix these two 'list_packed_objects' invocations by specifying the pack index files as parameters. Furthermore, eliminate the pipe in that function by replacing it with an &&-chained pair of commands using an intermediate file, so a failure of 'git show-index' or the shell redirection will fail the test. Signed-off-by: SZEDER Gábor <szeder.dev@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

We don't need to check if the oidset has our object before we insert it; that's done as part of the insertion. We can just rely on the return value from oidset_insert(), which saves one hash lookup per object. This measurable speedup is tiny and within the run-to-run noise, but the result is simpler to read, too. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

We use the "buf" strbuf for two things: to read incoming lines, and as a scratch space for test-expanding the user-provided format. Let's split this into two variables with descriptive names, which makes their purpose and lifetime more clear. It will also help in a future patch when we start using the "output" buffer for more expansions. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

When we're in batch mode, we end up in batch_object_write() for each object, which allocates its own strbuf for each call. Instead, we can provide a single "scratch" buffer that gets reused for each output. When running: git cat-file --batch-all-objects --batch-check='%(objectname)' on git.git, my best-of-five time drops from: real 0m0.171s user 0m0.159s sys 0m0.012s to: real 0m0.133s user 0m0.121s sys 0m0.012s Note that we could do this just by putting the "scratch" pointer into "struct expand_data", but I chose instead to add an extra parameter to the callstack. That's more verbose, but it makes it a bit more obvious what is going on, which in turn makes it easy to see where we need to be releasing the string in the caller (right after the loop which uses it in each case). Based-on-a-patch-by: René Scharfe <l.s.r@web.de> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The for_each_loose_object() and for_each_packed_object() functions are meant to be part of a unified interface: they use the same set of for_each_object_flags, and it's not inconceivable that we might one day add a single for_each_object() wrapper around them. Let's put them together in a single file, so we can avoid awkwardness like saying "the flags for this function are over in cache.h". Moving the loose functions to packfile.h is silly. Moving the packed functions to cache.h works, but makes the "cache.h is a kitchen sink" problem worse. The best place is the recently-created object-store.h, since these are quite obviously related to object storage. The for_each_*_in_objdir() functions do not use the same flags, but they are logically part of the same interface as for_each_loose_object(), and share callback signatures. So we'll move those, as well, as they also make sense in object-store.h. Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>

In a56771a (builtin/pull: respect verbosity settings in submodules, 2018-01-25), we made sure to pass on both quiet and verbose flag from builtin/pull.c to the submodule shell script. However git-submodule doesn't understand a verbose flag, which results in a bug when invoking git pull --recurse-submodules -v [...] There are a few different approaches to fix this bug: 1) rewrite 'argv_push_verbosity' or its caller in builtin/pull.c to cap opt_verbosity at 0. Then 'argv_push_verbosity' would only add '-q' if any. 2) Have a flag in 'argv_push_verbosity' that specifies if we allow adding -q or -v (or both). 3) Add -v to git-submodule.sh and make it a no-op (1) seems like a maintenance burden: What if we add code after the submodule operations or move submodule operations higher up, then we have altered the opt_verbosity setting further down the line in builtin/pull.c. (2) seems like it could work reasonably well without more regressions (3) seems easiest to implement as well as actually is a feature with the last-one-wins rule of passing flags to Git commands. Reported-by: Jochen Kühner Signed-off-by: Stefan Beller <sbeller@google.com> Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The tests added in 63e95be ("submodule: port resolve_relative_url from shell to C", 2016-04-15) didn't do a good job of testing various up-path invocations where the up-path would bring us beyond even the URL in question without emitting an error. These results look nonsensical, but it's worth exhaustively testing them before fixing any of this code, so we can see which of these cases were changed. Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Acked-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Rendered documentation can be easier to read than raw text because headings and emphasized phrases stand out. Add the missing markup and Makefile rule required to render this design document using asciidoc. Tested by running make -C Documentation technical/partial-clone.html and viewing the output in a browser. Signed-off-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The `chainlint` target compares actual output to expected output, where the actual output is generated from files that are specifically checked out with LF-only line endings. So the expected output needs to be checked out with LF-only line endings, too. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Acked-by: Eric Sunshine <sunshine@sunshineco.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Commit e12a7ef ("rebase -i: Handle "combination of <n> commits" with GETTEXT_POISON", 2018-04-27) changed the way that individual commit messages are labelled when squashing commits together. In doing so a regression was introduced where the numbering of the messages is off by one. This commit fixes that and adds a test for the numbering. Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk> Signed-off-by: Junio C Hamano <gitster@pobox.com>

I looped over the toplevel header files, creating a temporary two-line C program for each consisting of #include "git-compat-util.h" #include $HEADER This patch is the result of manually fixing errors in compiling those tiny programs. Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Since both functions are using the same data type, they should either both refer to it as void *, or both use the real type (struct alloc_state *). Opt for the latter. Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

'branch_track' feels more closely related to branching, and it is needed later in branch.h; rather than #include'ing cache.h in branch.h for this small enum, just move the enum and the external declaration for git_branch_track to branch.h. Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Reviewed-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

According to http://c-faq.com/null/machexamp.html, sizeof(char*) != sizeof(int*) on some platforms. Since an enum could be a char or int (or long or...), knowing the size of the enum thus is important to knowing the size of a pointer to an enum, so we cannot just forward declare an enum the way we can a struct. (Also, modern C++ compilers apparently define forward declarations of an enum to either be useless because the enum was defined, or require an explicit size specifier, or be a compilation error.) Helped-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Elijah Newren <newren@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Doc formatting fix. * jk/ui-color-always-to-auto: Documentation: fix --color option formatting

Doc update. * sb/trailers-docfix: Documentation/git-interpret-trailers: explain possible values

Test update. * sg/fast-import-dump-refs-on-checkpoint-fix: t9300: wait for background fast-import process to die after killing it

i18n updates. * bw/fetch-pack-i18n: fetch-pack: mark die strings for translation

Test fix. * js/t7406-recursive-submodule-update-order-fix: t7406: avoid failures solely due to timing issues

Update the way we run static analysis tool at TravisCI to make it easier to use its findings. * sg/travis-cocci-diagnose-failure: travis-ci: fail if Coccinelle static analysis found something to transform travis-ci: run Coccinelle static analysis with two parallel jobs

Doc update. * bw/protocol-v2: pack-protocol: mention and point to docs for protocol v2

One of the "diff --color-moved" mode "dimmed_zebra" that was named in an unusual way has been deprecated and replaced by "dimmed-zebra". * es/diff-color-moved-fix: diff: --color-moved: rename "dimmed_zebra" to "dimmed-zebra"

"make DEVELOPER=1 DEVOPTS=pedantic" allows developers to compile with -pedantic option, which may catch more problematic program constructs and potential bugs. * bb/make-developer-pedantic: Makefile: add a DEVOPTS flag to get pedantic compilation

Preparatory code to later add json output for telemetry data. * jh/json-writer: json_writer: new routines to create JSON data

A new configuration variable core.usereplacerefs has been added, primarily to help server installations that want to ignore the replace mechanism altogether. * jk/core-use-replace-refs: add core.usereplacerefs config option check_replace_refs: rename to read_replace_refs check_replace_refs: fix outdated comment

The wire-protocol v2 relies on the client to send "ref prefixes" to limit the bandwidth spent on the initial ref advertisement. "git clone" when learned to speak v2 forgot to do so, which has been corrected. * bw/clone-ref-prefixes: clone: send ref-prefixes when using protocol v2

Teach "git tag -s" etc. a few configuration variables (gpg.format that can be set to "openpgp" or "x509", and gpg.<format>.program that is used to specify what program to use to deal with the format) to allow x.509 certs with CMS via "gpgsm" to be used instead of openpgp via "gnupg". * hs/gpgsm: gpg-interface t: extend the existing GPG tests with GPGSM gpg-interface: introduce new signature format "x509" using gpgsm gpg-interface: introduce new config to select per gpg format program gpg-interface: do not hardcode the key string len anymore gpg-interface: introduce an abstraction for multiple gpg formats t/t7510: check the validation of the new config gpg.format gpg-interface: add new config to select how to sign a commit

Many more strings are prepared for l10n. * nd/i18n: (23 commits) transport-helper.c: mark more strings for translation transport.c: mark more strings for translation sha1-file.c: mark more strings for translation sequencer.c: mark more strings for translation replace-object.c: mark more strings for translation refspec.c: mark more strings for translation refs.c: mark more strings for translation pkt-line.c: mark more strings for translation object.c: mark more strings for translation exec-cmd.c: mark more strings for translation environment.c: mark more strings for translation dir.c: mark more strings for translation convert.c: mark more strings for translation connect.c: mark more strings for translation config.c: mark more strings for translation commit-graph.c: mark more strings for translation builtin/replace.c: mark more strings for translation builtin/pack-objects.c: mark more strings for translation builtin/grep.c: mark strings for translation builtin/config.c: mark more strings for translation ...

"git diff --histogram" had a bad memory usage pattern, which has been rearranged to reduce the peak usage. * sb/histogram-less-memory: xdiff/histogram: remove tail recursion xdiff/xhistogram: move index allocation into find_lcs xdiff/xhistogram: factor out memory cleanup into free_index() xdiff/xhistogram: pass arguments directly to fall_back_to_classic_diff

Update the way we use Coccinelle to find out-of-style code that need to be modernised. * sg/coccicheck-updates: coccinelle: extract dedicated make target to clean Coccinelle's results coccinelle: put sane filenames into output patches coccinelle: exclude sha1dc source files from static analysis coccinelle: use $(addsuffix) in 'coccicheck' make target coccinelle: mark the 'coccicheck' make target as .PHONY

Code clean-up to use size_t/ssize_t when they are the right type. * jk/size-t: strbuf_humanise: use unsigned variables pass st.st_size as hint for strbuf_readlink() strbuf_readlink: use ssize_t strbuf: use size_t for length in intermediate variables reencode_string: use size_t for string lengths reencode_string: use st_add/st_mult helpers

The wire-protocol v2 relies on the client to send "ref prefixes" to limit the bandwidth spent on the initial ref advertisement. "git fetch $remote branch:branch" that asks tags that point into the history leading to the "branch" automatically followed sent to narrow prefix and broke the tag following, which has been fixed. * jt/tag-following-with-proto-v2-fix: fetch: send "refs/tags/" prefix upon CLI refspecs t5702: test fetch with multiple refspecs at a time

When the sparse checkout feature is in use, "git cherry-pick" and other mergy operations lost the skip_worktree bit when a path that is excluded from checkout requires content level merge, which is resolved as the same as the HEAD version, without materializing the merge result in the working tree, which made the path appear as deleted. This has been corrected by preserving the skip_worktree bit (and not materializing the file in the working tree). * en/merge-recursive-skip-fix: merge-recursive: preserve skip_worktree bit when necessary t3507: add a testcase showing failure with sparse checkout

It is too easy to misuse system API functions such as strcat(); these selected functions are now forbidden in this codebase and will cause a compilation failure. * jk/banned-function: banned.h: mark strncpy() as banned banned.h: mark sprintf() as banned banned.h: mark strcat() as banned automatically ban strcpy()

Compilation fix. * bb/redecl-enum-fix: packfile: ensure that enum object_type is defined

Add a script (in contrib/) to help users of VSCode work better with our codebase. * js/vscode: vscode: let cSpell work on commit messages, too vscode: add a dictionary for cSpell vscode: use 8-space tabs, no trailing ws, etc for Git's source code vscode: wrap commit messages at column 72 by default vscode: only overwrite C/C++ settings mingw: define WIN32 explicitly cache.h: extract enum declaration from inside a struct declaration vscode: hard-code a couple defines contrib: add a script to initialize VS Code configuration

"git p4 submit" learns to ask its own pre-submit hook if it should continue with submitting. * cb/p4-pre-submit-hook: git-p4: add the `p4-pre-submit` hook

Test fix. * jn/subtree-test-fixes: subtree test: simplify preparation of expected results subtree test: add missing && to &&-chain

Doc fix. * nd/pack-objects-threading-doc: pack-objects: document about thread synchronization

Doc fix. * ms/http-proto-doc: doc: fix want-capability separator

Test fix. * es/mw-to-git-chain-fix: mw-to-git/t9360: fix broken &&-chain

Leakfix. * rs/remote-mv-leakfix: remote: clear string_list after use in mv()

The Travis CI scripts were taught to ship back the test data from failed tests. * sg/travis-retrieve-trash-upon-failure: travis-ci: include the trash directories of failed tests in the trace log

"git fetch" sometimes failed to update the remote-tracking refs, which has been corrected. * jt/connectivity-check-after-unshallow: fetch-pack: unify ref in and out param

Signed-off-by: Junio C Hamano <gitster@pobox.com>

Move the creation of conflicting-G from a test to the setup so that it can be used in subsequent tests without creating the kind of implicit dependencies that plague t3404. While we're at it simplify the arguments to the test_commit() call the creates the conflicting commit. Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk> Signed-off-by: Junio C Hamano <gitster@pobox.com>

If a merge command in the todo list specifies just a branch to merge with no -C/-c argument then item->commit is NULL. This means that if there are merge conflicts error_with_patch() is passed a NULL commit which causes a segmentation fault when make_patch() tries to look it up. This commit implements a minimal fix which fixes the crash and allows the user to successfully commit a conflict resolution with 'git rebase --continue'. It does not write .git/rebase-merge/patch, .git/rebase-merge/stopped-sha or update REBASE_HEAD. To sensibly get the hashes of the merge parents would require refactoring do_merge() to extract the code that parses the merge parents into a separate function which error_with_patch() could then use to write the parents into the stopped-sha file. To create meaningful output make_patch() and 'git rebase --show-current-patch' would also need to be modified to diff the merge parent and merge base in this case. Signed-off-by: Phillip Wood <phillip.wood@dunelm.org.uk> Acked-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Junio C Hamano <gitster@pobox.com>

--quit is supposed to be --abort but without restoring HEAD. Leaving CHERRY_PICK_HEAD behind could make other commands mistake that cherry-pick is still ongoing (e.g. "git commit --amend" will refuse to work). Clean it too. For --abort, this job of deleting CHERRY_PICK_HEAD is on "git reset" so we don't need to do anything else. But let's add extra checks in --abort tests to confirm. Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

While at it fix a typo (s/independed/independent) and make sure git is not in a chain of pipes. Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Signed-off-by: Stefan Beller <sbeller@google.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

The test performed at the receiving end of "git push" to prevent bad objects from entering repository can be customized via receive.fsck.* configuration variables; we now have gained a counterpart to do the same on the "git fetch" side, with fetch.fsck.* configuration variables. * ab/fsck-transfer-updates: fsck: test and document unknown fsck.<msg-id> values fsck: add stress tests for fsck.skipList fsck: test & document {fetch,receive}.fsck.* config fallback fetch: implement fetch.fsck.* transfer.fsckObjects tests: untangle confusing setup config doc: elaborate on fetch.fsckObjects security config doc: elaborate on what transfer.fsckObjects does config doc: unify the description of fsck.* and receive.fsck.* config doc: don't describe *.fetchObjects twice receive.fsck.<msg-id> tests: remove dead code

The "author-script" file "git rebase -i" creates got broken when we started to move the command away from shell script, which is getting fixed now. * es/rebase-i-author-script-fix: sequencer: don't die() on bogus user-edited timestamp sequencer: fix "rebase -i --root" corrupting author header timestamp sequencer: fix "rebase -i --root" corrupting author header timezone sequencer: fix "rebase -i --root" corrupting author header

Test updates. * ab/test-must-be-empty: tests: make use of the test_must_be_empty function

The automatic tree-matching in "git merge -s subtree" was broken 5 years ago and nobody has noticed since then, which is now fixed. * jk/merge-subtree-heuristics: score_trees(): fix iteration over trees with missing entries

"git fetch $there refs/heads/s" ought to fetch the tip of the branch 's', but when "refs/heads/refs/heads/s", i.e. a branch whose name is "refs/heads/s" exists at the same time, fetched that one instead by mistake. This has been corrected to honor the usual disambiguation rules for abbreviated refnames. * jt/refspec-dwim-precedence-fix: remote: make refspec follow the same disambiguation rule as local refs

Update to a few other topics around 'git fetch'. * ab/fetch-nego: fetch doc: cross-link two new negotiation options negotiator: unknown fetch.negotiationAlgorithm should error out

The parse-options machinery learned to refrain from enclosing placeholder string inside a "<bra" and "ket>" pair automatically without PARSE_OPT_LITERAL_ARGHELP. Existing help text for option arguments that are not formatted correctly have been identified and fixed. * rs/parse-opt-lithelp: parse-options: automatically infer PARSE_OPT_LITERAL_ARGHELP shortlog: correct option help for -w send-pack: specify --force-with-lease argument help explicitly pack-objects: specify --index-version argument help explicitly difftool: remove angular brackets from argument help add, update-index: fix --chmod argument help push: use PARSE_OPT_LITERAL_ARGHELP instead of unbalanced brackets

AIX portability update for the SHA1DC hash, imported from upstream. * ab/sha1dc: sha1dc: update from upstream

Futureproofing a helper function that can easily be misused. * es/want-color-fd-defensive: color: protect against out-of-bounds reads and writes

Noiseword "extern" has been removed from function decls in the header files. * nd/no-extern: submodule.h: drop extern from function declaration revision.h: drop extern from function declaration repository.h: drop extern from function declaration rerere.h: drop extern from function declaration line-range.h: drop extern from function declaration diff.h: remove extern from function declaration diffcore.h: drop extern from function declaration convert.h: drop 'extern' from function declaration cache-tree.h: drop extern from function declaration blame.h: drop extern on func declaration attr.h: drop extern from function declaration apply.h: drop extern on func declaration

A few atoms like %(objecttype) and %(objectsize) in the format specifier of "for-each-ref --format=<format>" can be filled without getting the full contents of the object, but just with the object header. These cases have been optimized by calling oid_object_info() API (instead of reading and inspecting the data). * ot/ref-filter-object-info: ref-filter: use oid_object_info() to get object ref-filter: merge get_obj and get_object ref-filter: initialize eaten variable ref-filter: fill empty fields with empty values ref-filter: add info_source to valid_atom

The http-backend (used for smart-http transport) used to slurp the whole input until EOF, without paying attention to CONTENT_LENGTH that is supplied in the environment and instead expecting the Web server to close the input stream. This has been fixed. * mk/http-backend-content-length: t5562: avoid non-portable "export FOO=bar" construct http-backend: respect CONTENT_LENGTH for receive-pack http-backend: respect CONTENT_LENGTH as specified by rfc3875 http-backend: cleanup writing to child process

"git merge --abort" etc. did not clean things up properly when there were conflicted entries in the index in certain order that are involved in D/F conflicts. This has been corrected. * en/abort-df-conflict-fixes: read-cache: fix directory/file conflict handling in read_index_unmerged() t1015: demonstrate directory/file conflict recovery failures

"git diff --indent-heuristic" had a bad corner case performance. * sb/indent-heuristic-optim: xdiff: reduce indent heuristic overhead

Test fix. * en/t3031-title-fix: t3031: update test description to mention desired behavior

Doc fix. * nd/config-blame-sort: config.txt: reorder blame stuff to keep config keys sorted

Header update. * hn/config-in-code-comment: config: document git config getter return value

The end result of documentation update has been made to be inspected more easily to help developers. * jk/diff-rendered-docs: add a script to diff rendered documentation

"git pull --rebase=interactive" learned "i" as a short-hand for "interactive". * js/pull-rebase-type-shorthand: pull --rebase=<type>: allow single-letter abbreviations for the type

Test fix. * ar/t4150-am-scissors-test-fix: t4150: fix broken test for am --scissors

Build fix. * nd/complete-config-vars: Makefile: add missing dependency for command-list.h

Signed-off-by: Junio C Hamano <gitster@pobox.com>

The caller of maybe_colorize_sideband() gives a counted buffer <src, n>, but the callee checked src[] as if it were a NUL terminated buffer. If src[] had all isspace() bytes in it, we would have made n negative, and then (1) made number of strncasecmp() calls to see if the remaining bytes in src[] matched keywords, reading beyond the end of the array (this actually happens even if n does not go negative), and/or (2) called strbuf_add() with negative count, most likely triggering the "you want to use way too much memory" error due to unsigned integer overflow. Fix both issues by making sure we do not go beyond &src[n]. In the longer term we may want to accept size_t as parameter for clarity (even though we know that a sideband message we are painting typically would fit on a line on a terminal and int is sufficient). Write it down as a NEEDSWORK comment. Helped-by: Jonathan Nieder <jrnieder@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Checkout with the -p switch uses the "add interactive" framework which is written in Perl. One test added in 8d7b558 ("checkout & worktree: introduce checkout.defaultRemote", 2018-06-05) didn't declare the PERL prerequisite, breaking the test when built with NO_PERL. Reported-by: CB Bailey <cb@hashpling.org> Signed-off-by: CB Bailey <cb@hashpling.org> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com> Signed-off-by: Junio C Hamano <gitster@pobox.com>

Documentation update. * ab/newhash-is-sha256: doc hash-function-transition: pick SHA-256 as NewHash doc hash-function-transition: note the lack of a changelog

Test updates. * ab/test-must-be-empty-for-master: tests: make use of the test_must_be_empty function

The "--exec" option to "git rebase --rebase-merges" placed the exec commands at wrong places, which has been corrected. * js/rebase-merges-exec-fix: rebase --exec: make it work with --rebase-merges t3430: demonstrate what -r, --autosquash & --exec should do

Test fixes. * en/t7406-fixes: t7406: avoid using test_must_fail for commands other than git t7406: prefer test_* helper functions to test -[feds] t7406: avoid having git commands upstream of a pipe t7406: simplify by using diff --name-only instead of diff --raw t7406: fix call that was failing for the wrong reason

"git instaweb" has been adjusted to run better with newer Apache on RedHat based distros. * sk/instaweb-rh-update: git-instaweb: fix apache2 config with apache >= 2.4 git-instaweb: support Fedora/Red Hat apache module path

Comment update. * js/typofixes: remote-curl: remove spurious period git-compat-util.h: fix typo

Doc update. * en/update-index-doc: git-update-index.txt: reword possibly confusing example

Doc update. * jc/update-index-doc: update-index: there no longer is `apply --index-info`

"git verify-tag" and "git verify-commit" have been taught to use the exit status of underlying "gpg --verify" to signal bad or untrusted signature they found. * jc/gpg-status: gpg-interface: propagate exit status from gpg back to the callers

Test fix. * jt/fetch-negotiator-skipping: t5552: suppress upload-pack trace output

Test fix. * ds/commit-graph-fsck: t5318: use 'test_cmp_bin' to compare commit-graph files

Test update. * jt/commit-graph-per-object-store: t5318: avoid unnecessary command substitutions

"git mergetool" stopped and gave an extra prompt to continue after the last path has been handled, which did not make much sense. * ng/mergetool-lose-final-prompt: mergetool: don't suggest to continue after last file

Test and doc clean-ups. * ab/fetch-tags-noclobber: pull doc: fix a long-standing grammar error fetch tests: correct a comment "remove it" -> "remove them" push tests: assert re-pushing annotated tags push tests: add more testing for forced tag pushing push tests: fix logic error in "push" test assertion push tests: remove redundant 'git push' invocation fetch tests: change "Tag" test tag to "testTag"

The API to iterate over all objects learned to optionally list objects in the order they appear in packfiles, which helps locality of access if the caller accesses these objects while as objects are enumerated. * jk/for-each-object-iteration: for_each_*_object: move declarations to object-store.h cat-file: use a single strbuf for all output cat-file: split batch "buf" into two variables cat-file: use oidset check-and-insert cat-file: support "unordered" output for --batch-all-objects cat-file: rename batch_{loose,packed}_object callbacks t1006: test cat-file --batch-all-objects with duplicates for_each_packed_object: support iterating in pack-order for_each_*_object: give more comprehensive docstrings for_each_*_object: take flag arguments as enum for_each_*_object: store flag definitions in a single location

Among the three codepaths we use O_APPEND to open a file for appending, one used for writing GIT_TRACE output requires O_APPEND implementation that behaves sensibly when multiple processes are writing to the same file. POSIX emulation used in the Windows port has been updated to improve in this area. * js/mingw-o-append: mingw: enable atomic O_APPEND

Test fix. * sg/t5310-empty-input-fix: t5310-pack-bitmaps: fix bogus 'pack-objects to file can use bitmap' test

Improve built-in facility to catch broken &&-chain in the tests. * es/chain-lint-more: chainlint: add test of pathological case which triggered false positive chainlint: recognize multi-line quoted strings more robustly chainlint: let here-doc and multi-line string commence on same line chainlint: recognize multi-line $(...) when command cuddled with "$(" chainlint: match 'quoted' here-doc tags chainlint: match arbitrary here-docs tags rather than hard-coded names

The more library-ish parts of the codebase learned to work on the in-core index-state instance that is passed in by their callers, instead of always working on the singleton "the_index" instance. * nd/no-the-index: (24 commits) blame.c: remove implicit dependency on the_index apply.c: remove implicit dependency on the_index apply.c: make init_apply_state() take a struct repository apply.c: pass struct apply_state to more functions resolve-undo.c: use the right index instead of the_index archive-*.c: use the right repository archive.c: avoid access to the_index grep: use the right index instead of the_index attr: remove index from git_attr_set_direction() entry.c: use the right index instead of the_index submodule.c: use the right index instead of the_index pathspec.c: use the right index instead of the_index unpack-trees: avoid the_index in verify_absent() unpack-trees: convert clear_ce_flags* to avoid the_index unpack-trees: don't shadow global var the_index unpack-trees: add a note about path invalidation unpack-trees: remove 'extern' on function declaration ls-files: correct index argument to get_convert_attr_ascii() preload-index.c: use the right index instead of the_index dir.c: remove an implicit dependency on the_index in pathspec code ...

"git tbdiff" that lets us compare individual patches in two iterations of a topic has been rewritten and made into a built-in command. * js/range-diff: (21 commits) range-diff: use dim/bold cues to improve dual color mode range-diff: make --dual-color the default mode range-diff: left-pad patch numbers completion: support `git range-diff` range-diff: populate the man page range-diff --dual-color: skip white-space warnings range-diff: offer to dual-color the diffs diff: add an internal option to dual-color diffs of diffs color: add the meta color GIT_COLOR_REVERSE range-diff: use color for the commit pairs range-diff: add tests range-diff: do not show "function names" in hunk headers range-diff: adjust the output of the commit pairs range-diff: suppress the diff headers range-diff: indent the diffs just like tbdiff range-diff: right-trim commit messages range-diff: also show the diff between patches range-diff: improve the order of the shown commits range-diff: first rudimentary implementation Introduce `range-diff` to compare iterations of a topic branch ...

"git pull --rebase -v" in a repository with a submodule barfed as an intermediate process did not understand what "-v(erbose)" flag meant, which has been fixed. * sb/pull-rebase-submodule: git-submodule.sh: accept verbose flag in cmd_update to be non-quiet

Test fix. * js/chain-lint-attrfix: chainlint: fix for core.autocrlf=true

Doc updates. * jh/partial-clone-doc: partial-clone: render design doc using asciidoc

A test prerequisite defined by various test scripts with slightly different semantics has been consolidated into a single copy and made into a lazily defined one. * wc/make-funnynames-shared-lazy-prereq: t: factor out FUNNYNAMES as shared lazy prereq

After a partial clone, repeated fetches from promisor remote would have accumulated many packfiles marked with .promisor bit without getting them coalesced into fewer packfiles, hurting performance. "git repack" now learned to repack them. * jt/repack-promisor-packs: repack: repack promisor objects if -a or -A is set repack: refactor setup of pack-objects cmd

Code hygiene improvement for the header files. * en/incl-forward-decl: Remove forward declaration of an enum compat/precompose_utf8.h: use more common include guard style urlmatch.h: fix include guard Move definition of enum branch_track from cache.h to branch.h alloc: make allocate_alloc_state and clear_alloc_state more consistent Add missing includes and forward declarations

Test updates. * ab/submodule-relative-url-tests: submodule: add more exhaustive up-path testing

Recent update to "git config" broke updating variable in a subsection, which has been corrected. * sb/config-write-fix: git-config: document accidental multi-line setting in deprecated syntax config: fix case sensitive subsection names on writing t1300: document current behavior of setting options

When "git rebase -i" is told to squash two or more commits into one, it labeled the log message for each commit with its number. It correctly called the first one "1st commit", but the next one was "commit #1", which was off-by-one. This has been corrected. * pw/rebase-i-squash-number-fix: rebase -i: fix numbering in squash message

"git rebase -i", when a 'merge <branch>' insn in its todo list fails, segfaulted, which has been (minimally) corrected. * pw/rebase-i-merge-segv-fix: rebase -i: fix SIGSEGV when 'merge <branch>' fails t3430: add conflicting commit

A few preliminary minor clean-ups in the area around submodules. * sb/submodule-cleanup: builtin/submodule--helper: remove stray new line t7410: update to new style

"git cherry-pick --quit" failed to remove CHERRY_PICK_HEAD even though we won't be in a cherry-pick session after it returns, which has been corrected. * nd/cherry-pick-quit-fix: cherry-pick: fix --quit not deleting CHERRY_PICK_HEAD

The sideband code learned to optionally paint selected keywords at the beginning of incoming lines on the receiving end. * hn/highlight-sideband-keywords: sideband: do not read beyond the end of input sideband: highlight keywords in remote sideband output

* ab/checkout-default-remote: t2024: mark test using "checkout -p" with PERL prerequisite

Signed-off-by: Junio C Hamano <gitster@pobox.com>

In a recent update in 2.18 era, "git pack-objects" started producing a larger than necessary packfiles by missing opportunities to use large deltas. * nd/pack-deltify-regression-fix: pack-objects: fix performance issues on packing large deltas

This patch helps address the CPU cost of loading the index by creating multiple threads to divide the work of loading and converting the cache entries across all available CPU cores. It accomplishes this by having the primary thread loop across the index file tracking the offset and (for V4 indexes) expanding the name. It creates a thread to process each block of entries as it comes to them. Once the threads are complete and the cache entries are loaded, the rest of the extensions can be loaded and processed normally on the primary thread. I used p0002-read-cache.sh to generate some performance data: 100,000 entries Test HEAD~3 HEAD~2 --------------------------------------------------------------------------- read_cache/discard_cache 1000 times 14.02(0.01+0.12) 9.81(0.01+0.07) -30.0% 1,000,000 entries Test HEAD~3 HEAD~2 ------------------------------------------------------------------------------ read_cache/discard_cache 1000 times 202.06(0.06+0.09) 155.72(0.03+0.06) -22.9% Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>

This patch helps address the CPU cost of loading the index by loading the cache extensions on a worker thread in parallel with loading the cache entries. This is possible because the current extensions don't access the cache entries in the index_state structure so are OK that they don't all exist yet. The CACHE_EXT_TREE, CACHE_EXT_RESOLVE_UNDO, and CACHE_EXT_UNTRACKED extensions don't even get a pointer to the index so don't have access to the cache entries. CACHE_EXT_LINK only uses the index_state to initialize the split index. CACHE_EXT_FSMONITOR only uses the index_state to save the fsmonitor last update and dirty flags. I used p0002-read-cache.sh to generate some performance data on the cumulative impact: 100,000 entries Test HEAD~3 HEAD~2 --------------------------------------------------------------------------- read_cache/discard_cache 1000 times 14.08(0.01+0.10) 9.72(0.03+0.06) -31.0% 1,000,000 entries Test HEAD~3 HEAD~2 ------------------------------------------------------------------------------ read_cache/discard_cache 1000 times 202.95(0.01+0.07) 154.14(0.03+0.06) -24.1% Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>

…arsing. - strbuf_remove() in expand_name_field() is not exactly a good fit for stripping a part at the end, _setlen() would do the same job and is much cheaper. - the open-coded loop to find the end of the string in expand_name_field() can't beat an optimized strlen() I used p0002-read-cache.sh to generate some performance data on the cumulative impact: 100,000 files Test HEAD~3 HEAD --------------------------------------------------------------------------- read_cache/discard_cache 1000 times 14.08(0.03+0.09) 8.71(0.01+0.09) -38.1% 1,000,000 files Test HEAD~3 HEAD ------------------------------------------------------------------------------ read_cache/discard_cache 1000 times 201.77(0.03+0.07) 149.68(0.04+0.07) -25.8% Suggested by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com> Signed-off-by: Ben Peart <Ben.Peart@microsoft.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TEST PR: speed up index load through parallelization #20

TEST PR: speed up index load through parallelization #20

Commits on Aug 2, 2018

Commits on Aug 3, 2018

Commits on Aug 6, 2018

Commits on Aug 7, 2018

Commits on Aug 8, 2018

Commits on Aug 9, 2018

Commits on Aug 10, 2018

Commits on Aug 13, 2018

Commits on Aug 14, 2018

Commits on Aug 15, 2018

Commits on Aug 16, 2018

Commits on Aug 17, 2018

Commits on Aug 18, 2018

Commits on Aug 20, 2018

Commits on Aug 22, 2018

Commits on Aug 29, 2018