Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci, gha: Add Windows jobs based on Linux image #1398

Merged
merged 3 commits into from
Aug 18, 2023

Conversation

hebasto
Copy link
Member

@hebasto hebasto commented Aug 10, 2023

Solves one item in #1392 partially.

@hebasto
Copy link
Member Author

hebasto commented Aug 10, 2023

@achow101

Could you please explicitly enable the actions as follows:

  • docker/setup-buildx-action@*
  • docker/build-push-action@*
  • addnab/docker-run-action@*

?

In the meantime, reviewers can view the builds in my personal repo: https://github.com/hebasto/secp256k1/actions/runs/5818492202.

@real-or-random
Copy link
Contributor

In the meantime, reviewers can view the builds in my personal repo: hebasto/secp256k1/actions/runs/5818492202.

Caching doesn't seem to work properly. I suggest trying to remove the mode=min.

@hebasto
Copy link
Member Author

hebasto commented Aug 10, 2023

In the meantime, reviewers can view the builds in my personal repo: hebasto/secp256k1/actions/runs/5818492202.

Caching doesn't seem to work properly.

Which job?

@real-or-random
Copy link
Contributor

real-or-random commented Aug 10, 2023

In the meantime, reviewers can view the builds in my personal repo: hebasto/secp256k1/actions/runs/5818492202.

Caching doesn't seem to work properly.

Which job?

Sorry, all jobs, but I meant to include a link to the job. But I think I was wrong. The "Building container" stages takes ~5min, but all of this time seems to be spent downloading from the cache?! I didn't expect that it's that slow.

edit: Fetching from ghcr is similarly slow, ok... https://github.com/hebasto/secp256k1/actions/runs/5783560280/job/15672591894#step:2:54

@hebasto
Copy link
Member Author

hebasto commented Aug 10, 2023

The "Building container" stages takes ~5min, but all of this time seems to be spent downloading from the cache?!

Correct.

I didn't expect that it's that slow.

That is what I meant in #1396 (comment) :)

@real-or-random
Copy link
Contributor

Okay! But do you agree with my observation that GHA cache vs. GHCR doesn't make a significant difference here? Then we can ignore that piece when deciding what to use for Linux tasks.

(Heck, for Linux native tasks with standard gcc and clang, using Docker image at all may be faster... But I'd say let's worry about optimizations later.)

@hebasto
Copy link
Member Author

hebasto commented Aug 10, 2023

Okay! But do you agree with my observation that GHA cache vs. GHCR doesn't make a significant difference here? Then we can ignore that piece when deciding what to use for Linux tasks.

I agree.

The advantage of GHA cache over GHCR is not requiring write permissions for a workflow.

@achow101
Copy link
Member

Updated the actions settings

@hebasto hebasto force-pushed the 230810-gha-win branch 2 times, most recently from 2dc7abe to dba39d0 Compare August 10, 2023 23:32
@hebasto
Copy link
Member Author

hebasto commented Aug 11, 2023

Reworked. The third-party addnab/docker-run-action has been replaced with own reusable workflow.

@hebasto
Copy link
Member Author

hebasto commented Aug 11, 2023

@real-or-random

Do you have permissions to re-run workflows and jobs?

@real-or-random
Copy link
Contributor

Hm, sad to see that it's already starting to get flaky: https://github.com/bitcoin-core/secp256k1/actions/runs/5827144103/job/15802793632?pr=1398 Perhaps this helps: moby/buildkit#3969

Anyway, I like the changes.

@hebasto
Copy link
Member Author

hebasto commented Aug 11, 2023

Hm, sad to see that it's already starting to get flaky: bitcoin-core/secp256k1/actions/runs/5827144103/job/15802793632?pr=1398

From my experience, I would estimate the rate of such kind of failure is about 1 (one) per cent of total runs.

@hebasto
Copy link
Member Author

hebasto commented Aug 11, 2023

Reworked with using a composite action to avoid environment variables hassles.

Using compression=estargz,force-compression=true for caching as suggested by @real-or-random offline.

@hebasto
Copy link
Member Author

hebasto commented Aug 11, 2023

The only thing I'm not sure about is that whether PR branches will hit image cache from the master branch.

@hebasto
Copy link
Member Author

hebasto commented Aug 11, 2023

The only thing I'm not sure about is that whether PR branches will hit image cache from the master branch.

I've just finished this check. It works:

image

Docker layers caches are re-used from the base/default branch.

@real-or-random
Copy link
Contributor

Using compression=estargz,force-compression=true for caching as suggested by @real-or-random offline.

Okay, as @hebasto figured out that doesn't work with the GHA cache backend for now (see https://docs.docker.com/build/cache/backends/#cache-compression). Well fine.

@hebasto
Copy link
Member Author

hebasto commented Aug 11, 2023

Using compression=estargz,force-compression=true for caching as suggested by @real-or-random offline.

Okay, as @hebasto figured out that doesn't work with the GHA cache backend for now (see docs.docker.com/build/cache/backends/#cache-compression). Well fine.

Dropped.

@real-or-random
Copy link
Contributor

This looks pretty clean now. Will perform a closer review soon.

One nit: Perhaps it's now time to rename cirrus.ci to ci.sh.

Some observations:

  • All the Docker stuff is a bit more complex on GHA than on Cirrus CI, but I still believe Docker is a good idea: It makes it easier to reproduce failures locally, and in principle, it makes it easy to switch between different CI providers. For example, we can easily switch back to Cirrus (self-hosted) runners if we keep the Docker images.
  • Store/restoring the Docker build cache is still a bit slow. It's a bit sad that downloading the cache takes almost as long as running the actual CI script. But we may be able to improve this later. And even if we don't manage to improve, I think the build times are acceptable. (Moreover, the native Windows builds don't use Docker, so we still get some immediate feedback for the most obvious problems.)

@hebasto
Copy link
Member Author

hebasto commented Aug 11, 2023

One nit: Perhaps it's now time to rename cirrus.ci to ci.sh.

Done.

Copy link
Contributor

@real-or-random real-or-random left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK mod nits

Do you think it makes sense to retain the Cirrus config commented out, like sometimes done in Core? This would require a bit more work here to make sure it still works now that we split the image.

.github/actions/run-in-docker-action/action.yml Outdated Show resolved Hide resolved
for var in "$(echo '${{ toJSON(matrix.configuration.env_vars) }}' | jq -r 'to_entries | .[] | "\(.key)=\(.value)"')"; do
echo "$var" >> "$GITHUB_ENV"
done
echo "MAKEFLAGS=-j$(($(nproc) + 1))" >> "$GITHUB_ENV"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: This will probably override MAKEFLAGS if in env_vars. Perhaps it's a bit cleaner to simply prepend MAKEFLAGS with $(($(nproc) + 1)), and do this inside ci.sh?

Copy link
Member Author

@hebasto hebasto Aug 11, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To keep the scope of this PR limited, is it OK to just set mingw_debian.env.MAKEFLAGS: '-j3'?

UPD. Or rely on the global env.MAKEFLAGS: '-j4'?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or rely on the global env.MAKEFLAGS: '-j4'?

Implemented.

.github/actions/run-in-docker-action/action.yml Outdated Show resolved Hide resolved
Comment on lines +95 to +119
- run: cat tests.log || true
if: ${{ always() }}
- run: cat noverify_tests.log || true
if: ${{ always() }}
- run: cat exhaustive_tests.log || true
if: ${{ always() }}
- run: cat ctime_tests.log || true
if: ${{ always() }}
- run: cat bench.log || true
if: ${{ always() }}
- run: cat config.log || true
if: ${{ always() }}
- run: cat test_env.log || true
if: ${{ always() }}
- name: CI env
run: env
if: ${{ always() }}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be refactored into an action? Or is this overkill?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this be refactored into an action? Or is this overkill?

In that case it won't show separated entries in the log.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok yes, let's leave it like this for now.

(I'm not happy with this and it should be reworked in a separate PR. I had introduced this printing of files to make sure that we always have all logs, and I think that's a good idea in general. But it's not a good idea to pipe it only to files. We should pipe it to files but also keep stdout/stderr, e.g., using tee. People get confused about this over and over because they can't find the error messages in the failing tasks etc.)

@real-or-random
Copy link
Contributor

@MarcoFalke if you want to have a look

@hebasto
Copy link
Member Author

hebasto commented Aug 14, 2023

Done in #1399.

@real-or-random
Copy link
Contributor

real-or-random commented Aug 14, 2023

I'm not entirely sure how to proceed. I think I'd like to see at least one of the two aforementioned issues addressed. If we can reduce the size of the images for now, that will probably good enough. Even if we then have issues with concurrent CI runs, that's not the end of the world. (One can just re-run manually...)

I'd say let's first try and see how much space we can gain. If we reduce the images far enough, then this may indeed be good enough, and at least prevent the issues of caused by the actual CI job having to rebuild and then running out of disk space.

edit: This comment and some others above apply more to #1396 than to this PR.

Copy link
Contributor

@jonasnick jonasnick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK, thanks @hebasto!

After @real-or-random explained this approach to me I was able to understand what happens without doing too much research into the github actions semantics. In general, it would be nice to have some comments that document what happens (e.g. what exactly is the purpose of docker_wine_cache and what it does if the image already exists), but we can add them over time in follow-up PRs.

@hebasto
Copy link
Member Author

hebasto commented Aug 17, 2023

Converting this PR to a draft for now.

Please review #1401 first.

@hebasto hebasto marked this pull request as draft August 17, 2023 13:40
@hebasto hebasto marked this pull request as ready for review August 17, 2023 20:50
@hebasto
Copy link
Member Author

hebasto commented Aug 17, 2023

Rebased.

Ready for review.

@hebasto
Copy link
Member Author

hebasto commented Aug 18, 2023

Rebased.

@real-or-random
Copy link
Contributor

I believe it's possible to shave off several minutes from the tasks that run in Docker by setting the docker build driver to docker, see docker/buildx#107 (comment) ff . This should avoid the long "sending tarball" phases. It can be done by setting driver: docker, see https://github.com/docker/setup-buildx-action#inputs . But let's experiment with this after this PR. :)

Copy link
Contributor

@real-or-random real-or-random left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 87d35f3

@hebasto
Copy link
Member Author

hebasto commented Aug 18, 2023

I believe it's possible to shave off several minutes from the tasks that run in Docker by setting the docker build driver to docker, see docker/buildx#107 (comment) ff . This should avoid the long "sending tarball" phases. It can be done by setting driver: docker, see docker/setup-buildx-action#inputs . But let's experiment with this after this PR. :)

Unfortunately, GitHub Actions cache does not work with the docker build driver.

Copy link
Contributor

@jonasnick jonasnick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ACK 87d35f3

@jonasnick jonasnick merged commit 6b9507a into bitcoin-core:master Aug 18, 2023
97 checks passed
@hebasto hebasto deleted the 230810-gha-win branch August 18, 2023 12:48
sipa added a commit to sipa/bitcoin that referenced this pull request Sep 4, 2023
199d27c Merge bitcoin-core/secp256k1#1415: release: Prepare for 0.4.0
1633980 release: Prepare for 0.4.0
d9a8506 changelog: Catch up in preparation of release
0b4640a Merge bitcoin-core/secp256k1#1413: ci: Add `release` job
8659a01 ci: Add `release` job
f9b3889 ci: Update `actions/checkout` version
727bec5 Merge bitcoin-core/secp256k1#1414: ci/gha: Add ARM64 QEMU jobs for clang and clang-snapshot
2635068 ci/gha: Let MSan continue checking after errors in all jobs
e78c7b6 ci/Dockerfile: Reduce size of Docker image further
2f0d3bb ci/Dockerfile: Warn if `ulimit -n` is too high when running Docker
4b8a647 ci/gha: Add ARM64 QEMU jobs for clang and clang-snapshot
6ebe7d2 ci/Dockerfile: Always use versioned clang packages
65c79fe Merge bitcoin-core/secp256k1#1412: ci: Switch macOS from Ventura to Monterey and add Valgrind
c223d7e ci: Switch macOS from Ventura to Monterey and add Valgrind
ea26b71 Merge bitcoin-core/secp256k1#1411: ci: Make repetitive command the default one
cce0456 ci: Make repetitive command the default one
317a4c4 ci: Move `git config ...` to `run-in-docker-action`
4d7fe60 Merge bitcoin-core/secp256k1#1409: ci: Move remained task from Cirrus to GitHub Actions
676ed8f ci: Move "C++ (public headers)" from Cirrus to GitHub Actions
61fc3a2 ci: Move "C++ -fpermissive..." from Cirrus to GitHub Actions
d51fb0a ci: Move "MSan" from Cirrus to GitHub Actions
c22ac27 ci: Move sanitizers task from Cirrus to GitHub Actions
26a9899 Merge bitcoin-core/secp256k1#1410: ci: Use concurrency for pull requests only
ee1be62 ci: Use concurrency for pull requests only
6ee1455 Merge bitcoin-core/secp256k1#1406: ci, gha: Move more non-x86_64 tasks from Cirrus CI to GitHub Actions
fc3dea2 ci: Move "ppc64le: Linux..." from Cirrus to GitHub Actions
7782dc8 ci: Move "ARM64: Linux..." from Cirrus to GitHub Actions
0a16de6 ci: Move "ARM32: Linux..." from Cirrus to GitHub Actions
ea33914 ci: Move "s390x (big-endian): Linux..." from Cirrus to GitHub Actions
880be8a ci: Move "i686: Linux (Debian stable)" from Cirrus to GiHub Actions
2e6cf9b Merge bitcoin-core/secp256k1#1396: ci, gha: Add "x86_64: Linux (Debian stable)" GitHub Actions job
5373693 Merge bitcoin-core/secp256k1#1405: ci: Drop no longer needed workaround
ef9fe95 ci: Drop no longer needed workaround
e10878f ci, gha: Drop `driver-opts.network` input for `setup-buildx-action`
4ad4914 ci, gha: Add `retry_builder` Docker image builder
6617a62 ci: Remove "x86_64: Linux (Debian stable)" task from Cirrus CI
03c9e65 ci, gha: Add "x86_64: Linux (Debian stable)" GitHub Actions job
ad3e65d ci: Remove GCC build files and sage to reduce size of Docker image
6b9507a Merge bitcoin-core/secp256k1#1398: ci, gha: Add Windows jobs based on Linux image
87d35f3 ci: Rename `cirrus.sh` to more general `ci.sh`
d6281dd ci: Remove Windows tasks from Cirrus CI
2b6f9cd ci, gha: Add Windows jobs based on Linux image
48b1d93 Merge bitcoin-core/secp256k1#1403: ci, gha: Ensure only a single workflow processes `github.ref` at a time
0ba2b94 Merge bitcoin-core/secp256k1#1373: Add invariant checking for scalars
060e32c Merge bitcoin-core/secp256k1#1401: ci, gha: Run all MSVC tests on Windows natively
de657c2 Merge bitcoin-core/secp256k1#1062: Removes `_fe_equal_var`, and unwanted `_fe_normalize_weak` calls (in tests)
bcffeb1 Merge bitcoin-core/secp256k1#1404: ci: Remove "arm64: macOS Ventura" task from Cirrus CI
c2f6435 ci: Add comment about switching macOS to M1 on GHA later
4a24fae ci: Remove "arm64: macOS Ventura" task from Cirrus CI
b0886fd ci, gha: Ensure only a single workflow processes `github.ref` at a time
3d05c86 Merge bitcoin-core/secp256k1#1394: ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions
d78bec7 ci: Remove Windows MSVC tasks from Cirrus CI
3545dc2 ci, gha: Run all MSVC tests on Windows natively
5d8fa82 Merge bitcoin-core/secp256k1#1274: test: Silent noisy clang warnings about Valgrind code on macOS x86_64
8e54a34 ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions
b327abf Merge bitcoin-core/secp256k1#1402: ci: Use Homebrew's gcc in native macOS task
d62db57 ci: Use Homebrew's gcc in native macOS task
54058d1 field: remove `secp256k1_fe_equal_var`
bb4efd6 tests: remove unwanted `secp256k1_fe_normalize_weak` call
eedd781 Merge bitcoin-core/secp256k1#1348: tighten group magnitude limits, save normalize_weak calls in group add methods (revival of bitcoin#1032)
b2f6712 Merge bitcoin-core/secp256k1#1400: ctimetests: Use new SECP256K1_CHECKMEM macros also for ellswift
9c91ea4 ci: Enable ellswift module where it's missing
db32a24 ctimetests: Use new SECP256K1_CHECKMEM macros also for ellswift
ce765a5 Merge bitcoin-core/secp256k1#1399: ci, gha: Run "SageMath prover" job on GitHub Actions
8408dfd Revert "ci: Run sage prover on CI"
c8d9914 ci, gha: Run "SageMath prover" job on GitHub Actions
8d2960c Merge bitcoin-core/secp256k1#1397: ci: Remove "Windows (VS 2022)" task from Cirrus CI
f1774e5 ci, gha: Make MSVC job presentation more explicit
5ee039b ci: Remove "Windows (VS 2022)" task from Cirrus CI
96294c0 Merge bitcoin-core/secp256k1#1389: ci: Run "Windows (VS 2022)" job on GitHub Actions
a2f7ccd ci: Run "Windows (VS 2022)" job on GitHub Actions
374e2b5 Merge bitcoin-core/secp256k1#1290: cmake: Set `ENVIRONMENT` property for examples on Windows
1b13415 Merge bitcoin-core/secp256k1#1391: refactor: take use of `secp256k1_scalar_{zero,one}` constants (part 2)
a1bd497 refactor: take use of `secp256k1_scalar_{zero,one}` constants (part 2)
b7c685e Save _normalize_weak calls in group add methods
c83afa6 Tighten group magnitude limits
26392da Merge bitcoin-core/secp256k1#1386: ci: print $ELLSWIFT in cirrus.sh
d23da6d use secp256k1_scalar_verify checks
4692478 ci: print $ELLSWIFT in cirrus.sh
c7d0454 add verification for scalars
c734c64 Merge bitcoin-core/secp256k1#1384: build: enable ellswift module via SECP_CONFIG_DEFINES
ad15215 update max scalar in scalar_cmov_test and fix schnorrsig_verify exhaustive test
78ca880 build: enable ellswift module via SECP_CONFIG_DEFINES
0e00fc7 Merge bitcoin-core/secp256k1#1383: util: remove unused checked_realloc
b097a46 util: remove unused checked_realloc
2bd5f3e Merge bitcoin-core/secp256k1#1382: refactor: Drop unused cast
4f8c5bd refactor: Drop unused cast
173e8d0 Implement current magnitude assumptions
49afd2f Take use of _fe_verify_magnitude in field_impl.h
4e9661f Add _fe_verify_magnitude (no-op unless VERIFY is enabled)
690b0fc add missing group element invariant checks
175db31 ci: Drop no longer needed `PATH` variable update on Windows
116d2ab cmake: Set `ENVIRONMENT` property for examples on Windows
cef3739 cmake, refactor: Use helper function instead of interface library
747ada3 test: Silent noisy clang warnings about Valgrind code on macOS x86_64

git-subtree-dir: src/secp256k1
git-subtree-split: 199d27c
Retropex added a commit to Retropex/bitcoin that referenced this pull request Oct 4, 2023
Release note for changing -permitbaremultisig config default to false

Remove unused MessageStartChars parameters from BlockManager methods

tracepoints: Disables `-Wgnu-zero-variadic-macro-arguments` to compile without warnings

Fixes #26916 by disabling the warning `-Wgnu-zero-variadic-macro-arguments` when clang is used as the compiler.

lint: drop DIR_IWYU global

lint: remove lint-logs.py

lint: remove  /* Continued */ markers from codebase

refactor: fix unterminated LogPrintf()s

tidy: Integrate bicoin-tidy clang-tidy plugin

Enable `bitcoin-unterminated-logprintf`.

Co-authored-by: Cory Fields <cory-nospam-@coryfields.com>

doc: remove Fedora libdb4-*-devel install docs

These are no-longer installable on any recent Fedora (33+).
Remove the install instructions.
Fix the typo in the Ubuntu/Debian instructions.

refactor: Wrap DestroyDB in dbwrapper helper

Wrap leveldb::DestroyDB in a helper function without exposing
leveldb-specifics.

Also, add missing optional include.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Split dbwrapper CDBBatch::Write implementation

Keep the generic serialization in the header, while moving
leveldb-specifics to the implementation file.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Split dbwrapper CDBatch::Erase implementation

Keep the generic serialization in the header, while moving
leveldb-specifics to the implementation file.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Pimpl leveldb::batch for CDBBatch

Hide the leveldb::WriteBatch member variable with a pimpl in order not
to expose it directly in the header.

Also move CDBBatch::Clear to the dbwrapper implementation to use the new
impl_batch.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Split dbwrapper CDBIterator::Seek implementation

Keep the generic serialization in the header, while moving
leveldb-specifics to the implementation file.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Split dbwrapper CDBIterator::GetKey implementation

Keep the generic serialization in the header, while moving
leveldb-specifics to the implementation file.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Split dbwrapper CDBIterator::GetValue implementation

Keep the generic serialization in the header, while moving
leveldb-specifics to the implementation file.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Pimpl leveldb::Iterator for CDBIterator

Hide the leveldb::Iterator member variable with a pimpl in order not to
expose it directly in the header.

Also, move CDBWrapper::NewIterator to the dbwrapper implementation to
use the pimpl for CDBIterator initialziation.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Split dbwrapper CDBWrapper::Read implementation

Keep the generic serialization in the header, while moving
leveldb-specifics to the implementation file.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Fix logging.h includes

These were uncovered as missing by the next commit.

refactor: Split dbwrapper CDBWrapper::Exists implementation

Keep the generic serialization in the header, while moving
leveldb-specifics to the implementation file.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Move HandleError to dbwrapper implementation

Make it a static function in dbwrapper.cpp, since it is not used
elsewhere and when left in the header, would expose a leveldb type.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Split dbwrapper CDBWrapper::EstimateSize implementation

Keep the generic serialization in the header, while moving
leveldb-specifics to the implementation file.

Since CharCast is no longer needed in the header, move it to the
implementation file.

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

refactor: Move CDBWrapper leveldb members to their own context struct

The context of this commit is an effort to decouple the dbwrapper header
file from leveldb includes. To this end, the includes are moved to the
dbwrapper implementation file. This is done as part of the kernel
project to reduce the number of required includes for users of the
kernel.

build: Remove leveldb from BITCOIN_INCLUDES

Since leveldb is no longer in our header tree, move its include flags to
whereever dbwrapper.cpp is built.

refactor: Correct dbwrapper key naming

The ss- prefix should connotate a DataStream variable. Now that these
variables are byte spans, drop the prefix.

ci: Use qemu-user through container engine

doc: document PeerManager::Options members

net processing: clamp -maxorphantx to uint32_t bounds

net processing: clamp -blockreconstructionextratxn to uint32_t bounds

Also changes max_extra_txs into a uint32_t to avoid platform-specific
behaviour

doc: Add release note

crypto: remove outdated variant of ChaCha20Poly1305 AEAD

Remove the variant of ChaCha20Poly1305 AEAD that was previously added in
anticipation of BIP324 using it. BIP324 was updated to instead use rekeying
wrappers around otherwise unmodified versions of the ChaCha20 stream cipher
and the ChaCha20Poly1305 AEAD as specified in RFC8439.

crypto: add the ChaCha20Poly1305 AEAD as specified in RFC8439

This adds an implementation of the ChaCha20Poly1305 AEAD exactly matching
the version specified in RFC8439 section 2.8, including tests and official
test vectors.

crypto: add FSChaCha20, a rekeying wrapper around ChaCha20

This adds the FSChaCha20 stream cipher as specified in BIP324, a
wrapper around the ChaCha20 stream cipher (specified in RFC8439
section 2.4) which automatically rekeys every N messages, and
manages the nonces used for encryption.

Co-authored-by: dhruv <856960+dhruv@users.noreply.github.com>

crypto: add FSChaCha20Poly1305, rekeying wrapper around ChaCha20Poly1305

This adds the FSChaCha20Poly1305 AEAD as specified in BIP324, a wrapper
around the ChaCha20Poly1305 AEAD (as specified in RFC8439 section 2.8) which
automatically rekeys every N messages, and automatically increments the nonce
every message.

bench: add benchmark for FSChaCha20Poly1305

Add a benchmark for FSChaCha20Poly1305 encryption, so the overhead of key
generation and authentication can be observed for various message sizes.

crypto: support split plaintext in ChaCha20Poly1305 Encrypt/Decrypt

Add BIP324Cipher, encapsulating key agreement, derivation, and stream/AEAD ciphers

Co-authored-by: dhruv <856960+dhruv@users.noreply.github.com>

tests: add decryption test to bip324_tests

doc: use llvm-config for bitcoin-tidy example

An LLVM installation will have `llvm-config` available to query for
info. Ask it for the `--cmakedir`, and use that in our bitcoin-tidy
example, rather than listing multiple different (potential) paths per
distro/OS etc.

bitcoin-tidy: fix macOS build

LLVM uses these options for building as well, so there's precedent.

Also fix the shared library extension which was incorrectly being set to dylib

test: locked_wallet, skip default fee estimation

Same as we do with the nodes default wallets.
No test case on this file is meant to exercise fee estimation.

fuzz: coins_view: correct an incorrect assertion

It is incorrect to assert that `cache.HaveCoin()` will always be `true`
if `backend.HaveCoin()` is. The coin could well have been marked as
spent in the cache but not yet flushed, in which case `cache.HaveCoin()`
would return `false`.

Note this was never hit because `exists_using_have_coin_in_backend` is
currently never `true` (it's the default implementation of `CCoinsView`.
However this might change if we were to add a target where the backend
is a `CCoinsViewDB`.

fuzz: coins_view: remove an incorrect assertion

Again, this was not hit because the default implementation of
`CCoinsView` return `false` for `GetCoin`.

crypto: BIP324 ciphersuite follow-up

follow-up to #28008.
* move `dummy_tag` variable in FSChaCha20Poly1305 crypto_tests
outside of the loop to be reused every time
* use easy to read `cipher.last()` in `AEADChaCha20Poly1305::Decrypt()`
* comment for initiator in `BIP324Cipher::Initialize()`
* systematically damage ciphertext with bit positions in bip324_tests
* use 4095 max bytes for aad in bip324 fuzz test

refactor: add missing headers for BIP324 ciphersuite

ci: Drop no longer needed `macos_sdk_cache`

It has been cached in the Docker image since https://github.com/bitcoin/bitcoin/pull/27028.

ci: Drop BASE_SCRATCH_DIR from LIBCXX_DIR

Using a hard-coded path avoids non-determinism issues and improves CI
UX.

ci: Only create folders when needed

Now that container volumes are used, the folders are no longer mounted.
They are only needed when running without a container engine (docker,
podman).

ci: Use hard-coded root path for CI containers

ci: Run "macOS native x86_64" job on GitHub Actions

Also, the "macOS native arm64" task has been removed from Cirrus CI.

Remove unused boost signals2 from torcontrol

iwyu on torcontrol

Replace LocaleIndependentAtoi with ToIntegral

No need for saturating behavior when the int is composed of 3 digits.

remove unused limits.h include in compat.h

Sort includes in compat.h

Can be reviewed with:
--color-moved=blocks  --color-moved-ws=ignore-all-space --ignore-all-space

test: check backup from `migratewallet` can be successfully restored

ci: Fix macOS-cross SDK rsync

This should fix the macOS-cross build on Cirrus CI containers.

Locally this was already working, because the SDK was cached in
/ci_container_base/ in the image, which is also the folder used for a
later CI run.

However, on Cirrus CI, when using an image *and* a custom BASE_ROOT_DIR,
the SDK will not be found in /ci_base_install/, nor in BASE_ROOT_DIR.

Fix this by normalizing *all* folders to /ci_container_base/.

ci: Avoid error on macOS native

This avoids "mkdir: /ci_container_base: Read-only file system"

mempool_entry: add mempool entry sequence number

validation: when adding txs due to a block reorg, allow immediate relay

net_processing: drop m_recently_announced_invs bloom filter

Rather than using a bloom filter to track announced invs, simply allow
a peer to request any tx that entered the mempool prior to the last INV
message we sent them. This also obsoletes the UNCONDITIONAL_RELAY_DELAY.

net_processing: don't add txids to m_tx_inventory_known_filter

We no longer have m_recently_announced_invs, so there is no need to add
txids to m_tx_inventory_known_filter to dedupe that filter.

test: Check tx from disconnected block is immediately requestable

Check that peers can immediately request txs from blocks that have been
reorged out and are now in our mempool.

net_processing: Clean up INVENTORY_BROADCAST_MAX constants

mempool_entry: improve struct packing

ci: Move tidy to persistent worker

refactor: Remove PERSISTENT_WORKER_* yaml templates

* PERSISTENT_WORKER_TEMPLATE_ENV is not needed at all, because
  RESTART_CI_DOCKER_BEFORE_RUN is already set on the persistent worker.
* PERSISTENT_WORKER_TEMPLATE can be replaced by pinning the
  previous_releases task to a type of worker. This should make the CI
  performance more consistent.

Move MANDATORY_SCRIPT_VERIFY_FLAGS from script/standard.h to policy/policy.h

Remove ScriptHash from CScriptID constructor

Replaces the constructor in CScriptID that converts a ScriptHash with a
function ToScriptID that does the same. This prepares for a move of
CScriptID to avoid a circular dependency.

Move CScriptID to script.{h/cpp}

CScriptID should be next to CScript just as CKeyID is next to CPubKey

Move Taproot{SpendData/Builder} to signingprovider.{h/cpp}

TaprootSpendData and TaprootBuilder are used in signing in
SigningProvider contexts, so they should live near that.

Move CTxDestination to its own file

CTxDestination is really our internal representation of an address and
doesn't really have anything to do with standard script types, so move
them to their own file.

MOVEONLY: Move datacarrier defaults to policy.h

Clean up things that include script/standard.h

Remove standard.h from files that don't use anything in it, and include
it in files that do.

Clean up script/standard.{h/cpp} includes

Rename script/standard.{cpp/h} to script/solver.{cpp/h}

Since script/standard only contains things that are used by the Solver
and its callers, rename the files to script/solver.

Rework receive buffer pushback

Co-authored-by: Anthony Towns <aj@erisian.com.au>

test: Fix intermittent issue in mining_getblocktemplate_longpoll.py

Bugfix: RPC: Remove quotes from non-string oneline descriptions

RPC/rpcdoccheck: Error if a oneline_description has a quote for a non-string

RPC/Mining: Document template_request better for getblocktemplate

ci: Ensure that only a single workflow processes `github.ref` at a time

ci: Refactor: Remove CI_USE_APT_INSTALL

rpc: remove one more quote from non-string oneline description

This fixes a silent conflict betwen #28123 and #27460

crypto: refactor ChaCha20 classes to use Span<std::byte> interface

random: simplify FastRandomContext::randbytes using fillrand

crypto: require key on ChaCha20 initialization

fuzz: support std::byte in Consume{Fixed,Variable}LengthByteVector

tests: miscellaneous hex / std::byte improvements

crypto: make ChaCha20::SetKey wipe buffer

doc: Fix bitcoin-unterminated-logprintf tidy comments

* Move module description from test to LogPrintfCheck
* Add test doc
* Remove unused comment, see https://github.com/bitcoin/bitcoin/pull/26296/files#r1279351539

refactor: Enforce C-str fmt strings in WalletLogPrintf()

refactor: Enable all clang-tidy plugin bitcoin tests

This makes it easier to add new ones without having to modify this file
every time.

bitcoin-tidy: Apply bitcoin-unterminated-logprintf to spkm as well

ci: Add missing ${CI_RETRY_EXE} before curl

ci: Add missing amd64 to win64-cross task

Also, do the same for android, which also fails.

assumeutxo cleanup: Move IsInitialBlockDownload & NotifyHeaderTip to ChainstateManager

This change makes IsInitialBlockDownload and NotifyHeaderTip functions no
longer tied to individual Chainstate objects. It makes them work with the
ChainstateManager object instead so code is simpler and it is no longer
possible to call them incorrectly with an inactive Chainstate.

This change also makes m_cached_finished_ibd caching easier to reason about,
because now there is only one cached value instead of two (for background and
snapshot chainstates) so the cached IBD state now no longer gets reset when a
snapshot is loaded.

There should be no change in behavior because these functions were always
called on the active ChainState objects.

These changes were discussed previously
https://github.com/bitcoin/bitcoin/pull/27746#discussion_r1246868905 and
https://github.com/bitcoin/bitcoin/pull/27746#discussion_r1237552792 as
possible followups for that PR.

ci: Disable cache save for pull requests in GitHub Actions

Otherwise, multiple pull requests fill GitHub Actions cache quota
shortly.

tx fees, policy: doc: update and delete unnecessary comment

test: ensure acceptstalefeeestimates is supported only on regtest chain

test: fix 'unknown named parameter' test in `wallet_basic`

Fixes loop when testing an unknown named parameter.

Remove unused includes from txmempool.h

... and move them to where they are really needed.

This was found by IWYU:

txmempool.h should remove these lines:
- #include <random.h>  // lines 29-29
- class CBlockIndex;  // lines 43-43
- class Chainstate;  // lines 45-45

Also, move the stdlib section to the right place. Can be reviewed with:
--color-moved=dimmed-zebra

move-only: Create src/kernel/mempool_removal_reason.h

This is needed for a future commit. Can be reviewed with:
--color-moved=dimmed-zebra

Remove unused includes from blockfilter.h

This removes unused includes, primitives/block found manually, and the
others by iwyu:

blockfilter.h should remove these lines:
- #include <serialize.h>  // lines 16-16
- #include <undo.h>  // lines 18-18

Remove unused includes from wallet.cpp

This removes unused includes, such as undo.h or txmempool.h from
wallet.cpp.

Also, add missing ones, according to IWYU.

test: Support riscv64 in get_previous_releases.py

refactor: Add missing includes

Refactor: Remove confusing static_cast

guix: pre time-machine bump changes (Windows)

Split out of #27897. This is some refactoring to the Windows Guix build
that facilitates bumping our Guix time-machine. Namely, avoiding
`package-with-extra-configure-variable`, which is non-functional in the
newer time-machine, see https://issues.guix.gnu.org/64436.

At the same time, consolidate our Windows GCC build into mingw-w64-base-gcc.
Rename `gcc-10-remap-guix-store.patch` to avoid changing it whenever GCC changes.

We move the old `building-on` inside `explicit-cross-configure`, so that
non-windows builds continue to work. Note that `explicit-cross-configure`
will be going away entirely (see #27897).

[test framework] make it easier to fast-forward setmocktime

Have each TestNode keep track of the last timestamp it called
setmocktime with, and add a bumpmocktime() function to bump by a
number of seconds. Makes it easy to fast forward n seconds without
keeping track of what the last timestamp was.

[functional test] transaction orphan handling

ci: Switch remaining tasks to self-hosted

This allows to drop unused templates, such as
cirrus_ephemeral_worker_template_env, or container_depends_template.

Also, ccache_cache, previous_releases_cache, and
base_depends_built_cache can be dropped, because the caching is done in
container volumes on the self-hosted runners.

ci: Remove distro-name from task name

The exact distro name should not be important. Also, it is easy to find
out, if needed. Thus, remove it to avoid bloat and maintenance overhead
having to keep it in sync.

doc, policy: Clarify comment on STANDARD_SCRIPT_VERIFY_FLAGS

Make post-p2sh consensus rules mandatory for tx relay

Update help text for spend and rawtransaction rpcs

fixing typo

guix: consolidate glibc 2.27 package

Refactor our glibc 2.27 to be a single 'package', and avoid the use of
`package-with-extra-configure-variable`. This also lets us drop the
`enable_werror` workaround, and just use --disable-werror directly.

Employ the same workaround as the Guix glibc, to avoid a "permission
denied" failure during build:
```bash
make  subdir=sunrpc -C sunrpc ..=../ subdir_install
make[2]: Entering directory '/tmp/guix-build-glibc-cross-x86_64-linux-gnu-2.27.drv-0/source/sunrpc'
.././scripts/mkinstalldirs /gnu/store/ga8jciqrd5lh52m572x3mk4q1smf5agq-glibc-cross-x86_64-linux-gnu-2.27/include/rpc
mkdir -p -- /gnu/store/ga8jciqrd5lh52m572x3mk4q1smf5agq-glibc-cross-x86_64-linux-gnu-2.27/include/rpc
/gnu/store/kvpvk5wh70wdbjnr83hh85rg22ysxm9h-coreutils-8.32/bin/install -c -m 644 rpc/netdb.h /gnu/store/ga8jciqrd5lh52m572x3mk4q1smf5agq-glibc-cross-x86_64-linux-gnu-2.27/include/rpc/netdb.h
.././scripts/mkinstalldirs /gnu/store/ga8jciqrd5lh52m572x3mk4q1smf5agq-glibc-cross-x86_64-linux-gnu-2.27/include/nfs
mkdir -p -- /gnu/store/ga8jciqrd5lh52m572x3mk4q1smf5agq-glibc-cross-x86_64-linux-gnu-2.27/include/nfs
/gnu/store/kvpvk5wh70wdbjnr83hh85rg22ysxm9h-coreutils-8.32/bin/install -c -m 644 ../sysdeps/unix/sysv/linux/nfs/nfs.h /gnu/store/ga8jciqrd5lh52m572x3mk4q1smf5agq-glibc-cross-x86_64-linux-gnu-2.27/include/nfs/nfs.h
/gnu/store/kvpvk5wh70wdbjnr83hh85rg22ysxm9h-coreutils-8.32/bin/install -c -m 644 /tmp/guix-build-glibc-cross-x86_64-linux-gnu-2.27.drv-0/build/gnu/lib-names-64.h /gnu/store/ga8jciqrd5lh52m572x3mk4q1smf5agq-glibc-cross-x86_64-linux-gnu-2.27/include/gnu/lib-names-64.h
/gnu/store/kvpvk5wh70wdbjnr83hh85rg22ysxm9h-coreutils-8.32/bin/install -c -m 644 etc.rpc /etc/rpc
/gnu/store/kvpvk5wh70wdbjnr83hh85rg22ysxm9h-coreutils-8.32/bin/install: cannot create regular file '/etc/rpc': Permission denied
make[2]: *** [Makefile:197: /etc/rpc] Error 1
```

guix: consolidate Linux GCC package

Refactor our Linux GCC to be a single 'package', and avoid the use of
`package-with-extra-configure-variable`.

ci: Limit scope of some env vars

No need to have a larger scope than needed.

Can be reviewed with --color-moved=dimmed-zebra

ci: Start with clean env

This should help to avoid non-determinism.

ci: Remove no longer applicable section

This fails with:

"Error: determining starting point for build: no FROM statement found"

test: Fix intermittent issue in mempool_reorg

ci: Use concurrency for pull requests only

Otherwise, any previously pending workflow will be canceled on the
following push.

test: wallet_backup.py, fix intermittent failure in "restore using dumped wallet"

The failure arises because the test expects 'init_wallet()' (the test
framework function) creating a wallet with no keys. However, the function
also imports the deterministic private key used to receive the coinbase coins.

This causes a race within the "restore using dumped wallet" case, where we
intend to have a new wallet (with no existing keys) to test the
'importwallet()' RPC result.
The reason behind the intermittent failures might be other peers delivering
the chain right after node2 startup (sync of the validation queue included)
and prior to the 'node2.getbalance()' check.

test: previous releases: speed up fetching sources with shallow clone

For the sake of building previous releases, fetching the whole history
of the repository for each version seems to be overkill as it takes much
more time, bandwidth and disk space than necessary. Create a shallow
clone instead with history truncated to the one commit of the version
tag, which is directly checked out in the same command. This has the
nice side-effect that we can remove the extra `git checkout` step after
as it's not needed anymore.

Note that it might look confusing to pass a _tag_ to a parameter named
`--branch`, but the git-clone manpage explicitly states that this is
supported.

ci: Add missing docker.io prefix to CI_IMAGE_NAME_TAG

rpc: Add MaybeArg() and Arg() default helper

refactor: merge transport serializer and deserializer into Transport class

This allows state that is shared between both directions to be encapsulated
into a single object. Specifically the v2 transport protocol introduced by
BIP324 has sending state (the encryption keys) that depends on received
messages (the DH key exchange). Having a single object for both means it can
hide logic from callers related to that key exchange and other interactions.

net: add V1Transport lock protecting receive state

Rather than relying on the caller to prevent concurrent calls to the
various receive-side functions of Transport, introduce a private m_cs_recv
inside the implementation to protect the lock state.

Of course, this does not remove the need for callers to synchronize calls
entirely, as it is a stateful object, and e.g. the order in which Receive(),
Complete(), and GetMessage() are called matters. It seems impossible to use
a Transport object in a meaningful way in a multi-threaded way without some
form of external synchronization, but it still feels safer to make the
transport object itself responsible for protecting its internal state.

refactor: rename Transport class receive functions

Now that the Transport class deals with both the sending and receiving side
of things, make the receive side have function names that clearly indicate
they're about receiving.

* Transport::Read() -> Transport::ReceivedBytes()
* Transport::Complete() -> Transport::ReceivedMessageComplete()
* Transport::GetMessage() -> Transport::GetReceivedMessage()
* Transport::SetVersion() -> Transport::SetReceiveVersion()

Further, also update the comments on these functions to (among others) remove
the "deserialization" terminology. That term is better reserved for just the
serialization/deserialization between objects and bytes (see serialize.h), and
not the conversion from/to wire bytes as performed by the Transport.

net: abstract sending side of transport serialization further

This makes the sending side of P2P transports mirror the receiver side: caller provides
message (consisting of type and payload) to be sent, and then asks what bytes must be
sent. Once the message has been fully sent, a new message can be provided.

This removes the assumption that P2P serialization of messages follows a strict structure
of header (a function of type and payload), followed by (unmodified) payload, and instead
lets transports decide the structure themselves.

It also removes the assumption that a message must always be sent at once, or that no
bytes are even sent on the wire when there is no message. This opens the door for
supporting traffic shaping mechanisms in the future.

net: make V1Transport implicitly use current chainparams

The rest of net.cpp already uses Params() to determine chainparams in many
places (and even V1Transport itself does so in some places).

Since the only chainparams dependency is through the message start characters,
just store those directly in the transport.

fuzz: add bidirectional fragmented transport test

This adds a simulation test, with two V1Transport objects, which send messages
to each other, with sending and receiving fragmented into multiple pieces that
may be interleaved. It primarily verifies that the sending and receiving side
are compatible with each other, plus a few sanity checks.

net: measure send buffer fullness based on memory usage

This more accurately captures the intent of limiting send buffer size, as
many small messages can have a larger overhead that is not counted with the
current approach.

It also means removing the dependency on the header size (which will become
a function of the transport choice) from the send buffer calculations.

net: move message conversion to wire bytes from PushMessage to SocketSendData

This furthers transport abstraction by removing the assumption that a message
can always immediately be converted to wire bytes. This assumption does not hold
for the v2 transport proposed by BIP324, as no messages can be sent before the
handshake completes.

This is done by only keeping (complete) CSerializedNetMsg objects in vSendMsg,
rather than the resulting bytes (for header and payload) that need to be sent.
In SocketSendData, these objects are handed to the transport as permitted by it,
and sending out the bytes the transport tells us to send. This also removes the
nSendOffset member variable in CNode, as keeping track of how much has been sent
is now a responsability of the transport.

This is not a pure refactor, and has the following effects even for the current
v1 transport:

* Checksum calculation now happens in SocketSendData rather than PushMessage.
  For non-optimistic-send messages, that means this computation now happens in
  the network thread rather than the message handler thread (generally a good
  thing, as the message handler thread is more of a computational bottleneck).
* Checksum calculation now happens while holding the cs_vSend lock. This is
  technically unnecessary for the v1 transport, as messages are encoded
  independent from one another, but is untenable for the v2 transport anyway.
* Statistics updates about per-message sent bytes now happen when those bytes
  are actually handed to the OS, rather than at PushMessage time.

refactor: make Transport::ReceivedBytes just return success/fail

fuzz: coinselection, add `CreateCoins`

Move coins creation for a specific function. It
allows us to use it in other parts of the code.

fuzz: coinselection, add coverage for `EligibleForSpending`

fuzz: coinselection, add coverage for `AddInputs`

fuzz: coinselection, add coverage for `GetShuffledInputVector`/`GetInputSet`

fuzz: coinselection, add coverage for `Merge`

fuzz: coinselection, improve `ComputeAndSetWaste`

Instead of using `cost_of_change` for `min_viable_change`
and `change_cost`, and 0 for `change_fee`, use values from
`coin_params`. The previous values don't generate any effects
that is relevant for that context.

fuzz: coinselection, compare `GetSelectedValue` with target

The valid results should have a target below the sum of
the selected inputs amounts. Also, it increases the
minimum value for target to make it more realistic.

fuzz: coinselection, BnB should never produce change

fuzz: coinselection, fix `m_cost_of_change`

`m_cost_of_change` must not be generated randomly
independent from m_change_fee. This commit changes
it to set it up according to `wallet/spend`.

doc: Improve documentation of rpcallowip rpchelp

Closes #21070

v21.0 introduced a behaviour changed noted in #21070 where using a config value
`rpcallowip=::0` no longer also permitted ipv4 ip addresses.

The rpc_bind.py functional test covers this new behaviour already by checking
that the list of bind addresses exactly matches what is expected so this
commit only updates the documentation.

rpc: add test-only sendmsgtopeer rpc

This rpc can be used when we want a node to send a message, but
cannot use a python P2P object, for example for testing of low-level
net transport behavior.

test: add basic tests for sendmsgtopeer to rpc_net.py

test: add functional test for deadlock situation

guix: backport glibc patch to fix powerpc build

Do this prior to bumping the time-machine, to avoid the following build
failure:
```bash
 /tmp/guix-build-glibc-cross-powerpc64le-linux-gnu-2.27.drv-0/build/string/memset-power8.o.dt -MT /tmp/guix-build-glibc-cross-powerpc64le-linux-gnu-2.27.drv-0/build/string/memset-power8.o
../sysdeps/powerpc/powerpc64/power4/memcmp.S: Assembler messages:
../sysdeps/powerpc/powerpc64/power4/memcmp.S:87: Error: unrecognized opcode: `ldbrx'
../sysdeps/powerpc/powerpc64/power4/memcmp.S:88: Error: unrecognized opcode: `ldbrx'
../sysdeps/powerpc/powerpc64/power4/memcmp.S:112: Error: unrecognized opcode: `ldbrx'
```

See:
https://sourceware.org/git/?p=glibc.git;a=commit;h=9250e6610fdb0f3a6f238d2813e319a41fb7a810.
https://github.com/gcc-mirror/gcc/commit/e154242724b084380e3221df7c08fcdbd8460674.

guix: update python-oscrypto to 1.3.0

This is required for bumping the time-machine, for compatibility with
OpenSSL:

oscrypto: openssl backend, 1.2.1, /tmp/guix-build-python-oscrypto-1.2.1.drv-0/source/oscrypto
Traceback (most recent call last):
  File "/tmp/guix-build-python-oscrypto-1.2.1.drv-0/source/oscrypto/_openssl/_libcrypto_ctypes.py", line 304, in <module>
    libcrypto.EVP_PKEY_size.argtypes = [
  File "/gnu/store/9dkl9fnidcdpw19ncw5pk0p7dljx7ijb-python-3.10.7/lib/python3.10/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/gnu/store/9dkl9fnidcdpw19ncw5pk0p7dljx7ijb-python-3.10.7/lib/python3.10/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: /gnu/store/2hr7w64zhr6jjznidyc2xi40d5ynhj9c-openssl-3.0.8/lib/libcrypto.so.3: undefined symbol: EVP_PKEY_size. Did you mean: 'EVP_PKEY_free'?

guix: update time-machine to 160f78a4d92205df986ed9efcce7d3aac188cb24

In our time-machine environment this changes the following:

GCC 10.3.0 -> 10.4.0
Binutils 2.37 -> 2.38
Linux Libre Headers 5.15.37 -> 5.15.127
git 2.36.0 -> 2.41.0
mingw-w64 8.0.0 -> 11.0.1
NSIS 3.05 -> 3.09
xorriso 1.5.2 -> 1.5.6.pl02
Python 3.9 -> 3.10.7
Python-asn1crypto 1.4.0 -> 1.5.1

GCC 12.3.0 becomes available.
LLVM 15.0.7 becomes available.

guix: use cross-* keyword arguments

Using the new time-machine results in warnings about consistently using
keyword arguments:
```bash
guix environment: warning: 'cross-kernel-headers' must be used with keyword arguments
guix environment: warning: 'cross-libc' must be used with keyword arguments
```

guix: drop NSIS patch now that we use 3.09

See https://sourceforge.net/p/nsis/bugs/1283/.

guix: drop Windows broken-longjmp.patch

This is no-longer required, now that we are building using GCC 10.4.0.

depends: use LLVM/Clang 15.0.6 for macOS cross-compile

There is no x86_64 binaries for 15.0.7.

guix: use clang-toolchain-15 for macOS compilation

ci: Run "Win64 native" job on GitHub Actions

refactor: Use HashWriter over legacy CHashWriter

refactor: Use HashWriter over legacy CHashWriter (via SerializeHash)

config: default acceptnonstdtxn=0 on all chains

Previously, the default for acceptnonstdtxn defaulted to 0 on all
chains except testnet. Change this to be consistent across all
chains, and remove the parameter from chainparams entirely.

doc: Release notes for testnet defaulting to -acceptnonstdtxn=0

script: replace deprecated pkg_resources with importlib.metadata

in our python linter:

```
./test/lint/lint-python.py:12: DeprecationWarning: pkg_resources is deprecated as an API.
  See https://setuptools.pypa.io/en/latest/pkg_resources.html
  import pkg_resources
```

The importlib.metadata library was added in Python 3.8, which is currently our
minimum-supported Python version.

For more details about importlib.metadata, see https://docs.python.org/3/library/importlib.metadata.html

ci, windows: Do not run extended functional tests for pull requests

This change is intended to speed up the CI feedback for pull requests.

test: Support powerpc64le in get_previous_releases.py

guix: remove GCC 10 workaround from NSIS

Fixed upstream in 3.06, see
https://github.com/kichik/nsis/commit/229b6136c41ba5caba25936f4927476d20aa283f.
https://sourceforge.net/p/nsis/bugs/1248/

doc: Remove sudo from command that is already run as root

ci: Remove /ro_base bind mount

Just set the bind mount to BASE_READ_ONLY_DIR, which allows to drop one
line of code and makes the code easier to understand.

ci: Remove unused TEST_RUNNER_ENV="LC_ALL=C" from s390x task

gui: make '-min' minimize wallet loading dialog

When '-min' is enabled, no loading dialog should
be presented on screen during startup.

doc: Fill in the required skills in the good_first_issue template

[log] include wtxid in tx {relay,validation,orphanage} logging

[log] add category TXPACKAGES for orphanage and package relay

[log] add more logs related to orphan handling

- Whenever a tx is erased. Allows somebody to see which transactions
  have been erased due to expiry/overflow, not just how many.
- Whenever a tx is added to a peer's workset.
- AcceptToMemoryPool when a tx is accepted, mirroring the one logged for
  a tx received from a peer. This allows someone to see all of the
  transactions that are accepted to mempool just by looking for ATMP logs.
- MEMPOOLREJ when a tx is rejected, mirroring the one logged for
  a tx received from a peer. This allows someone to see all of the
  transaction rejections by looking at MEMPOOLREJ logs.

[doc] move comment about AlreadyHaveTx DoS score to the right place

This comment isn't in the right place, as detection of a tx in
recent_rejects would cause the function to exit much earlier.
Move the comment to the right place and tweak the first sentence for
accuracy.

ci: Avoid oversubscription in functional tests on Windows

ci: Avoid saving the same Ccache cache

This occurred when a job was being rerun.

log: Print error message when coindb is in inconsistent state

qt: Translation updates from Transifex

The diff is generated by executing the `update-translations.py` script.

qt: Bump Transifex slug for 26.x

qt: Update translation source file

The diff is generated by executing `make -C src translate`.

removed StrFormatInternalBug quote delimitation

ci: Bump `actions/checkout` version

See: https://github.com/actions/checkout/releases/tag/v4.0.0

test: p2p: check that `getaddr` msgs are only responded once per connection

test: remove fixed timeouts from feature_config_args

They cannot be scaled by the timeout_factor option and can
therefore cause timeouts in slow environments.
They are also not necessary for the test, since they measure time
frome startup until a debug message is encountered, which
is not restricted to 1 minute by any internal logic within bitcoind.

ci: Asan with -ftrivial-auto-var-init=pattern

Squashed 'src/secp256k1/' changes from c545fdc374..199d27cea3

199d27cea3 Merge bitcoin-core/secp256k1#1415: release: Prepare for 0.4.0
16339804c9 release: Prepare for 0.4.0
d9a85065a9 changelog: Catch up in preparation of release
0b4640aedd Merge bitcoin-core/secp256k1#1413: ci: Add `release` job
8659a01714 ci: Add `release` job
f9b38894ba ci: Update `actions/checkout` version
727bec5bc2 Merge bitcoin-core/secp256k1#1414: ci/gha: Add ARM64 QEMU jobs for clang and clang-snapshot
2635068abf ci/gha: Let MSan continue checking after errors in all jobs
e78c7b68eb ci/Dockerfile: Reduce size of Docker image further
2f0d3bbffb ci/Dockerfile: Warn if `ulimit -n` is too high when running Docker
4b8a647ad3 ci/gha: Add ARM64 QEMU jobs for clang and clang-snapshot
6ebe7d2bb3 ci/Dockerfile: Always use versioned clang packages
65c79fe2d0 Merge bitcoin-core/secp256k1#1412: ci: Switch macOS from Ventura to Monterey and add Valgrind
c223d7e33d ci: Switch macOS from Ventura to Monterey and add Valgrind
ea26b71c3a Merge bitcoin-core/secp256k1#1411: ci: Make repetitive command the default one
cce0456304 ci: Make repetitive command the default one
317a4c48f0 ci: Move `git config ...` to `run-in-docker-action`
4d7fe60905 Merge bitcoin-core/secp256k1#1409: ci: Move remained task from Cirrus to GitHub Actions
676ed8f9cf ci: Move "C++ (public headers)" from Cirrus to GitHub Actions
61fc3a2dc8 ci: Move "C++ -fpermissive..." from Cirrus to GitHub Actions
d51fb0a533 ci: Move "MSan" from Cirrus to GitHub Actions
c22ac27529 ci: Move sanitizers task from Cirrus to GitHub Actions
26a989924b Merge bitcoin-core/secp256k1#1410: ci: Use concurrency for pull requests only
ee1be62d84 ci: Use concurrency for pull requests only
6ee14550c8 Merge bitcoin-core/secp256k1#1406: ci, gha: Move more non-x86_64 tasks from Cirrus CI to GitHub Actions
fc3dea29ea ci: Move "ppc64le: Linux..." from Cirrus to GitHub Actions
7782dc8276 ci: Move "ARM64: Linux..." from Cirrus to GitHub Actions
0a16de671c ci: Move "ARM32: Linux..." from Cirrus to GitHub Actions
ea33914e00 ci: Move "s390x (big-endian): Linux..." from Cirrus to GitHub Actions
880be8af99 ci: Move "i686: Linux (Debian stable)" from Cirrus to GiHub Actions
2e6cf9bae5 Merge bitcoin-core/secp256k1#1396: ci, gha: Add "x86_64: Linux (Debian stable)" GitHub Actions job
5373693e45 Merge bitcoin-core/secp256k1#1405: ci: Drop no longer needed workaround
ef9fe959de ci: Drop no longer needed workaround
e10878f58e ci, gha: Drop `driver-opts.network` input for `setup-buildx-action`
4ad4914bd1 ci, gha: Add `retry_builder` Docker image builder
6617a620d9 ci: Remove "x86_64: Linux (Debian stable)" task from Cirrus CI
03c9e6508c ci, gha: Add "x86_64: Linux (Debian stable)" GitHub Actions job
ad3e65d9fe ci: Remove GCC build files and sage to reduce size of Docker image
6b9507adf6 Merge bitcoin-core/secp256k1#1398: ci, gha: Add Windows jobs based on Linux image
87d35f30c0 ci: Rename `cirrus.sh` to more general `ci.sh`
d6281dd008 ci: Remove Windows tasks from Cirrus CI
2b6f9cd546 ci, gha: Add Windows jobs based on Linux image
48b1d939b5 Merge bitcoin-core/secp256k1#1403: ci, gha: Ensure only a single workflow processes `github.ref` at a time
0ba2b94551 Merge bitcoin-core/secp256k1#1373: Add invariant checking for scalars
060e32cb60 Merge bitcoin-core/secp256k1#1401: ci, gha: Run all MSVC tests on Windows natively
de657c2044 Merge bitcoin-core/secp256k1#1062: Removes `_fe_equal_var`, and unwanted `_fe_normalize_weak` calls (in tests)
bcffeb14bc Merge bitcoin-core/secp256k1#1404: ci: Remove "arm64: macOS Ventura" task from Cirrus CI
c2f6435802 ci: Add comment about switching macOS to M1 on GHA later
4a24fae0bc ci: Remove "arm64: macOS Ventura" task from Cirrus CI
b0886fd35c ci, gha: Ensure only a single workflow processes `github.ref` at a time
3d05c86d63 Merge bitcoin-core/secp256k1#1394: ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions
d78bec7001 ci: Remove Windows MSVC tasks from Cirrus CI
3545dc2b9b ci, gha: Run all MSVC tests on Windows natively
5d8fa825e2 Merge bitcoin-core/secp256k1#1274: test: Silent noisy clang warnings about Valgrind code on macOS x86_64
8e54a346d2 ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions
b327abfcea Merge bitcoin-core/secp256k1#1402: ci: Use Homebrew's gcc in native macOS task
d62db57427 ci: Use Homebrew's gcc in native macOS task
54058d16fe field: remove `secp256k1_fe_equal_var`
bb4efd6404 tests: remove unwanted `secp256k1_fe_normalize_weak` call
eedd781085 Merge bitcoin-core/secp256k1#1348: tighten group magnitude limits, save normalize_weak calls in group add methods (revival of #1032)
b2f6712dd3 Merge bitcoin-core/secp256k1#1400: ctimetests: Use new SECP256K1_CHECKMEM macros also for ellswift
9c91ea41b1 ci: Enable ellswift module where it's missing
db32a24761 ctimetests: Use new SECP256K1_CHECKMEM macros also for ellswift
ce765a5b8e Merge bitcoin-core/secp256k1#1399: ci, gha: Run "SageMath prover" job on GitHub Actions
8408dfdc4c Revert "ci: Run sage prover on CI"
c8d9914fb1 ci, gha: Run "SageMath prover" job on GitHub Actions
8d2960c8e2 Merge bitcoin-core/secp256k1#1397: ci: Remove "Windows (VS 2022)" task from Cirrus CI
f1774e5ec4 ci, gha: Make MSVC job presentation more explicit
5ee039bb58 ci: Remove "Windows (VS 2022)" task from Cirrus CI
96294c00fb Merge bitcoin-core/secp256k1#1389: ci: Run "Windows (VS 2022)" job on GitHub Actions
a2f7ccdecc ci: Run "Windows (VS 2022)" job on GitHub Actions
374e2b54e2 Merge bitcoin-core/secp256k1#1290: cmake: Set `ENVIRONMENT` property for examples on Windows
1b13415df9 Merge bitcoin-core/secp256k1#1391: refactor: take use of `secp256k1_scalar_{zero,one}` constants (part 2)
a1bd4971d6 refactor: take use of `secp256k1_scalar_{zero,one}` constants (part 2)
b7c685e74a Save _normalize_weak calls in group add methods
c83afa66e0 Tighten group magnitude limits
26392da2fb Merge bitcoin-core/secp256k1#1386: ci: print $ELLSWIFT in cirrus.sh
d23da6d557 use secp256k1_scalar_verify checks
4692478853 ci: print $ELLSWIFT in cirrus.sh
c7d0454932 add verification for scalars
c734c64278 Merge bitcoin-core/secp256k1#1384: build: enable ellswift module via SECP_CONFIG_DEFINES
ad152151b0 update max scalar in scalar_cmov_test and fix schnorrsig_verify exhaustive test
78ca880788 build: enable ellswift module via SECP_CONFIG_DEFINES
0e00fc7d10 Merge bitcoin-core/secp256k1#1383: util: remove unused checked_realloc
b097a466c1 util: remove unused checked_realloc
2bd5f3e618 Merge bitcoin-core/secp256k1#1382: refactor: Drop unused cast
4f8c5bd761 refactor: Drop unused cast
173e8d061a Implement current magnitude assumptions
49afd2f5d8 Take use of _fe_verify_magnitude in field_impl.h
4e9661fc42 Add _fe_verify_magnitude (no-op unless VERIFY is enabled)
690b0fc05a add missing group element invariant checks
175db31149 ci: Drop no longer needed `PATH` variable update on Windows
116d2ab3df cmake: Set `ENVIRONMENT` property for examples on Windows
cef373997c cmake, refactor: Use helper function instead of interface library
747ada3587 test: Silent noisy clang warnings about Valgrind code on macOS x86_64

git-subtree-dir: src/secp256k1
git-subtree-split: 199d27cea32203b224b208627533c2e813cd3b21

scripted-diff: Use blocks_path where possible

-BEGIN VERIFY SCRIPT-
  sed -i 's|].chain_path, .blocks.|].blocks_path|g' $(git grep -l chain_path)
-END VERIFY SCRIPT-

index: Drop legacy -txindex check

move-only: Move CBlockTreeDB to node/blockstorage

The block index (CBlockTreeDB) is required to write and read blocks, so
move it to blockstorage. This allows to drop the txdb.h include from
`node/blockstorage.h`.

Can be reviewed with:

--color-moved=dimmed-zebra  --color-moved-ws=ignore-all-space

Fixup style of moved code

Can be reviewed with --word-diff-regex=.

scripted-diff: Rename CBlockTreeDB -> BlockTreeDB

-BEGIN VERIFY SCRIPT-
 sed -i 's|CBlockTreeDB|BlockTreeDB|g' $( git grep -l CBlockTreeDB )
-END VERIFY SCRIPT-

build: use -muse-unaligned-vector-move for Windows

We currently work around a longstanding GCC issue with aligned vector
instructions, in our release builds, by patching the behaviour we want
into GCC (see discussion in #24736).

A new option now exists in the binutils assembler,
`-muse-unaligned-vector-move`, which should also achieve the behaviour
we want (at least for our code). This was added in the 2.38 release,
see
https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=c8480b58e1968f209b6365af7422678f348222c2.
```bash
x86: Add -muse-unaligned-vector-move to assembler

Unaligned load/store instructions on aligned memory or register are as
fast as aligned load/store instructions on modern Intel processors.  Add
a command-line option, -muse-unaligned-vector-move, to x86 assembler to
encode encode aligned vector load/store instructions as unaligned
vector load/store instructions.
```

Even if we introduce this option into our build system, we'll have to
maintain our GCC patching, as we want all code that ends up in the
binary, to avoid these instructions. However, there may be some value in
adding the option, as it could be an improvement for someone building
(bitcoind.exe) with an unpatched compiler.

test: Combine sync_send_with_ping and sync_with_ping

miniscript: make GetStackSize independent of P2WSH context

It was taking into account the P2WSH script push in the number of stack
elements.

miniscript: introduce a helper to get the maximum witness size

Similarly to how we compute the maximum stack size.

Also note how it would be quite expensive to recompute it recursively
by accounting for different ECDSA signature sizes. So we just assume
high-R everywhere. It's only a trivial difference anyways.

descriptor: introduce a method to get the satisfaction size

In the wallet code, we are currently estimating the size of a signed
input by doing a dry run of the signing logic. This is unnecessary as
all outputs we are able to sign for can be represented by a descriptor,
and we can derive the size of a satisfaction ("signature") from the
descriptor itself directly.
In addition, this approach does not scale: getting the size of a
satisfaction through a dry run of the signing logic is only possible for
the most basic scripts.

This commit introduces the computation of the size of satisfaction per
descriptor. It's a bit intricate for 2 main reasons:
- We want to conserve the behaviour of the current dry-run logic used by
  the wallet that sometimes assumes ECDSA signatures will be low-r,
  sometimes not (when we don't create them).
- We need to account for the witness discount. A single descriptor may
  sometimes benefit of it, sometimes not (for instance `pk()` if used as
  top-level versus if used inside `wsh()`).

script/signingprovider: introduce a MultiSigningProvider

It is sometimes useful to interface with multiple signing providers at
once. For instance when inferring a descriptor with solving information
being provided from multiple sources (see next commit).

Instead of inneficiently copying the information from one provider into
the other, introduce a new signing provider that takes a list of
pointers to existing providers.

wallet: use descriptor satisfaction size to estimate inputs size

Instead of using the dummysigner to compute a placeholder satisfaction,
infer a descriptor on the scriptPubKey of the coin being spent and use
the estimation of the satisfaction size given by the descriptor
directly.

Note this (almost, see next paragraph) exactly conserves the previous
behaviour. For instance CalculateMaximumSignedInputSize was previously
assuming the input to be spent in a transaction that spends at least one
Segwit coin, since it was always accounting for the serialization of the
number of witness elements.

In this commit we use a placeholder for the size of the serialization of
the witness stack size (1 byte). Since the logic in this commit is
already tricky enough to review, and that it is only a very tiny
approximation not observable through the existing tests, it is addressed
in the next commit.

wallet: accurately account for the size of the witness stack

When estimating the maximum size of an input, we were assuming the
number of elements on the witness stack could be encode in a single
byte. This is a valid approximation for all the descriptors we support
(including P2WSH Miniscript ones), but may not hold anymore once we
support Miniscript within Taproot descriptors (since the max standard
witness stack size of 100 gets lifted).

It's a low-hanging fruit to account for it correctly, so just do it now.

fuzz: introduce and use `ConsumePrivateKey` helper

doc: s/--no-substitute/--no-substitutes in guix/INSTALL

Replace READWRITEAS macro with AsBase wrapping function

Co-authored-by: Pieter Wuille <pieter@wuille.net>

Rename CSerAction* to Action*

This allows new code, added in the next commit, to conform to the coding
guideline: No C-prefix for class names.

Support for serialization parameters

The moved part can be reviewed with the git options
 --ignore-all-space --color-moved=dimmed-zebra --color-moved-ws=ignore-all-space

(Modified by Marco Falke)

Co-authored-by: Pieter Wuille <pieter@wuille.net>

Use serialization parameters for CAddress serialization

This also cleans up the addrman (de)serialization code paths to only
allow `Disk` serialization. Some unit tests previously forced a
`Network` serialization, which does not make sense, because Bitcoin Core
in production will always `Disk` serialize.
This cleanup idea was suggested by Pieter Wuille and implemented by Anthony
Towns.

Co-authored-by: Pieter Wuille <pieter@wuille.net>
Co-authored-by: Anthony Towns <aj@erisian.com.au>

test: add tests that exercise WithParams()

Co-authored-by: Vasil Dimov <vd@FreeBSD.org>

Remove unused legacy CHashVerifier

depends: libtapi 1300.0.6.5

depends: cctools 986 & ld64 711

test: remove unused variables in `p2p_invalid_block`

fuzz: add ConstructPubKeyBytes function

Today, this code only has one spot where it needs well-formed pubkeys,
but future PRs will want to reuse this code.

Add a function which creates a well-formed byte array that can be turned
into a pubkey. It is not required that the pubkey is valid, just that it
can be recognized as a compressed or uncompressed pubkey.

Note: while the main intent of this commit is to wrap the existing
logic into a function, it also switches to `PickValueFromArray` so that
we are only choosing one of 0x04, 0x06, or 0x07. The previous code,
`ConsumeIntegralInRange` would have also picked 0x05, which is not
definied in the context of compressed vs uncompressed keys.

See https://bitcoin.stackexchange.com/questions/57855/c-secp256k1-what-do-prefixes-0x06-and-0x07-in-an-uncompressed-public-key-signif
for more details.

net: add have_next_message argument to Transport::GetBytesToSend()

Before this commit, there are only two possibly outcomes for the "more" prediction
in Transport::GetBytesToSend():
* true: the transport itself has more to send, so the answer is certainly yes.
* false: the transport has nothing further to send, but if vSendMsg has more message(s)
         left, that still will result in more wire bytes after the next
         SetMessageToSend().

For the BIP324 v2 transport, there will arguably be a third state:
* definitely not: the transport has nothing further to send, but even if vSendMsg has
                  more messages left, they can't be sent (right now). This happens
                  before the handshake is complete.

To implement this, we move the entire decision logic to the Transport, by adding a
boolean to GetBytesToSend(), called have_next_message, which informs the transport
whether more messages are available. The return values are still true and false, but
they mean "definitely yes" and "definitely no", rather than "yes" and "maybe".

net: remove unused Transport::SetReceiveVersion

crypto: Spanify EllSwiftPubKey constructor

net: add V2Transport class with subset of BIP324 functionality

This introduces a V2Transport with a basic subset of BIP324 functionality:
* no ability to send garbage (but receiving is supported)
* no ability to send decoy packets (but receiving them is supported)
* no support for short message id encoding (neither encoding or decoding)
* no waiting until 12 non-V1 bytes have been received
* (and thus) no detection of V1 connections on the responder side
  (on the sender side, detecting V1 is not supported either, but that needs
  to be dealt with at a higher layer, by reconnecting)

net: make V2Transport auto-detect incoming V1 and fall back to it

net: add short message encoding/decoding support to V2Transport

net: make V2Transport send uniformly random number garbage bytes

net: make V2Transport preallocate receive buffer space

test: add unit tests for V2Transport

net: detect wrong-network V1 talking to V2Transport

Remove version/hashing options from CBlockLocator/CDiskBlockIndex

refactor: Use DataStream now that version/type are unused

refactor: remove clientversion include from dbwrapper.h

consensus/validation.h: remove needless GetTransactionOutputWeight helper

Introduced in 9b7ec393b82ca9d7ada77d06e0835df0386a8b85. This copied the format of the other Get.*Weight helpers but it's useless for a CTxOut.

test: refactor: remove unnecessary blocks_checked counter

Since we already store all the blocks in `events`, keeping an
additional counter is redundant.

test: refactor:  rename inbound to is_inbound

Makes it easier to recognize this variable represents a flag.

test: refactor: deduplicate handle_utxocache_* logic

Carve out the comparison logic into a helper function to avoid
code duplication.

test: store utxocache events

By storing the events instead of doing the comparison inside the
handle_utxocache_* functions, we simplify the overall logic and
potentially making debugging easier, by allowing pdb to access the
events.

Mostly a refactor, but changes logging behaviour slightly by not
raising and not calling self.log.exception("Assertion failed")

test: log sanity check assertion failures

test: refactor: remove unnecessary nonlocal

Since we're only mutating, and not reassigning, we don't need to
declare `events` as `nonlocal`.

test: refactor: usdt_mempool: store all events

Even though we expect these functions to only produce one event,
we still keep a counter to check if that's true. By simply storing
all the events, we can remove the counters and make debugging
easier, by allowing pdb to access the events.

net: merge V2Transport constructors, move key gen

This removes the ability for BIP324Cipher to generate its own key, moving that
responsibility to the caller (mostly, V2Transport). This allows us to write
the random-key V2Transport constructor by delegating to the explicit-key one.

net: do not use send buffer to store/cache garbage

Before this commit the V2Transport::m_send_buffer is used to store the
garbage:
* During MAYBE_V1 state, it's there despite not being sent.
* During AWAITING_KEY state, while it is being sent.
* At the end of the AWAITING_KEY state it cannot be wiped as it's still
  needed to compute the garbage authentication packet.

Change this by introducing a separate m_send_garbage field, taking over
the first and last role listed above. This means the garbage is only in
the send buffer when it's actually being sent, removing a few special
cases related to this.

doc: fix typos and mistakes in BIP324 code comments

index: coinstats reorg, fail when block cannot be reversed

During a reorg, continuing execution when a block cannot be
reversed leaves the coinstats index in an inconsistent state,
which was surely overlooked when 'CustomRewind' was implemented.

index: add [nodiscard] attribute to functions writing to the db

rpc: Deprecate rpcserialversion=0

doc: Clarify that -fstack-reuse=all bugs exist on all versions of GCC

wallet rpc: return final tx hex from walletprocesspsbt if complete

test: remove unnecessary finalizepsbt rpc calls

doc: add release note for PR #28414

doc, refactor: Changing -torcontrol help to specify that a default port is used

Right now when we get the help for -torcontrol it says that there is a
default ip and port we dont specify if there is a specified ip that we
would also use port 9051 as default

[policy] check for duplicate txids in package

Duplicates of normal transactions would be found by looking for
conflicting inputs, but this doesn't catch identical empty transactions.
These wouldn't be valid but exiting early is good and AcceptPackage's
result sanity checks assume non-duplicate transactions.

[CCoinsViewMemPool] track non-base coins and allow Reset

Temporary coins should not be available in separate subpackage submissions.
Any mempool coins that are cached in m_view should be removed whenever
mempool contents change, as they may be spent or no longer exist.

[validation] add AcceptSubPackage to delegate Accept* calls and clean up m_view

(1) Call AcceptSingleTransaction when there is only 1 transaction in the
  subpackage. This avoids calling PackageMempoolChecks() which enforces
rules that don't need to be applied for a single transaction, i.e.
disabling CPFP carve out.
There is a slight change in the error type returned, as shown in the
txpackage_tests change.  When a transaction is the last one left in the
package and its fee is too low, this returns a PCKG_TX instead of
PCKG_POLICY. This interface is clearer; "package-fee-too-low" for 1
transaction would be a bit misleading.

(2) Clean up m_view and m_viewmempool so that coins created in this
sub-package evaluation are not available for other sub-package
evaluations. The contents of the mempool may change, so coins that are
available now might not be later.

[validation] make PackageMempoolAcceptResult members mutable

After the PackageMempoolAcceptResult is returned from
AcceptMultipleTransactions, leave room for results to change due to
LimitMempool() eviction.

[refactor] back-fill results in AcceptPackage

Instead of populating the last PackageMempoolAcceptResult with stuff
from results_final and individual_results_nonfinal, fill results_final
and create a PackageMempoolAcceptResult using that one.

A future commit will add LimitMempoolSize() which may change the status
of each of these transactions from "already in mempool" or "submitted to
mempool" to "no longer in mempool". We will change those transactions'
results here.

A future commit also gets rid of the last AcceptSubPackage outside of
the loop. It makes more sense to use results_final as the place where
all results end up.

[validation] return correct result when already-in-mempool tx gets evicted

Bug fix: a transaction may be in the mempool when package evaluation
begins (so it is added to results_final with MEMPOOL_ENTRY or
DIFFERENT_WITNESS), but get evicted due to another transaction
submission.

[validation] don't LimitMempoolSize in any subpackage submissions

Don't do any mempool evictions until package validation is done,
preventing the mempool minimum feerate from changing. Whether we submit
transactions separately or as a package depends on whether they meet the
mempool minimum feerate threshold, so it's best that the value not
change while we are evaluating a package.
This avoids a situation where we have a CPFP package in which
the parents meet the mempool minimum feerate and are submitted by
themselves, but they are evicted before we have submitted the child.

[test framework] add ability to spend only confirmed utxos

Useful to ensure that the topologies of packages/transactions are as
expected, preventing bugs caused by having unexpected mempool ancestors.

[refactor] split setup in mempool_limit test

We want to be able to re-use fill_mempool so that none of the tests
affect each other.

Change the logs from info to debug because they are otherwise repeated
many times in the test output.

[test] mempool coins disappearing mid-package evaluation

Test for scenario(s) outlined in PR 28251.
Test what happens when a package transaction spends a mempool coin which
is fetched and then disappears mid-package evaluation due to eviction or
replacement.

dbwrapper: Use DataStream for batch operations

Remove unused GetType() from CBufferedFile and CAutoFile

GetType() is only called in tests, so it is unused and can be removed.

scripted-diff: Rename CBufferedFile to BufferedFile

While touching all constructors in the previous commit, the class name
can be adjusted to comply with the style guide.

-BEGIN VERIFY SCRIPT-
 sed -i 's/CBufferedFile/BufferedFile/g' $( git grep -l CBufferedFile )
-END VERIFY SCRIPT-

ci: Add test-each-commit task

ci: Limit test-each-commit to --max-count=6

[refactor] Allow std::array<std::byte, N> in serialize.h

This is already possible for C-style arrays, so allow it for C++11
std::array as well.

[refactor] Define MessageStartChars as std::array

kernel: Move MessageStartChars to its own file

The protocol.h file contains many non-consensus related definitions and
should thus not be part of the libbitcoinkernel. This commit makes
protocol.h no longer a required include for users of the
libbitcoinkernel.

This commit is part of the libbitcoinkernel project, namely its stage 1
step 3: Decouple most non-consensus headers from libbitcoinkernel.

Co-Authored-By: Cory Fields <cory-nospam-@coryfields.com>

[refactor] Add missing includes for next commit

[refactor] Add CChainParams member to CConnman

This is done in preparation to the next commit, but has the nice
effect of removing one further data structure relying on the global
`Params()`.

[refactor] Remove netaddress.h from kernel headers

Move functions requiring the netaddress.h include out of
libbitcoinkernel source files.

The netaddress.h file contains many non-consensus related definitions
and should thus not be part of the libbitcoinkernel. This commit makes
netaddress.h no longer a required include for users of the
libbitcoinkernel.

This commit is part of the libbitcoinkernel project, namely its stage 1
step 3: Decouple most non-consensus headers from libbitcoinkernel.

[refactor] Remove compat.h from kernel headers

This commit makes compat.h no longer a required include for users of the
libbitcoinkernel. Including compat.h imports a bunch of
platform-specific definitions.

This commit is part of the libbitcoinkernel project, namely its stage 1
step 3: Decouple most non-consensus headers from libbitcoinkernel.

ci: clang-17 for fuzz and tsan

[fuzz] Use afl++ shared-memory fuzzing

Using shared-memory is faster than reading from stdin, see
https://github.com/AFLplusplus/AFLplusplus/blob/7d2122e0596132f9344a5d0896020ebc79cd33db/instrumentation/README.persistent_mode.md

Revert "Merge bitcoin/bitcoin#28279: ci: Add test-each-commit task"

This reverts commit…
real-or-random added a commit to BlockstreamResearch/secp256k1-zkp that referenced this pull request Oct 12, 2023
…02, 1274, 1394, 1404, 1062, 1401, 1373, 1403, 1398, 1405, 1396, 1406, 1410, 1409, 1411, 1412, 1414, 1413, 1415, 1417, 1390, 1416, 1422, 1424, 1395

b41caaa bppp: replace memcmp in tests with secp256k1_memcmp_var (Jonas Nick)
6a3aae8 group_parse: use secp256k1_memcmp_var instead of memcmp (Jonas Nick)
e9d522f ci: turn on -zkp modules in macos-native job (Jonas Nick)
c45b7c4 refactor: introduce testutil.h (deduplicate `random_fe_`, `ge_equals_` helpers) (Sebastian Falbesoner)
dc55141 tests: simplify `random_fe_non_zero` (remove loop limit and unneeded normalize) (Sebastian Falbesoner)
d9d80fd ci: Bump major versions for docker actions (Hennadii Stepanov)
421d848 ci: Align Autotools/CMake `CI_INSTALL` directory names (Hennadii Stepanov)
9f005c6 cmake: Install `libsecp256k1.pc` file (Hennadii Stepanov)
b0f7bfe doc: Do not mention soname in CHANGELOG.md "ABI Compatibility" section (Hennadii Stepanov)
bd9d98d doc: Align documented scripts with CI ones (Hennadii Stepanov)
7030364 tests: add CHECK_ERROR_VOID and use it in scratch tests (Jonas Nick)
f8d7ea6 tests: Replace counting_illegal_callbacks with CHECK_ILLEGAL_VOID (Jonas Nick)
a1d52e3 tests: remove unnecessary test in run_ec_pubkey_parse_test (Jonas Nick)
875b0ad tests: remove unnecessary set_illegal_callback (Jonas Nick)
9b118bc release cleanup: bump version after 0.4.0 (Jonas Nick)
1633980 release: Prepare for 0.4.0 (Tim Ruffing)
d9a8506 changelog: Catch up in preparation of release (Tim Ruffing)
8659a01 ci: Add `release` job (Hennadii Stepanov)
f9b3889 ci: Update `actions/checkout` version (Hennadii Stepanov)
2635068 ci/gha: Let MSan continue checking after errors in all jobs (Tim Ruffing)
e78c7b6 ci/Dockerfile: Reduce size of Docker image further (Tim Ruffing)
2f0d3bb ci/Dockerfile: Warn if `ulimit -n` is too high when running Docker (Tim Ruffing)
4b8a647 ci/gha: Add ARM64 QEMU jobs for clang and clang-snapshot (Tim Ruffing)
6ebe7d2 ci/Dockerfile: Always use versioned clang packages (Tim Ruffing)
c223d7e ci: Switch macOS from Ventura to Monterey and add Valgrind (Hennadii Stepanov)
cce0456 ci: Make repetitive command the default one (Hennadii Stepanov)
317a4c4 ci: Move `git config ...` to `run-in-docker-action` (Hennadii Stepanov)
676ed8f ci: Move "C++ (public headers)" from Cirrus to GitHub Actions (Hennadii Stepanov)
61fc3a2 ci: Move "C++ -fpermissive..." from Cirrus to GitHub Actions (Hennadii Stepanov)
d51fb0a ci: Move "MSan" from Cirrus to GitHub Actions (Hennadii Stepanov)
c22ac27 ci: Move sanitizers task from Cirrus to GitHub Actions (Hennadii Stepanov)
ee1be62 ci: Use concurrency for pull requests only (Hennadii Stepanov)
fc3dea2 ci: Move "ppc64le: Linux..." from Cirrus to GitHub Actions (Hennadii Stepanov)
7782dc8 ci: Move "ARM64: Linux..." from Cirrus to GitHub Actions (Hennadii Stepanov)
0a16de6 ci: Move "ARM32: Linux..." from Cirrus to GitHub Actions (Hennadii Stepanov)
ea33914 ci: Move "s390x (big-endian): Linux..." from Cirrus to GitHub Actions (Hennadii Stepanov)
880be8a ci: Move "i686: Linux (Debian stable)" from Cirrus to GiHub Actions (Hennadii Stepanov)
e10878f ci, gha: Drop `driver-opts.network` input for `setup-buildx-action` (Hennadii Stepanov)
4ad4914 ci, gha: Add `retry_builder` Docker image builder (Hennadii Stepanov)
6617a62 ci: Remove "x86_64: Linux (Debian stable)" task from Cirrus CI (Hennadii Stepanov)
03c9e65 ci, gha: Add "x86_64: Linux (Debian stable)" GitHub Actions job (Hennadii Stepanov)
ad3e65d ci: Remove GCC build files and sage to reduce size of Docker image (Tim Ruffing)
ef9fe95 ci: Drop no longer needed workaround (Hennadii Stepanov)
87d35f3 ci: Rename `cirrus.sh` to more general `ci.sh` (Hennadii Stepanov)
d6281dd ci: Remove Windows tasks from Cirrus CI (Hennadii Stepanov)
2b6f9cd ci, gha: Add Windows jobs based on Linux image (Hennadii Stepanov)
b0886fd ci, gha: Ensure only a single workflow processes `github.ref` at a time (Hennadii Stepanov)
d23da6d use secp256k1_scalar_verify checks (stratospher)
c7d0454 add verification for scalars (stratospher)
ad15215 update max scalar in scalar_cmov_test and fix schnorrsig_verify exhaustive test (stratospher)
d78bec7 ci: Remove Windows MSVC tasks from Cirrus CI (Hennadii Stepanov)
3545dc2 ci, gha: Run all MSVC tests on Windows natively (Hennadii Stepanov)
54058d1 field: remove `secp256k1_fe_equal_var` (siv2r)
bb4efd6 tests: remove unwanted `secp256k1_fe_normalize_weak` call (siv2r)
c2f6435 ci: Add comment about switching macOS to M1 on GHA later (Tim Ruffing)
4a24fae ci: Remove "arm64: macOS Ventura" task from Cirrus CI (Hennadii Stepanov)
8e54a34 ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions (Hennadii Stepanov)
747ada3 test: Silent noisy clang warnings about Valgrind code on macOS x86_64 (Hennadii Stepanov)
d62db57 ci: Use Homebrew's gcc in native macOS task (Hennadii Stepanov)
b7c685e Save _normalize_weak calls in group add methods (Peter Dettman)
c83afa6 Tighten group magnitude limits (Peter Dettman)
173e8d0 Implement current magnitude assumptions (Peter Dettman)
49afd2f Take use of _fe_verify_magnitude in field_impl.h (Sebastian Falbesoner)
4e9661f Add _fe_verify_magnitude (no-op unless VERIFY is enabled) (Peter Dettman)
690b0fc add missing group element invariant checks (Sebastian Falbesoner)
9c91ea4 ci: Enable ellswift module where it's missing (Tim Ruffing)
db32a24 ctimetests: Use new SECP256K1_CHECKMEM macros also for ellswift (Tim Ruffing)
8408dfd Revert "ci: Run sage prover on CI" (Hennadii Stepanov)
c8d9914 ci, gha: Run "SageMath prover" job on GitHub Actions (Hennadii Stepanov)
f1774e5 ci, gha: Make MSVC job presentation more explicit (Hennadii Stepanov)
5ee039b ci: Remove "Windows (VS 2022)" task from Cirrus CI (Hennadii Stepanov)
a2f7ccd ci: Run "Windows (VS 2022)" job on GitHub Actions (Hennadii Stepanov)
175db31 ci: Drop no longer needed `PATH` variable update on Windows (Hennadii Stepanov)
116d2ab cmake: Set `ENVIRONMENT` property for examples on Windows (Hennadii Stepanov)
cef3739 cmake, refactor: Use helper function instead of interface library (Hennadii Stepanov)
a1bd497 refactor: take use of `secp256k1_scalar_{zero,one}` constants (part 2) (Sebastian Falbesoner)

Pull request description:

  [bitcoin-core/secp256k1#1391]: refactor: take use of `secp256k1_scalar_{zero,one}` constants (part 2)
  [bitcoin-core/secp256k1#1290]: cmake: Set `ENVIRONMENT` property for examples on Windows
  [bitcoin-core/secp256k1#1389]: ci: Run "Windows (VS 2022)" job on GitHub Actions
  [bitcoin-core/secp256k1#1397]: ci: Remove "Windows (VS 2022)" task from Cirrus CI
  [bitcoin-core/secp256k1#1399]: ci, gha: Run "SageMath prover" job on GitHub Actions
  [bitcoin-core/secp256k1#1400]: ctimetests: Use new SECP256K1_CHECKMEM macros also for ellswift
  [bitcoin-core/secp256k1#1348]: tighten group magnitude limits, save normalize_weak calls in group add methods (revival of #1032)
  [bitcoin-core/secp256k1#1402]: ci: Use Homebrew'\''s gcc in native macOS task
  [bitcoin-core/secp256k1#1274]: test: Silent noisy clang warnings about Valgrind code on macOS x86_64
  [bitcoin-core/secp256k1#1394]: ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions
  [bitcoin-core/secp256k1#1404]: ci: Remove "arm64: macOS Ventura" task from Cirrus CI
  [bitcoin-core/secp256k1#1062]: Removes `_fe_equal_var`, and unwanted `_fe_normalize_weak` calls (in tests)
  [bitcoin-core/secp256k1#1401]: ci, gha: Run all MSVC tests on Windows natively
  [bitcoin-core/secp256k1#1373]: Add invariant checking for scalars
  [bitcoin-core/secp256k1#1403]: ci, gha: Ensure only a single workflow processes `github.ref` at a time
  [bitcoin-core/secp256k1#1398]: ci, gha: Add Windows jobs based on Linux image
  [bitcoin-core/secp256k1#1405]: ci: Drop no longer needed workaround
  [bitcoin-core/secp256k1#1396]: ci, gha: Add "x86_64: Linux (Debian stable)" GitHub Actions job
  [bitcoin-core/secp256k1#1406]: ci, gha: Move more non-x86_64 tasks from Cirrus CI to GitHub Actions
  [bitcoin-core/secp256k1#1410]: ci: Use concurrency for pull requests only
  [bitcoin-core/secp256k1#1409]: ci: Move remained task from Cirrus to GitHub Actions
  [bitcoin-core/secp256k1#1411]: ci: Make repetitive command the default one
  [bitcoin-core/secp256k1#1412]: ci: Switch macOS from Ventura to Monterey and add Valgrind
  [bitcoin-core/secp256k1#1414]: ci/gha: Add ARM64 QEMU jobs for clang and clang-snapshot
  [bitcoin-core/secp256k1#1413]: ci: Add `release` job
  [bitcoin-core/secp256k1#1415]: release: Prepare for 0.4.0
  [bitcoin-core/secp256k1#1417]: release cleanup: bump version after 0.4.0
  [bitcoin-core/secp256k1#1390]: tests: Replace counting_illegal_callbacks with CHECK_ILLEGAL_VOID
  [bitcoin-core/secp256k1#1416]: doc: Align documented scripts with CI ones
  [bitcoin-core/secp256k1#1422]: cmake: Install `libsecp256k1.pc` file
  [bitcoin-core/secp256k1#1424]: ci: Bump major versions for docker actions
  [bitcoin-core/secp256k1#1395]: tests: simplify `random_fe_non_zero` (remove loop limit and unneeded normalize)

  This PR can be recreated with `./contrib/sync-upstream.sh -b master range ee7aaf2`.
  Tip: Use `git show --remerge-diff` to show the changes manually added to the merge commit.' --web

  - Replace fe_equal_var with fe_equal
  - Use CHECK_ILLEGAL instead of CHECK/ecount
  - Turn on secp256k1-zkp specific modules in CI

ACKs for top commit:
  real-or-random:
    ACK b41caaa

Tree-SHA512: e8994fa2590d6e277d60a130d3f11b0256dc7fbe67e8a2b85463d9849d3f9fbfccf366b17b3904f1974f8b7b27a24128d2a04a3337c1b8409d000d136714bf90
janus pushed a commit to BitgesellOfficial/bitgesell that referenced this pull request Apr 1, 2024
199d27cea3 Merge bitcoin-core/secp256k1#1415: release: Prepare for 0.4.0
16339804c9 release: Prepare for 0.4.0
d9a85065a9 changelog: Catch up in preparation of release
0b4640aedd Merge bitcoin-core/secp256k1#1413: ci: Add `release` job
8659a01714 ci: Add `release` job
f9b38894ba ci: Update `actions/checkout` version
727bec5bc2 Merge bitcoin-core/secp256k1#1414: ci/gha: Add ARM64 QEMU jobs for clang and clang-snapshot
2635068abf ci/gha: Let MSan continue checking after errors in all jobs
e78c7b68eb ci/Dockerfile: Reduce size of Docker image further
2f0d3bbffb ci/Dockerfile: Warn if `ulimit -n` is too high when running Docker
4b8a647ad3 ci/gha: Add ARM64 QEMU jobs for clang and clang-snapshot
6ebe7d2bb3 ci/Dockerfile: Always use versioned clang packages
65c79fe2d0 Merge bitcoin-core/secp256k1#1412: ci: Switch macOS from Ventura to Monterey and add Valgrind
c223d7e33d ci: Switch macOS from Ventura to Monterey and add Valgrind
ea26b71c3a Merge bitcoin-core/secp256k1#1411: ci: Make repetitive command the default one
cce0456304 ci: Make repetitive command the default one
317a4c48f0 ci: Move `git config ...` to `run-in-docker-action`
4d7fe60905 Merge bitcoin-core/secp256k1#1409: ci: Move remained task from Cirrus to GitHub Actions
676ed8f9cf ci: Move "C++ (public headers)" from Cirrus to GitHub Actions
61fc3a2dc8 ci: Move "C++ -fpermissive..." from Cirrus to GitHub Actions
d51fb0a533 ci: Move "MSan" from Cirrus to GitHub Actions
c22ac27529 ci: Move sanitizers task from Cirrus to GitHub Actions
26a989924b Merge bitcoin-core/secp256k1#1410: ci: Use concurrency for pull requests only
ee1be62d84 ci: Use concurrency for pull requests only
6ee14550c8 Merge bitcoin-core/secp256k1#1406: ci, gha: Move more non-x86_64 tasks from Cirrus CI to GitHub Actions
fc3dea29ea ci: Move "ppc64le: Linux..." from Cirrus to GitHub Actions
7782dc8276 ci: Move "ARM64: Linux..." from Cirrus to GitHub Actions
0a16de671c ci: Move "ARM32: Linux..." from Cirrus to GitHub Actions
ea33914e00 ci: Move "s390x (big-endian): Linux..." from Cirrus to GitHub Actions
880be8af99 ci: Move "i686: Linux (Debian stable)" from Cirrus to GiHub Actions
2e6cf9bae5 Merge bitcoin-core/secp256k1#1396: ci, gha: Add "x86_64: Linux (Debian stable)" GitHub Actions job
5373693e45 Merge bitcoin-core/secp256k1#1405: ci: Drop no longer needed workaround
ef9fe959de ci: Drop no longer needed workaround
e10878f58e ci, gha: Drop `driver-opts.network` input for `setup-buildx-action`
4ad4914bd1 ci, gha: Add `retry_builder` Docker image builder
6617a620d9 ci: Remove "x86_64: Linux (Debian stable)" task from Cirrus CI
03c9e6508c ci, gha: Add "x86_64: Linux (Debian stable)" GitHub Actions job
ad3e65d9fe ci: Remove GCC build files and sage to reduce size of Docker image
6b9507adf6 Merge bitcoin-core/secp256k1#1398: ci, gha: Add Windows jobs based on Linux image
87d35f30c0 ci: Rename `cirrus.sh` to more general `ci.sh`
d6281dd008 ci: Remove Windows tasks from Cirrus CI
2b6f9cd546 ci, gha: Add Windows jobs based on Linux image
48b1d939b5 Merge bitcoin-core/secp256k1#1403: ci, gha: Ensure only a single workflow processes `github.ref` at a time
0ba2b94551 Merge bitcoin-core/secp256k1#1373: Add invariant checking for scalars
060e32cb60 Merge bitcoin-core/secp256k1#1401: ci, gha: Run all MSVC tests on Windows natively
de657c2044 Merge bitcoin-core/secp256k1#1062: Removes `_fe_equal_var`, and unwanted `_fe_normalize_weak` calls (in tests)
bcffeb14bc Merge bitcoin-core/secp256k1#1404: ci: Remove "arm64: macOS Ventura" task from Cirrus CI
c2f6435802 ci: Add comment about switching macOS to M1 on GHA later
4a24fae0bc ci: Remove "arm64: macOS Ventura" task from Cirrus CI
b0886fd35c ci, gha: Ensure only a single workflow processes `github.ref` at a time
3d05c86d63 Merge bitcoin-core/secp256k1#1394: ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions
d78bec7001 ci: Remove Windows MSVC tasks from Cirrus CI
3545dc2b9b ci, gha: Run all MSVC tests on Windows natively
5d8fa825e2 Merge bitcoin-core/secp256k1#1274: test: Silent noisy clang warnings about Valgrind code on macOS x86_64
8e54a346d2 ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions
b327abfcea Merge bitcoin-core/secp256k1#1402: ci: Use Homebrew's gcc in native macOS task
d62db57427 ci: Use Homebrew's gcc in native macOS task
54058d16fe field: remove `secp256k1_fe_equal_var`
bb4efd6404 tests: remove unwanted `secp256k1_fe_normalize_weak` call
eedd781085 Merge bitcoin-core/secp256k1#1348: tighten group magnitude limits, save normalize_weak calls in group add methods (revival of #1032)
b2f6712dd3 Merge bitcoin-core/secp256k1#1400: ctimetests: Use new SECP256K1_CHECKMEM macros also for ellswift
9c91ea41b1 ci: Enable ellswift module where it's missing
db32a24761 ctimetests: Use new SECP256K1_CHECKMEM macros also for ellswift
ce765a5b8e Merge bitcoin-core/secp256k1#1399: ci, gha: Run "SageMath prover" job on GitHub Actions
8408dfdc4c Revert "ci: Run sage prover on CI"
c8d9914fb1 ci, gha: Run "SageMath prover" job on GitHub Actions
8d2960c8e2 Merge bitcoin-core/secp256k1#1397: ci: Remove "Windows (VS 2022)" task from Cirrus CI
f1774e5ec4 ci, gha: Make MSVC job presentation more explicit
5ee039bb58 ci: Remove "Windows (VS 2022)" task from Cirrus CI
96294c00fb Merge bitcoin-core/secp256k1#1389: ci: Run "Windows (VS 2022)" job on GitHub Actions
a2f7ccdecc ci: Run "Windows (VS 2022)" job on GitHub Actions
374e2b54e2 Merge bitcoin-core/secp256k1#1290: cmake: Set `ENVIRONMENT` property for examples on Windows
1b13415df9 Merge bitcoin-core/secp256k1#1391: refactor: take use of `secp256k1_scalar_{zero,one}` constants (part 2)
a1bd4971d6 refactor: take use of `secp256k1_scalar_{zero,one}` constants (part 2)
b7c685e74a Save _normalize_weak calls in group add methods
c83afa66e0 Tighten group magnitude limits
26392da2fb Merge bitcoin-core/secp256k1#1386: ci: print $ELLSWIFT in cirrus.sh
d23da6d557 use secp256k1_scalar_verify checks
4692478853 ci: print $ELLSWIFT in cirrus.sh
c7d0454932 add verification for scalars
c734c64278 Merge bitcoin-core/secp256k1#1384: build: enable ellswift module via SECP_CONFIG_DEFINES
ad152151b0 update max scalar in scalar_cmov_test and fix schnorrsig_verify exhaustive test
78ca880788 build: enable ellswift module via SECP_CONFIG_DEFINES
0e00fc7d10 Merge bitcoin-core/secp256k1#1383: util: remove unused checked_realloc
b097a466c1 util: remove unused checked_realloc
2bd5f3e618 Merge bitcoin-core/secp256k1#1382: refactor: Drop unused cast
4f8c5bd761 refactor: Drop unused cast
173e8d061a Implement current magnitude assumptions
49afd2f5d8 Take use of _fe_verify_magnitude in field_impl.h
4e9661fc42 Add _fe_verify_magnitude (no-op unless VERIFY is enabled)
690b0fc05a add missing group element invariant checks
175db31149 ci: Drop no longer needed `PATH` variable update on Windows
116d2ab3df cmake: Set `ENVIRONMENT` property for examples on Windows
cef373997c cmake, refactor: Use helper function instead of interface library
747ada3587 test: Silent noisy clang warnings about Valgrind code on macOS x86_64

git-subtree-dir: src/secp256k1
git-subtree-split: 199d27cea32203b224b208627533c2e813cd3b21
hebasto added a commit to hebasto/secp256k1-CMake-example that referenced this pull request May 11, 2024
e3a885d Merge bitcoin-core/secp256k1#1522: release: prepare for 0.5.0
c0e4ec3 release: prepare for 0.5.0
bb528cf Merge bitcoin-core/secp256k1#1518: Add secp256k1_pubkey_sort
7d2591c Add secp256k1_pubkey_sort
da51507 Merge bitcoin-core/secp256k1#1058: Signed-digit multi-comb ecmult_gen algorithm
4c341f8 Add changelog entry for SDMC
a043940 Permit COMB_BITS < 256 for exhaustive tests
39b2f2a Add test case for ecmult_gen recoded = {-1,0,1}
644e86d Reintroduce projective blinding
07810d9 Reduce side channels from single-bit reads
a0d32b5 Optimization: use Nx32 representation for recoded bits
e03dcc4 Make secp256k1_scalar_get_bits support 32-bit reads
5005abe Rename scalar_get_bits -> scalar_get_bits_limb32; return uint32_t
6247f48 Optimization: avoid unnecessary doublings in precomputation
15d0cca Optimization: first table lookup needs no point addition
7a33db3 Optimization: move (2^COMB_BITS-1)/2 term into ctx->scalar_offset
ed2a056 Provide 3 configurations accessible through ./configure
5f7be9f Always generate tables for current (blocks,teeth) config
fde1dfc Signed-digit multi-comb ecmult_gen algorithm
486518b Make exhaustive tests's scalar_inverse(&x,&x) work
ab45c3e Initial gej blinding -> final ge blinding
aa00a6b Introduce CEIL_DIV macro and use it
d831168 Merge bitcoin-core/secp256k1#1515: ci: Note affected clangs in comment on ASLR quirk
a85e223 ci: Note affected clangs in comment on ASLR quirk
4b77fec Merge bitcoin-core/secp256k1#1512: msan: notate more variable assignments from assembly code
f7f0184 msan: notate more variable assignments from assembly code
a613391 change inconsistent array param to pointer
05bfab6 Merge bitcoin-core/secp256k1#1507: ci: Add workaround for ASLR bug in sanitizers
a5e8ab2 ci: Add sanitizer env variables to debug output
84a93de ci: Add workaround for ASLR bug in sanitizers
427e86b Merge bitcoin-core/secp256k1#1490: tests: improve fe_sqr test (issue #1472)
2028069 doc: clarify input requirements for secp256k1_fe_mul
11420a7 tests: improve fe_sqr test
cdc9a62 Merge bitcoin-core/secp256k1#1489: tests: add missing fe comparison checks for inverse field test cases
d926510 Merge bitcoin-core/secp256k1#1496: msan: notate variable assignments from assembly code
31ba404 msan: notate variable assignments from assembly code
e7ea32e msan: Add SECP256K1_CHECKMEM_MSAN_DEFINE which applies to memory sanitizer and not valgrind
e7bdddd refactor: rename `check_fe_equal` -> `fe_equal`
00111c9 tests: add missing fe comparison checks for inverse field test cases
0653a25 Merge bitcoin-core/secp256k1#1486: ci: Update cache action
94a14d5 ci: Update cache action
2483627 Merge bitcoin-core/secp256k1#1483: cmake: Recommend native CMake commands in README
5ad3aa3 Merge bitcoin-core/secp256k1#1484: tests: Drop redundant _scalar_check_overflow calls
51df2d9 tests: Drop redundant _scalar_check_overflow calls
3777e3f cmake: Recommend native CMake commands in README
e4af41c Merge bitcoin-core/secp256k1#1249: cmake: Add `SECP256K1_LATE_CFLAGS` configure option
3bf4d68 Merge bitcoin-core/secp256k1#1482: build: Clean up handling of module dependencies
e682267 build: Error if required module explicitly off
89ec583 build: Clean up handling of module dependencies
4437886 Merge bitcoin-core/secp256k1#1468: v0.4.1 release aftermath
a9db9f2 Merge bitcoin-core/secp256k1#1480: Get rid of untested sizeof(secp256k1_ge_storage) == 64 code path
74b7c3b Merge bitcoin-core/secp256k1#1476: include: make docs more consistent
b37fdb2 check-abi: Minor UI improvements
ad5f589 check-abi: Default to HEAD for new version
9fb7e2f release process: Style and formatting nits
ba5d72d assumptions: Use new STATIC_ASSERT macro
e53c2d9 Require that sizeof(secp256k1_ge_storage) == 64
d0ba2ab util: Add STATIC_ASSERT macro
da7bc1b include: in doc, remove article in front of "pointer"
aa3dd52 include: make doc about ctx more consistent
e3f6900 include: remove obvious "cannot be NULL" doc
d373bf6 Merge bitcoin-core/secp256k1#1474: tests: restore scalar_mul test
79e0945 Merge bitcoin-core/secp256k1#1473: Fix typos
3dbfb48 tests: restore scalar_mul test
d77170a Fix typos
e7053d0 release process: Add email step
429d21d release process: Run sanity checks on release PR
efe85c7 Merge bitcoin-core/secp256k1#1466: release cleanup: bump version after 0.4.1
4b2e06f release cleanup: bump version after 0.4.1
1ad5185 Merge bitcoin-core/secp256k1#1465: release: prepare for 0.4.1
672053d release: prepare for 0.4.1
1a81df8 Merge bitcoin-core/secp256k1#1380: Add ABI checking tool for release process
74a4d97 doc: Add ABI checking with `check-abi.sh` to the Release Process
e7f830e Add `tools/check-abi.sh`
77af1da Merge bitcoin-core/secp256k1#1455: doc: improve secp256k1_fe_set_b32_mod doc
3928b7c doc: improve secp256k1_fe_set_b32_mod doc
5e9a4d7 Merge bitcoin-core/secp256k1#990: Add comment on length checks when parsing ECDSA sigs
4197d66 Merge bitcoin-core/secp256k1#1431: Add CONTRIBUTING.md
0e5ea62 CONTRIBUTING: add some coding and style conventions
e2c9888 Merge bitcoin-core/secp256k1#1451: changelog: add entry for "field: Remove x86_64 asm"
d2e36a2 changelog: add entry for "field: Remove x86_64 asm"
1a432cb README: update first sentence
0922a04 docs: move coverage report instructions to CONTRIBUTING
76880e4 Add CONTRIBUTING.md including scope and guidelines for new code
d3e29db Merge bitcoin-core/secp256k1#1450: Add group.h ge/gej equality functions
04af0ba Replace ge_equals_ge[,j] calls with group.h equality calls
60525f6 Add unit tests for group.h equality functions
a47cd97 Add group.h ge/gej equality functions
10e6d29 Merge bitcoin-core/secp256k1#1446: field: Remove x86_64 asm
07687e8 Merge bitcoin-core/secp256k1#1393: Implement new policy for VERIFY_CHECK and #ifdef VERIFY (issue #1381)
bb46723 remove VERIFY_SETUP define
a3a3e11 remove unneeded VERIFY_SETUP uses in ECMULT_CONST_TABLE_GET_GE macro
a0fb68a introduce and use SECP256K1_SCALAR_VERIFY macro
cf25c86 introduce and use SECP256K1_{FE,GE,GEJ}_VERIFY macros
5d89bc0 remove superfluous `#ifdef VERIFY`/`#endif` preprocessor conditions
c2688f8 redefine VERIFY_CHECK to empty in production (non-VERIFY) mode
5814d84 Merge bitcoin-core/secp256k1#1438: correct assertion for secp256k1_fe_mul_inner
c1b4966 Merge bitcoin-core/secp256k1#1445: bench: add --help option to bench_internal
f07cead build: Don't call assembly an optimization
2f0762f field: Remove x86_64 asm
1ddd76a bench: add --help option to bench_internal
e721039 Merge bitcoin-core/secp256k1#1441: asm: add .note.GNU-stack section for non-exec stack
ea47c82 Merge bitcoin-core/secp256k1#1442: Return temporaries to being unsigned in secp256k1_fe_sqr_inner
dcdda31 Tighten secp256k1_fe_mul_inner's VERIFY_BITS checks
1027135 Return temporaries to being unsigned in secp256k1_fe_sqr_inner
33dc7e4 asm: add .note.GNU-stack section for non-exec stack
c891c5c Merge bitcoin-core/secp256k1#1437: ci: Ignore internal errors of snapshot compilers
8185e72 ci: Ignore internal errors in snapshot compilers
40f50d0 Merge bitcoin-core/secp256k1#1184: Signed-digit based ecmult_const algorithm
8e2a5fe correct assertion for secp256k1_fe_mul_inner
355bbdf Add changelog entry for signed-digit ecmult_const algorithm
21f49d9 Remove unused secp256k1_scalar_shr_int
115fdc7 Remove unused secp256k1_wnaf_const
aa9f3a3 ecmult_const: add/improve tests
4d16e90 Signed-digit based ecmult_const algorithm
ba523be make SECP256K1_SCALAR_CONST reduce modulo exhaustive group order
2140da9 Add secp256k1_scalar_half for halving scalars (+ tests/benchmarks).
1f1bb78 Merge bitcoin-core/secp256k1#1430: README: remove CI badge
5dab0ba README: remove CI badge
b314cf2 Merge bitcoin-core/secp256k1#1426: ci/cirrus: Add native ARM64 jobs
fa4d6c7 ci/cirrus: Add native ARM64 persistent workers
ee7aaf2 Merge bitcoin-core/secp256k1#1395: tests: simplify `random_fe_non_zero` (remove loop limit and unneeded normalize)
ba9cb6f Merge bitcoin-core/secp256k1#1424: ci: Bump major versions for docker actions
d9d80fd ci: Bump major versions for docker actions
4fd00f4 Merge bitcoin-core/secp256k1#1422: cmake: Install `libsecp256k1.pc` file
421d848 ci: Align Autotools/CMake `CI_INSTALL` directory names
9f005c6 cmake: Install `libsecp256k1.pc` file
2262d0e ci/cirrus: Bring back skeleton .cirrus.yml without jobs
b10ddd2 Merge bitcoin-core/secp256k1#1416: doc: Align documented scripts with CI ones
49be5be Merge bitcoin-core/secp256k1#1390: tests: Replace counting_illegal_callbacks with CHECK_ILLEGAL_VOID
cbf3053 Merge bitcoin-core/secp256k1#1417: release cleanup: bump version after 0.4.0
9b118bc release cleanup: bump version after 0.4.0
199d27c Merge bitcoin-core/secp256k1#1415: release: Prepare for 0.4.0
7030364 tests: add CHECK_ERROR_VOID and use it in scratch tests
f8d7ea6 tests: Replace counting_illegal_callbacks with CHECK_ILLEGAL_VOID
1633980 release: Prepare for 0.4.0
d9a8506 changelog: Catch up in preparation of release
b0f7bfe doc: Do not mention soname in CHANGELOG.md "ABI Compatibility" section
bd9d98d doc: Align documented scripts with CI ones
0b4640a Merge bitcoin-core/secp256k1#1413: ci: Add `release` job
8659a01 ci: Add `release` job
f9b3889 ci: Update `actions/checkout` version
a1d52e3 tests: remove unnecessary test in run_ec_pubkey_parse_test
875b0ad tests: remove unnecessary set_illegal_callback
727bec5 Merge bitcoin-core/secp256k1#1414: ci/gha: Add ARM64 QEMU jobs for clang and clang-snapshot
2635068 ci/gha: Let MSan continue checking after errors in all jobs
e78c7b6 ci/Dockerfile: Reduce size of Docker image further
2f0d3bb ci/Dockerfile: Warn if `ulimit -n` is too high when running Docker
4b8a647 ci/gha: Add ARM64 QEMU jobs for clang and clang-snapshot
6ebe7d2 ci/Dockerfile: Always use versioned clang packages
65c79fe Merge bitcoin-core/secp256k1#1412: ci: Switch macOS from Ventura to Monterey and add Valgrind
c223d7e ci: Switch macOS from Ventura to Monterey and add Valgrind
ea26b71 Merge bitcoin-core/secp256k1#1411: ci: Make repetitive command the default one
cce0456 ci: Make repetitive command the default one
317a4c4 ci: Move `git config ...` to `run-in-docker-action`
4d7fe60 Merge bitcoin-core/secp256k1#1409: ci: Move remained task from Cirrus to GitHub Actions
676ed8f ci: Move "C++ (public headers)" from Cirrus to GitHub Actions
61fc3a2 ci: Move "C++ -fpermissive..." from Cirrus to GitHub Actions
d51fb0a ci: Move "MSan" from Cirrus to GitHub Actions
c22ac27 ci: Move sanitizers task from Cirrus to GitHub Actions
26a9899 Merge bitcoin-core/secp256k1#1410: ci: Use concurrency for pull requests only
ee1be62 ci: Use concurrency for pull requests only
6ee1455 Merge bitcoin-core/secp256k1#1406: ci, gha: Move more non-x86_64 tasks from Cirrus CI to GitHub Actions
fc3dea2 ci: Move "ppc64le: Linux..." from Cirrus to GitHub Actions
7782dc8 ci: Move "ARM64: Linux..." from Cirrus to GitHub Actions
0a16de6 ci: Move "ARM32: Linux..." from Cirrus to GitHub Actions
ea33914 ci: Move "s390x (big-endian): Linux..." from Cirrus to GitHub Actions
880be8a ci: Move "i686: Linux (Debian stable)" from Cirrus to GiHub Actions
2e6cf9b Merge bitcoin-core/secp256k1#1396: ci, gha: Add "x86_64: Linux (Debian stable)" GitHub Actions job
5373693 Merge bitcoin-core/secp256k1#1405: ci: Drop no longer needed workaround
ef9fe95 ci: Drop no longer needed workaround
e10878f ci, gha: Drop `driver-opts.network` input for `setup-buildx-action`
4ad4914 ci, gha: Add `retry_builder` Docker image builder
6617a62 ci: Remove "x86_64: Linux (Debian stable)" task from Cirrus CI
03c9e65 ci, gha: Add "x86_64: Linux (Debian stable)" GitHub Actions job
ad3e65d ci: Remove GCC build files and sage to reduce size of Docker image
6b9507a Merge bitcoin-core/secp256k1#1398: ci, gha: Add Windows jobs based on Linux image
87d35f3 ci: Rename `cirrus.sh` to more general `ci.sh`
d6281dd ci: Remove Windows tasks from Cirrus CI
2b6f9cd ci, gha: Add Windows jobs based on Linux image
48b1d93 Merge bitcoin-core/secp256k1#1403: ci, gha: Ensure only a single workflow processes `github.ref` at a time
0ba2b94 Merge bitcoin-core/secp256k1#1373: Add invariant checking for scalars
c45b7c4 refactor: introduce testutil.h (deduplicate `random_fe_`, `ge_equals_` helpers)
dc55141 tests: simplify `random_fe_non_zero` (remove loop limit and unneeded normalize)
060e32c Merge bitcoin-core/secp256k1#1401: ci, gha: Run all MSVC tests on Windows natively
de657c2 Merge bitcoin-core/secp256k1#1062: Removes `_fe_equal_var`, and unwanted `_fe_normalize_weak` calls (in tests)
bcffeb1 Merge bitcoin-core/secp256k1#1404: ci: Remove "arm64: macOS Ventura" task from Cirrus CI
c2f6435 ci: Add comment about switching macOS to M1 on GHA later
4a24fae ci: Remove "arm64: macOS Ventura" task from Cirrus CI
b0886fd ci, gha: Ensure only a single workflow processes `github.ref` at a time
3d05c86 Merge bitcoin-core/secp256k1#1394: ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions
d78bec7 ci: Remove Windows MSVC tasks from Cirrus CI
3545dc2 ci, gha: Run all MSVC tests on Windows natively
5d8fa82 Merge bitcoin-core/secp256k1#1274: test: Silent noisy clang warnings about Valgrind code on macOS x86_64
8e54a34 ci, gha: Run "x86_64: macOS Ventura" job on GitHub Actions
b327abf Merge bitcoin-core/secp256k1#1402: ci: Use Homebrew's gcc in native macOS task
d62db57 ci: Use Homebrew's gcc in native macOS task
54058d1 field: remove `secp256k1_fe_equal_var`
bb4efd6 tests: remove unwanted `secp256k1_fe_normalize_weak` call
eedd781 Merge bitcoin-core/secp256k1#1348: tighten group magnitude limits, save normalize_weak calls in group add methods (revival of #1032)
b2f6712 Merge bitcoin-core/secp256k1#1400: ctimetests: Use new SECP256K1_CHECKMEM macros also for ellswift
9c91ea4 ci: Enable ellswift module where it's missing
db32a24 ctimetests: Use new SECP256K1_CHECKMEM macros also for ellswift
ce765a5 Merge bitcoin-core/secp256k1#1399: ci, gha: Run "SageMath prover" job on GitHub Actions
8408dfd Revert "ci: Run sage prover on CI"
c8d9914 ci, gha: Run "SageMath prover" job on GitHub Actions
8d2960c Merge bitcoin-core/secp256k1#1397: ci: Remove "Windows (VS 2022)" task from Cirrus CI
f1774e5 ci, gha: Make MSVC job presentation more explicit
5ee039b ci: Remove "Windows (VS 2022)" task from Cirrus CI
96294c0 Merge bitcoin-core/secp256k1#1389: ci: Run "Windows (VS 2022)" job on GitHub Actions
a2f7ccd ci: Run "Windows (VS 2022)" job on GitHub Actions
374e2b5 Merge bitcoin-core/secp256k1#1290: cmake: Set `ENVIRONMENT` property for examples on Windows
1b13415 Merge bitcoin-core/secp256k1#1391: refactor: take use of `secp256k1_scalar_{zero,one}` constants (part 2)
a1bd497 refactor: take use of `secp256k1_scalar_{zero,one}` constants (part 2)
b7c685e Save _normalize_weak calls in group add methods
c83afa6 Tighten group magnitude limits
26392da Merge bitcoin-core/secp256k1#1386: ci: print $ELLSWIFT in cirrus.sh
d23da6d use secp256k1_scalar_verify checks
4692478 ci: print $ELLSWIFT in cirrus.sh
c7d0454 add verification for scalars
c734c64 Merge bitcoin-core/secp256k1#1384: build: enable ellswift module via SECP_CONFIG_DEFINES
ad15215 update max scalar in scalar_cmov_test and fix schnorrsig_verify exhaustive test
78ca880 build: enable ellswift module via SECP_CONFIG_DEFINES
0e00fc7 Merge bitcoin-core/secp256k1#1383: util: remove unused checked_realloc
b097a46 util: remove unused checked_realloc
2bd5f3e Merge bitcoin-core/secp256k1#1382: refactor: Drop unused cast
4f8c5bd refactor: Drop unused cast
173e8d0 Implement current magnitude assumptions
49afd2f Take use of _fe_verify_magnitude in field_impl.h
4e9661f Add _fe_verify_magnitude (no-op unless VERIFY is enabled)
690b0fc add missing group element invariant checks
175db31 ci: Drop no longer needed `PATH` variable update on Windows
116d2ab cmake: Set `ENVIRONMENT` property for examples on Windows
cef3739 cmake, refactor: Use helper function instead of interface library
747ada3 test: Silent noisy clang warnings about Valgrind code on macOS x86_64
42f8c51 cmake: Add `SECP256K1_LATE_CFLAGS` configure option
e02f313 Add comment on length checks when parsing ECDSA sigs

git-subtree-dir: src/secp256k1
git-subtree-split: e3a885d42a7800c1ccebad94ad1e2b82c4df5c65
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants