Rollup merge of #104051 - RalfJung:miri, r=RalfJung

update Miri

Notable PRs:
- rust-lang/miri#2636
- rust-lang/miri#2641
- rust-lang/miri#2638

src/tools/miri/.github/workflows/ci.yml

@@ -67,9 +67,9 @@ jobs:
         shell: bash
         run: |
           if [[ ${{ github.event_name }} == 'schedule' ]]; then
-            ./rustup-toolchain HEAD --host ${{ matrix.host_target }}
+            ./miri toolchain HEAD --host ${{ matrix.host_target }}
           else
-            ./rustup-toolchain "" --host ${{ matrix.host_target }}
+            ./miri toolchain "" --host ${{ matrix.host_target }}
           fi
 
       - name: Show Rust version
@@ -118,7 +118,7 @@ jobs:
       - name: Install "master" toolchain
         shell: bash
         run: |
-          ./rustup-toolchain "" -c clippy
+          ./miri toolchain
 
       - name: Show Rust version
         run: |

src/tools/miri/.gitpod.yml

@@ -4,6 +4,6 @@ tasks:
   - before: echo "..."
     init: |
       cargo install rustup-toolchain-install-master
-      ./rustup-toolchain
+      ./miri toolchain
       ./miri build
-    command: echo "Run tests with ./miri test"
+    command: echo "Run tests with ./miri test"

src/tools/miri/CONTRIBUTING.md

@@ -23,13 +23,13 @@ tested against. Other versions will likely not work. After installing
 [`rustup-toolchain-install-master`], you can run the following command to
 install that exact version of rustc as a toolchain:
 ```
-./rustup-toolchain
+./miri toolchain
 ```
 This will set up a rustup toolchain called `miri` and set it as an override for
 the current directory.
 
 You can also create a `.auto-everything` file (contents don't matter, can be empty), which
-will cause any `./miri` command to automatically call `rustup-toolchain`, `clippy` and `rustfmt`
+will cause any `./miri` command to automatically call `./miri toolchain`, `clippy` and `rustfmt`
 for you. If you don't want all of these to happen, you can add individual `.auto-toolchain`,
 `.auto-clippy` and `.auto-fmt` files respectively.
 
@@ -132,12 +132,15 @@ development version of Miri using
 and then you can use it as if it was installed by `rustup`.  Make sure you use
 the same toolchain when calling `cargo miri` that you used when installing Miri!
 Usually this means you have to write `cargo +miri miri ...` to select the `miri`
-toolchain that was installed by `./rustup-toolchain`.
+toolchain that was installed by `./miri toolchain`.
 
 There's a test for the cargo wrapper in the `test-cargo-miri` directory; run
 `./run-test.py` in there to execute it. Like `./miri test`, this respects the
 `MIRI_TEST_TARGET` environment variable to execute the test for another target.
 
+Note that installing Miri like this will "take away" Miri management from `rustup`.
+If you want to later go back to a rustup-installed Miri, run `rustup update`.
+
 ### Using a modified standard library
 
 Miri re-builds the standard library into a custom sysroot, so it is fairly easy
@@ -214,7 +217,7 @@ for changes in rustc. In both cases, `rustc-version` needs updating.
 
 To update the `rustc-version` file and install the latest rustc, you can run:
 ```
-./rustup-toolchain HEAD
+./miri toolchain HEAD
 ```
 
 Now edit Miri until `./miri test` passes, and submit a PR. Generally, it is
@@ -290,16 +293,14 @@ cargo run --release -p josh-proxy -- --local=$(pwd)/local --remote=https://githu
 
 ### Importing changes from the rustc repo
 
+Josh needs to be running, as described above.
 We assume we start on an up-to-date master branch in the Miri repo.
 
 ```sh
-# Fetch rustc side of the history. Takes ca 5 min the first time.
-# Do NOT change that commit ID, it needs to be exactly this!
-git fetch http://localhost:8000/rust-lang/rust.git:at_commit=75dd959a3a40eb5b4574f8d2e23aa6efbeb33573[:prefix=src/tools/miri]:/src/tools/miri.git master
-# Include that history into ours.
-git merge FETCH_HEAD -m "merge rustc history"
+# Fetch and merge rustc side of the history. Takes ca 5 min the first time.
+./miri rustc-pull
 # Update toolchain reference and apply formatting.
-./rustup-toolchain HEAD && ./miri fmt
+./miri toolchain HEAD && ./miri fmt
 git commit -am "rustup"
 ```
 
@@ -310,16 +311,15 @@ needed.
 
 ### Exporting changes to the rustc repo
 
-We will use the josh proxy to push to your fork of rustc. You need to make sure
-that the master branch of your fork is up-to-date. Also make sure that there
-exists no branch called `miri` in your fork. Then run the following in the Miri
-repo, assuming we are on an up-to-date master branch:
+Josh needs to be running, as described above. We will use the josh proxy to push
+to your fork of rustc. Run the following in the Miri repo, assuming we are on an
+up-to-date master branch:
 
 ```sh
 # Push the Miri changes to your rustc fork (substitute your github handle for YOUR_NAME).
-# Do NOT change that commit ID, it needs to be exactly this!
-git push http://localhost:8000/YOUR_NAME/rust.git:at_commit=75dd959a3a40eb5b4574f8d2e23aa6efbeb33573[:prefix=src/tools/miri]:/src/tools/miri.git -o base=master HEAD:miri
+./miri rustc-push YOUR_NAME miri
 ```
 
-This will create a new branch in your fork, and the output should include a link
-to create a rustc PR that will integrate those changes into the main repository.
+This will create a new branch called 'miri' in your fork, and the output should
+include a link to create a rustc PR that will integrate those changes into the
+main repository.

src/tools/miri/README.md

@@ -374,14 +374,15 @@ to Miri failing to detect cases of undefined behavior in a program.
   application instead of raising an error within the context of Miri (and halting
   execution). Note that code might not expect these operations to ever panic, so
   this flag can lead to strange (mis)behavior.
-* `-Zmiri-retag-fields` changes Stacked Borrows retagging to recurse into fields.
+* `-Zmiri-retag-fields` changes Stacked Borrows retagging to recurse into *all* fields.
   This means that references in fields of structs/enums/tuples/arrays/... are retagged,
   and in particular, they are protected when passed as function arguments.
+  (The default is to recurse only in cases where rustc would actually emit a `noalias` attribute.)
 * `-Zmiri-retag-fields=<all|none|scalar>` controls when Stacked Borrows retagging recurses into
   fields. `all` means it always recurses (like `-Zmiri-retag-fields`), `none` means it never
-  recurses (the default), `scalar` means it only recurses for types where we would also emit
+  recurses, `scalar` (the default) means it only recurses for types where we would also emit
   `noalias` annotations in the generated LLVM IR (types passed as indivudal scalars or pairs of
-  scalars).
+  scalars). Setting this to `none` is **unsound**.
 * `-Zmiri-tag-gc=<blocks>` configures how often the pointer tag garbage collector runs. The default
   is to search for and remove unreachable tags once every `10000` basic blocks. Setting this to
   `0` disables the garbage collector, which causes some programs to have explosive memory usage
@@ -419,9 +420,9 @@ Some native rustc `-Z` flags are also very relevant for Miri:
 
 Moreover, Miri recognizes some environment variables:
 
-* `MIRI_AUTO_OPS` indicates whether the automatic execution of rustfmt, clippy and rustup-toolchain
-  should be skipped. If it is set to any value, they are skipped. This is used for avoiding
-  infinite recursion in `./miri` and to allow automated IDE actions to avoid the auto ops.
+* `MIRI_AUTO_OPS` indicates whether the automatic execution of rustfmt, clippy and toolchain setup
+  should be skipped. If it is set to any value, they are skipped. This is used for avoiding infinite
+  recursion in `./miri` and to allow automated IDE actions to avoid the auto ops.
 * `MIRI_LOG`, `MIRI_BACKTRACE` control logging and backtrace printing during
   Miri executions, also [see "Testing the Miri driver" in `CONTRIBUTING.md`][testing-miri].
 * `MIRIFLAGS` (recognized by `cargo miri` and the test suite) defines extra

src/tools/miri/cargo-miri/src/phases.rs

@@ -528,7 +528,7 @@ pub fn phase_runner(mut binary_args: impl Iterator<Item = String>, phase: Runner
     cmd.args(binary_args);
 
     // Make sure we use the build-time working directory for interpreting Miri/rustc arguments.
-    // But then we need to switch to the run-time one, which we instruct Miri do do by setting `MIRI_CWD`.
+    // But then we need to switch to the run-time one, which we instruct Miri to do by setting `MIRI_CWD`.
     cmd.current_dir(info.current_dir);
     cmd.env("MIRI_CWD", env::current_dir().unwrap());
 

src/tools/miri/miri

@@ -42,6 +42,22 @@ many different seeds.
 Runs the benchmarks from bench-cargo-miri in hyperfine. hyperfine needs to be installed.
 <benches> can explicitly list the benchmarks to run; by default, all of them are run.
 
+./miri rustc-pull:
+Pull and merge Miri changes from the rustc repo.
+
+./miri rustc-push <github user> <branch>:
+Push Miri changes back to the rustc repo. This will update the 'master' branch
+in the Rust fork of the given user to upstream. It will also pull a copy of the
+rustc history into the Miri repo, unless you set the RUSTC_GIT env var to an
+existing clone of the rustc repo.
+
+./miri toolchain <commit> <flags>:
+Update and activate the rustup toolchain 'miri'. If no commit is given, updates
+to the commit given in the `rust-version` file. If the commit is `HEAD`, updates
+to the latest upstream rustc commit.
+`rustup-toolchain-install-master` must be installed for this to work. Any extra
+flags are passed to `rustup-toolchain-install-master`.
+
   ENVIRONMENT VARIABLES
 
 MIRI_SYSROOT:
@@ -52,37 +68,98 @@ Pass extra flags to all cargo invocations. (Ignored by `./miri cargo`.)
 EOF
 )
 
-## We need to know where we are.
+## We need to know which command to run and some global constants.
+COMMAND="$1"
+if [ -z "$COMMAND" ]; then
+    echo "$USAGE"
+    exit 1
+fi
+shift
 # macOS does not have a useful readlink/realpath so we have to use Python instead...
 MIRIDIR=$(python3 -c 'import os, sys; print(os.path.dirname(os.path.realpath(sys.argv[1])))' "$0")
+# Used for rustc syncs.
+JOSH_FILTER=":at_commit=75dd959a3a40eb5b4574f8d2e23aa6efbeb33573[:prefix=src/tools/miri]:/src/tools/miri"
+# Needed for `./miri bench`.
+TOOLCHAIN=$(cd "$MIRIDIR"; rustup show active-toolchain | head -n 1 | cut -d ' ' -f 1)
 
-## Run the auto-things.
-if [ -z "$MIRI_AUTO_OPS" ]; then
-    export MIRI_AUTO_OPS=42
-
-    # Run this first, so that the toolchain doesn't change after
-    # other code has run.
-    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-toolchain" ] ; then
-        (cd "$MIRIDIR" && ./rustup-toolchain)
+## Early commands, that don't do auto-things and don't want the environment-altering things happening below.
+case "$COMMAND" in
+toolchain)
+    cd "$MIRIDIR"
+    # Make sure rustup-toolchain-install-master is installed.
+    if ! which rustup-toolchain-install-master >/dev/null; then
+        echo "Please install rustup-toolchain-install-master by running 'cargo install rustup-toolchain-install-master'"
+        exit 1
     fi
-
-    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-fmt" ] ; then
-        $0 fmt
+    # Determine new commit.
+    if [[ "$1" == "" ]]; then
+        NEW_COMMIT=$(cat rust-version)
+    elif [[ "$1" == "HEAD" ]]; then
+        NEW_COMMIT=$(git ls-remote https://github.com/rust-lang/rust/ HEAD | cut -f 1)
+    else
+        NEW_COMMIT="$1"
     fi
-
-    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-clippy" ] ; then
-        $0 clippy -- -D warnings
+    echo "$NEW_COMMIT" > rust-version
+    shift || true # don't fail if shifting fails because no commit was given
+    # Check if we already are at that commit.
+    CUR_COMMIT=$(rustc +miri --version -v 2>/dev/null | grep "^commit-hash: " | cut -d " " -f 2)
+    if [[ "$CUR_COMMIT" == "$NEW_COMMIT" ]]; then
+        echo "miri toolchain is already at commit $CUR_COMMIT."
+        rustup override set miri
+        exit 0
     fi
-fi
-
-## Determine command and toolchain.
-COMMAND="$1"
-[ $# -gt 0 ] && shift
-# Doing this *after* auto-toolchain logic above, since that might change the toolchain.
-TOOLCHAIN=$(cd "$MIRIDIR"; rustup show active-toolchain | head -n 1 | cut -d ' ' -f 1)
-
-## Handle some commands early, since they should *not* alter the environment.
-case "$COMMAND" in
+    # Install and setup new toolchain.
+    rustup toolchain uninstall miri
+    rustup-toolchain-install-master -n miri -c cargo -c rust-src -c rustc-dev -c llvm-tools -c rustfmt -c clippy "$@" -- "$NEW_COMMIT"
+    rustup override set miri
+    # Cleanup.
+    cargo clean
+    # Call 'cargo metadata' on the sources in case that changes the lockfile
+    # (which fails under some setups when it is done from inside vscode).
+    cargo metadata --format-version 1 --manifest-path "$(rustc --print sysroot)/lib/rustlib/rustc-src/rust/compiler/rustc/Cargo.toml" >/dev/null
+    # Done!
+    exit 0
+    ;;
+rustc-pull)
+    cd "$MIRIDIR"
+    git fetch http://localhost:8000/rust-lang/rust.git$JOSH_FILTER.git master
+    git merge FETCH_HEAD --no-ff -m "Merge from rustc"
+    exit 0
+    ;;
+rustc-push)
+    USER="$1"
+    BRANCH="$2"
+    if [ -z "$USER" ] || [ -z "$BRANCH" ]; then
+        echo "Usage: $0 rustc-push <github user> <branch>"
+        exit 1
+    fi
+    if [ -n "$RUSTC_GIT" ]; then
+        # Use an existing fork for the branch updates.
+        cd "$RUSTC_GIT"
+    else
+        # Do this in the local Miri repo.
+        echo "This will pull a copy of the rust-lang/rust history into this Miri checkout, growing it by about 1GB."
+        read -r -p "To avoid that, abort now and set the RUSTC_GIT environment variable to an existing rustc checkout. Proceed? [y/N] "
+        if [[ ! $REPLY =~ ^[Yy]$ ]]; then
+            exit 1
+        fi
+        cd "$MIRIDIR"
+    fi
+    # Prepare the branches. For reliable pushing we need to push to a non-existent branch
+    # and set `-o base` to a branch that holds current rustc master.
+    echo "Preparing $USER/rust..."
+    if git fetch https://github.com/$USER/rust $BRANCH &>/dev/null; then
+        echo "The '$BRANCH' seems to already exist in $USER/rust. Please delete it and try again."
+        exit 1
+    fi
+    git fetch https://github.com/rust-lang/rust master
+    git push https://github.com/$USER/rust FETCH_HEAD:master
+    # Do the actual push.
+    cd "$MIRIDIR"
+    echo "Pushing Miri changes..."
+    git push http://localhost:8000/$USER/rust.git$JOSH_FILTER.git HEAD:$BRANCH -o base=master
+    exit 0
+    ;;
 many-seeds)
     for SEED in $({ echo obase=16; seq 0 255; } | bc); do
         echo "Trying seed: $SEED"
@@ -106,9 +183,29 @@ bench)
     ;;
 esac
 
+## Run the auto-things.
+if [ -z "$MIRI_AUTO_OPS" ]; then
+    export MIRI_AUTO_OPS=42
+
+    # Run this first, so that the toolchain doesn't change after
+    # other code has run.
+    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-toolchain" ] ; then
+        $0 toolchain
+        # Let's make sure to actually use that toolchain, too.
+        TOOLCHAIN=miri
+    fi
+
+    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-fmt" ] ; then
+        $0 fmt
+    fi
+
+    if [ -f "$MIRIDIR/.auto-everything" ] || [ -f "$MIRIDIR/.auto-clippy" ] ; then
+        $0 clippy -- -D warnings
+    fi
+fi
+
 ## Prepare the environment
 # Determine some toolchain properties
-# export the target so its available in miri
 TARGET=$(rustc +$TOOLCHAIN --version --verbose | grep "^host:" | cut -d ' ' -f 2)
 SYSROOT=$(rustc +$TOOLCHAIN --print sysroot)
 LIBDIR=$SYSROOT/lib/rustlib/$TARGET/lib
@@ -227,10 +324,7 @@ cargo)
     $CARGO "$@"
     ;;
 *)
-    if [ -n "$COMMAND" ]; then
-      echo "Unknown command: $COMMAND"
-      echo
-    fi
-    echo "$USAGE"
+    echo "Unknown command: $COMMAND"
     exit 1
+    ;;
 esac

src/tools/miri/rust-version

@@ -1 +1 @@
-85d089b41e2a0c0f07ab34f6c5a7c451389f25e6
+b03502b35d111bef0399a66ab3cc765f0802e8ba