Move the query engine out of rustc_middle #70951

cjgillot · 2020-04-09T11:02:07Z

The handling of queries is moved to a trait QueryEngine.
It replaces query::Queries in the TyCtxt, allowing to move the query engine out of librustc_middle.

There are 2 modes to access the query engine: through TyCtxt and dynamic dispatch,
or through a QueryCtxt. The QueryCtxt is required for everything touching the OnDiskCache.

For now, I put it in librustc_incremental, which is very small.
This may not be the best place.

A significant part of the codegen time for librustc_middle is moved to the recipient crate.

This PR may require a perf run.

cc #65031
r? @Zoxc

rust-highfive · 2020-04-09T12:49:33Z

The job x86_64-gnu-llvm-7 of your PR failed (pretty log, raw log). Through arcane magic we have determined that the following fragments from the build log may contain information about the problem.

Click to expand the log.

2020-04-09T11:14:16.2515150Z ========================== Starting Command Output ===========================
2020-04-09T11:14:16.2521043Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/e0ed475d-ae9c-4baa-baa5-0a1ab9787d1f.sh
2020-04-09T11:14:16.2521572Z 
2020-04-09T11:14:16.2525738Z ##[section]Finishing: Disable git automatic line ending conversion
2020-04-09T11:14:16.2541788Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/70951/merge to s
2020-04-09T11:14:16.2545038Z Task         : Get sources
2020-04-09T11:14:16.2545277Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-04-09T11:14:16.2545561Z Version      : 1.0.0
2020-04-09T11:14:16.2545718Z Author       : Microsoft
---
2020-04-09T11:14:17.6943329Z ##[command]git remote add origin https://github.com/rust-lang/rust
2020-04-09T11:14:17.6951184Z ##[command]git config gc.auto 0
2020-04-09T11:14:17.6956111Z ##[command]git config --get-all http.https://github.com/rust-lang/rust.extraheader
2020-04-09T11:14:17.6960134Z ##[command]git config --get-all http.proxy
2020-04-09T11:14:17.6968908Z ##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --force --tags --prune --progress --no-recurse-submodules --depth=2 origin +refs/heads/*:refs/remotes/origin/* +refs/pull/70951/merge:refs/remotes/pull/70951/merge
---
2020-04-09T11:17:30.5619206Z Looks like docker image is the same as before, not uploading
2020-04-09T11:17:37.9402204Z [CI_JOB_NAME=x86_64-gnu-llvm-7]
2020-04-09T11:17:37.9730092Z [CI_JOB_NAME=x86_64-gnu-llvm-7]
2020-04-09T11:17:37.9754643Z == clock drift check ==
2020-04-09T11:17:37.9763274Z   local time: Thu Apr  9 11:17:37 UTC 2020
2020-04-09T11:17:38.0071927Z   network time: Thu, 09 Apr 2020 11:17:38 GMT
2020-04-09T11:17:38.0093855Z Starting sccache server...
2020-04-09T11:17:38.0924925Z configure: processing command line
2020-04-09T11:17:38.0925677Z configure: 
2020-04-09T11:17:38.0926644Z configure: rust.dist-src        := False
---
2020-04-09T11:22:25.0351593Z    Compiling rustc_feature v0.0.0 (/checkout/src/librustc_feature)
2020-04-09T11:22:26.5119845Z    Compiling fmt_macros v0.0.0 (/checkout/src/libfmt_macros)
2020-04-09T11:22:28.0565236Z    Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
2020-04-09T11:22:29.9412656Z    Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
2020-04-09T11:22:37.4414157Z    Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
2020-04-09T11:22:40.8331679Z    Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
2020-04-09T11:22:44.9935010Z    Compiling rustc_attr v0.0.0 (/checkout/src/librustc_attr)
2020-04-09T11:22:48.8444668Z    Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
2020-04-09T11:22:56.7221321Z    Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
2020-04-09T11:40:43.8760206Z    Compiling rustc_feature v0.0.0 (/checkout/src/librustc_feature)
2020-04-09T11:40:45.4758654Z    Compiling fmt_macros v0.0.0 (/checkout/src/libfmt_macros)
2020-04-09T11:40:47.2887338Z    Compiling rustc_ast_pretty v0.0.0 (/checkout/src/librustc_ast_pretty)
2020-04-09T11:40:49.5512841Z    Compiling rustc_hir v0.0.0 (/checkout/src/librustc_hir)
2020-04-09T11:40:59.0803127Z    Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
2020-04-09T11:41:02.7107331Z    Compiling rustc_hir_pretty v0.0.0 (/checkout/src/librustc_hir_pretty)
2020-04-09T11:41:07.6359279Z    Compiling rustc_attr v0.0.0 (/checkout/src/librustc_attr)
2020-04-09T11:41:12.7119532Z    Compiling rustc_parse v0.0.0 (/checkout/src/librustc_parse)
2020-04-09T11:41:21.9777454Z    Compiling rustc_ast_lowering v0.0.0 (/checkout/src/librustc_ast_lowering)
---
2020-04-09T12:01:37.2155944Z .................................................................................................... 1700/9879
2020-04-09T12:01:40.8878646Z .................................................................................................... 1800/9879
2020-04-09T12:01:49.0196755Z ..................................................................................................i. 1900/9879
2020-04-09T12:01:56.5353403Z .................................................................................................... 2000/9879
2020-04-09T12:02:02.5455562Z ........................................................................................iiiii....... 2100/9879
2020-04-09T12:02:23.2501260Z .................................................................................................... 2300/9879
2020-04-09T12:02:25.1657947Z .................................................................................................... 2400/9879
2020-04-09T12:02:27.3480825Z .................................................................................................... 2500/9879
2020-04-09T12:02:33.0898795Z .................................................................................................... 2600/9879
---
2020-04-09T12:05:21.2523207Z ..............................................................i...............i..................... 5000/9879
2020-04-09T12:05:27.6536288Z .................................................................................................... 5100/9879
2020-04-09T12:05:34.5714912Z .................................................................................................... 5200/9879
2020-04-09T12:05:39.4171953Z .......i............................................................................................ 5300/9879
2020-04-09T12:05:48.9979146Z ................................................................................................ii.i 5400/9879
2020-04-09T12:05:53.6475678Z i........i...i...................................................................................... 5500/9879
2020-04-09T12:06:01.6933917Z .........................................i.......................................................... 5700/9879
2020-04-09T12:06:10.8922800Z .............................................................ii..................................... 5800/9879
2020-04-09T12:06:17.8782743Z i................................................................................................... 5900/9879
2020-04-09T12:06:22.8891063Z .................................................................................................... 6000/9879
2020-04-09T12:06:22.8891063Z .................................................................................................... 6000/9879
2020-04-09T12:06:31.9688848Z ..............................................................................................ii...i 6100/9879
2020-04-09T12:06:43.2086658Z ..ii...........i.................................................................................... 6200/9879
2020-04-09T12:06:58.0479432Z .................................................................................................... 6400/9879
2020-04-09T12:07:03.6576587Z .................................................................................................... 6500/9879
2020-04-09T12:07:03.6576587Z .................................................................................................... 6500/9879
2020-04-09T12:07:18.5401703Z ........................i..ii....................................................................... 6600/9879
2020-04-09T12:07:39.2231619Z .................................................................................................... 6800/9879
2020-04-09T12:07:41.2827494Z ........................i........................................................................... 6900/9879
2020-04-09T12:07:43.3995489Z .................................................................................................... 7000/9879
2020-04-09T12:07:45.5770244Z ...............................................................i.................................... 7100/9879
---
2020-04-09T12:09:17.9300517Z .................................................................................................... 7800/9879
2020-04-09T12:09:22.1056339Z .................................................................................................... 7900/9879
2020-04-09T12:09:27.7963662Z .................................................................................................... 8000/9879
2020-04-09T12:09:35.1806134Z ............................i....................................................................... 8100/9879
2020-04-09T12:09:42.9379453Z ............................................................................iiiiii.iiii.i........... 8200/9879
2020-04-09T12:09:57.5965878Z .....................i......i....................................................................... 8400/9879
2020-04-09T12:10:01.9206072Z .................................................................................................... 8500/9879
2020-04-09T12:10:11.9025104Z .................................................................................................... 8600/9879
2020-04-09T12:10:23.0433625Z .................................................................................................... 8700/9879
---
2020-04-09T12:12:33.9407490Z Suite("src/test/codegen") not skipped for "bootstrap::test::Codegen" -- not in ["src/tools/tidy"]
2020-04-09T12:12:33.9616787Z Check compiletest suite=codegen mode=codegen (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-04-09T12:12:34.1784194Z 
2020-04-09T12:12:34.1784967Z running 185 tests
2020-04-09T12:12:36.5564918Z iiii......i............ii.i..iiii....i....i...........i............i..i..................i....i..... 100/185
2020-04-09T12:12:39.0197685Z .......i.i.i...iii..iiiiiiiiiiiiiiii........................iii..............ii......
2020-04-09T12:12:39.0201795Z 
2020-04-09T12:12:39.0208826Z  finished in 5.059
2020-04-09T12:12:39.0227218Z Suite("src/test/codegen-units") not skipped for "bootstrap::test::CodegenUnits" -- not in ["src/tools/tidy"]
2020-04-09T12:12:39.0416613Z Check compiletest suite=codegen-units mode=codegen-units (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
2020-04-09T12:12:41.0701777Z Suite("src/test/assembly") not skipped for "bootstrap::test::Assembly" -- not in ["src/tools/tidy"]
2020-04-09T12:12:41.0897117Z Check compiletest suite=assembly mode=assembly (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-04-09T12:12:41.2300312Z 
2020-04-09T12:12:41.2300621Z running 9 tests
2020-04-09T12:12:41.2302584Z iiiiiiiii
2020-04-09T12:12:41.2303382Z 
2020-04-09T12:12:41.2303522Z  finished in 0.140
2020-04-09T12:12:41.2304226Z Suite("src/test/incremental") not skipped for "bootstrap::test::Incremental" -- not in ["src/tools/tidy"]
2020-04-09T12:12:41.2467108Z Check compiletest suite=incremental mode=incremental (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
2020-04-09T12:12:59.9681395Z Suite("src/test/debuginfo") not skipped for "bootstrap::test::Debuginfo" -- not in ["src/tools/tidy"]
2020-04-09T12:12:59.9955369Z Check compiletest suite=debuginfo mode=debuginfo (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-04-09T12:13:00.1801502Z 
2020-04-09T12:13:00.1802215Z running 115 tests
2020-04-09T12:13:13.9471962Z iiiii..i.....i..i...i..i.i.i..i..i..ii....i.i....ii..........iiii.........i.....i..i.......ii.i.ii.. 100/115
2020-04-09T12:13:15.8477365Z ...iiii.....ii.
2020-04-09T12:13:15.8479015Z 
2020-04-09T12:13:15.8479199Z  finished in 15.852
2020-04-09T12:13:15.8535181Z Suite("src/test/ui-fulldeps") not skipped for "bootstrap::test::UiFullDeps" -- not in ["src/tools/tidy"]
2020-04-09T12:13:15.8535898Z Uplifting stage1 rustc (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
---
2020-04-09T12:24:47.2109950Z 
2020-04-09T12:24:47.2110953Z    Doc-tests core
2020-04-09T12:24:51.5282569Z 
2020-04-09T12:24:51.5283572Z running 2490 tests
2020-04-09T12:24:59.8942490Z ......iiiii......................................................................................... 100/2490
2020-04-09T12:25:07.9490412Z .....................................................................................ii............. 200/2490
2020-04-09T12:25:26.7288412Z ....................i............................................................................... 400/2490
2020-04-09T12:25:26.7288412Z ....................i............................................................................... 400/2490
2020-04-09T12:25:35.7247108Z ..........................................................................i..i..................iiii 500/2490
2020-04-09T12:25:50.3299882Z .................................................................................................... 700/2490
2020-04-09T12:25:57.7453999Z .................................................................................................... 800/2490
2020-04-09T12:26:05.1467046Z .................................................................................................... 900/2490
2020-04-09T12:26:12.6711187Z .................................................................................................... 1000/2490
---
2020-04-09T12:29:29.4456347Z 
2020-04-09T12:29:29.4457409Z running 1019 tests
2020-04-09T12:29:45.8720146Z i................................................................................................... 100/1019
2020-04-09T12:29:54.8851758Z .................................................................................................... 200/1019
2020-04-09T12:30:01.6575162Z ..................iii......i......i...i......i...................................................... 300/1019
2020-04-09T12:30:12.1614615Z ...................................................i....i......................................ii... 500/1019
2020-04-09T12:30:18.9622458Z .................................................................................................... 600/1019
2020-04-09T12:30:23.3503669Z .................................................................................................... 700/1019
2020-04-09T12:30:23.3503669Z .................................................................................................... 700/1019
2020-04-09T12:30:29.5502263Z .............................................iiii................................................... 800/1019
2020-04-09T12:30:43.5403577Z .................................................................................................... 900/1019
2020-04-09T12:30:48.1439834Z ...................................................................iiii............................. 1000/1019
2020-04-09T12:30:49.3021398Z test result: ok. 999 passed; 0 failed; 20 ignored; 0 measured; 0 filtered out
2020-04-09T12:30:49.3021592Z 
2020-04-09T12:30:49.3112058Z  finished in 151.423
2020-04-09T12:30:49.3120077Z Set({"src/libterm"}) not skipped for "bootstrap::test::Crate" -- not in ["src/tools/tidy"]
---
2020-04-09T12:33:48.8265037Z 
2020-04-09T12:33:48.8265259Z test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
2020-04-09T12:33:48.8265463Z 
2020-04-09T12:33:48.8315887Z  finished in 0.909
2020-04-09T12:33:48.8321986Z Set({"/checkout/src/librustc_query_system"}) not skipped for "bootstrap::test::CrateLibrustc" -- not in ["src/tools/tidy"]
2020-04-09T12:33:48.8350221Z Testing rustc_query_system stage1 (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-04-09T12:33:49.0170226Z    Compiling rustc_query_system v0.0.0 (/checkout/src/librustc_query_system)
2020-04-09T12:33:49.9834045Z      Running build/x86_64-unknown-linux-gnu/stage1-rustc/x86_64-unknown-linux-gnu/release/deps/rustc_query_system-dc8c35248385a45c
2020-04-09T12:33:49.9858840Z 
2020-04-09T12:33:49.9859042Z running 0 tests
2020-04-09T12:33:49.9859151Z 
---
2020-04-09T12:47:24.2198286Z Set({"/checkout/src/librustc_parse"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
2020-04-09T12:47:24.2204511Z Set({"/checkout/src/librustc_passes"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
2020-04-09T12:47:24.2205460Z Set({"/checkout/src/librustc_plugin_impl"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
2020-04-09T12:47:24.2206737Z Set({"/checkout/src/librustc_privacy"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
2020-04-09T12:47:24.2207699Z Set({"/checkout/src/librustc_query_system"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
2020-04-09T12:47:24.2209048Z Set({"/checkout/src/librustc_save_analysis"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
2020-04-09T12:47:24.2209690Z Set({"/checkout/src/librustc_session"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
2020-04-09T12:47:24.2210343Z Set({"/checkout/src/librustc_span"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
2020-04-09T12:47:24.2211151Z Set({"/checkout/src/librustc_symbol_mangling"}) not skipped for "bootstrap::doc::Rustc" -- not in ["src/tools/tidy"]
---
2020-04-09T12:48:22.2785292Z Suite("src/test/run-make-fulldeps") not skipped for "bootstrap::test::RunMakeFullDeps" -- not in ["src/tools/tidy"]
2020-04-09T12:48:22.3090884Z Check compiletest suite=run-make-fulldeps mode=run-make (x86_64-unknown-linux-gnu -> x86_64-unknown-linux-gnu)
2020-04-09T12:48:22.5067561Z 
2020-04-09T12:48:22.5068084Z running 210 tests
2020-04-09T12:48:50.3094755Z ......................i...ii.................................F.....................................i 100/210
2020-04-09T12:49:26.3486631Z ........................................iiiiii......i..............iii.............................. 200/210
2020-04-09T12:49:27.4149990Z failures:
2020-04-09T12:49:27.4150095Z 
2020-04-09T12:49:27.4150529Z ---- [run-make] run-make-fulldeps/hotplug_codegen_backend stdout ----
2020-04-09T12:49:27.4150719Z 
2020-04-09T12:49:27.4150719Z 
2020-04-09T12:49:27.4150847Z error: make failed
2020-04-09T12:49:27.4151013Z status: exit code: 2
2020-04-09T12:49:27.4151189Z command: "make"
2020-04-09T12:49:27.4151324Z stdout:
2020-04-09T12:49:27.4151652Z ------------------------------------------
2020-04-09T12:49:27.4151920Z /bin/echo || exit 0 # This test requires /bin/echo to exist
2020-04-09T12:49:27.4152115Z 
2020-04-09T12:49:27.4153951Z LD_LIBRARY_PATH="/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/hotplug_codegen_backend/hotplug_codegen_backend:/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/checkout/obj/build/x86_64-unknown-linux-gnu/stage0/lib" '/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/hotplug_codegen_backend/hotplug_codegen_backend -L /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/hotplug_codegen_backend/hotplug_codegen_backend  the_backend.rs --crate-name the_backend --crate-type dylib \
2020-04-09T12:49:27.4155569Z  -o /checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps/hotplug_codegen_backend/hotplug_codegen_backend/the_backend.dylib
2020-04-09T12:49:27.4156103Z Makefile:6: recipe for target 'all' failed
2020-04-09T12:49:27.4156571Z ------------------------------------------
2020-04-09T12:49:27.4156743Z stderr:
2020-04-09T12:49:27.4157074Z ------------------------------------------
2020-04-09T12:49:27.4157074Z ------------------------------------------
2020-04-09T12:49:27.4157457Z warning: ignoring --out-dir flag due to -o flag
2020-04-09T12:49:27.4158239Z error[E0053]: method `codegen_crate` has an incompatible type for trait
2020-04-09T12:49:27.4158703Z   --> the_backend.rs:67:5
2020-04-09T12:49:27.4158864Z    |
2020-04-09T12:49:27.4159196Z 67 | /     fn codegen_crate<'a, 'tcx>(
2020-04-09T12:49:27.4159196Z 67 | /     fn codegen_crate<'a, 'tcx>(
2020-04-09T12:49:27.4159391Z 68 | |         &self,
2020-04-09T12:49:27.4159732Z 69 | |         tcx: TyCtxt<'tcx>,
2020-04-09T12:49:27.4159971Z 70 | |         _metadata: EncodedMetadata,
2020-04-09T12:49:27.4160171Z ...  |
2020-04-09T12:49:27.4160386Z 75 | |         Box::new(tcx.crate_name(LOCAL_CRATE) as Symbol)
2020-04-09T12:49:27.4160608Z 76 | |     }
2020-04-09T12:49:27.4160974Z    | |_____^ expected struct `rustc_incremental::query::plumbing::QueryCtxt`, found struct `rustc_middle::ty::TyCtxt`
2020-04-09T12:49:27.4161302Z    |
2020-04-09T12:49:27.4162194Z    = note: expected fn pointer `fn(&TheBackend, rustc_incremental::query::plumbing::QueryCtxt<'tcx>, rustc_middle::middle::cstore::EncodedMetadata, _) -> std::boxed::Box<_>`
2020-04-09T12:49:27.4163339Z               found fn pointer `fn(&TheBackend, rustc_middle::ty::TyCtxt<'_>, rustc_middle::middle::cstore::EncodedMetadata, _) -> std::boxed::Box<_>`
2020-04-09T12:49:27.4164098Z error: aborting due to previous error
2020-04-09T12:49:27.4164261Z 
2020-04-09T12:49:27.4164679Z For more information about this error, try `rustc --explain E0053`.
2020-04-09T12:49:27.4164679Z For more information about this error, try `rustc --explain E0053`.
2020-04-09T12:49:27.4164923Z make: *** [all] Error 1
2020-04-09T12:49:27.4165541Z ------------------------------------------
2020-04-09T12:49:27.4165687Z 
2020-04-09T12:49:27.4165769Z 
2020-04-09T12:49:27.4165849Z 
---
2020-04-09T12:49:27.4167718Z thread 'main' panicked at 'Some tests failed', src/tools/compiletest/src/main.rs:348:22
2020-04-09T12:49:27.4168075Z note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
2020-04-09T12:49:27.4168286Z 
2020-04-09T12:49:27.4168366Z 
2020-04-09T12:49:27.4176391Z command did not execute successfully: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage0-tools-bin/compiletest" "--compile-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib" "--run-lib-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" "--rustc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "--rustdoc-path" "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" "--src-base" "/checkout/src/test/run-make-fulldeps" "--build-base" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/run-make-fulldeps" "--stage-id" "stage2-x86_64-unknown-linux-gnu" "--mode" "run-make" "--target" "x86_64-unknown-linux-gnu" "--host" "x86_64-unknown-linux-gnu" "--llvm-filecheck" "/usr/lib/llvm-7/bin/FileCheck" "--nodejs" "/usr/bin/node" "--host-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--target-rustcflags" "-Crpath -O -Cdebuginfo=0 -Zunstable-options  -Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "--docck-python" "/usr/bin/python2.7" "--lldb-python" "/usr/bin/python2.7" "--gdb" "/usr/bin/gdb" "--quiet" "--llvm-version" "7.0.0" "--system-llvm" "--cc" "cc" "--cxx" "c++" "--cflags" "-ffunction-sections -fdata-sections -fPIC -m64" "--llvm-components" "aarch64 aarch64asmparser aarch64asmprinter aarch64codegen aarch64desc aarch64disassembler aarch64info aarch64utils aggressiveinstcombine all all-targets amdgpu amdgpuasmparser amdgpuasmprinter amdgpucodegen amdgpudesc amdgpudisassembler amdgpuinfo amdgpuutils analysis arm armasmparser armasmprinter armcodegen armdesc armdisassembler arminfo armutils asmparser asmprinter avr avrasmparser avrasmprinter avrcodegen avrdesc avrdisassembler avrinfo binaryformat bitreader bitwriter bpf bpfasmparser bpfasmprinter bpfcodegen bpfdesc bpfdisassembler bpfinfo codegen core coroutines coverage debuginfocodeview debuginfodwarf debuginfomsf debuginfopdb demangle dlltooldriver engine executionengine fuzzmutate globalisel hexagon hexagonasmparser hexagoncodegen hexagondesc hexagondisassembler hexagoninfo instcombine instrumentation interpreter ipo irreader lanai lanaiasmparser lanaiasmprinter lanaicodegen lanaidesc lanaidisassembler lanaiinfo libdriver lineeditor linker lto mc mcdisassembler mcjit mcparser mips mipsasmparser mipsasmprinter mipscodegen mipsdesc mipsdisassembler mipsinfo mirparser msp430 msp430asmprinter msp430codegen msp430desc msp430info native nativecodegen nvptx nvptxasmprinter nvptxcodegen nvptxdesc nvptxinfo objcarcopts object objectyaml option orcjit passes perfjitevents powerpc powerpcasmparser powerpcasmprinter powerpccodegen powerpcdesc powerpcdisassembler powerpcinfo profiledata runtimedyld scalaropts selectiondag sparc sparcasmparser sparcasmprinter sparccodegen sparcdesc sparcdisassembler sparcinfo support symbolize systemz systemzasmparser systemzasmprinter systemzcodegen systemzdesc systemzdisassembler systemzinfo tablegen target transformutils vectorize webassembly webassemblyasmparser webassemblyasmprinter webassemblycodegen webassemblydesc webassemblydisassembler webassemblyinfo windowsmanifest x86 x86asmparser x86asmprinter x86codegen x86desc x86disassembler x86info x86utils xcore xcoreasmprinter xcorecodegen xcoredesc xcoredisassembler xcoreinfo" "--ar" "ar" "--llvm-bin-dir" "/usr/lib/llvm-7/bin" "--adb-path" "adb" "--adb-test-dir" "/data/tmp/work" "--android-cross-path" "" "--color" "always"
2020-04-09T12:49:27.4181989Z 
2020-04-09T12:49:27.4182070Z 
2020-04-09T12:49:27.4182527Z failed to run: /checkout/obj/build/bootstrap/debug/bootstrap test --exclude src/tools/tidy
2020-04-09T12:49:27.4182853Z Build completed unsuccessfully in 1:30:25
2020-04-09T12:49:27.4182853Z Build completed unsuccessfully in 1:30:25
2020-04-09T12:49:27.4183056Z == clock drift check ==
2020-04-09T12:49:27.4183275Z   local time: Thu Apr  9 12:49:27 UTC 2020
2020-04-09T12:49:27.4904269Z   network time: Thu, 09 Apr 2020 12:49:27 GMT
2020-04-09T12:49:28.2311682Z 
2020-04-09T12:49:28.2311682Z 
2020-04-09T12:49:28.2376797Z ##[error]Bash exited with code '1'.
2020-04-09T12:49:28.2392065Z ##[section]Finishing: Run build
2020-04-09T12:49:28.2439231Z ##[section]Starting: Checkout rust-lang/rust@refs/pull/70951/merge to s
2020-04-09T12:49:28.2443832Z Task         : Get sources
2020-04-09T12:49:28.2444103Z Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
2020-04-09T12:49:28.2444364Z Version      : 1.0.0
2020-04-09T12:49:28.2444536Z Author       : Microsoft
2020-04-09T12:49:28.2444536Z Author       : Microsoft
2020-04-09T12:49:28.2444810Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
2020-04-09T12:49:28.2445144Z ==============================================================================
2020-04-09T12:49:28.5520017Z Cleaning any cached credential from repository: rust-lang/rust (GitHub)
2020-04-09T12:49:28.5559272Z ##[section]Finishing: Checkout rust-lang/rust@refs/pull/70951/merge to s
2020-04-09T12:49:28.5646322Z Cleaning up task key
2020-04-09T12:49:28.5647566Z Start cleaning up orphan processes.
2020-04-09T12:49:28.5819729Z Terminate orphan process: pid (3777) (python)
2020-04-09T12:49:28.6058935Z ##[section]Finishing: Finalize Job

I'm a bot! I can only do what humans tell me to, so if this was not helpful or you have suggestions for improvements, please ping or otherwise contact @rust-lang/infra. (Feature Requests)

andjo403 · 2020-04-09T14:39:42Z

for comparison with the measurement from 11days ago: #65031 (comment)
linux (wsl2)

cjgillot · 2020-04-09T15:12:35Z

Thank you.
TBH, this is underwhelming. I had another impression on my laptop, because of the memory usage.
I should document that.

This means the query system corresponds to 140s of codegen time. This is huge. I will try to find a way to reduce that.

nikomatsakis · 2020-04-10T17:34:00Z

r? @eddyb

eddyb · 2020-04-10T18:24:54Z

IMO a better approach to our monolithic rustc_middle::{query, ty::query}, inspired by Salsa but closer to the current system, would be this:

In crate rustc_hir (as an example), we add a mod query, containing a declare_queries! macro invocation, with only the queries that can be described in terms of rustc_hir, and the macro defines:

a struct Providers<Cx> where each field is foo: fn(Cx, FooKey) -> FooResult
- downstream, we'll have hir: hir::query::Providers<TyCtxt<'tcx>>, in struct Providers<'tcx>
- instead of fn provide methods that mutate a Providers, we could also use a trait that we then take fn pointers to the methods (implemented in several crates and brought together by rustc_interface, presumably?), but this is harder and more of a future extension
a macro to define methods (on the context type) for using the query (e.g. tcx.foo(key))
- downstream, we'd invoke hir::query::define_methods!(TyCtxt<'tcx>)
- alternatively, a trait relying on some helper traits being implemented
  - potentially harder to make work, and maybe slower w/o tuning
  - even closer to Salsa, and allows functions to be written that are generic over the context, instead of needing to know about TyCtxt

cc @rust-lang/compiler if anyone has any thoughts on that approach (it should let us break up the compiler more, and decentralize the queries)

As for this PR specifically, I haven't reviewed it yet and I don't know if I'll get to it any time soon, it seems like a pretty complex change. Maybe the new MCP process should be involved?

nikomatsakis · 2020-04-10T18:30:22Z

I agree that the MCP process would be a great fit here! I'm not sure I fully understand what you're proposing, have to read the comment a time or two more, but then again that's kind of the point of the MCP process, isn't it (too encourage enough up-front documentation until people understand what's being proposed).

eddyb · 2020-04-10T18:41:19Z

I'm not sure I fully understand what you're proposing

I was referring to MCP for this PR, not my suggestion. Although both are MCP-relevant.

nikomatsakis · 2020-04-10T19:10:43Z

I see. Well, I agree with that. =)

cjgillot · 2020-04-11T17:08:32Z

I agree with the end goal of decentralizing queries. The point of this PR is to see what needs to stay in librustc_middle, and what does not. From where I stand, the definitions of the DepKind and Providers types.
The latter may be easier to split up, as you described. About the DepKinds, I am less confident. The usage of the DepGraph would need refactoring.

nikomatsakis · 2020-04-13T14:04:39Z

@cjgillot Have you read about the new MCP process (see rust-lang/rfcs#2904)? The idea would be that you can open and issue and summarize the changes made in the PR and the goals with them. This does seem like a large enough change to merit an MCP, although I think it would probably be best to step back a bit and discuss the larger aims (e.g., I too think it'd be good to decentralize the query system. Long term, I would like to extract the query system that the compiler uses into a re-usable library, whether that be salsa or something else, but I don't think we have to start there.)

bors · 2020-04-13T16:16:56Z

☔ The latest upstream changes (presumably #70961) made this pull request unmergeable. Please resolve the merge conflicts.

joelpalmer · 2020-04-21T09:11:00Z

Ping from Triage: Hi @cjgillot - any updates?

cjgillot · 2020-04-21T09:52:26Z

I submitted a MCP at rust-lang/compiler-team#277
Some design needs to be done as part of this process.

bors · 2020-05-02T00:25:30Z

☔ The latest upstream changes (presumably #69808) made this pull request unmergeable. Please resolve the merge conflicts.

bors · 2021-02-20T13:17:42Z

📌 Commit 9823c2c has been approved by oli-obk

bors · 2021-02-20T14:36:25Z

⌛ Testing commit 9823c2c with merge 3439cb289114f9e715d32f6a854f2cdc74e40688...

bors · 2021-02-20T18:51:25Z

💥 Test timed out

cjgillot · 2021-02-20T18:58:01Z

@bors retry

bors · 2021-02-20T18:58:10Z

⌛ Testing commit 9823c2c with merge 83b30a6...

rust-log-analyzer · 2021-02-20T19:00:29Z

A job failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

bjorn3 · 2021-02-20T19:52:41Z

Hanging test:

2021-02-20T15:11:58.6669755Z test src/hash/mod.rs - hash::Hash (line 118) ... test src/hash/mod.rs - hash::Hash (line 118) has been running for over 60 seconds
2021-02-20T18:59:20.4859234Z ##[error]The operation was canceled.

bors · 2021-02-20T21:38:48Z

☀️ Test successful - checks-actions
Approved by: oli-obk
Pushing 83b30a6 to master...

rust-log-analyzer · 2021-02-20T21:39:58Z

A job failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 83b30a639 Auto merge of #70951 - cjgillot:anarchy, r=oli-obk
##[group]Run src/ci/publish_toolstate.sh
src/ci/publish_toolstate.sh
env:
  SCCACHE_BUCKET: rust-lang-ci-sccache2
  DEPLOY_BUCKET: rust-lang-ci2
  TOOLSTATE_REPO: https://github.com/rust-lang-nursery/rust-toolstate
---
  CACHE_DOMAIN: ci-caches.rust-lang.org
  TOOLSTATE_REPO_ACCESS_TOKEN: ***
##[endgroup]
Cloning into 'rust-toolstate'...
/home/runner/work/rust/rust/src/tools/publish_toolstate.py:121: DeprecationWarning: 'U' mode is deprecated
📣 Toolstate changed by rust-lang/rust#70951!
  with open(path, 'rU') as f:

Tested on commit rust-lang/rust@83b30a639d5abd1270ade35d9bd92271f5a5ba18.
Direct link to PR: <https://github.com/rust-lang/rust/pull/70951>

💔 miri on windows: test-fail → build-fail (cc @eddyb @oli-obk @RalfJung).
💔 miri on linux: test-fail → build-fail (cc @eddyb @oli-obk @RalfJung).
Traceback (most recent call last):
Traceback (most recent call last):
  File "/home/runner/work/rust/rust/src/tools/publish_toolstate.py", line 338, in <module>
    response = urllib2.urlopen(urllib2.Request(
  File "/usr/lib/python3.8/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.8/urllib/request.py", line 522, in open
    req = meth(req)
  File "/usr/lib/python3.8/urllib/request.py", line 1281, in do_request_
    raise TypeError(msg)
TypeError: POST data should be bytes, an iterable of bytes, or a file object. It cannot be of type str.
##[error]Process completed with exit code 1.

rylev · 2021-02-24T12:34:32Z

I looked over some of the perf results, and I agree we should merge it. There are two main regressions that should be looked into in follow-ups: typeck and const_eval. Both may again be due to subtle changes in inlining behaviour.

@oli-obk @cjgillot Shall we at least open an issue to track this? It would be nice to see if we can gain back the lost performance. The post-merge perf run doesn't seem to point to any obvious culprits though the evaluate_obligation and normalize_projection_ty queries did show up a few times as slowing down.

cjgillot · 2021-02-24T14:56:20Z

@rylev In the post-merge perf run, wall-time, cpu-clock and max-rss are overall green. I am not sure there is a regression to investigate.

rylev · 2021-02-24T18:16:36Z

The instruction count and wall-time have regressed . We usually investigate when there are regressions in instruction count.

tgnottingham · 2021-02-24T20:38:19Z

IIUC, some part of the instruction count regression will be attributable to virtual method calls resulting from now invoking queries through the QueryEngine trait object.

If that's the case, a direct way of addressing it would be to implement devirtualization of applicable trait method calls during LTO, and enable full thin LTO when building rustc. IIUC, devirtualizing should apply here, since there is only one implementor of QueryEngine.

There is an open issue for full program devirtualization, #68262.

rust-highfive assigned Zoxc Apr 9, 2020

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Apr 9, 2020

nikomatsakis unassigned Zoxc Apr 10, 2020

rust-highfive assigned eddyb Apr 10, 2020

cjgillot mentioned this pull request Apr 18, 2020

Decentralize queries rust-lang/compiler-team#277

Closed

wesleywiser added S-blocked Status: Blocked on something else such as an RFC or other implementation work. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels Apr 21, 2020

cjgillot force-pushed the anarchy branch from 552e47d to 87d4735 Compare April 29, 2020 16:59

This comment has been minimized.

Sign in to view

cjgillot force-pushed the anarchy branch from 87d4735 to 1cad322 Compare May 1, 2020 14:46

cjgillot force-pushed the anarchy branch from 1cad322 to 50b4753 Compare May 16, 2020 12:09

jyn514 added the I-compiletime Issue: Problems and improvements with respect to compile times. label Sep 15, 2020

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 20, 2021

bors added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. labels Feb 20, 2021

bors added S-waiting-on-bors Status: Waiting on bors to run and complete tests. Bors will change the label on completion. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Feb 20, 2021

bors added the merged-by-bors This PR was explicitly merged by bors. label Feb 20, 2021

bors merged commit 83b30a6 into rust-lang:master Feb 20, 2021

rustbot added this to the 1.52.0 milestone Feb 20, 2021

bors mentioned this pull request Feb 20, 2021

Make the Query enum a simple struct. #80891

Merged

cjgillot deleted the anarchy branch February 20, 2021 21:52

bors mentioned this pull request Feb 20, 2021

Only keep one dep-graph in memory #82345

Closed

tgnottingham mentioned this pull request Feb 22, 2021

rustc_codegen_ssa: tune codegen according to available concurrency #82127

Merged

matthiaskrgr mentioned this pull request Mar 20, 2021

query system no longer lives inside rustc_middle ? rust-lang/rustc-dev-guide#1091

Open

jyn514 mentioned this pull request Jul 2, 2021

The rustc_query_impl crate is too big, which hurts compile times for the compiler itself #65031

Open

tgnottingham mentioned this pull request Sep 29, 2021

Move OnDiskCache to rustc_query_impl. #86698

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Move the query engine out of rustc_middle #70951

Move the query engine out of rustc_middle #70951

cjgillot commented Apr 9, 2020

rust-highfive commented Apr 9, 2020

andjo403 commented Apr 9, 2020

cjgillot commented Apr 9, 2020

nikomatsakis commented Apr 10, 2020

eddyb commented Apr 10, 2020

nikomatsakis commented Apr 10, 2020

eddyb commented Apr 10, 2020

nikomatsakis commented Apr 10, 2020

cjgillot commented Apr 11, 2020

nikomatsakis commented Apr 13, 2020

bors commented Apr 13, 2020

joelpalmer commented Apr 21, 2020

cjgillot commented Apr 21, 2020

This comment has been minimized.

bors commented May 2, 2020

bors commented Feb 20, 2021

bors commented Feb 20, 2021

bors commented Feb 20, 2021

cjgillot commented Feb 20, 2021

bors commented Feb 20, 2021

rust-log-analyzer commented Feb 20, 2021

bjorn3 commented Feb 20, 2021

bors commented Feb 20, 2021

rust-log-analyzer commented Feb 20, 2021

rylev commented Feb 24, 2021

cjgillot commented Feb 24, 2021

rylev commented Feb 24, 2021

tgnottingham commented Feb 24, 2021

Move the query engine out of rustc_middle #70951

Move the query engine out of rustc_middle #70951

Conversation

cjgillot commented Apr 9, 2020

rust-highfive commented Apr 9, 2020

andjo403 commented Apr 9, 2020

cjgillot commented Apr 9, 2020

nikomatsakis commented Apr 10, 2020

eddyb commented Apr 10, 2020

nikomatsakis commented Apr 10, 2020

eddyb commented Apr 10, 2020

nikomatsakis commented Apr 10, 2020

cjgillot commented Apr 11, 2020

nikomatsakis commented Apr 13, 2020

bors commented Apr 13, 2020

joelpalmer commented Apr 21, 2020

cjgillot commented Apr 21, 2020

This comment has been minimized.

bors commented May 2, 2020

bors commented Feb 20, 2021

bors commented Feb 20, 2021

bors commented Feb 20, 2021

cjgillot commented Feb 20, 2021

bors commented Feb 20, 2021

rust-log-analyzer commented Feb 20, 2021

bjorn3 commented Feb 20, 2021

bors commented Feb 20, 2021

rust-log-analyzer commented Feb 20, 2021

rylev commented Feb 24, 2021

cjgillot commented Feb 24, 2021

rylev commented Feb 24, 2021

tgnottingham commented Feb 24, 2021