Replace threads with a job manager in minicargo #316

thepowersgang · 2023-09-22T13:05:50Z

This simplifies the minicargo code (as it avoids threading issues), and opens the way for a pretty progress indicator.

Fixes build system issues where libstd was being built twice with different features (happend with libstd on >1.19)
Fix a performance issue in evaluating cfg() in Cargo.toml files
Optimise tool debugging messages
Add a GNU make jobserver client/server implementation (intended to allow multi-threaded mrustc)

…compilation

…iple builds with different features)

thepowersgang · 2023-09-22T13:09:49Z

Should fix #307

catap · 2023-09-22T13:19:23Z

Yay! I’ll try to run a few builds via MacPorts port file

thepowersgang · 2023-09-22T13:59:15Z

Let me know how it goes, I'm pretty sure the branch is working but not 100% sure.

catap · 2023-09-22T15:29:05Z

@thepowersgang it fails quite fast on

:info:build os.cpp:203:13: error: use of undeclared identifier 'pipe2'
:info:build         if( pipe2(stderr_streams, O_CLOEXEC) != 0 ) {
:info:build             ^

thepowersgang · 2023-09-23T01:42:46Z

Should be fixed now.

thepowersgang · 2023-09-23T04:11:13Z

@catap Let me know if that fixes the issue you're seeing? Don't want to break things too badly.

catap · 2023-09-23T08:26:01Z

@thepowersgang now it's failing as:

:info:build [CXX] -o .obj/mir/borrow_check.o
:info:build src/mir/borrow_check.cpp:82:9: error: non-void function does not return a value in all control paths [-Werror,-Wreturn-type]
:info:build         }
:info:build         ^
:info:build src/mir/borrow_check.cpp:99:9: error: non-void function does not return a value in all control paths [-Werror,-Wreturn-type]
:info:build         }
:info:build         ^
:info:build src/mir/borrow_check.cpp:429:25: warning: binding reference member 'state' to stack allocated parameter 'state' [-Wdangling-field]
:info:build                 : state(state)
:info:build                         ^~~~~
:info:build src/mir/borrow_check.cpp:426:41: note: reference member declared here
:info:build             const ::MIR::TypeResolve&   state;
:info:build                                         ^
:info:build src/mir/borrow_check.cpp:553:30: warning: unused variable 'maybe_monomorph' [-Wunused-variable]
:info:build                         auto maybe_monomorph = [&](const auto& ty)->const HIR::TypeRef& {
:info:build                              ^
:info:build 2 warnings and 2 errors generated.
:info:build make[1]: *** [.obj/mir/borrow_check.o] Error 1

thepowersgang · 2023-09-23T08:30:40Z

Ah, the fun of different compilers having different ability to see returns.
Pushed new version.

catap · 2023-09-23T10:36:46Z

@thepowersgang mrust is build, anyway bootstrap rust fails as:

--- BUILDING rustc_mir v0.0.0 (89.0% 1r,5w,4b/91t)
> /opt/local/libexec/mrustc/bin/mrustc rustc-1.54.0-src/compiler/rustc_mir/src/lib.rs -o output-1.54.0/rustc-build/librustc_mir.rlib -C emit-depfile=output-1.54.0/rustc-build/librustc_mir.rlib.d --cfg debug_assertions -O -L output-1.54.0 -L output-1.54.0/rustc-build --crate-name rustc_mir --crate-type rlib --crate-tag 0_0_0 --edition 2018 --extern either=output-1.54.0/rustc-build/libeither-1_6_1_H2.rlib --extern rustc_graphviz=output-1.54.0/rustc-build/librustc_graphviz.rlib --extern gsgdt=output-1.54.0/rustc-build/libgsgdt-0_1_2.rlib --extern itertools=output-1.54.0/rustc-build/libitertools-0_9_0_H1.rlib --extern tracing=output-1.54.0/rustc-build/libtracing-0_1_26_H30002.rlib --extern polonius_engine=output-1.54.0/rustc-build/libpolonius_engine-0_12_1.rlib --extern regex=output-1.54.0/rustc-build/libregex-1_5_4_H1fffb.rlib --extern rustc_middle=output-1.54.0/rustc-build/librustc_middle.rlib --extern rustc_attr=output-1.54.0/rustc-build/librustc_attr.rlib --extern rustc_data_structures=output-1.54.0/rustc-build/librustc_data_structures.rlib --extern rustc_errors=output-1.54.0/rustc-build/librustc_errors.rlib --extern rustc_hir=output-1.54.0/rustc-build/librustc_hir.rlib --extern rustc_index=output-1.54.0/rustc-build/librustc_index.rlib --extern rustc_infer=output-1.54.0/rustc-build/librustc_infer.rlib --extern rustc_lexer=output-1.54.0/rustc-build/librustc_lexer-0_1_0.rlib --extern rustc_macros=output-1.54.0/rustc-build/librustc_macros-0_1_0-plugin --extern rustc_serialize=output-1.54.0/rustc-build/librustc_serialize.rlib --extern rustc_session=output-1.54.0/rustc-build/librustc_session.rlib --extern rustc_target=output-1.54.0/rustc-build/librustc_target.rlib --extern rustc_trait_selection=output-1.54.0/rustc-build/librustc_trait_selection.rlib --extern rustc_ast=output-1.54.0/rustc-build/librustc_ast.rlib --extern rustc_span=output-1.54.0/rustc-build/librustc_span.rlib --extern rustc_apfloat=output-1.54.0/rustc-build/librustc_apfloat.rlib --extern smallvec=output-1.54.0/rustc-build/libsmallvec-1_6_1_H12.rlib > output-1.54.0/rustc-build/librustc_mir.rlib_dbg.txt
 (89.0% 1r,5w,4b/91t): rustc_mir v0.0.0
rustc-1.54.0-src/compiler/rustc_mir/src/borrow_check/region_infer/mod.rs:1864:50-58 warn:0:Unexpected attribute inline on expression
Process was terminated with signal 11
FAILING COMMAND: /opt/local/libexec/mrustc/bin/mrustc rustc-1.54.0-src/compiler/rustc_mir/src/lib.rs -o output-1.54.0/rustc-build/librustc_mir.rlib -C emit-depfile=output-1.54.0/rustc-build/librustc_mir.rlib.d --cfg debug_assertions -O -L output-1.54.0 -L output-1.54.0/rustc-build --crate-name rustc_mir --crate-type rlib --crate-tag 0_0_0 --edition 2018 --extern either=output-1.54.0/rustc-build/libeither-1_6_1_H2.rlib --extern rustc_graphviz=output-1.54.0/rustc-build/librustc_graphviz.rlib --extern gsgdt=output-1.54.0/rustc-build/libgsgdt-0_1_2.rlib --extern itertools=output-1.54.0/rustc-build/libitertools-0_9_0_H1.rlib --extern tracing=output-1.54.0/rustc-build/libtracing-0_1_26_H30002.rlib --extern polonius_engine=output-1.54.0/rustc-build/libpolonius_engine-0_12_1.rlib --extern regex=output-1.54.0/rustc-build/libregex-1_5_4_H1fffb.rlib --extern rustc_middle=output-1.54.0/rustc-build/librustc_middle.rlib --extern rustc_attr=output-1.54.0/rustc-build/librustc_attr.rlib --extern rustc_data_structures=output-1.54.0/rustc-build/librustc_data_structures.rlib --extern rustc_errors=output-1.54.0/rustc-build/librustc_errors.rlib --extern rustc_hir=output-1.54.0/rustc-build/librustc_hir.rlib --extern rustc_index=output-1.54.0/rustc-build/librustc_index.rlib --extern rustc_infer=output-1.54.0/rustc-build/librustc_infer.rlib --extern rustc_lexer=output-1.54.0/rustc-build/librustc_lexer-0_1_0.rlib --extern rustc_macros=output-1.54.0/rustc-build/librustc_macros-0_1_0-plugin --extern rustc_serialize=output-1.54.0/rustc-build/librustc_serialize.rlib --extern rustc_session=output-1.54.0/rustc-build/librustc_session.rlib --extern rustc_target=output-1.54.0/rustc-build/librustc_target.rlib --extern rustc_trait_selection=output-1.54.0/rustc-build/librustc_trait_selection.rlib --extern rustc_ast=output-1.54.0/rustc-build/librustc_ast.rlib --extern rustc_span=output-1.54.0/rustc-build/librustc_span.rlib --extern rustc_apfloat=output-1.54.0/rustc-build/librustc_apfloat.rlib --extern smallvec=output-1.54.0/rustc-build/libsmallvec-1_6_1_H12.rlib
Env:  OUT_DIR=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_mrustc/mrustc-rust/work/thepowersgang-mrustc-ca1096a/output-1.54.0/rustc-build/build_rustc_mir CARGO_MANIFEST_DIR=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_lang_mrustc/mrustc-rust/work/thepowersgang-mrustc-ca1096a/rustc-1.54.0-src/compiler/rustc_mir CARGO_PKG_NAME=rustc_mir CARGO_PKG_VERSION=0.0.0 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=0 CARGO_PKG_VERSION_PATCH=0
 (90.1% 0r,5w,4b/91t):
BUILD FAILED

thepowersgang · 2023-09-23T13:33:08Z

hmm... segfault, I'm assuming that would also be present in master, so I'm just going to merge this branch to get it in,
@catap Could you open another issue with debugging info for that crash?

thepowersgang added 20 commits September 9, 2023 12:13

minicargo - Experimental job system

789c351

minicargo - Minimially-tested new build job system

ad82eac

minicargo - New build system seems to be working

e2a1257

minicargo - Fix windows build

df71e16

minicargo - Fix compilation error on linux

9a5f8d3

minicargo - Add dry run (-j 0) support to new job system

3a0fbfb

minicargo.mk - Tweak compilation of standard library to avoid double-…

ad2c4f5

…compilation

minicargo.mk - Another tweak to libstd build

87f8d50

windows build - Build libtest to get unwind/std for 1.54 (avoids mult…

808c821

…iple builds with different features)

common - Draft (windows only) version of a make jobserver client/server

fde18f9

common - Make jobserver compile on linux

140004c

minicargo - Optimise cfg checking

cf5f981

common - Save debug state when entering a new phase (optimisation)

075f065

common - Test results for jobserver on linux

98de700

minicargo - Use the jobserver!

4336e95

minicargo.mk - Invoke minicargo such that the jobserver is passed on

b1aea89

minicargo - Use jobserver if -j isn't passed

d4f6734

common jobserver - Fix linux compilation error

3503df0

minicargo - Remove the old build file.

97bdc0f

minicargo - Rename build2.cpp to build.cpp

180f48d

thepowersgang mentioned this pull request Sep 22, 2023

Parallel building has random fails #225

Closed

thepowersgang added 4 commits September 23, 2023 08:50

minicargo - Fix non-linux compilation (use pipe)

5e82727

minicargo - Fix non-blocking wait_one on linux

58174c0

minicargo - Set CLOEXEC on jobserver named pipe fds

dca98be

Makefile - Avoid phantom recompiles of mrustc

e90e4eb

MIR - Fix warnings in borrowcheck

ca1096a

thepowersgang merged commit 0728995 into master Sep 23, 2023
16 checks passed

This was referenced Sep 23, 2023

Parallel builds #307

Closed

SIGSEGV compiling 1.54.0 librustc_mir on OSX #319

Closed

thepowersgang deleted the minicargo_job_system branch December 4, 2023 09:54

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace threads with a job manager in minicargo #316

Replace threads with a job manager in minicargo #316

thepowersgang commented Sep 22, 2023 •

edited

Loading

thepowersgang commented Sep 22, 2023

catap commented Sep 22, 2023

thepowersgang commented Sep 22, 2023

catap commented Sep 22, 2023

thepowersgang commented Sep 23, 2023

thepowersgang commented Sep 23, 2023

catap commented Sep 23, 2023

thepowersgang commented Sep 23, 2023

catap commented Sep 23, 2023

thepowersgang commented Sep 23, 2023

Replace threads with a job manager in minicargo #316

Replace threads with a job manager in minicargo #316

Conversation

thepowersgang commented Sep 22, 2023 • edited Loading

thepowersgang commented Sep 22, 2023

catap commented Sep 22, 2023

thepowersgang commented Sep 22, 2023

catap commented Sep 22, 2023

thepowersgang commented Sep 23, 2023

thepowersgang commented Sep 23, 2023

catap commented Sep 23, 2023

thepowersgang commented Sep 23, 2023

catap commented Sep 23, 2023

thepowersgang commented Sep 23, 2023

thepowersgang commented Sep 22, 2023 •

edited

Loading