poky + remote-test-client = test failures?

[Yashinde145]

After building and testing rustc bootstrap build via cmd python3 src/bootstrap/bootstrap.py test --exclude<some excluded test cases list> --no-fail-fast --bless --target x86_64-poky-linux-gnu, some unit test fail as follows and the build test is unsuccessful.

---- markdown::term::tests::test_output stdout ----
thread 'markdown::term::tests::test_output' panicked at compiler/rustc_errors/src/markdown/tests/term.rs:75:55:
called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::result::unwrap_failed
   3: rustc_errors::markdown::term::tests::test_output
   4: <rustc_errors::markdown::term::tests::test_output::{closure#0} as core::ops::function::FnOnce<()>>::call_once
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


Running unittests src/lib.rs (/home/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.76.0/rustc-1.76.0-src/build/bootstrap/debug/deps/bootstrap-214111e112557758)
uploaded "/home/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.76.0/rustc-1.76.0-src/build/bootstrap/debug/deps/bootstrap-214111e112557758", waiting for result
thread 'main' panicked at src/tools/remote-test-client/src/main.rs:310:9:
client.read_exact(&mut header) failed with failed to fill whole buffer
stack backtrace:
   0: rust_begin_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
   2: remote_test_client::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: test failed, to rerun pass `--lib`
error: 1 target failed:
    `--lib`
	finished in 16.730 seconds

This failing cases should be fixed to successfully run the rustc build.
Full log for more details: log_1_76_without_doc_flag.txt

https://gist.github.com/Yashinde145/036a934f0523307859f7c855b83ecfd6/raw/f7bc4558eed66a10b65bb915f292ae166d3bb52a/log_1_76_without_doc_flag.txt

[Yashinde145]

The unwrap() function doesn't seems to match with output at line 79:
79 if std::fs::read(blessed).unwrap() != output {

The piece of code corresponding to above failure is present at compiler/rustc_errors/src/markdown/tests/term.rs

62 #[test]
63 fn test_output() {
64     // Capture `--bless` when run via ./x
65     let bless = std::env::var_os("RUSTC_BLESS").is_some_and(|v| v != "0");
66     let ast = MdStream::parse_str(INPUT);
67     let bufwtr = BufferWriter::stderr(ColorChoice::Always);
68     let mut buffer = bufwtr.buffer();
69     ast.write_termcolor_buf(&mut buffer).unwrap();
70 
71     let mut blessed = PathBuf::new();
72     blessed.extend(OUTPUT_PATH);
73 
74     if bless {
75         std::fs::write(&blessed, buffer.into_inner()).unwrap();
76         eprintln!("blessed output at {}", blessed.display());
77     } else {
78         let output = buffer.into_inner();
79         if std::fs::read(blessed).unwrap() != output {
80             // hack: I don't know any way to write bytes to the captured stdout
81             // that cargo test uses
82             let mut out = std::io::stdout();
83             out.write_all(b"\n\nMarkdown output did not match. Expected:\n").unwrap();
84             out.write_all(&output).unwrap();
85             out.write_all(b"\n\n").unwrap();
86             panic!("markdown output mismatch");
87         }
88     }
89 }

More details of backtrace are as follows:

failures:                                                                                                                                                                                                       
                                                                                                                                                                                                                
---- markdown::term::tests::test_output stdout ----                                                                                                                                                             
thread 'markdown::term::tests::test_output' panicked at compiler/rustc_errors/src/markdown/tests/term.rs:79:35:                                                                                                 
called `Result::unwrap()` on an `Err` value: Os { code: 2, kind: NotFound, message: "No such file or directory" }                                                                                               
stack backtrace:                                                                                                                                                                                                
   0:     0x7f0b971ed525 - <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt::hb5b8b23e1d3117df                                                                                 
   1:     0x7f0b97250a50 - core::fmt::write::h77cf7ecd13e24cdb                                                                                                                                                  
   2:     0x7f0b971e1a54 - std::io::Write::write_fmt::h929abddc424c1c03                                                                                                                                         
   3:     0x7f0b971ed304 - std::sys_common::backtrace::print::h6ef0f6b3ba2dff15                                                                                                                                 
   4:     0x7f0b971d679a - std::panicking::default_hook::{{closure}}::hf334a0ed2d582f71                                                                                                                         
   5:     0x7f0b971d646d - std::panicking::default_hook::h94ada5f696a4da09                                                                                                                                      
   6:     0x7f0b97315597 - test::test_main::{{closure}}::h982c6c7f4d49b186                                                                                                                                      
   7:     0x7f0b971d7a30 - std::panicking::rust_panic_with_hook::h8ac3130610deff09                                                                                                                              
   8:     0x7f0b971eda98 - std::panicking::begin_panic_handler::{{closure}}::h5daaeb2c8c5b5ae2                                                                                                                  
   9:     0x7f0b971ed756 - std::sys_common::backtrace::__rust_end_short_backtrace::h36eb27ebe1130b62                                                                                                            
  10:     0x7f0b971d75ef - rust_begin_unwind                                                                                                                                                                    
  11:     0x7f0b971c08d4 - core::panicking::panic_fmt::he2f151b5ff136851                                                                                                                                        
  12:     0x7f0b971c0ee2 - core::result::unwrap_failed::hd3774fef199fac57                                                                                                                                       
  13:     0x55e27d031c42 - rustc_errors[c1187c563773e00b]::markdown::term::tests::test_output                                                                                                                   
  14:     0x55e27d021fc9 - <rustc_errors[c1187c563773e00b]::markdown::term::tests::test_output::{closure#0} as core[6271f49ed3764161]::ops::function::FnOnce<()>>::call_once                                    
  15:     0x7f0b9731b87f - test::__rust_begin_short_backtrace::h219f60de6750c236                                                                                                                                
  16:     0x7f0b97300b8e - test::types::RunnableTest::run::h627f218ba9dc470b                                                                                                                                    
  17:     0x7f0b9731ba8b - test::run_test_in_process::h31596d3c9413cc59                                                                                                                                         
  18:     0x7f0b9730dfd7 - std::sys_common::backtrace::__rust_begin_short_backtrace::h1421b686c341a724                                                                                                          
  19:     0x7f0b97304175 - core::ops::function::FnOnce::call_once{{vtable.shim}}::ha0c922f0b66be45e                                                                                                             
  20:     0x7f0b971f0965 - std::sys::unix::thread::Thread::new::thread_start::hee2d40b57cf79590                                                                                                                 
  21:     0x7f0b970007f2 - <unknown>                                                                                                                                                                            
  22:     0x7f0b97078a3c - <unknown>                                                                                                                                                                            
  23:                0x0 - <unknown>

[Yashinde145]

Is there any update regarding this issue?
@rustbot label +I-prioritize

[apiraino]

By looking at the full log, IIUC there's one test failing: markdown::term::tests::test_output, PathBuf::new() seems to fail at writing in the file.

I've tried running that test alone (with a rustc fresh checkout) and it does not fail (note: removed the --target):
python3 src/bootstrap/bootstrap.py test markdown::term::tests::test_output --no-fail-fast --bless Building bootstrap. I'm now running the full test suite, will take a while but so far, I can't reproduce.

@Yashinde145 can you reproduce on any other target? (Just making sure it's not related to your poky target, which seems unlikely).

EDIT: replying to myself. That test fails also for me but in a different way, so I guess unrelated

Error details

---- [run-make] tests/run-make/libtest-junit stdout ----

error: make failed
status: exit status: 2
command: cd "/home/me/rust/tests/run-make/libtest-junit" && env -u CARGO_MAKEFLAGS -u MAKEFLAGS -u MFLAGS -u RUSTFLAGS AR="ar" CC="cc -ffunction-sections -fdata-sections -fPIC -m64" CXX="c++ -ffunction-sections -fdata-sections -fPIC -m64" HOST_RPATH_DIR="/home/me/rust/build/x86_64-unknown-linux-gnu/stage2/lib" LD_LIB_PATH_ENVVAR="LD_LIBRARY_PATH" LLVM_BIN_DIR="/home/me/rust/build/x86_64-unknown-linux-gnu/llvm/bin" LLVM_COMPONENTS="aarch64 aarch64asmparser aarch64codegen aarch64desc aarch64disassembler aarch64info aarch64utils aggressiveinstcombine all all-targets analysis arm armasmparser armcodegen armdesc armdisassembler arminfo armutils asmparser asmprinter avr avrasmparser avrcodegen avrdesc avrdisassembler avrinfo binaryformat bitreader bitstreamreader bitwriter bpf bpfasmparser bpfcodegen bpfdesc bpfdisassembler bpfinfo cfguard codegen codegentypes core coroutines coverage csky cskyasmparser cskycodegen cskydesc cskydisassembler cskyinfo debuginfobtf debuginfocodeview debuginfodwarf debuginfogsym debuginfologicalview debuginfomsf debuginfopdb demangle dlltooldriver dwarflinker dwarflinkerclassic dwarflinkerparallel dwp engine executionengine extensions filecheck frontenddriver frontendhlsl frontendoffloading frontendopenacc frontendopenmp fuzzercli fuzzmutate globalisel hexagon hexagonasmparser hexagoncodegen hexagondesc hexagondisassembler hexagoninfo hipstdpar instcombine instrumentation interfacestub interpreter ipo irprinter irreader jitlink libdriver lineeditor linker loongarch loongarchasmparser loongarchcodegen loongarchdesc loongarchdisassembler loongarchinfo lto m68k m68kasmparser m68kcodegen m68kdesc m68kdisassembler m68kinfo mc mca mcdisassembler mcjit mcparser mips mipsasmparser mipscodegen mipsdesc mipsdisassembler mipsinfo mirparser msp430 msp430asmparser msp430codegen msp430desc msp430disassembler msp430info native nativecodegen nvptx nvptxcodegen nvptxdesc nvptxinfo objcarcopts objcopy object objectyaml option orcdebugging orcjit orcshared orctargetprocess passes powerpc powerpcasmparser powerpccodegen powerpcdesc powerpcdisassembler powerpcinfo profiledata remarks riscv riscvasmparser riscvcodegen riscvdesc riscvdisassembler riscvinfo riscvtargetmca runtimedyld scalaropts selectiondag sparc sparcasmparser sparccodegen sparcdesc sparcdisassembler sparcinfo support symbolize systemz systemzasmparser systemzcodegen systemzdesc systemzdisassembler systemzinfo tablegen target targetparser textapi textapibinaryreader transformutils vectorize webassembly webassemblyasmparser webassemblycodegen webassemblydesc webassemblydisassembler webassemblyinfo webassemblyutils windowsdriver windowsmanifest x86 x86asmparser x86codegen x86desc x86disassembler x86info x86targetmca xray" LLVM_FILECHECK="/home/me/rust/build/x86_64-unknown-linux-gnu/llvm/build/bin/FileCheck" NODE="/usr/bin/node" PYTHON="/usr/bin/python3" RUSTC="/home/me/rust/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" RUSTC_BLESS_TEST="--bless" RUSTDOC="/home/me/rust/build/x86_64-unknown-linux-gnu/stage2/bin/rustdoc" RUST_BUILD_STAGE="stage2-x86_64-unknown-linux-gnu" S="/home/me/rust" TARGET="x86_64-unknown-linux-gnu" TARGET_RPATH_DIR="/home/me/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib" TMPDIR="/home/me/rust/build/x86_64-unknown-linux-gnu/test/run-make/libtest-junit/libtest-junit" "make"
--- stdout -------------------------------
LD_LIBRARY_PATH="/home/me/rust/build/x86_64-unknown-linux-gnu/test/run-make/libtest-junit/libtest-junit:/home/me/rust/build/x86_64-unknown-linux-gnu/stage2/lib:/home/me/rust/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/home/me/rust/build/x86_64-unknown-linux-gnu/stage0/lib" '/home/me/rust/build/x86_64-unknown-linux-gnu/stage2/bin/rustc' --out-dir /home/me/rust/build/x86_64-unknown-linux-gnu/test/run-make/libtest-junit/libtest-junit -L /home/me/rust/build/x86_64-unknown-linux-gnu/test/run-make/libtest-junit/libtest-junit  -Ainternal_features --test f.rs
RUST_BACKTRACE=0 LD_LIBRARY_PATH="/home/me/rust/build/x86_64-unknown-linux-gnu/test/run-make/libtest-junit/libtest-junit:/home/me/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/me/rust/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/home/me/rust/build/x86_64-unknown-linux-gnu/stage0/lib" /home/me/rust/build/x86_64-unknown-linux-gnu/test/run-make/libtest-junit/libtest-junit/f -Z unstable-options --test-threads=1 --format=junit > /home/me/rust/build/x86_64-unknown-linux-gnu/test/run-make/libtest-junit/libtest-junit/libtest-junit-output-default.xml || true
RUST_BACKTRACE=0 LD_LIBRARY_PATH="/home/me/rust/build/x86_64-unknown-linux-gnu/test/run-make/libtest-junit/libtest-junit:/home/me/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib:/home/me/rust/build/x86_64-unknown-linux-gnu/stage0-bootstrap-tools/x86_64-unknown-linux-gnu/release/deps:/home/me/rust/build/x86_64-unknown-linux-gnu/stage0/lib" /home/me/rust/build/x86_64-unknown-linux-gnu/test/run-make/libtest-junit/libtest-junit/f -Z unstable-options --test-threads=1 --format=junit --show-output > /home/me/rust/build/x86_64-unknown-linux-gnu/test/run-make/libtest-junit/libtest-junit/libtest-junit-output-stdout-success.xml || true
cat /home/me/rust/build/x86_64-unknown-linux-gnu/test/run-make/libtest-junit/libtest-junit/libtest-junit-output-default.xml | "/usr/bin/python3" validate_junit.py
Invalid xml: 'print from successful test\n'
------------------------------------------
--- stderr -------------------------------
thread 'b' panicked at f.rs:10:5:
assertion failed: false
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'c' panicked at f.rs:16:5:
assertion failed: false
thread 'b' panicked at f.rs:10:5:
assertion failed: false
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'c' panicked at f.rs:16:5:
assertion failed: false
Traceback (most recent call last):
  File "/home/me/rust/tests/run-make/libtest-junit/validate_junit.py", line 9, in 
    ET.fromstring(line)
  File "/usr/lib/python3.11/xml/etree/ElementTree.py", line 1345, in XML
    parser.feed(text)
xml.etree.ElementTree.ParseError: syntax error: line 1, column 38
make: *** [Makefile:14: all] Error 1
------------------------------------------

[Yashinde145]

@apiraino , Thanks for looking into it.

I've tried running that test alone (with a rustc fresh checkout) and it does not fail (note: removed the --target):
python3 src/bootstrap/bootstrap.py test markdown::term::tests::test_output --no-fail-fast --bless Building bootstrap. I'm now running the full test suite, will take a while but so far, I can't reproduce.

@Yashinde145 can you reproduce on any other target? (Just making sure it's not related to your poky target, which seems unlikely).

When I checked with other target (x86_64-unknown-linux-gnu), I was unable to run the test alone using python3 src/bootstrap/bootstrap.py test markdown::term::tests::test_output --no-fail-fast --bless. Got the following error, (let me know if I am missing something):

Building bootstrap
    Finished dev [unoptimized] target(s) in 0.36s
WARNING: The `change-id` is missing in the `config.toml`. This means that you will not be able to track the major changes made to the bootstrap configurations.
NOTE: to silence this warning, add `change-id = 102579` at the top of `config.toml`
ERROR: no `test` rules matched ["markdown::term::tests::test_output"]
HELP: run `x.py test --help --verbose` to show a list of available paths
NOTE: if you are adding a new Step to bootstrap itself, make sure you register it with `describe!`
Build completed unsuccessfully in 0:00:00

Then I ran the complete test suite but the logs didn't show any details about the test case or even that test module.

EDIT: @apiraino, any updates? Thanks.

[Yashinde145]

@onur-ozkan , Can you please assist me with this?

I can ignore the test at compiler/rustc_errors/src/markdown/tests/term.rs:75:55 using the #[ignore] tag but need to fix other failure, that is

Running unittests src/lib.rs (/home/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.76.0/rustc-1.76.0-src/build/bootstrap/debug/deps/bootstrap-214111e112557758)
uploaded "/home/poky/build-st/tmp/work/core2-64-poky-linux/rust/1.76.0/rustc-1.76.0-src/build/bootstrap/debug/deps/bootstrap-214111e112557758", waiting for result
thread 'main' panicked at src/tools/remote-test-client/src/main.rs:310:9:
client.read_exact(&mut header) failed with failed to fill whole buffer
stack backtrace:
   0: rust_begin_unwind
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/std/src/panicking.rs:645:5
   1: core::panicking::panic_fmt
             at /rustc/82e1608dfa6e0b5569232559e3d385fea5a93112/library/core/src/panicking.rs:72:14
   2: remote_test_client::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
error: test failed, to rerun pass `--lib`
error: 1 target failed:
    `--lib`
	finished in 16.730 seconds

I tried a few ways to fix this but no luck.

[onur-ozkan]

I don't have enough time to assist you with this task at the moment. However, I would prefer to fix the tests rather than ignore them.

[Yashinde145]

Is there someone in the rust team whom I can contact regarding this issue?

[onur-ozkan]

compiler and bootstrap teams can help.

[workingjubilee]

In the original test you cited an issue with, there is a panic because of this:

let mut blessed = PathBuf::new();
blessed.extend(OUTPUT_PATH);

Yes, this fails:

if std::fs::read(blessed).unwrap() != output {

But it fails because there's no file there.

It's looking at:

const OUTPUT_PATH: &[&str] =
    &[env!("CARGO_MANIFEST_DIR"), "src", "markdown", "tests", "output.stdout"];

What is the environment variable CARGO_MANIFEST_DIR being set to at the time of building the compiler/rustc_errors crate for testing?

[workingjubilee]

This seems to be another issue with the yocto/poky build code. It has not been correctly identified as a regression, as we track regressions relative to our repository, not yours.

@rustbot label: -I-prioritize -regression-untriaged

[workingjubilee]

There are two problems here:

• The failure in rustc_errors::markdown::term::tests::test_output may be a problem only with the build instructions that Yocto's poky uses, but we have still failed to reproduce the issue outside of the poky build environment. We do not run Yocto's software in CI, so we do not guarantee their tests will pass. Unfortunate, but expected.
• The remote-test-client failure concerns some deserialization code, which is often notoriously touchy with respect to codegen correctness. Subtle ABI inaccuracies are very likely to manifest as problems here. It has only been demonstrated to occur in a custom build target, x86_64-poky-linux-gnu, with a Rust compiler patched according to poky's patchset, which has undermined the integrity of rustc's codegen by disabling checking the data layout of a target against LLVM's understanding of the same, and other questionable choices. Also see:
  • yoctoproject/poky@c08c522
  • yoctoproject/poky@d419566
  • poky-patched cargo segfaults on cargo build --bin helloworld #128492

This combination of facts removes the remaining evidence that this is a bug, at least with respect to our source code. It seems to, instead, be software that is working according to the command of those that have patched it. 🎵 If you replace the working parts, what you get is a different machine. 🎵

I am closing this issue. I can think of two situations which justify reopening it:

• If we get a reproducer that works with our upstream codebase, then we can reopen this issue focused on the rustc_errors::markdown::term::tests::test_output failure.
• If Yocto starts shipping a rustc in poky that has not had significant parts of its codegen correctness undermined, then we can reopen the issue focused on the remote-test-client. A reproducer that would work with our upstream codebase might be quite contorted-seeming in any case, as it requires a particular setup to do remote testing to begin with, so any resulting "minimal" repro might be somewhat strange-looking anyways.

There may be others. In any case, even if something changes that justifies revisiting this issue, it might be best to open a new issue, or pair of issues, or trio of issues or more, so as to keep these requests focused and minimize unnecessary overlap with other issues.

@rustbot label: -C-bug +S-needs-repro