library compilation problem on s390x

[pazolka]

I'm getting the following error when building Parity Ethereum:

LLVM ERROR: Cannot select: 0x3ff58cd1598: i32 = SystemZISD::GET_CCMASK 0x3ff58cd13f8, Constant:i32<15>, Constant:i32<3>
  0x3ff58cd13f8: i32 = srl 0x3ff58a47040, Constant:i32<31>
    0x3ff58a47040: i32 = add 0x3ff58cd1a10, Constant:i32<-536870912>
      0x3ff58cd1a10: i32 = SystemZISD::IPM 0x3ff58cd1870:1
        0x3ff58cd1870: i64,i32 = SystemZISD::USUBO 0x3ff58cd12c0, Constant:i64<1>
          0x3ff58cd12c0: i64,ch = CopyFromReg 0x3ff58837308, Register:i64 %1
            0x3ff58a46f70: i64 = Register %1
          0x3ff58cd1f90: i64 = Constant<1>
      0x3ff58cd18d8: i32 = Constant<-536870912>
    0x3ff589c8260: i32 = Constant<31>
  0x3ff58cd1c80: i32 = Constant<15>
  0x3ff589c8740: i32 = Constant<3>
In function: _ZN15primitive_types4U12814saturating_sub17haa8aa831da279cf8E
error: Could not compile `primitive-types`.
warning: build failed, waiting for other jobs to finish...
error: build failed

Rust version: 1.34.2

[nikita-fuchs]

The source can be found here: https://bitbucket.org/nikita-fuchs/parity-ring-removal/src/master/

It is a fork of https://github.com/paritytech/parity-ethereum with two libraries removed that are not supported on s390x architecture. The forked code compiles just fine on x64.

[mati865]

Was the Rust installed with official package (rustup, installers) or using Linux package manager?

[pazolka]

Rustup

[mati865]

https://reviews.llvm.org/D59822 should fix it.
I'll see if I can reproduce it by cross compilation and prepare backports.

[mati865]

I can reproduce it by cross compiling from x86_64 and unfortunately backporting https://reviews.llvm.org/D59822 did not solve the issue.
https://github.com/paritytech/parity-common/tree/9d2de86e70f8e84358d0819a92666acb4f6ab9c4/primitive-types is what exposes this LLVM bug and can be used as smaller repro.

[nikic]

@mati865 Can you provide the --emit=llvm-ir output from primitive-types?

[mati865]

@nikic I think it's too big to be useful but here you go:
IR.zip

Obtained with: CARGO_INCREMENTAL=0 cargo rustc --target s390x-unknown-linux-gnu --release -- -o ir --emit=llvm-ir -C codegen-units=1 -C debuginfo=0 -C panic=abort

[nikic]

Bugpoint reduced:

; ModuleID = '<stdin>'
source_filename = "primitive_types.arilfrs7-cgu.0"
target datalayout = "E-m:e-i1:8:16-i8:8:16-i64:64-f128:64-a:8:16-n32:64"
target triple = "s390x-unknown-linux-gnu"

define void @_ZN15primitive_types4U51211checked_sub17h5ff0d8bcee5c2f13E() unnamed_addr #0 {
start:
  %_5.sroa.4.0.copyload = load i64, i64* undef, align 8
  %0 = tail call { i64, i1 } @llvm.usub.with.overflow.i64(i64 undef, i64 %_5.sroa.4.0.copyload) #2
  %1 = extractvalue { i64, i1 } %0, 1
  %2 = zext i1 %1 to i8
  br i1 undef, label %bb21.i, label %bb34.i

bb21.i:                                           ; preds = %start
  %3 = tail call { i64, i1 } @llvm.usub.with.overflow.i64(i64 undef, i64 1) #2
  %4 = extractvalue { i64, i1 } %3, 1
  %5 = zext i1 %4 to i8
  %6 = add nuw nsw i8 %5, %2
  br label %bb34.i

bb34.i:                                           ; preds = %bb21.i, %start
  %carry.1.in.i = phi i8 [ %6, %bb21.i ], [ %2, %start ]
  ret void
}

; Function Attrs: nounwind readnone speculatable
declare { i64, i1 } @llvm.usub.with.overflow.i64(i64, i64) #1

attributes #0 = { "target-features"="-vector" }
attributes #1 = { nounwind readnone speculatable }
attributes #2 = { nounwind }

!llvm.module.flags = !{!0}

!0 = !{i32 2, !"RtLibUseGOT", i32 1}

[nikic]

LLVM bug report: https://bugs.llvm.org/show_bug.cgi?id=42512

[nikic]

Fixed upstream via llvm/llvm-project@a2a09cb.

[nikita-fuchs]

Thank you @nikic , what does that mean for our case? As I'm not particularly familiar with LLVM and it came bundled with rust anyway I think, how do we obtain the latest fix? Just manually install/upgrade LLVM ?

[mati865]

@nikita-fuchs Rust have to build against LLVM with the fix.

[nikita-fuchs]

@mati865 any idea/estimate on when this might happen?

[mati865]

@nikita-fuchs when somebody tests the fix, backports it to Rust's LLVM fork and updates LLVM fork in Rust repo.

[nikic]

This should be fixed in the latest nightly. Can you confirm that the issue is resolved?

[pazolka]

when trying to install nightly:

~$ rustup default nightly
info: syncing channel updates for 'nightly-s390x-unknown-linux-gnu'
info: latest update on 2019-07-19, rust version 1.38.0-nightly (311376d30 2019-07-18)
info: downloading component 'rustc'
info: downloading component 'rust-std'
218.0 MiB / 218.0 MiB (100 %) 188.9 MiB/s in  1s ETA:  0s
info: downloading component 'cargo'
info: installing component 'rustc'
info: rolling back changes
error: failed to extract package (perhaps you ran out of disk space?)
info: caused by: lzma data error

Any other version (beta/stable) gets installed corrrectly.

[nikita-fuchs]

@nikic generally speaking, do you guys have sufficient access to System Z mainframes to try things out or should we ask IBM for some support for the Rust community ?

[nikita-fuchs]

hey @mati865 , @nikic , @jonas-schievink , @nikic , the error persists in version rustc 1.37.0 (eae3437df 2019-08-13) , installed through curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh . Sorry for spamming all of you, but it is a crucial major blocker to our project, currently in the fundraising phase, so it would be absolutely awesome if the LLVM issue could be resolved somehow. Thanks for your efforts, reach out if we can help anyhow ! :)

The error message changes only very slightly:

LLVM ERROR: Cannot select: 0x3ff947b5610: i32 = SystemZISD::GET_CCMASK 0x3ff943a9258, Constant:i32<15>, Constant:i32<3>
  0x3ff943a9258: i32 = srl 0x3ff947b5af0, Constant:i32<31>
    0x3ff947b5af0: i32 = add 0x3ff943a9870, Constant:i32<-536870912>
      0x3ff943a9870: i32 = SystemZISD::IPM 0x3ff943a96d0:1
        0x3ff943a96d0: i64,i32 = SystemZISD::USUBO 0x3ff943a9120, Constant:i64<1>
          0x3ff943a9120: i64,ch = CopyFromReg 0x3ff947c35e8, Register:i64 %1
            0x3ff948ee890: i64 = Register %1
          0x3ff94d3f7d0: i64 = Constant<1>
      0x3ff943a9738: i32 = Constant<-536870912>
    0x3ff947b5200: i32 = Constant<31>
  0x3ff943a8e48: i32 = Constant<15>
  0x3ff947b5880: i32 = Constant<3>
In function: _ZN15primitive_types4U12814saturating_sub17hbfa9e41384c02873E

[jonas-schievink]

s390x is not a well-supported architecture at the moment, I can not recommend building a business around it. If this issue is that important to you you can contract people with LLVM/s390x expertise to fix it.

[mati865]

@nikita-fuchs the fix is not in the stable release yet. It should be fixed in beta and nightly versions though.

[nikita-fuchs]

@jonas-schievink what would it theoretically need to support it better?
@mati865 there is unfortunately no way that I can download a nightly or beta with rustup:

root@zla19054:~/.cargo/bin#  ./rustup default nightly

gives always the following error:

info: syncing channel updates for 'nightly-s390x-unknown-linux-gnu'
info: latest update on 2019-08-20, rust version 1.39.0-nightly (29a54035c 2019-08-19)
info: downloading component 'rustc'
info: downloading component 'rust-std'
221.6 MiB / 221.6 MiB (100 %) 197.8 MiB/s in  1s ETA:  0s
info: downloading component 'cargo'
info: installing component 'rustc'
info: rolling back changes
error: failed to extract package (perhaps you ran out of disk space?)
info: caused by: lzma data error

although there is enough space on the drive.

That's what stacktrace says around this error:

...
write(2, "error: ", 7error: )                  = 7
ioctl(2, TCGETS, 0x3ffe647dd54)         = -1 ENOTTY (Inappropriate ioctl for device)
write(2, "failed to extract package (perha"..., 62failed to extract package (perhaps you ran out of disk space?)) = 62
write(2, "\n", 1
)                       = 1
ioctl(2, TCGETS, 0x3ffe647dd54)         = -1 ENOTTY (Inappropriate ioctl for device)
write(2, "info: ", 6info: )                   = 6
ioctl(2, TCGETS, 0x3ffe647dd54)         = -1 ENOTTY (Inappropriate ioctl for device)
write(2, "caused by: ", 11caused by: )             = 11
write(2, "lzma data error", 15lzma data error)         = 15
write(2, "\n", 1
...

[jonas-schievink]

A more robust LLVM backend would be a start, and there's little to no in-tree tests and no CI for the target. The extraction error you're seeing might also be caused by some other toolchain issue.

[mati865]

@nikita-fuchs I have no idea what is wrong but if it doesn't get fixed you won't be able to install next stable via rustup.
You should report it to https://github.com/rust-lang/rustup.rs/

[nikita-fuchs]

Thanks both of you @jonas-schievink and @mati865 , I've opened an issue about this package extraction error rust-lang/rustup#1970 , happens only with nightly and beta release though, the stable installs fine. Do you have an estimate on when this LLVM fix will make it into stable? I'll talk to IBM about considering some support, Rust is definitely nothing they would want to support sloppily.

[mati865]

@nikita-fuchs it should be included in the next release, that would be 26th September.

happens only with nightly and beta release though, the stable installs fine.

If it doesn't get fixed before next release stable will have the same error.

[jonas-schievink]

It looks like this is a legit miscompilation, not an issue with rustup or the archives. The fix will need a beta backport within the next 5-6 weeks, or the next stable will have the same problem.

[nikita-fuchs]

@jonas-schievink it looks like the culprit that's causing the package extraction issues were identified: rust-lang/rustup#1970 (comment) - hopefully someone can take care of it.

[sanxiyn]

rustup issue is now closed. Can this be tested again?

[nikita-fuchs]

hey @sanxiyn , as you can see rust-lang/rustup#1970 they celebrated a bit too early on that one, so it will take some more time. Getting back on this as soon as they've figured it out.

[nikita-fuchs]

Fixed, can be closed. Thanks for all your awesome support.

[sanxiyn]

Thanks for testing! Closing.