[BUG]: User getting illegal instructions after upgrading from 0.3.1 to 0.4.0

[jackos]

Bug description

After a user upgraded from 0.3.1 to 0.4.0, trying to run a hello world seg faults:

PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace.
Stack dump:
0.      Program arguments: mojo
 #0 0x0000564da0d06fd7 (/home/vilson/.modular/pkg/packages.modular.com_mojo/bin/mojo+0x5b7fd7)
 #1 0x0000564da0d04bae (/home/vilson/.modular/pkg/packages.modular.com_mojo/bin/mojo+0x5b5bae)
 #2 0x0000564da0d076af (/home/vilson/.modular/pkg/packages.modular.com_mojo/bin/mojo+0x5b86af)
 #3 0x00007fa68354f520 (/lib/x86_64-linux-gnu/libc.so.6+0x42520)
 #4 0x0000564da2ea0a80 (/home/vilson/.modular/pkg/packages.modular.com_mojo/bin/mojo+0x2751a80)
 #5 0x0000564da2ea0ceb (/home/vilson/.modular/pkg/packages.modular.com_mojo/bin/mojo+0x2751ceb)
 #6 0x0000564da10cc0e9 (/home/vilson/.modular/pkg/packages.modular.com_mojo/bin/mojo+0x97d0e9)
 #7 0x0000564da0cb346a (/home/vilson/.modular/pkg/packages.modular.com_mojo/bin/mojo+0x56446a)
 #8 0x00007fa683536d90 (/lib/x86_64-linux-gnu/libc.so.6+0x29d90)
 #9 0x00007fa683536e40 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e40)
#10 0x0000564da0cb335e (/home/vilson/.modular/pkg/packages.modular.com_mojo/bin/mojo+0x56435e)
Illegal instruction (core dumped)

Tried installing from scratch didn't fix it, also checked all system libraries being linked are the same as a working instance.

System information:

Linux awbox 5.15.0-67-generic #74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.2 LTS"

ldd mojo

        linux-vdso.so.1 (0x00007fff735d1000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7f580fb000)
        libLLCLRuntimeGlobals.so.18git => /home/vilson/.modular/pkg/packages.modular.com_mojo/bin/./../lib/libLLCLRuntimeGlobals.so.18git (0x00007f7f580f6000)
        libMSupportGlobals.so.18git => /home/vilson/.modular/pkg/packages.modular.com_mojo/bin/./../lib/libMSupportGlobals.so.18git (0x00007f7f580f1000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7f580ec000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7f580e7000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f7f57ffe000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7f57fe2000)
        libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f7f57fb0000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f7f57d84000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f7f57d64000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f7f57b3c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f7f5ca12000)

Might be something to do with the CPU.

lscpu

Architecture:            x86_64
  CPU op-mode(s):        32-bit, 64-bit
  Address sizes:         40 bits physical, 48 bits virtual
  Byte Order:            Little Endian
CPU(s):                  8
  On-line CPU(s) list:   0-7
Vendor ID:               GenuineIntel
  Model name:            DO-Premium-Intel
    CPU family:          6
    Model:               85
    Thread(s) per core:  1
    Core(s) per socket:  8
    Socket(s):           1
    Stepping:            7
    BogoMIPS:            3990.62
    Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_
                         freq pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_sing
                         le ssbd ibrs ibpb ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb xsaveopt xsavec xgetbv1 arat pku
                          ospke arch_capabilities
Virtualization features:
  Virtualization:        VT-x
  Hypervisor vendor:     KVM
  Virtualization type:   full
Caches (sum of all):
  L1d:                   256 KiB (8 instances)
  L1i:                   256 KiB (8 instances)
  L2:                    32 MiB (8 instances)
NUMA:
  NUMA node(s):          1
  NUMA node0 CPU(s):     0-7
Vulnerabilities:
  Itlb multihit:         KVM: Mitigation: VMX disabled
  L1tf:                  Not affected
  Mds:                   Not affected
  Meltdown:              Not affected
  Mmio stale data:       Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
  Retbleed:              Mitigation; Enhanced IBRS
  Spec store bypass:     Mitigation; Speculative Store Bypass disabled via prctl and seccomp
  Spectre v1:            Mitigation; usercopy/swapgs barriers and __user pointer sanitization
  Spectre v2:            Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
  Srbds:                 Not affected

From discord: https://discord.com/channels/1087530497313357884/1159586707415502999/1159681161094766597

[Mogball]

Could be an installer issue

[ematejska]

Perhaps related to: #993

[ematejska]

@Mogball, looked at this with @zbowling and he asked if the top frame could be symbolicated to see where it failed and if it's a compiler issue.

[Mogball]

I wouldn't know how to do that. I believe there's some process by which the symbols are stripped, but that's a release process thing

[jackos]

Another user with same issue here: #1057

triple = x86_64-unknown-linux
cpu = tigerlake
features = +64bit
data_layout = e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128
simd_bit_width = 512

adx
aes
avx
avx2
avx512bitalg
avx512bw
avx512cd
avx512dq
avx512f
avx512ifma
avx512vbmi
avx512vbmi2
avx512vl
avx512vnni
avx512vp2intersect
avx512vpopcntdq
bmi
bmi2
clflushopt
clwb
cmov
crc32
cx16
cx8
f16c
fma
fsgsbase
fxsr
gfni
invpcid
kl
lzcnt
mmx
movbe
movdir64b
movdiri
pclmul
pku
popcnt
prfchw
rdpid
rdrnd
rdseed
sahf
sgx
sha
shstk
sse
sse2
sse3
sse4.1
sse4.2
ssse3
vaes
vpclmulqdq
widekl
x87
xsave
xsavec
xsaveopt
xsaves

[Mogball]

With no readable stack trace or way to reproduce this on one of our machines, this is impossible to debug. We cannot even use the user's machine for debugging for obvious reasons.

[Mogball]

Update from the discussion: subsequent releases will include a way to symbolicate the stack traces. For now, the suspected issue is a mis-install from the CLI.

[jackos]

Another user with this issue on Discord here Leaving here as a reminder to respond to user when the issue is closed.

[ematejska]

Will close this in favor of a symbolication issue.

[jackos]

Users have reported in other tickets that this fixes the above problem:

removed ~/.modular directory and installed python3.10-venv, then used "modular auth XXXX && modular install mojo"