OS, compiler minimum versions and minimum processor requirements for running Julia

[ViralBShah]

It's been a while since we bumped minimum OS requirements and compiler versions.

OSes for running:

• For Windows, pick Windows 10, since Windows 7 is no longer supported by Microsoft. https://support.microsoft.com/en-in/help/4057281/windows-7-support-ended-on-january-14-2020 (Drop Windows 7 official support #35974)
• For Mac, what are the right considerations? Is there a benefit to bumping the min supported version? Even if just to simplify our build process? I suspect Mac should continue with 10.8.
• For Linux, we say kernel 2.6.18+. That sounds specific enough.

CPUs for running:

• Define clearly the oldest processor we support. (E.g. Julia 1.4 fails on startup (AMD Phenom on Linux) #35215 for AMD phenom pentium 4 for x86)
• Pick something similar for an equivalent Intel processor

For building:

• For compiling Julia, LLVM 9 leads to a new lower bound on gcc - julia fails to build #34555 (comment)

[ViralBShah]

Also some of the installation and compilation related documentation in docs/build should probably be shown more prominently, perhaps in the manual.

[PallHaraldsson]

[EDIT: if this is just to build, not run Julia, just ignore my comment]

pick Windows 10, since Windows 7 is no longer supported

Is there a real need? 2/3 are on Windows 10, and the rest more or less on Windows 7.

I'm not using Windows so don't care much personally, just seems premature. Next LTS will probably be Julia 1.5 right? Change after that?

[musm]

[EDIT: if this is just to build, not run Julia, just ignore my comment]

pick Windows 10, since Windows 7 is no longer supported

Is there a real need? 2/3 are on Windows 10, and the rest more or less on Windows 7.

I'm not using Windows so don't care much personally, just seems premature. Next LTS will probably be Julia 1.5 right? Change after that?

Julia being not supported on Windows 7, doesn't mean it'll stop working on Windows 7, just that it won't be supported by the Julia language team.

[ViralBShah]

At the moment it basically means that we can make assumptions about some minimum capabilities available by default - a reasonable version of powershell, maybe the windows terminal, etc. For Windows 7 users, they'll probably have to install some extra things or upgrades.

[ViralBShah]

#36123 updates the build dependencies

[ViralBShah]

Based on what @staticfloat said in #35215, we'll adopt the Core 2 instruction set as the minimum for Intel and Bulldozer for AMD.

Is there something we should do for ARM?

[yuyichao]

If anything that should at Most be what the official binary is compiled for and should not be the required feature. And there's also very little point of raising that requirement since basically no performance sensitive code is affected.

Nothing should change for arm.

[ViralBShah]

It is the required minimum CPU - in that older than those may not work - as discussed in #35215. This is documenting the current practice -not setting a policy for a new practice.

[ViralBShah]

@yuyichao Clearly our binaries do not work on armv6 processors. So presumably there is something we need to say about what works and what does not.

[yuyichao]

As I did mention in that issue, it is indeed a new practice. The old one was sse2, ie no additional requirements for x64. It was only broken recently.

And that’s why I said nothing should change for arm, since the binary already says armv7a and nothing should have changed about it. Armv6 has always been partially supported for source build and the binary from respian.

[ViralBShah]

As far as I understand, that decision is deliberate.

[yuyichao]

How?

It appears that what's happened here is that the LLVM rebuild for 1.4.0 was done with a newer GCC version, which ended up using SSSE3 instructions where it previously didn't,

AFAICT that means it's unintentional.

[ViralBShah]

My understanding is that all that was done to address performance issues with gmp and mpfr in the stock Julia distribution. @giordano and @staticfloat can say more.

If you think this can be done without using SSSE3 instructions, and can provide a patch, that would be great.

[giordano]

Not sure why that happened, probably @vchuravy can tell more

[yuyichao]

Well, that is inconsisitent with the observation since the issue, as mentioned in the commenet i quoted above, is on the LLVM. Not gmp or mpfr.

[giordano]

Yes, I think it was LLVM who ended up using a newer version of GCC, that's why I pinged Valentin

[yuyichao]

Also, as for BigInt, gmp specificially supports fat binary so it shouldn't need any elevated minimum hardware requirement on x64 (that it does seem to be enabled in the gmp builder script --enable-fat).

[staticfloat]

In the past, within BB we compiled with a GCC that had no default architecture. That meant it never used anything other than the most very basic of CPU instructions. By applying a default architecture of core2 on x86_64 we saw a 2x performance increase within GMP and MPFR. Because this change is being made BB-wide, all binary dependencies that are built using BB will start to use these instructions.

As stated in the AMD SSSE3 support issue, while we had verbally stated that "core2" was the required minimum chip support, we hadn't explicitly written it down in easy to find places, so we're trying to rectify that here. Julia can be built to run on older architectures, but our binaries are not going to support that, because there's too much performance being left on the table for the vast majority of our users who have processors that support SSSE3.

[yuyichao]

It shouldn't be needed for GMP, since they have dispatch. The issue linked does not seem to mention anything about GMP test result.

while we had verbally stated that "core2" was the required minimum chip support

The point is that this is clearly not stated anywhere and it clearly affects real users. In fact, it was explicitly stated in codegen, and then later processor*.cpp that the requirement was sse2, not anything higher. It is also clearly not the case since the old binary works and things were setup to support it.

---

It's important to make things work correct first before making it fast. If no user actually uses the binary on old hardware it'll be somewhat different but given how quickly the issue got reported it is certainly not the case.

---

And then after the correctness issue is fixed, i.e. lower the base requirement back to what it was, the performance issue can be figured out afterwards. It is not that hard to add target dispatch support for a library for a particular build. I've done it here for openlibm. A few caveated,

1. Compiler version/arch detection isi more anony, which is the main reason I haven't bother upstreaming the openlibm change yet.
2. You need a fairly new GCC. 9 should be fine.
3. mpfr does not have as easy a macro to patch as openlibm so more manual modification is needed.

I can help with it for sure, but at any rate, this is an optimization problem and it should never get before correctness.

[yuyichao]

And see https://github.com/yuyichao/arch-pkg/blob/be9b02aed257ba08c2160da9ae8dca4efc7ad06c/pkg/all/mpfr/target-h.patch and https://github.com/yuyichao/arch-pkg/blob/be9b02aed257ba08c2160da9ae8dca4efc7ad06c/pkg/all/mpfr/PKGBUILD#L22-L32 for the MPFR patch for testing. (be careful with quoting, see a few lines below, when passing in the target clone attribute).

[yuyichao]

BTW, with GCC 10, I see no difference at all for the benchmark in #31759 with generic target or core2 as target.

[yuyichao]

Based on local testing of the generic binaries and #35215 (comment) as well as bits from #31759 (comment) and #31759 (comment) it seems that.

1. GCC 7 + generic arch (up to 1.1)

   Works and fastest

2. GCC 4.8 + core2 (1.4 +)

   Slower and doesn't work for old systems that people are still using.

3. GCC 4.8 + generic arch (1.2 to 1.3)

   Slowest

So the best solution is to use GCC 7+. The comment from #35215 (comment)

As an example, GMP and MPFR, the libraries behind our BigInt capabilities, recently had a 2x performance difference when restricted to only older instructions

is also misleading/wrong since the change from gcc-7 -march=generic to gcc-4.8 -march=generic does not restrict anything to "only older instructions". It was purely a compiler optimization difference.

The reason for prefering (2) rather than the obviously better (1) appears to be

We do not yet have a nice, automatic way to determine that a binary dependency does not rely upon libstdc++ and auto-build it with GCC8, but we should.

as well as wanting to use BB rather than source compilation, which is something completely internal and therefore should have the lowest priority.

---

The correct action for mpfr/gmp/generic binary right now should be to revert to either gcc 4.8 + generic, if it is determined that using BB is more important, or to use 7+ + generic, if it is determined that performance is more important.

As for the documentation, this and this are the minimum requirement (edit: effectively added in 930e6b5), which should include everything from pentium4+, including all x64 chips from AMD and Intel. This requirement is for ABI/IEEE floating point reason. (edit: which is basically also the reason for the minimum requirement on ARM.)

---

Additionally, IIUC, the BB compiler change has caused all new BB binary to stop working on older systems. Therefore, once a fix of the BB compiler is implemented, all binaries built after the change in compiler version should be rebuilt to remove the bad binaries.

---

Edit: and if you want to stick to BB and 4.8, I recommend trying to specify -mtune. Looking at the assembly for mpfr_add_d, the function benchmarked in #31759, it does not seem to use much/any vector instructions so the new instruction really shouldn't be needed. I can imagine, however, that GCC 4.8 defaults to an older scheduling model that could have a bigger effect.