Is runtime.linux-x64.Microsoft.NETCore.Runtime.ObjWriter necessary for source-build? #69611
Comments
|
I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label. |
ghost
added
the
filipnavara
added
area-NativeAOT-coreclr
question
|
objwriter is not testing related. It needs to be included in source-build, assuming we want NativeAOT packages to be produced by source-build. |
|
I think that means we need to include https://github.com/dotnet/llvm-project in source-build as well, correct? |
|
Yes |
|
I think this is an important question:
There have been some other design problems with NativeAOT and the source build restrictions. In particular, NativeAOT uses pre-built static libraries from the runtime to compile packages. If we use the source build version, things break because they reference specific versions of openssl instead of dynamically loading what's available. See #66859. There seem to be some fundamental conflicts between source build and NativeAOT in terms of compilation model. |
Source build today conflates building from a big pile of sources with applying distro specific policies (#66859 is caused by distro specific policy for dynamic linking). It means that the source build produces product with different characteristics than regular build. It is by design. I do not think that these are fundamental conflicts. #66859 can be fixed by creating .props file with the additional libraries to reference when the distro specific policies are in affect. It is still work that would have to do be done. |
|
I agree with the comment that source-build - as it currently exists - mixes both "build everything from source" as well as "use the common Linux distribution practices" (including dynamic linking). There's no real reason that both have to be tied together. I could see someone using source-build to build on an "old" platform like CentOS 7 (and leave out the distribution policies) to get something close to Microsoft's portable builds of .NET SDK. The OpenJDK packages in Fedora are actually exploring something like this (build from source, but violate distro policies and bundle everything), and there's a long discussion thread about that here: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/IR5C3YVVABP6LVCXQL3UY5VZPSYWDAF5/. So it's not totally out of the world that source-build could be used this way too. I would much prefer that the source-build SDK have the same set of (possible) features as the Microsoft-built SDK, including NativeAOT. Taking a step back, how does NativeAOT deal with bundled vs system libraries (
Is there no way to use an upstream llvm directly? Rebuilding all of llvm, aside from taking a very long time every build, is also duplicating the efforts of each distro's existing llvm packages. |
We are using llvm with custom changes: https://github.com/dotnet/llvm-project/tree/objwriter/12.x. We are building a subset, not the whole thing. |
…ow to handle it. See dotnet#69611 for discussion.
|
It looks like this uses a newer version of CMake than is currently available on CentOS7, which is our least-common-denominator image: Is the newer version necessary? Is this a change we made? I'm going to try just a straight downgrade and hope there's no actual CMake 3.13 features used but it'd be good to know what our options are. |
|
The documented dotnet/runtime cmake version is 3.14.5 or higher: https://github.com/dotnet/runtime/blob/main/docs/workflow/requirements/linux-requirements.md#toolchain-setup . Is the source build really using 3.6.2? |
|
Sigh, yes - it's the RHEL 7 requirement. We'd bumped it to 3.14 and it blocked us. I thought we had added a sourcebuild validation leg? The downgrade was done in #39044 and then we had to polyfill a lot of the usages. It's also not great since we know some CMake versions broke our build at some point, so it makes sense that always relying on provided build tools is not always feasible (granted, the one I am thinking of was in Windows). cc @jkoritzinsky. |
|
This is the one I was thinking of: #65798 (comment) |
Does the source build still need to support RHEL 7?
This is coming from LLVM 12.0 minimum requirements that this component is based on: https://github.com/dotnet/llvm-project/blob/objwriter/12.x/llvm/docs/CMake.rst#quick-start . Lowering this requirement for LLVM is non-starter. |
cc @omajid to comment
We do, this issue surfaced while tracking down the ObjWriter prebuilt that was introduced in 7.0 |
|
I think to handle this correctly, we might need to build a version of objwriter against an older LLVM version that supports the right CMake version (likely the same version whose toolset config we use in our source-build image). Hopefully we don't need to support RHEL 7 any more and we can bump our minimum CMake to something more modern. |
|
Downgrading LLVM is non-trivial because we have customizations and use LLVM APIs in the objwriter component that are not stable. I recently did an LLVM 5 -> 12 upgrade because we were tired of hitting bugs that were fixed years ago. It's a pretty extensive change: dotnet/runtimelab#1461. If building the subset of LLVM 12 that we need with older CMake is not feasible, we'll have to take this out of source build and Native AOT won't work in source build. We need to use pretty modern versions of LLVM if we want to support modern platforms in .NET because I doubt anyone is backporting e.g. arm64 Mac support to some obsolete version of LLVM. |
We should try to make source-build include Native AOT on platforms that have a recent enough version of LLVM. If RHEL7 doesn't meet the bar, it won't have Native AOT. |
Maybe? Two things to consider:
|
|
If we decide to drop support for RHEL 7, what would be min CMake version we would need to support building with? |
|
I compiled a list of distros and cmake versions at #38755 a few years ago. Does that help? (Also, RHEL 7 via the llvm-toolset-12.0-cmake has a newer version of cmake now) |
If we could move to CMake 3.18 for source-build, that would allow us to significantly simplify our CMake build (and remove the polyfills we needed to add earlier in #39044 for source-build). Can we try updating the source-build docker image to point to the correct toolset (an equivalent of what you built .NET 6 for source-build on)?
Given that RHEL 7 can support building with CMake 3.18, I'd assume whatever newer system we build against would have the same if not higher minimum. Our current recommended minimum CMake version on Windows is 3.16.4, which is the highest version of any of our target platforms (IIRC due to some bug with the VS or MSVC integration), so if CMake 3.18 or newer is available, we can definitely set our minimum at 3.16.4 unless there are new features that can significantly simplify or speed up our build process. |
|
OpenSuse 15 which is in the lifecycle policy is at 3.10 - although not sure if that matters since they don't participate in SourceBuild. Ubuntu 18.04 is part of the support matrix also at 3.10 and I think it's one of those that we might care? Not sure if always needing the inbox version is tenable like this. |
|
From my understanding, Clang doesn't build with the in-box version. They provide their own toolset that we could optionally enable to get a newer CMake version that is also part of the distro. This mechanism is how we get the CMake 3.6 version in our source-build Docker image today. Maybe Canonical has something similar for Ubuntu that we could use? We need Canonical to tell us what version of CMake compatibility they need for their source-build scenario before we can bump the minimum. Also, are they going to do source-build for .NET 8+ on 18.04 since that goes out of support in April? 20.04 won't be an issue for us, so 18.04 is the only concern. |
|
cc @dotnet/distro-maintainers |
|
I did some digging and here are the cmake versions I can see that are available in some common distributions:
|
|
In terms of distributions that have cmake older than 3.18, we have:
Are any of these using (or planning to use) source-build to ship .NET at the moment? I don't think so. cc @dotnet/distro-maintainers to correct me if I have it wrong. For a first attempt, how about we just don't support anything with too old cmake? While it would be ideal to implement things the way @tmds suggested (NativeAOT conditioned on a recent-enough version of cmake), I understand that there are lots of unknowns (eg, what does the final UX look like?) that make it challenging to implement right now. |
|
@omajid, Did you mean to include Ubuntu 20.04 instead of 22.04 in the unsupported cmake like? |
|
Yes. Sorry, was a typo. |
|
I'm often the person pushing on OS support policies. One of my philosophies is making decisions in terms of an STS/LTS pair. We want to avoid offering support for an STS release and then pulled support for the next LTS release. Customers get wedged that way. Something similar happened with .NET Core 2.2, and that was bad. There's also the question of whether we've spending our time and effort wisely. We made the decision to not support Windows 7 for .NET 7/8. RHEL 7 feels similar. Clearly, Red Hat gets to make its own decisions on which RHEL versions to support, however, the overall project doesn't necessarily need to take on the extra burden for that. From what I'm reading, the official policy of the .NET 7 project should be RHEL 8+ in terms of taking on burden and leave RHEL 7 as a topic for RH exclusively. Is that fair? Separately, I believe that including Native AOT in source build is a good plan. If there is a reasonable way to split source-build so that Native AOT is included for RHEL 8 and not RHEL 7, I'm neutral on that. On portable vs distro-specific, I like having both options. I see distro-specific as the "optimized version". For folks like RH, that seems like the virtuous choice. For Microsoft building for "all the Linuxes everywhere", portable seems like the virtuous choice. Going forward, we should enable anyone to build portable as well. Only Microsoft can reasonably do that today, which is NOT a goal, just a point-in-time situation. |
|
Hi all, sorry for the delay, as I was collecting the info internally. Thank you @omajid for the versions compilation, and yes, for Ubuntu there are two LTS versions under 3.18 (Bionic 18.04 and Focal 20.04). The following are the current cmake versions included in the latest supported Ubuntu series: 3.10.2-1ubuntu2.18.04.2 | bionic-updates Which will be the oldest LTS that will include dotnet is still an ongoing internal discussion, taking into account the EOL ( not only the EOSS) for the Extended Security Maintenance period of each Ubuntu series. @jkoritzinsky About cmake being fetched via clang to get higher versions or any different toolchain where this may occur, I did not find any scenario on our side where this happens that way, as we provide cmake separate from clang and it is not supported at build time to retrieve anything apart from the debs packages in the corresponding Ubuntu series pocket being built (i.e. we cannot grab the 22.04 cmake if we are building on 18.04) nor outside of that pocket neither. Regarding the use of llvm and trying to summarize what is being discussed (iiuc), the actual situation is that we use llvm package (in combination with cmake, clang and ldc packages) for building dotnet from source. The oldest version of llvm is llvm-10 on Bionic 18.04. In our case, using vendorized software is always a handicap (and could become a blocker) for pursuing the inclusion of the dotnet package in main pocket, where vendorized code is allowed at building time but not at runtime (i.e. the use of |
|
Corresponding libc versions, thanks for the script @omajid:
So it looks like CentOS Stream 8 would probably be our new least-common-denominator version for the base tarball. Do the Stream versions get updated or should that be pretty stable @omajid? |
|
We seem to have switched from a "should source-build support building on old distros where the distro-default cmake is too old?" to "what distros should .NET drop support for?" Or am I mixing up source-build and (Microsoft-build of) .NET here? Or, to put it another way, are we thinking of moving the minimum requirements to build .NET from source or to run .NET? Moving the runtime baseline from "minimum of RHEL 7 or equivalent" to something newer would also affect other distributions, like Ubuntu 16.04 (which is currently supported for .NET 6) but wont be if we move the baseline higher.
That's a very good point I hadn't thought of.
The EOL dates are a bit different, though. Windows 7, if I understand correctly was EOL'ed in 2020 so it makes sense to drop that in .NET 7. RHEL 7's similar EOL date is in 2024. It seems a little too early to say ".NET 7 won't run on RHEL 7 at all".
That policy does make a lot of sense. I think there are some practical issues that make it very difficult to run .NET on platforms that are not supported by Microsoft, so I am hesitant to fully agree with this policy at this time. |
I asked our libc maintainers, and their stance is that glibc is backwards compatible, not forward compatible. If you build against a RHEL 8.x build of glibc (CentOS Stream 8 roughly corresponds to RHEL 8.7 right now), .NET will run against that version of glibc or newer. In other words, building against glibc 2.28 in CentOS Stream 8 today may produce binaries that don't work on RHEL 8.6 or older. However, this has been the glibc stance since forever. It was also true in the RHEL/CentOS 7 era. However, I don't think this ever became an issue in CentOS 7, so perhaps it will be the same way in CentOS Stream 8? |
|
This was resolved with dotnet/installer#14314. |
This is a new prebuilt in 7.0 that we're seeing being used by coreclr/ILCompiler. Is this a component that needs to be included in source-build? It seems to be related to Native AOT but there's also some notes where it's being used that make it seem testing-related.
The text was updated successfully, but these errors were encountered: