[ci] Use Ubuntu 22.04 as ubuntu-latest at CI (fixes #5186)

[StrikerRUS]

Fixes #5186.

Depends on actions/runner-images#5490 when ubuntu-22.04 will become the default Ubuntu on GitHub Actions.

[StrikerRUS]

/gha run r-configure

[jameslamb]

Updated this to latest master of LightGBM tonight. It still failed on R jobs looking for a specific autoconf version, because ubuntu-latest on GitHub Actions is still Ubuntu 20.04.

But only for a few more days!

From actions/runner-images#6399

Ubuntu 22.04 is ready to be the default version for the "ubuntu-latest" label in GitHub Actions and Azure DevOps.

This change will be rolled out over a period of several weeks beginning in October, 3. We plan to complete the migration by December, 1.

[jameslamb]

The date for "Ubuntu 22.04 is the default ubuntu-latest on GitHub Actions" just got pushed back from December 1, 2022 to March 1, 2023: actions/runner-images#6399 (comment).

[jameslamb]

Based on some recent comments on that upstream tracking issue, it seems like GitHub's plan is "from now until March 2023, slowly increase the number of runners for which ubuntu-latest gives you Ubuntu 22.04".

For example:

• Ubuntu-latest workflows will use Ubuntu-22.04 actions/runner-images#6399 (comment)
• Ubuntu-latest workflows will use Ubuntu-22.04 actions/runner-images#6399 (comment)

I see evidence of this on recent builds in #5619 too...some ubuntu-latest GitHub Actions jobs are getting Ubuntu 22.04.

(build link)

This is a REALLY long time to not know which environment a job is going to get 😕 .

So for now, to minimize disruption to LightGBM's CI, I've pushed commits here replacing all uses of ubuntu-latest on Azure DevOps and GitHub Actions to ubuntu-22.04.

[jmoralez]

Is the IN_UBUNTU_LATEST_CONTAINER variable only set in Azure?

[jameslamb]

Is the IN_UBUNTU_LATEST_CONTAINER variable only set in Azure?

Yep! That's a LightGBM-specific thing, and only used on Azure. Maybe we should rethink that.

Specifically, maybe we should try to switch the R jobs to running in an ubuntu:22.04 container instead of just bare on the GitHub Actions ubuntu-latest VM.

[jmoralez]

I think the issue is with clang 14, I'm able to reproduce this locally installing that version from conda:

mamba install -y "clang==14.0"
conda deactivate && conda activate $ENV # to refresh the path
export CXX=$(which clang)
export CC=$CXX
mkdir build && cd build # from repo root
cmake ..

I get:

CMake Error at /home/jose/mambaforge/envs/lightgbm/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES)
Call Stack (most recent call first):
  /home/jose/mambaforge/envs/lightgbm/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /home/jose/mambaforge/envs/lightgbm/share/cmake-3.24/Modules/FindOpenMP.cmake:545 (find_package_handle_standard_args)
  CMakeLists.txt:163 (find_package)

I'll keep looking, just wanted to share in case this helps you.

[jmoralez]

I think we need apt-get install libomp-14-dev for clang14 llvm page.

[jameslamb]

@jmoralez I don't think that's correct.

The most recent runs I pushed just installing libomp-dev (link) failed like this:

E: Could not open lock file /var/lib/dpkg/lock-frontend - open (13: Permission denied)
E: Unable to acquire the dpkg frontend lock (/var/lib/dpkg/lock-frontend), are you root?

Which means we haven't yet actually tested that installing libomp-dev (no version number) works.

I appreciate the help, but could you please experiment on a different branch instead of pushing commits here? I'm in the middle of testing here on this one.

[jmoralez]

Sure, sorry. Can you please just let those two runs finish to see if they succeed?

[jameslamb]

Can you please just let those two runs finish to see if they succeed?

Sure, I won't cancel them.

[jmoralez]

Done, I've canceled the rest. Sorry again, thought you were done for the day.

[jameslamb]

No prob no prob! We're both here visiting a @StrikerRUS PR anyway 😂

[jameslamb]

Looks like just installing libomp-dev on the clang R builds worked!

Builds: https://github.com/microsoft/LightGBM/actions/runs/3701050842/jobs/6270018989

I think that's preferable to using something with a version number like libomp-14-dev, because it improves the changes of that line not needing to be changed to run this script on other Linux distributions (e.g., on a future Ubuntu 24.04 or something).

---

I'm going to open this up for review, since I know it's blocking: #5612 (comment)

I'm also going to open another draft PR to work on trying to move the R package Linux jobs into containers. Will explain in that other PR why I think that's preferable.

[jameslamb]

Ah wait sorry, the Linux_latest jobs on Azure are all failing: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=14008&view=logs&j=275189f9-c769-596a-7ef9-49fb48a9ab70&t=1fb0a3dd-ad11-527b-ef3d-485324b146c4

I'll move this back to draft and keep experimenting.

[jmoralez]

Maybe we could set an env variable in the R package workflow to hold the value of ubuntu latest and just use that everywhere (example), that way the next time we'd just change that. Approving because that's just nice to have and can be done later.

[jmoralez]

Oh, took too long to review and missed the comment.

[jameslamb]

Ha yeah sorry!

The most recent commit I pushed got us a bit closer. Now just the Linux_latest gpu jobs are failing.

I see this error further up in the setup.sh logs for them:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/softwareproperties/shortcuthandler.py", line 423, in add_key
    subprocess.run(cmd.split(), check=True, input=keys)
  File "/usr/lib/python3.10/subprocess.py", line 524, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['gpg', '-q', '--no-options', '--no-default-keyring', '--batch', '--keyring', '/etc/apt/trusted.gpg.d/mhier-ubuntu-libboost-latest.gpg', '--homedir', '/tmp/tmprqsrju7p', '--import']' returned non-zero exit status 2.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/add-apt-repository", line 364, in <module>
    sys.exit(0 if addaptrepo.main() else 1)
  File "/usr/bin/add-apt-repository", line 357, in main
    shortcut.add()
  File "/usr/lib/python3/dist-packages/softwareproperties/shortcuthandler.py", line 222, in add
    self.add_key()
  File "/usr/lib/python3/dist-packages/softwareproperties/shortcuthandler.py", line 425, in add_key
    raise ShortcutException(e)
softwareproperties.shortcuthandler.ShortcutException: Command '['gpg', '-q', '--no-options', '--no-default-keyring', '--batch', '--keyring', '/etc/apt/trusted.gpg.d/mhier-ubuntu-libboost-latest.gpg', '--homedir', '/tmp/tmprqsrju7p', '--import']' returned non-zero exit status 2.

Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease
Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease
Hit:3 http://security.ubuntu.com/ubuntu jammy-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease
Ign:5 https://ppa.launchpadcontent.net/mhier/libboost-latest/ubuntu jammy InRelease
Err:6 https://ppa.launchpadcontent.net/mhier/libboost-latest/ubuntu jammy Release
  404  Not Found [IP: 185.125.190.52 443]
Reading package lists...
E: The repository 'https://ppa.launchpadcontent.net/mhier/libboost-latest/ubuntu jammy Release' does not have a Release file.

Which eventually results in

-- OpenCL include directory: /usr/include
CMake Error at /usr/lib/x86_64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake:141 (find_package):
  Could not find a package configuration file provided by "boost_filesystem"
  (requested version 1.74.0) with any of the following names:

    boost_filesystemConfig.cmake
    boost_filesystem-config.cmake

  Add the installation prefix of "boost_filesystem" to CMAKE_PREFIX_PATH or
  set "boost_filesystem_DIR" to a directory containing one of the above
  files.  If "boost_filesystem" provides a separate development package or
  SDK, be sure it has been installed.

Build link: https://dev.azure.com/lightgbm-ci/lightgbm-ci/_build/results?buildId=14009&view=logs&j=582743ec-83fe-58c4-937e-4197b5816801&t=63b016c8-8965-547a-fab6-395af921c96a

Will push a fix shortly.

Each time I touch one of these setup.sh lines that failed, I'm adding a || exit -1 on the end of it, so that future builds will just fail immediately when those commands fail. Having to find these things much further up in the logs after the fact is a bit difficult.

[jameslamb]

Alright I'm gonna stop for tonight. Hopefully the CUDA, QEMU, and Appveyor jobs will end up succeeding.

@jmoralez @shiyu1994 if you get back to this PR before me and see everything passing, merge it. Otherwise, I'll do that tomorrow. Thanks for the help!

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.