Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inline apt source generation in rootfs script #14849

Merged
merged 8 commits into from
Jun 21, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions eng/common/cross/arm/sources.list.bionic

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/arm/sources.list.focal

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/arm/sources.list.jammy

This file was deleted.

3 changes: 0 additions & 3 deletions eng/common/cross/arm/sources.list.jessie

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/arm/sources.list.xenial

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/arm/sources.list.zesty

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/arm64/sources.list.bionic

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/arm64/sources.list.buster

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/arm64/sources.list.focal

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/arm64/sources.list.jammy

This file was deleted.

12 changes: 0 additions & 12 deletions eng/common/cross/arm64/sources.list.stretch

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/arm64/sources.list.xenial

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/arm64/sources.list.zesty

This file was deleted.

3 changes: 0 additions & 3 deletions eng/common/cross/armel/sources.list.jessie

This file was deleted.

2 changes: 0 additions & 2 deletions eng/common/cross/armv6/sources.list.bookworm

This file was deleted.

2 changes: 0 additions & 2 deletions eng/common/cross/armv6/sources.list.buster

This file was deleted.

41 changes: 30 additions & 11 deletions eng/common/cross/build-rootfs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,8 @@ __IllumosArch=arm7
__HaikuArch=arm
__QEMUArch=arm
__UbuntuArch=armhf
__UbuntuRepo="http://ports.ubuntu.com/"
__UbuntuRepo=
__UbuntuSuites="updates security backports"
__LLDB_Package="liblldb-3.9-dev"
__SkipUnmount=0

Expand Down Expand Up @@ -129,9 +130,9 @@ __AlpineKeys='
616db30d:MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAnpUpyWDWjlUk3smlWeA0\nlIMW+oJ38t92CRLHH3IqRhyECBRW0d0aRGtq7TY8PmxjjvBZrxTNDpJT6KUk4LRm\na6A6IuAI7QnNK8SJqM0DLzlpygd7GJf8ZL9SoHSH+gFsYF67Cpooz/YDqWrlN7Vw\ntO00s0B+eXy+PCXYU7VSfuWFGK8TGEv6HfGMALLjhqMManyvfp8hz3ubN1rK3c8C\nUS/ilRh1qckdbtPvoDPhSbTDmfU1g/EfRSIEXBrIMLg9ka/XB9PvWRrekrppnQzP\nhP9YE3x/wbFc5QqQWiRCYyQl/rgIMOXvIxhkfe8H5n1Et4VAorkpEAXdsfN8KSVv\nLSMazVlLp9GYq5SUpqYX3KnxdWBgN7BJoZ4sltsTpHQ/34SXWfu3UmyUveWj7wp0\nx9hwsPirVI00EEea9AbP7NM2rAyu6ukcm4m6ATd2DZJIViq2es6m60AE6SMCmrQF\nwmk4H/kdQgeAELVfGOm2VyJ3z69fQuywz7xu27S6zTKi05Qlnohxol4wVb6OB7qG\nLPRtK9ObgzRo/OPumyXqlzAi/Yvyd1ZQk8labZps3e16bQp8+pVPiumWioMFJDWV\nGZjCmyMSU8V6MB6njbgLHoyg2LCukCAeSjbPGGGYhnKLm1AKSoJh3IpZuqcKCk5C\n8CM1S15HxV78s9dFntEqIokCAwEAAQ==
'
__Keyring=
__KeyringFile="/usr/share/keyrings/ubuntu-archive-keyring.gpg"
__SkipSigCheck=0
__UseMirror=0
__UseDeb822Format=0

__UnprocessedBuildArgs=
while :; do
Expand Down Expand Up @@ -163,17 +164,20 @@ while :; do
__UbuntuArch=armel
__UbuntuRepo="http://ftp.debian.org/debian/"
__CodeName=jessie
__KeyringFile="/usr/share/keyrings/debian-archive-keyring.gpg"
;;
armv6)
__BuildArch=armv6
__UbuntuArch=armhf
__QEMUArch=arm
__UbuntuRepo="http://raspbian.raspberrypi.org/raspbian/"
__CodeName=buster
__KeyringFile="/usr/share/keyrings/raspbian-archive-keyring.gpg"
__LLDB_Package="liblldb-6.0-dev"
__UbuntuSuites=

if [[ -e "/usr/share/keyrings/raspbian-archive-keyring.gpg" ]]; then
__Keyring="--keyring /usr/share/keyrings/raspbian-archive-keyring.gpg"
if [[ -e "$__KeyringFile" ]]; then
__Keyring="--keyring $__KeyringFile"
fi
;;
riscv64)
Expand All @@ -184,10 +188,6 @@ while :; do
__UbuntuArch=riscv64
__UbuntuPackages="${__UbuntuPackages// libunwind8-dev/}"
unset __LLDB_Package

if [[ -e "/usr/share/keyrings/debian-archive-keyring.gpg" ]]; then
__Keyring="--keyring /usr/share/keyrings/debian-archive-keyring.gpg --include=debian-archive-keyring"
fi
;;
ppc64le)
__BuildArch=ppc64le
Expand Down Expand Up @@ -292,10 +292,13 @@ while :; do
if [[ "$__CodeName" != "jessie" ]]; then
__CodeName=noble
fi
__UseDeb822Format=1
if [[ -n "$__LLDB_Package" ]]; then
__LLDB_Package="liblldb-18-dev"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand why and when we set __LLDB_Package for the various distros, would you mind explaining?

Copy link
Member Author

@am11 am11 Jun 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When we invoke build-rootfs.sh <arch> <codename> lldbX, it uses liblldb-X-dev. When we don't provide lldbX, it uses the default 3.9. llvm/lldb v3.9 is not available in 24.04 (noble), so I'm changing its default. We have used noble branch only for riscv64 until now, where lldb is not yet available in debian/ubuntu packages.

The current approach of setting defaults, parsing args, overriding defaults based on arguments, then figuring out the "order" of args (what makes sense for overriding and when) is very twisted. I think we should refactor this script and define presets: combination of mandatory arguments. And remove all default/overriding handing. It will be a bit verbose, but it will make things readable for someone trying to work with it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok thanks, I agree we should refactor the defaults.

fi
;;
jessie) # Debian 8
__CodeName=jessie
__KeyringFile="/usr/share/keyrings/debian-archive-keyring.gpg"

if [[ -z "$__UbuntuRepo" ]]; then
__UbuntuRepo="http://ftp.debian.org/debian/"
Expand All @@ -304,6 +307,7 @@ while :; do
stretch) # Debian 9
__CodeName=stretch
__LLDB_Package="liblldb-6.0-dev"
__KeyringFile="/usr/share/keyrings/debian-archive-keyring.gpg"

if [[ -z "$__UbuntuRepo" ]]; then
__UbuntuRepo="http://ftp.debian.org/debian/"
Expand All @@ -312,27 +316,31 @@ while :; do
buster) # Debian 10
__CodeName=buster
__LLDB_Package="liblldb-6.0-dev"
__KeyringFile="/usr/share/keyrings/debian-archive-keyring.gpg"

if [[ -z "$__UbuntuRepo" ]]; then
__UbuntuRepo="http://ftp.debian.org/debian/"
fi
;;
bullseye) # Debian 11
__CodeName=bullseye
__KeyringFile="/usr/share/keyrings/debian-archive-keyring.gpg"

if [[ -z "$__UbuntuRepo" ]]; then
__UbuntuRepo="http://ftp.debian.org/debian/"
fi
;;
bookworm) # Debian 12
__CodeName=bookworm
__KeyringFile="/usr/share/keyrings/debian-archive-keyring.gpg"

if [[ -z "$__UbuntuRepo" ]]; then
__UbuntuRepo="http://ftp.debian.org/debian/"
fi
;;
sid) # Debian sid
__CodeName=sid
__KeyringFile="/usr/share/keyrings/debian-archive-keyring.gpg"

if [[ -z "$__UbuntuRepo" ]]; then
__UbuntuRepo="http://ftp.debian.org/debian/"
Expand Down Expand Up @@ -442,6 +450,10 @@ fi

__UbuntuPackages+=" ${__LLDB_Package:-}"

if [[ -z "$__UbuntuRepo" ]]; then
__UbuntuRepo="http://ports.ubuntu.com/"
fi

if [[ -n "$__LLVM_MajorVersion" ]]; then
__UbuntuPackages+=" libclang-common-${__LLVM_MajorVersion}${__LLVM_MinorVersion:+.$__LLVM_MinorVersion}-dev"
fi
Expand Down Expand Up @@ -740,9 +752,16 @@ elif [[ -n "$__CodeName" ]]; then
# shellcheck disable=SC2086
echo running debootstrap "--variant=minbase" $__Keyring --arch "$__UbuntuArch" "$__CodeName" "$__RootfsDir" "$__UbuntuRepo"
debootstrap "--variant=minbase" $__Keyring --arch "$__UbuntuArch" "$__CodeName" "$__RootfsDir" "$__UbuntuRepo"

mkdir -p "$__RootfsDir/etc/apt/sources.list.d/"
grep -q "Types:" "$__CrossDir/$__BuildArch/sources.list.$__CodeName" && filename="$__CodeName.sources" || filename="$__CodeName.list"
cp "$__CrossDir/$__BuildArch/sources.list.$__CodeName" "$__RootfsDir/etc/apt/sources.list.d/$filename"
cat > "$__RootfsDir/etc/apt/sources.list.d/$__CodeName.sources" <<EOF
Types: deb
URIs: $__UbuntuRepo
Suites: $__CodeName $(echo $__UbuntuSuites | xargs -n 1 | xargs -I {} echo -n "$__CodeName-{} ")
Components: main universe
Signed-By: $__KeyringFile
EOF

chroot "$__RootfsDir" apt-get update
chroot "$__RootfsDir" apt-get -f -y install
# shellcheck disable=SC2086
Expand Down
11 changes: 0 additions & 11 deletions eng/common/cross/ppc64le/sources.list.bionic

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/riscv64/sources.list.noble

This file was deleted.

1 change: 0 additions & 1 deletion eng/common/cross/riscv64/sources.list.sid

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/s390x/sources.list.bionic

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/x64/sources.list.bionic

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/x64/sources.list.xenial

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/x86/sources.list.bionic

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/x86/sources.list.focal

This file was deleted.

11 changes: 0 additions & 11 deletions eng/common/cross/x86/sources.list.jammy

This file was deleted.

Loading
Loading