build: T3664: add support for building non-ISO flavors #550

dmbaturin · 2024-04-07T16:32:41Z

Change Summary

Add support for building flavors other than ISO images directly from vyos-build.

The diff is, unfortunately, unavoidably messy-looking due to a large change, sorry about that.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Code style update (formatting, renaming)
Refactoring (no functional changes)
Migration from an old Vyatta component to vyos-1x, please link to related PR inside obsoleted component
Other (please describe):

Related Task(s)

Component(s) name

Build scripts.

Proposed changes

The following new fields are available in flavor files:

# Image format: can be "raw" or anything supported by qemu-img
image_format = ["raw"]

# Post-build hook script for executing special procedures (such as OVA building signing).
# It gets the raw image path relative to build/ as its argument.
post_build_hook = "/path/to/script"

# Boot settings for GRUB configuration
[build_settings]
  timeout = 5
  console_type = 'tty',
  console_num = 0
  console_speed = 115200
  bootmode = 'normal'

The build script supports the following new options:

--disk-size — disk size in gigabytes (ignored for ISO)
--reuse-iso — path to an ISO image (relative to build/), if given, the script will use it instead of building it from scratch.

The script uses the following new environment variables:

VYOS_TEMPLATE_DIR — normally should point to the template dir of the vyos-1x submodule (the variable is passwd to vyos.template). This variable is currently required for the script to work, we may want to find a way to add a hardcoded default to it so simplify script calls.
VYOS_BUILD_FLAVORS_DIR — optional variable to specify a separate dir with flavors (e.g., the new vyos-community-flavors repository),

How to test

For example, put the following in ~/flavors/raw.toml:

# Generic (aka "universal") raw image
# Like the generic ISO, but raw ;)

image_format = "raw"

# Include these packages in the image regardless of the architecture
packages = [
  # QEMU and Xen guest tools exist for multiple architectures
  "qemu-guest-agent",
  "vyos-xe-guest-utilities",
]

[architectures.amd64]
  # Hyper-V and VMware guest tools are x86-only
  packages = ["hyperv-daemons", "vyos-1x-vmware"]

You can download an nightly image to speed up things.

Then run something like this:

sudo VYOS_BUILD_FLAVORS_DIR=/home/dmbaturin/flavors/  VYOS_TEMPLATE_DIR=/home/dmbaturin/vyos-build/vyos-1x/data/templates/ ./build-vyos-image --reuse-iso vyos-1.5-rolling-202404040019-amd64.iso raw

Checklist:

I have read the CONTRIBUTING document
I have linked this PR to one or more Phabricator Task(s)
My commit headlines contain a valid Task id
My change requires a change to the documentation
I have updated the documentation accordingly

to be able to use its functions in build scripts

scripts/image-build/build-vyos-image

sever-sever · 2024-04-13T09:26:20Z

Help should check if vyos-1x exists

vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$ sudo ./build-vyos-image --help
E: vyos-1x subdirectory does not exist, did you initialize submodules?
Traceback (most recent call last):
  File "/vyos/tmp/vyos-build/./build-vyos-image", line 56, in <module>
    import raw_image
  File "/vyos/tmp/vyos-build/scripts/image-build/raw_image.py", line 23, in <module>
    import vyos.utils.process
ModuleNotFoundError: No module named 'vyos'
vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$

Dependency

vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$ git clone https://github.com/vyos/vyos-1x

vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$ sudo ./build-vyos-image --help
I: Checking if packages required for VyOS image build are installed
E: E: There are missing system dependencies!
E: Missing packages: gdisk dosfstools
vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$

sever-sever · 2024-04-13T09:59:11Z

Unexpected flavor README

vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$ sudo ./build-vyos-image --build-by viacheslav@host.local --reuse-iso=./vyos-1.5-rolling-202404130016-amd64.iso 
I: Checking if packages required for VyOS image build are installed
I: using build flavors directory data/build-flavors
E: Build flavor is not specified!
E: For example, to build the generic ISO, run ./build-vyos-image iso
Available build flavors:

edgecore
dell-vep1400
README
xcpng
iso
dell-vep4600
aws-iso
azure-iso

sever-sever · 2024-04-13T10:11:38Z

It does nothing:

vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$ cat ../flavors/raw.toml 
# Generic (aka "universal") ISO image

image_format = "iso"

# Include these packages in the image regardless of the architecture
packages = [
  # QEMU and Xen guest tools exist for multiple architectures
  "qemu-guest-agent",
  "vyos-xe-guest-utilities",
]
vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$ 


vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$ sudo VYOS_BUILD_FLAVORS_DIR=../flavors/ ./build-vyos-image --build-by viacheslav@host.local --reuse-iso vyos-1.5-rolling-202404130016-amd64.iso raw
I: Checking if packages required for VyOS image build are installed
I: using build flavors directory ../flavors/
vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$

c-po · 2024-04-13T10:27:33Z

gdisk dosfstools should be added to the Docker container as build dependencies for vyos-build https://github.com/vyos/vyos-build/blob/current/docker/Dockerfile#L92-L93

Signed-off-by: Daniil Baturin <daniil@vyos.io>

c-po · 2024-04-13T16:09:36Z

It does nothing:

vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$ cat ../flavors/raw.toml 
# Generic (aka "universal") ISO image

image_format = "iso"

# Include these packages in the image regardless of the architecture
packages = [
  # QEMU and Xen guest tools exist for multiple architectures
  "qemu-guest-agent",
  "vyos-xe-guest-utilities",
]
vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$ 


vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$ sudo VYOS_BUILD_FLAVORS_DIR=../flavors/ ./build-vyos-image --build-by viacheslav@host.local --reuse-iso vyos-1.5-rolling-202404130016-amd64.iso raw
I: Checking if packages required for VyOS image build are installed
I: using build flavors directory ../flavors/
vyos_bld@83a9e41894e5:/vyos/tmp/vyos-build$

I can second this:

(16:08) vyos_bld 09c0e4e65e29:/vyos/vyos-build [current] # sudo VYOS_BUILD_FLAVORS_DIR=/vyos/vyos-build VYOS_TEMPLATE_DIR=/vyos/vyos-1x/data/templates/ ./build-vyos-image --reuse-iso vyos-999.202404131522-amd64.iso raw
I: Checking if packages required for VyOS image build are installed
I: using build flavors directory /vyos/vyos-build

to avoid mistakenly listing auixilliary files like README as flavors

dmbaturin · 2024-04-13T19:56:33Z

It does nothing

With image_format = "iso" and --reuse-iso, that's trying to build an ISO from an ISO. Since the ISO is already an ISO, the script doesn't have anything to do to produce an ISO.

and normalize it to a list if only one format is specified

Signed-off-by: Daniil Baturin <daniil@vyos.io>

to simplify the now-rather-tricky build script invocation

Signed-off-by: Daniil Baturin <daniil@vyos.io>

.gitmodules

to avoid permission problems

build: T3664: add vyos-1x as a submodule

7d2c84f

to be able to use its functions in build scripts

github-actions bot added the current label Apr 7, 2024

vyosbot requested review from a team and removed request for a team April 7, 2024 16:32

github-actions bot assigned dmbaturin Apr 7, 2024

vyosbot requested review from sarthurdev, zdc, jestabro, sever-sever and c-po April 7, 2024 16:32

build: T3664: add support for building non-ISO flavors

a896176

dmbaturin force-pushed the T3664-raw-flavors branch from 7d31ca4 to a896176 Compare April 10, 2024 16:36

sever-sever requested changes Apr 13, 2024

View reviewed changes

scripts/image-build/build-vyos-image Show resolved Hide resolved

scripts/image-build/build-vyos-image Outdated Show resolved Hide resolved

scripts/image-build/build-vyos-image Outdated Show resolved Hide resolved

Account for the working dir below the root in git invocation

7fa66c7

Signed-off-by: Daniil Baturin <daniil@vyos.io>

dmbaturin force-pushed the T3664-raw-flavors branch from e4e8c03 to 7fa66c7 Compare April 13, 2024 12:55

build: T3664: filter out non-TOML files from the build flavor dir

c072689

to avoid mistakenly listing auixilliary files like README as flavors

dmbaturin and others added 3 commits April 13, 2024 20:06

build: T3664: add a sanity check for image_format

72c7c7f

and normalize it to a list if only one format is specified

fix path to versions file

ed7bcbd

Signed-off-by: Daniil Baturin <daniil@vyos.io>

docker: T3664: add dependencies for raw disk building

c535510

sever-sever approved these changes Apr 15, 2024

View reviewed changes

dmbaturin and others added 2 commits April 15, 2024 17:17

build: T3664: add a parameterized make rule for build targets

2928865

to simplify the now-rather-tricky build script invocation

build: T3664: initialize git vyos-1x submodule before imports

867567e

Signed-off-by: Daniil Baturin <daniil@vyos.io>

c-po reviewed Apr 15, 2024

View reviewed changes

.gitmodules Show resolved Hide resolved

build: T3664: correct the parameterized make rule

008bb50

build: T3664: move sudo out of the make target

abb5b51

to avoid permission problems

c-po approved these changes Apr 20, 2024

View reviewed changes

c-po merged commit 671bbd0 into vyos:current Apr 20, 2024
3 of 4 checks passed

fvlaicu mentioned this pull request May 11, 2024

build: T6330: fix indentation for the version pinning file. #611

Closed

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

build: T3664: add support for building non-ISO flavors #550

build: T3664: add support for building non-ISO flavors #550

dmbaturin commented Apr 7, 2024 •

edited

Loading

sever-sever commented Apr 13, 2024 •

edited

Loading

sever-sever commented Apr 13, 2024 •

edited

Loading

sever-sever commented Apr 13, 2024

c-po commented Apr 13, 2024

c-po commented Apr 13, 2024

dmbaturin commented Apr 13, 2024

build: T3664: add support for building non-ISO flavors #550

build: T3664: add support for building non-ISO flavors #550

Conversation

dmbaturin commented Apr 7, 2024 • edited Loading

Change Summary

Types of changes

Related Task(s)

Component(s) name

Proposed changes

How to test

Checklist:

sever-sever commented Apr 13, 2024 • edited Loading

sever-sever commented Apr 13, 2024 • edited Loading

sever-sever commented Apr 13, 2024

c-po commented Apr 13, 2024

c-po commented Apr 13, 2024

dmbaturin commented Apr 13, 2024

dmbaturin commented Apr 7, 2024 •

edited

Loading

sever-sever commented Apr 13, 2024 •

edited

Loading

sever-sever commented Apr 13, 2024 •

edited

Loading