vagga fails to build container with subconfigs: can't read /usr/local/bin ?

[mkpankov]

Hi,
I need help with an issue I'm encountering consistently on my project.
I couldn't minimize it yet - maybe you can tell what's wrong already.
This issue occurs when directly building final container pavetta-sdk-custom, but if we first build pavetta-redist, it goes away.
There's also a warning I don't know how to handle:

 WARN 2019-08-19T12:26:09Z: vagga::builder::commands::subcontainer: Can't write image usage info: No such file or directory (os error 2)

➜  seidhe git:(master) ✗ vagga pavetta run Iss/libraries/sysid
<ubuntu setup logs>
Setting up g++ (4:7.4.0-1ubuntu2.3) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
update-alternatives: warning: skip creation of /usr/share/man/man1/c++.1.gz because associated file /usr/share/man/man1/g++.1.gz (of link group c++) doesn't exist
Setting up build-essential (12.4ubuntu1) ...
Processing triggers for libc-bin (2.27-3ubuntu1) ...
 WARN 2019-08-19T12:26:09Z: vagga::builder::commands::subcontainer: Can't write image usage info: No such file or directory (os error 2)
ERROR 2019-08-19T12:26:09Z: vagga::builder: Error building container "pavetta-sdk-custom": step SubConfig { source: Directory, path: "vagga.yaml", container: "pavetta-sdk-common", cache: None } failed: sub-step Build { container: "pavetta-redist", source: "/usr/local/bin", path: Some("/usr/local/bin"), temporary_mount: None, content_hash: false, rules: ["/pavetta"] } failed: can't read "/vagga/base/.roots/pavetta-redist.a9d21258/root/usr/local/bin": No such file or directory (os error 2)
ERROR 2019-08-19T12:26:09Z: vagga::wrapper: Error executing _build: Builder exited with code 1
Command <Command "/proc/self/exe" "__wrapper__" "_build" "pavetta-sdk-custom"; environ[7]; uid_map=[UidMap { inside_uid: 0, outside_uid: 1000, count: 1 }, UidMap { inside_uid: 1, outside_uid: 100000, count: 65535 }]; gid_map=[GidMap { inside_gid: 0, outside_gid: 1000, count: 1 }, GidMap { inside_gid: 1, outside_gid: 100000, count: 65535 }]> exited with code 124

Relevant configs are:

containers:
  pavetta-sdk-common:
    setup:
      - !Ubuntu bionic
      - !UbuntuUniverse
      - !Install [build-essential]
      - !Build
        container: pavetta-redist
        source: /usr/local/bin
        path: /usr/local/bin
        rules:
        - /pavetta
      - !Build
        container: ciri-redist
        source: /usr/local/bin
        path: /usr/local/bin
        rules:
        - /ciri
      - !Build
        container: ciri-lua
        source: /opt/lua
        path: /opt/lua
      - !Install [liblua5.2-0]
      - !Text
        /usr/local/bin/Config.toml: |
          test_suites = [
              "/work/ciri/test_suites/",
          ]

          [ciri]

          path = "/usr/local/bin/ciri"
          config_path = "/usr/local/bin/ciri-Config.toml"
      - !Text
        /usr/local/bin/ciri-Config.toml: |
          lock_file_path = "/tmp/lock"
          lua_prefix_path = "/opt/lua/"
      - !SubConfig
        container: sdk-deps

  pavetta-sdk-custom:
    environ:
      HOME: /work/.vagga-ws
    setup:
      - !SubConfig
        container: pavetta-sdk-common
      - !SubConfig
        container: sdk-deps
      - !Build
        container: sdk-custom
        source: /opt
        path: /opt

  sdk-deps:
    setup:
      - !Install
        - dosfstools
        - doxygen
        - jq
        - libasound2
        - libcurl3-gnutls
        - libglib2.0-0
        - libpixman-1-0
        - libpulse0
        - libsdl2-2.0-0
        - libx11-6
        - libxml2
        - make
        - mtd-utils
        - parted
        - python
        - python3
        - virtualenv

  pavetta-redist:
    environ:
      HOME: /tmp
    setup:
      - !Container rust-build
      - !Depends 
        path: pavetta
        rules:
        - /src
        - /Cargo.toml
      - !Sh |
         cd pavetta
         cargo build --target-dir /work/.vagga-ws/target/pavetta
      - !Sh "cp /work/.vagga-ws/target/pavetta/debug/pavetta /usr/local/bin"

  rust:
    environ:
      HOME: /work/.vagga-ws
    setup:
      - !Ubuntu bionic
      - !UbuntuUniverse
      - !TarInstall
        url: https://static.rust-lang.org/dist/rust-1.35.0-x86_64-unknown-linux-gnu.tar.gz
        script: ./install.sh --prefix=/usr
      - !TarInstall
        url: https://static.rust-lang.org/dist/rust-1.35.0-x86_64-unknown-linux-musl.tar.gz
        script: ./install.sh --prefix=/usr --components=rust-std-x86_64-unknown-linux-musl

  rust-build:
    environ:
      HOME: /work/.vagga-ws
    setup:
      - !Ubuntu bionic
      - !UbuntuUniverse
      - !Build
        container: rust
        source: /usr
        path: /usr
      - !Install [build-essential, curl, ca-certificates]
      - !CacheDirs
        /tmp/.cargo: cargo

  ciri-redist:
    environ:
      HOME: /tmp
    setup:
      - !Container ciri-build
      - !CacheDirs
        /tmp/.cargo: cargo
      - !Depends 
        path: ciri
        rules:
        - /src
        - /Cargo.toml
      - !Sh |
         cd ciri
         cargo build --target-dir /work/.vagga-ws/target/ciri
      - !Sh "cp /work/.vagga-ws/target/ciri/debug/ciri /usr/local/bin/ciri"

  ciri-build:
    environ:
      HOME: /work/.vagga-ws
    setup:
      - !Container rust-build
      - !Install [liblua5.2-dev, liblua5.2-0, pkg-config]

  ciri-lua:
    setup:
      - !Ubuntu bionic
      - !UbuntuUniverse
      - !Install
        # Lua and Lua rocks build dependencies
        - build-essential
        - ca-certificates
        - curl
        - libreadline-dev
        - unzip
        # Lua native rocks extensions
        - liblua5.2-dev
        # Luarocks installation
        - python-pip
        # Luarocks GitHub installation support
        - git
        # luasec native dependency
        - libssl-dev
      - !Sh "pip install hererocks"
      - !EnsureDir /opt
      - !Cmd
        - /bin/bash
        - -exc
        - |
          hererocks -l5.2 -rlatest /opt/lua
      - !CacheDirs
        /tmp/.cache/luarocks: luarocks-cache
      - !Depends
        path: ciri
        rules:
        - /rocks.sh
      - !Cmd
        - /bin/bash
        - -exc
        - |
          . /opt/lua/bin/activate
          /work/ciri/rocks.sh

  sdk-custom:
    setup:
      - !Container base
      - !Sh |
         echo "In case next step fails, look at .vagga/sdk-parser/sdk.yaml"
      - !SubConfig
        source: !Container sdk-spec-parser
        path: sdk.yaml
        container: sdk

  sdk-spec-parser-base:
    setup:
    - !Container empty
    - !Depends prepare-sdk-yaml.py
    - !PipConfig 
      dependencies: true
    - !Py3Install [PyYAML, pystache]

  sdk-spec-parser:
    setup:
    - !Container sdk-spec-parser-base
    - !Depends .vagga-ws/sdk-spec.yaml
    - !Sh |
       /work/prepare-sdk-yaml.py /work/.vagga-ws/sdk-spec.yaml /sdk.yaml

  empty:
    setup:
      - !Alpine v3.10

prepare-sdk-yaml.py:

#!/usr/bin/env python3

import sys
import yaml
import pystache

template_head = """
containers:
  sdk:
    setup:
"""

template_steps_base = """
      - !RunAs
        script: |
          if [ -f "/work/.vagga-ws/{{id}}/{{base_name}}-{{id}}_{{version}}_amd64.deb" ]; then
            mkdir /tmp/{{id}}
            cp /work/.vagga-ws/{{id}}/{{base_name}}-{{id}}_{{version}}_amd64.deb /tmp/{{id}}
          else
            jfrog rt download sdk/{{id}}/*_{{version}}_* | tee /work/.vagga-ws/jfrog-result | jq -e ".totals.success == 1 and .totals.failure == 0" || cat /work/.vagga-ws/jfrog-result
          fi
        work-dir: /tmp
      - !Sh |
          echo debconf {{base_name_lower}}-{{id_lower}}/eula select true | debconf-set-selections
          echo debconf {{base_name_lower}}-{{id_lower}}/eula seen true | debconf-set-selections
          dpkg -i /tmp/{{id}}/{{base_name}}-{{id}}_{{version}}_amd64.deb
"""

def main():
    spec_file_path = sys.argv[1]
    out_file_path = sys.argv[2]

    with open(spec_file_path, 'r') as spec_file:
        try:
            with open(out_file_path, 'w') as out_file:
                spec = yaml.safe_load(spec_file)

                result = pystache.render(template_head, spec)
                out_file.write(result)

                for s in spec.values():
                    s['id_lower'] = s['id'].lower()
                    if not 'base_name' in s:
                        s['base_name'] = "KasperskyOS"

                    s['base_name_lower'] = s['base_name'].lower()
                    
                    template_steps = pystache.render(template_steps_base, s)
                    result = pystache.render(template_steps, s)
                    out_file.write(result)

        except yaml.YAMLError as exc:
            print(exc)


if __name__ == "__main__":
    main()

sdk-spec.template.yaml:

# Copy this file to .vagga-ws/sdk-spec.yaml and change as you like
# Then use container 'sdk-custom'. It will have specified SDK installed

komset:
  id: Komset-001
  version: 0.11
myoffice:
  id: MyOffice-001
  version: 0.39
  # Initial part of deb package name
  # By default, it's KasperskyOS
  base_name: KasperskyOS-SDK

[anti-social]

Your config is not complete:

Container "ciri-redist" referenced from "pavetta-sdk-common" is not found

Also there is no ciri-lua and sdk-custom containers in the config.

[mkpankov]

@anti-social will post more complete config on work week, though it still references proprietary code...

[mkpankov]

I've updated the first comment and added those you mentioned, and two files used to generate a sub-config.

As for ciri, pavetta and SDK deb packages, I currently can't share them. In case it reproduces with dummy rust projects / dummy deb, it's gonna be great. Meanwhile I'll try to minimize the example.

[tailhook]

This error means that container version changes during the build of that container. At a glance, I don't see any obvious conflicts here. But maybe it's because of things you've stripped. The most common case is when you !Depend on some files in /work but modify them during the build.

In that case, you should put the result of the build into the container itself (i.e. in /usr/share of say container A), and container (B) that depends on it should depend on A rather than on the file in /work. (At a glance, it seems you do that right in most cases, but probably there is tiny one place that fails at it)

The !Depend command is either for things that are checked out from VCS, or for things that other external commands do, not as a part of build process of any container.

[mkpankov]

First two paragraphs make sense.

The last one perplexes me however. Is that correct that I !Depend on Cargo.toml and src in container, that builds a Rust project and puts the binary to /usr/local/bin in the container?..

There Rust sources are in the VCS, via a submodule repo.

[tailhook]

Is that correct that I !Depend on Cargo.toml and src in container, that builds a Rust project and puts the binary to /usr/local/bin in the container?..

Yes, it looks like perfectly correct.

By the way, there is a way to debug dependencies:

vagga _version_hash --debug-versioning pavetta-redist

Try it before and after bulding pavetta-redist container and it should show you differences. If output is the same, then referred files differ (you can use --dump-version-date to compare data itself, but most of the time, looking at file entries in --debug-versioning should be enough to find out which unexpected files are referred).

[mkpankov]

Okay so I tried that.
I've built pavetta-sdk-custom, which builds pavetta-sdk-common, which builds pavetta-redist. Dumped the version data (both --debug-versioning and --dump-version-data).
Then I've added a println!(""); to one function in src/main.rs in pavetta.
Dumped the version data - as expected, it's the same, except for that line and final hash.
Then I've build pavetta-redist and pavetta-sdk-custom.
Dumped the version data - and again, as expected, it's the same, except for that line and final hash.

Any more pointers on where to look?

Does vagga interact with git or submodules in particular?

Can you clarify the following from !Build docs?

Despite the name Build dependencies are not rebuilt.
The Build command itself depends only on the container but on on the individual files. You need to ensure that the source container is versioned well (sometimes you need Copy or Depends for the task)

Also, in docs for content-hash option:

The container in (1) is not binary reproducible (i.e. every build could possibly produce different checksums)

Might that be the case? Like, maybe cargo puts compilation date into binary or something?..

Tried building and then rebuilding the pavetta-redist with --force - hash of rust binary is the same.

[mkpankov]

I've been able to produce a minimal example using a simple "Hello, world" rust program.

https://github.com/mkpankov/vr

To reproduce the problem, you have to build once with vagga pavetta, then edit pavetta/src/main.rs - change one println to the other, then run vagga pavetta again.

Even though the pavetta-redist is properly rebuilt in the beginning, in the end there's

 WARN 2019-09-10T15:09:20Z: vagga::builder::commands::subcontainer: Can't write image usage info: No such file or directory (os error 2)
ERROR 2019-09-10T15:09:20Z: vagga::builder: Error building container "pavetta-sdk-common": step Build { container: "pavetta-redist", source: "/usr/local/bin", path: Some("/usr/local/bin"), temporary_mount: None, content_hash: false, rules: ["/pavetta"] } failed: can't read "/vagga/base/.roots/pavetta-redist.1c729ece/root/usr/local/bin": No such file or directory (os error 2)
ERROR 2019-09-10T15:09:20Z: vagga::wrapper: Error executing _build: Builder exited with code 1
Command <Command "/proc/self/exe" "__wrapper__" "_build" "pavetta-sdk-custom"; environ[7]; uid_map=[UidMap { inside_uid: 0, outside_uid: 1000, count: 1 }, UidMap { inside_uid: 1, outside_uid: 100000, count: 65535 }]; gid_map=[GidMap { inside_gid: 0, outside_gid: 1000, count: 1 }, GidMap { inside_gid: 1, outside_gid: 100000, count: 65535 }]> exited with code 124

I suspect the issue is that there's untracked state in /work/.vagga-ws/pavetta/target - it's cargo output directory, and I think it can touch it intermittently (didn't check that yet).

[tailhook]

Well, these are offending lines:

      - !Sh |
         cd pavetta
         cargo build --target-dir /work/.vagga-ws/target/pavetta
      - !Copy
        source: /work/.vagga-ws/target/pavetta/debug/pavetta
        path: /usr/local/bin/pavetta

Because source in !Copy are in /work, they are hashed before building container to make a container hash. Then after container is built the contents of the file change and new container hash is different.

The fix is easy: put --target-dir into a container itself (e.g. /tmp) or into a cache dir. Another hack would be to just cp [..snip..]/pavetta /usr/local/bin instead of using !Copy because you don't need to hash a binary if you already hashed sources (this happens when you !Copy not from a /work anyway).

[mkpankov]

I started with simple cp, but it didn't work (the same way as now). I'll try rolling it back in the minimal example.

[mkpankov]

So yeah, it did help in the minimal example... problem is, simple cp ... is what I started with in my actual project, and it didn't work... Will dig further.

[mkpankov]

Found the actual issue. This commit makes it fail in all cases (even initial build is not successful).

mkpankov/vr@2f17b70

[mkpankov]

I've also reduced the reproduction repository to not include any compilation. Problem is present when simply depending on files, copying them into container, using this container in !SubConfig via !Build. Changing !SubConfig to !Container fixes the issue but is not applicable in all cases.

[mkpankov]

If I were to try and fix this, where in the sources should I be looking?

[tailhook]

Well, thank you for finding this out! It's really a bug, but not trivial to fix:

1. The bug essentially is that SubConfig doesn't expose container it depends on, like !Container
2. The motivation is that vagga.yaml for !SubConfig might not be existent at the moment of initial versioning (this was the original intention of !SubConfig).

Now (1) requires changing internal API, which isn't very hard. But it might be non-trivial to keep use-case (2) functioning in all cases.

On the other hand, in trivial cases of subconfig where yaml isn't generated you can use sequence unpacking:

  pavetta-sdk-common:
    setup: &pavetta-sdk-common
      - !Ubuntu bionic
      - !UbuntuUniverse
      - !Install [build-essential]
      - !Build
        container: pavetta-redist
        source: /usr/local/bin
        path: /usr/local/bin
        rules:
        - /pavetta

  pavetta-sdk-custom:
    environ:
      HOME: /work/.vagga-ws
    setup:
      - !Unpack [*pavetta-sdk-common]
      - !Install [whatever, else, you, need]

[tailhook]

Extra thoughts:

It became easier to spot when I enabled debug log (RUST_LOG=info vagga pavette), there was no container pavetta-redist built before the pavetta-sdk-custom

If we are to fix the bug, the stumble block isn't non-existent vagga.yaml where !SubConfig sources data from (it can be achieved by returning Version::New when no config preset), but the fact that generated vagga.yaml may contain new dependencies. I'm not sure the latter is easy to make working.