remote build: odd errors for certain architectures: setups

[dilyn-corner]

Bug Description

The new remote-build method will fail under particular configurations of the architectures section in a snapcraft.yaml.

To Reproduce

dilyn@Ares:~/cross-test -> mkdir cross-test && cd cross-test
dilyn@Ares:~/cross-test -> snapcraft init

{add the below section to the snapcraft.yaml}
architectures:
  - build-on:  [amd64]
    build-for: [arm64]

dilyn@Ares:~/cross-test -> snapcraft                                                                                                            
dilyn@Ares:~/cross-test -> snapcraft remote-build
dilyn@Ares:~/cross-test -> snapcraft remote-build --build-for=arm64

Environment

Ubuntu 23.04
snapcraft 8.x (rev. 10143)

snapcraft.yaml

name: my-snap-name # you probably want to 'snapcraft register <name>'
base: core22 # the base snap is the execution environment for this snap
version: '0.1' # just for humans, typically '1.2+git' or '1.3.2'
summary: Single-line elevator pitch for your amazing snap # 79 char long summary
description: |
  This is my-snap's description. You have a paragraph or two to tell the
  most important story about your snap. Keep it under 100 words though,
  we live in tweetspace and your description wants to look good in the snap
  store.

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots

architectures:
  - build-on:  [amd64]
    build-for: [arm64]

parts:
  my-part:
    # See 'snapcraft plugins'
    plugin: nil

Relevant log output

Logs from three different attempts:

dilyn@Ares:~/cross-test -> snapcraft
Starting Snapcraft 8.0.0                                                                                                                                      
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20231212-201938.403999.log'                                                            
Running on amd64 for arm64                                                                                                                                    
Launching instance...                                                                                                                                         
Starting Snapcraft 8.0.0                                                                                                                                      
Logging execution to '/tmp/snapcraft.log'                                                                                                                     
Running on amd64 for arm64                                                                                                                                    
Initializing parts lifecycle                                                                                                                                  
Installing build-packages                                                                                                                                     
Installing build-snaps                                                                                                                                        
Pulling my-part                                                                                                                                               
Building my-part                                                                                                                                              
Staging my-part                                                                                                                                               
Priming my-part                                                                                                                                               
Extracting and updating metadata...                                                                                                                           
Copying snap assets...                                                                                                                                        
Generating snap metadata...                                                                                                                                   
Generated snap metadata                                                                                                                                       
Generating snap manifest...                                                                                                                                   
Generated snap manifest                                                                                                                                       
Reading snap metadata...                                                                                                                                      
Running linters...                                                                                                                                            
Running linter: classic                                                                                                                                       
Running linter: library                                                                                                                                       
Creating snap package...                                                                                                                                      
Created snap package my-snap-name_0.1_arm64.snap   

                                                                                                           
dilyn@Ares:~/cross-test -> snapcraft remote-build
Starting Snapcraft 8.0.0                                                                                                                                      
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20231212-202007.454275.log'                                                            
snapcraft remote-build is experimental and is subject to change - use with caution.                                                                           
All data sent to remote builders will be publicly available. Are you sure you want to continue? [y/N]: y
Issues while validating snapcraft.yaml: The 'architectures[0]' property does not match the required schema: additional properties are not allowed ('build-for' was unexpected) or orderedDict([('build-on', ['amd64']), ('build-for', ['arm64'])]) is not of type 'string'


dilyn@Ares:~/cross-test -> snapcraft remote-build --build-for=arm64
Starting Snapcraft 8.0.0                                                                                                                                      
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20231212-202017.786399.log'                                                            
snapcraft remote-build is experimental and is subject to change - use with caution.                                                                           
All data sent to remote builders will be publicly available. Are you sure you want to continue? [y/N]: y
Issues while validating snapcraft.yaml: The 'architectures[0]' property does not match the required schema: additional properties are not allowed ('build-for' was unexpected) or orderedDict([('build-on', ['amd64']), ('build-for', ['arm64'])]) is not of type 'string'

Additional context

Some additional behavior happens in case I expand the architectures section to include an additional architecture:

architectures:
  - build-on:  [amd64, arm64]
    build-for: [arm64]
  - build-on:  [amd64, armhf]
    build-for: [armhf]

Output:

dilyn@Ares:~/cross-test -> snapcraft remote-build  
Starting Snapcraft 8.0.0                                                                                                                                      
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20231212-202200.825177.log'                                                            
snapcraft remote-build is experimental and is subject to change - use with caution.                                                                           
All data sent to remote builders will be publicly available. Are you sure you want to continue? [y/N]: y
Issues while validating snapcraft.yaml: The 'architectures' property does not match the required schema: multiple items will build snaps that claim to run on 'amd64'

dilyn@Ares:~/cross-test -> snapcraft remote-build --build-for=arm64
Starting Snapcraft 8.0.0                                                                                                                                      
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20231212-202300.295288.log'                                                            
snapcraft remote-build is experimental and is subject to change - use with caution.                                                                           
All data sent to remote builders will be publicly available. Are you sure you want to continue? [y/N]: y
Issues while validating snapcraft.yaml: The 'architectures' property does not match the required schema: multiple items will build snaps that claim to run on 'amd64'

Some extra odd behavior happens if I disable fallback with SNAPCRAFT_REMOTE_BUILD_STRATEGY=disable-fallback and expand the architectures stanza as above:

dilyn@Ares:~/cross-test -> SNAPCRAFT_REMOTE_BUILD_STRATEGY=disable-fallback snapcraft remote-build
Starting Snapcraft 8.0.0                                                                                                                                      
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20231212-202415.233506.log'                                                            
snapcraft remote-build is experimental and is subject to change - use with caution.                                                                           
All data sent to remote builders will be publicly available. Are you sure you want to continue? [y/N]: y
Setting up launchpad environment                                                                                                                              
Looking for existing build                                                                                                                                    
No existing build found                                                                                                                                       
If interrupted, resume with: 'snapcraft remote-build --recover --build-id snapcraft-my-snap-name-78a0da0edb2a2b9b359107180b3914fe'                            
Starting build                                                                                                                                                
creating git repo: name=snapcraft-my-snap-name-78a0da0edb2a2b9b359107180b3914fe, owner=/~dilyn-corner, target=/~dilyn-corner                                  
Sending build data to Launchpad: https://dilyn-corner:<token>@git.launchpad.net/~dilyn-corner/+git/snapcraft-my-snap-name-78a0da0edb2a2b9b359107180b3914fe/   
Registering snap job on Launchpad...                                                                                                                          
Issuing build request on Launchpad...                                                                                                                         
Waiting on Launchpad build request...                                                                                                                         
Build request accepted: 84571241                                                                                                                              
Building                                                                                                                                                      
Building snap package for amd64, amd64, arm64, and armhf. This may take some time to finish.                                                                  
Building...                                                                                                                                                   
amd64: Needs building, amd64: Needs building                                                                                                                  
amd64: Currently building, amd64: Currently building - (8.0s)                                       amd64: Currently building, amd64: Currently building - (19.amd64: Currently building, amd64: Currently building                                                                                                          
amd64: Currently building, amd64: Currently building                                                                                                          
amd64: Currently building, amd64: Currently building                                                                                                          
amd64: Currently building, amd64: Currently building                                                                                                          
amd64: Currently building, amd64: Currently building                                                                                                          
amd64: Currently building, amd64: Currently building                                                                                                          
amd64: Currently building, amd64: Currently building                                                                                                          
amd64: Currently building, amd64: Currently building                                                                                                          
amd64: Currently building, amd64: Currently building                                                                                                          
amd64: Currently building, amd64: Currently building                                                                                                          
amd64: Currently building, amd64: Currently building                                                                                                          
amd64: Gathering build output, amd64: Gathering build output                                                                                                  
amd64: Uploading build, amd64: Uploading build                                                                                                                
amd64: Successfully built, amd64: Successfully built                                                                                                          
Downloading artifacts...                                                                                                                                      
Downloading: https://launchpad.net/~dilyn-corner/+snap/snapcraft-my-snap-name-78a0da0edb2a2b9b359107180b3914fe/+build/2326477/+files/my-snap-name_0.1_arm64.snap                                                                                                                                                             
Snapped my-snap-name_0.1_arm64.snap                                                                                                                           
Downloading: https://launchpad.net/~dilyn-corner/+snap/snapcraft-my-snap-name-78a0da0edb2a2b9b359107180b3914fe/+build/2326477/+files/buildlog_snap_ubuntu_jammy_amd64_snapcraft-my-snap-name-78a0da0edb2a2b9b359107180b3914fe_BUILDING.txt.gz                                                                                
Build log available at 'my-snap-name_amd64.txt'.                                                                                                              
Downloading: https://launchpad.net/~dilyn-corner/+snap/snapcraft-my-snap-name-78a0da0edb2a2b9b359107180b3914fe/+build/2326478/+files/my-snap-name_0.1_arm64.snap                                                                                                                                                             
Snapped my-snap-name_0.1_arm64.snap                                                                                                                           
Downloading: https://launchpad.net/~dilyn-corner/+snap/snapcraft-my-snap-name-78a0da0edb2a2b9b359107180b3914fe/+build/2326478/+files/my-snap-name_0.1_armhf.snap                                                                                                                                                             
Snapped my-snap-name_0.1_armhf.snap                                                                                                                           
Downloading: https://launchpad.net/~dilyn-corner/+snap/snapcraft-my-snap-name-78a0da0edb2a2b9b359107180b3914fe/+build/2326478/+files/buildlog_snap_ubuntu_jammy_amd64_snapcraft-my-snap-name-78a0da0edb2a2b9b359107180b3914fe_BUILDING.txt.gz                                                                                
Build log available at 'my-snap-name_amd64.1.txt'.                                                                                                            
Build complete.                                                                                                                                               
Cleaning                                                                                                                                                      
Cleaning existing builds and artefacts.                                                                                                                       
Removing snap job from Launchpad...                                                                                                                           
Deleting source repository from Launchpad...                                                                                                                  
Build completed                                                                                                                                               
dilyn@Ares:~/cross-test -> ls
my-snap-name_0.1_arm64.snap  my-snap-name_0.1_armhf.snap  my-snap-name_amd64.1.txt  my-snap-name_amd64.txt  snap

Note the repetition of building for amd64; especially odd as I don't expect a snap to be built for AMD64 at all given my architectures listed.

The two AMD64 build logs indicate that the builds happened on AMD64 machines, with the first log reporting that one snap was successfully built (an ARM64 one), and the second log indicating that two snaps were successfully built (both ARM64 and ARMHF). Both the snap files seem to have all the correct information for being installed on their respective architectures. However, if I had built a non-trivial snap, I'm not convinced they would actually function properly (e.g. AMD64 libs might be inside them).

Setting force-fallback instead results in the same error message as I mentioned earlier in this section (as expected).

[dilyn-corner]

I just attempted with 7.x and received the following:

dilyn@Ares:~/cross-test -> snapcraft_7 remote-build --build-for=arm64
Starting Snapcraft 7.5.4                                                                           
Logging execution to '/home/dilyn/.local/state/snapcraft/log/snapcraft-20231212-204009.665237.log' 
snapcraft remote-build is experimental and is subject to change - use with caution.                
All data sent to remote builders will be publicly available. Are you sure you want to continue? [y/N]: y
Issues while validating snapcraft.yaml: The 'architectures' property does not match the required schema: multiple items will build snaps that claim to run on 'amd64'

Using the below snapcraft.yaml:

name: my-snap-name # you probably want to 'snapcraft register <name>'
base: core22 # the base snap is the execution environment for this snap
version: '0.1' # just for humans, typically '1.2+git' or '1.3.2'
summary: Single-line elevator pitch for your amazing snap # 79 char long summary
description: |
  This is my-snap's description. You have a paragraph or two to tell the
  most important story about your snap. Keep it under 100 words though,
  we live in tweetspace and your description wants to look good in the snap
  store.

grade: devel # must be 'stable' to release into candidate/stable channels
confinement: devmode # use 'strict' once you have the right plugs and slots

architectures:
  - build-on:  [amd64, arm64]
    build-for: [arm64]
  - build-on:  [amd64, armhf]
    build-for: [armhf]

parts:
  my-part:
    # See 'snapcraft plugins'
    plugin: nil

[mr-cal]

I see 2 things here:

architectures:
  - build-on:  [amd64, arm64]
    build-for: [arm64]
  - build-on:  [amd64, armhf]
    build-for: [armhf]

giving strange progress and logs.

This line

Building snap package for amd64, amd64, arm64, and armhf. This may take some time to finish.

is certainly misleading. It's a list of all the build-ons. Snapcraft does some weird translation between build-on and build-for before launching the remote-builder and I think this is going to need some more investigation.

At the end of the day, Launchpad appears to build the correct snaps but I'm not sure why or how. This will require me to read the Launchpad API more and figure out what architectures we need to parse and pass to it.

architectures:
  - build-on:  [amd64]
    build-for: [arm64]

gives the error:

Issues while validating snapcraft.yaml: ... 'build-for' was unexpected ...

I was able to reproduce this on snapcraft 7.x and 8.x. I think it is a duplicate of https://bugs.launchpad.net/snapcraft/+bug/2007789.

The problem is that the legacy remote-builder is expecting run-on instead of build-for.
I'm not sure of the best path forward anymore. A workaround is to not use the legacy remote-builder (i.e. SNAPCRAFT_REMOTE_BUILD_STRATEGY="disable-fallback")

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-3283.

This message was autogenerated

[mr-cal]

We've made many improvements to the remote-builder that shoudl address problems like Launchpad reporting the wrong architectures. Other behaviors here, such as using --build-for while architectures are defined in the project are not allowed and give a user-friendly error via #4994.

I'm closing this the main problems here have been addressed or are tracked in a different issue. If anything comes up again, please make a new issue.