Added code for supporting arch system package #1143

proneon267 · 2023-03-20T21:17:41Z

Testing instructions:

If you're on an Arch Linux system, or any derivative of an Arch system (e.g., Manjaro), you can test this branch with:

mkdir bee-test
cd bee-test
python3 -m venv venv
source venv/bin/activate
pip install git+https://github.com/proneon267/briefcase.git@patch-1
briefcase new --template https://github.com/beeware/briefcase-template --template-branch arch

Accept all the default values from the template, then run:

cd helloworld
briefcase run

You should get a running Toga GTK app. Then, run:

briefcase package

This should generate a dist/helloworld-0.0.1-x86_64.pkg.tar.zst file, which you can install with pacman -U dist/helloworld-0.0.1-x86_64.pkg.tar.zst

If you have any problems building, running or installing this package, or the package that is generated doesn't match your expectations as an Arch* user, let us know in the comments below, including details of the specific distro that you are using for testing.

For creating arch packages, we need to build a PKGBUILD file which will be used by makepkg utility to create the .pkg.tar.zst file.
A sample PKGBUILD file:

# Maintainer: Your Name <youremail@domain.com>
pkgname=NAME
pkgver=VERSION
pkgrel=1
epoch=
pkgdesc=""
arch=()
url=""
license=('GPL')
groups=()
depends=()
makedepends=()
checkdepends=()
optdepends=()
provides=()
conflicts=()
replaces=()
backup=()
options=()
install=
changelog=
source=("$pkgname-$pkgver.tar.gz"
        "$pkgname-$pkgver.patch")
noextract=()
md5sums=()
validpgpkeys=()

prepare() {
	cd "$pkgname-$pkgver"
	patch -p1 -i "$srcdir/$pkgname-$pkgver.patch"
}

build() {
	cd "$pkgname-$pkgver"
	./configure --prefix=/usr
	make
}

check() {
	cd "$pkgname-$pkgver"
	make -k check
}

package() {
	cd "$pkgname-$pkgver"
	make DESTDIR="$pkgdir/" install
}

There are many parameters but we do not need to use them all. As you can see among all these functions, only package() is necessary, others are optional.
makepkg doesn't necessarily need to compile the stub binary. Since, briefcase build does the stub binary compilation, we can skip this step.
A simplified and tested PKGBUILD file:

pkgname=helloworld
pkgver=0.0.1
pkgrel=1
pkgdesc="This is a package"
arch=('x86_64')
url="https://github.com"
license=('GPL')
depends=('python' 'gobject-introspection' 'gobject-introspection-runtime' 'glibc' 'gtk3')
source=("$pkgname-$pkgver.tar.gz")
md5sums=('SKIP')
changelog=CHANGELOG
package() {
  cp -r "$srcdir/$pkgname-$pkgver"/usr/ "$pkgdir"/usr/
}

I will add more details later on.
This PR relates to an issue: #1064

PR Checklist:

All new features have been tested
All new features have been documented
I have read the CONTRIBUTING.md file
I will abide by the code of conduct

proneon267

corrected a typo

freakboy3742

I've updated your PR to fix some indentation and formatting issues, and tried it out; even with those changes, I get the following error:

[togatest] Building .pkg.tar.zst package...
Building source archive... done
Generating pkgbuild layout... done

Entering Docker context...
Docker| ------------------------------------------------------------------
Docker| ==> ERROR: togatest is not available for the 'x86_64' architecture.
Docker| ==> ERROR: changelog file (/Users/rkm/beeware/briefcase/local/togatest/CHANGELOG) does not exist or is not a regular file.
Docker| ------------------------------------------------------------------
Leaving Docker context.

Building Arch package...

Error while building .pkg.tar.zst package for togatest.

I'm also slightly dubious whether this will work in general. You're shipping the /usr/bin/<myapp> binary as part of the distribution artefact - won't that cause problems if the person installing the binary doesn't have exactly the same versions of the package? I thought Arch packages needed to be completely compiled from source - which means the distribution artefact needs to not include the binary, include the bootstrap folder, and perform a make as part of the installation process.

proneon267 · 2023-03-21T02:34:02Z

Hello
The first error seems to be due to togatest, which I guess needs to have test for this. But, i haven't written any tests.

The second error must be due, not being able to find the CHANGELOG file, which I will fix.

You are right the stub binary won't work if the libraries are not compatible. ( sorry, force of habit of distributing only for a single architecture)

I'll make the changes and update the pull request soon. Thanks.

freakboy3742 · 2023-03-21T02:39:55Z

Hello The first error seems to be due to togatest, which I guess needs to have test for this. But, i haven't written any tests.

Tests will be needed, but that's got nothing to do with this error. Togatest is the name of my test application. It's nothing more than the "Hello World" generated by Briefcase, but with a customised name (because I need to test against PySide, PPB, Pygame,...)

The second error must be due, not being able to find the CHANGELOG file, which I will fix.

For the record - the referenced filename exists, and is a regular file. It's the package builder that is failing.

You are right the stub binary won't work if the libraries are not compatible. ( sorry, force of habit of distributing only for a single architecture)

To be clear: Its not just "single architecture" that is the issue here - it's "incompatible version of libraries". Distributions like Ubuntu are able to provide guarantees on the specific version (and thus ABI) that various libraries will provide - locking a specific set of ABIs is the major reason for having tagged releases. AFAIK, Arch doesn't do this - every user is compiling everything from scratch, so there's no way to provide ABI guarantees.

Modified so that `makepkg` will compile the stub binary. Changed `arch` to `'any'` to prevent architecture error. Also, the `CHANGELOG` file will be added to the source archive so that `makepkg` will be able to find it while running the `PKGBUILD` file.

proneon267

Modified to tackle the errors.

freakboy3742 · 2023-03-21T07:27:23Z

src/briefcase/platforms/linux/system.py

-                    root_dir=self.bundle_path(app),
-                    base_dir=f"{app.app_name}-{app.version}",
+                    root_dir=self.build_path(app),
+                    base_dir=self.bundle_path(app),


This archive will include a pre-build binary unless you either exclude usr/bin/{app.app_name} as a path, or disable the call to make during the Build command.

Hello
I mean, doesn't briefcase build just run make and make install. So, we could just briefcase create and then briefcase package. If this is not the intended behavior, then I will modify it.

I can tell you for free - It's not going to be easy to remove the build step conditionally based on the packaging format, because there's a whole lot of logic that is shared across all backends that determines how to infer that a build is needed. It's going to be a lot easier to skip the specific call to make in the arch packaging case.

You are right, skipping the make call should solve the problem.

freakboy3742 · 2023-03-21T07:28:54Z

src/briefcase/platforms/linux/system.py

        # Generate the pkgbuild layout
-        pkgbuild_path = self.bundle_path(app) / "pkgbuild"
+        pkgbuild_path = self.build_path(app) / "pkgbuild"


This isn't correct - it definitely needs to be bundle_path, or you'll generate code in a directory outside the Arch bundle.

yes, we need to generate the code outside the bundle_path as the whole bundle_path will be archived for source tarball as bundle_path contains the bootstrap directory.

This setup is going to lead to a briefcase file layout that looks like:

build

myapp

arch

pkgbuild

...

rolling

bootstrap

...

myapp-1.2.3

usr

...

That's the wrong location for the pkgbuild folder. It should be inside the rolling folder. This is, at the very least, needed to support Docker builds, because Docker mounts build/myapp/arch/rolling as /app; so the pkgbuild folder won't be visible inside the Docker container. More generally, all the artefacts of the arch/rolling build should be in the arch/rolling folder.

Thanks! I was confused about the directory structure. I will modify it.

freakboy3742 · 2023-03-21T07:29:50Z

src/briefcase/platforms/linux/system.py

        with self.input.wait_bar("Generating pkgbuild layout..."):
            if pkgbuild_path.exists():
                self.tools.shutil.rmtree(pkgbuild_path)
            (pkgbuild_path).mkdir(parents=True)

            # Build the source archive
            with self.input.wait_bar("Building source archive..."):
+
+                # Copy the CHANGELOG file to the bundle_path so it can be included in the source archive
+                self.tools.shutil.copy(changelog_source, self.bundle_path(app) / "CHANGELOG")


Why does it need to be copied? Couldn't you add a single file directly to the archive?

Since we are archiving the whole bundle_path directory, I thought first copying and then archiving would be better.

Copying is a last resort; by copying, you're creating a duplicate file, which could lead to drift between the source and the packaged artefact. Packaging the source file directly is vastly preferred, and I can't see any reason it couldn't be done here.

Sure. I will change it.

proneon267 · 2023-03-21T08:32:14Z

Hello
About how the package system works.
makepkg reads PKGBUILD file and compiles the source(if you specify it to), and then installs(copies,) to the "$pkgdir". Then it packages them into a .pkg.tar.zst file.
Then the package can be installed with pacman. pacman copies the contents of the "$pkgdir" on to the root folder of the target system. That's it. It doesn't compile from source and is just a package manager.
I think you are confusing arch package system with gentoo package system, where everything needs to be compiled from source.

Moreover, probably you are confusing with the AUR system. In AUR repository, there are only PKGBUILD files. When someone installs a package from AUR, the system is actually downloading the PKGBUILD file and running makepkg, which may(if the source tar ball contains uncompiled source files) or may not compile the source (if the source tar ball contains pre built binaries instead), depending upon what is specified in the PKGBUILD file and then calls pacman to install it.

We do not necessarily need to run make and make install in the PKGBUILD file. As, the same is accomplished by running briefcase build.
And running make and make install from within the PKGBUILD file is the same as what briefcase build is doing. This is because they are running on the system on which the .pkg.tar.zst file is generated and will not be run on the target(user's) system.
Hence, we can skip the build() function in PKGBUILD file.

If you agree to let briefcase build the binary stub and makepkg to only create the .pkg.tar.zst file, then we can work inside bundle_path directory.
Else, if you want to run make and make install in PKGBUILD file, then briefcase build should not run make & make install, and the bundle_path directory needs to be archived. This is because we need the bootstrap directory. I guess, selectively adding directories to the source archive could also solve the problem which would allow us to build the package inside a directory within bundle_path.

Moreover, if you choose the former solution then we can also skip creating the source archive (source tarball) before creating the actual .pkg.tar.zst file and directly generate the pkg using the local directory.

freakboy3742 · 2023-03-21T23:07:50Z

(Apologies on the close - hit the wrong button)

Hello About how the package system works. makepkg reads PKGBUILD file and compiles the source(if you specify it to), and then installs(copies,) to the "$pkgdir". Then it packages them into a .pkg.tar.zst file. Then the package can be installed with pacman. pacman copies the contents of the "$pkgdir" on to the root folder of the target system. That's it. It doesn't compile from source and is just a package manager. I think you are confusing arch package system with gentoo package system, where everything needs to be compiled from source.

Sure - in that regard, you're describing a process that is functionally identical to both Debian and Redhat packages.

However, Debian and Redhat have a significant difference with Arch: they are able to guarantee binary compatibility. If I have Debian 11, you know exactly what version of (say) GTK I have, because that's part of the "GTK" package definition for the Debian 11 release. Arch doesn't provide that guarantee, because it doesn't tag releases in that way.

So - if you build a package for your Toga app, and you give it to me... how can I guarantee that the package will work? You know nothing about my system, other than the fact that I'm running Arch. If I'm running on an arch system where GTK was built 2 years ago, and you build your package on a "bleeding edge" Arch install... how do you know that your app will run on my machine?

Moreover, probably you are confusing with the AUR system.

I have no idea what I'm confusing with what, because I'm not an Arch user. :-)

Functionally, I want to be able to guarantee that user X can build a package, give it to user Y, and that package will Just Work. I was under the impression that the only way that would happen on Arch is if the compiled components were compiled on the installing system; but if that's not the case, then I'll gladly stand corrected.

freakboy3742 · 2023-03-21T23:10:31Z

src/briefcase/platforms/linux/system.py

+                            f'pkgdesc="{app.description}"',
+                            f"arch=('any')",  # We should use {self.linux_arch}, but doing this as the architecture error 
+                                              # was probably caused due to this parameter.
+                                              # Moreover arch wiki recommends 'any' for architecture independent packages.


This doesn't smell right to me.

If you have numpy (or any other binary dependency) in your app, Briefcase will be installing an architecture-specific binary wheel. Python is able to guarantee that the wheel will work because it adheres to the manylinux specification, which constraints the libraries that the wheel can link against; but it means that the artefacts in the wheel are platform specific. If you build your package on x86-64, you won't be able to deploy it on arm64.

The architecture error is usually caused if the host machine and target package architectures are different. That means, the previous error was caused due to mismatch of host architecture string. I have modified it to use self.tools.host_arch. Now, it should work fine.

proneon267

Modified for addressing issues. The architecture error is usually caused if the host machine and target package architectures are different. I will research more and post about it.

proneon267 · 2023-03-22T14:17:06Z

For dependencies compatibility
PKGBUILD file provides the depends=() argument, which we can use to setup the dependencies along with their specific version to create a hard version requirement or soft version requirement.
For example, we need webkit2gtk to be installed on the target system so that our app can work on the target system. Then, we can specify this using:
depends=('webkit2gtk') -> For latest version
depends=('webkit2gtk>=2.40.0-2') -> For version greater than or equal to 2.40.0-2
depends=('webkit2gtk<=2.40.0-2') -> For less than or equal to 2.40.0-2
depends=('webkit2gtk>2.40.0-2') -> For greater than 2.40.0-2
depends=('webkit2gtk<2.40.0-2') -> For less than 2.40.0-2

This will be checked by pacman to make sure the dependencies are installed or present before installing the actual package. So, if our app can only work with webkit2gtk version 2.40.0-2 then we can specify a hard version requirement like: depends=('webkit2gtk=2.40.0-2').

freakboy3742 · 2023-03-22T15:16:23Z

For dependencies compatibility PKGBUILD file provides the depends=() argument...

Yes - you've just given a description of how dependencies work in every environment. What you're missing are the practical application of these dependency specification for Arch as a distribution.

On every other Linux distro, we don't need to do any version pinning, because the version of GTK that is provided on Ubuntu 22.04 (or any other distro) is implicitly pinned. All Ubuntu 22.04 machines have binary compatible versions of GTK - that's why they tag releases. If I produce a binary package for Ubuntu 22.04 that depends on GTK, I know that package will still work in 2 years time.

On Arch, a description like "latest" isn't helpful when you're building an app for distribution. If I build a package with a dependency on the "latest" GTK, and 2 weeks pass, the "latest" isn't the same version any more, so there's no guarantee that someone who downloads a 2 week old package with a dependency on the "latest" GTK will be able to install it.

So, if an Arch package contains a pre-compiled binary, we need to guarantee the specific versions of every dependency - which means hard-pins on every library version (or, at best, very narrow ranges). This leads to a bad user experience as well - if I download a 2 year old package, I will discover that I have to downgrade my version of GTK just so I can support the binary requirements of this old package - something that may be mutually incompatible with other packages that I'm running.

This is why I'm suggesting that shipping a pre-compiled binary in an Arch package isn't a viable option. At the very least, we would need to have specific (or very narrow) version pins on every dependency, which is a maintenance overhead for app developers - and for Briefcase as well - as we need to take on the burden of ensuring that the version pins in the templates we provide are up to date with what Arch is currently publishing.

Modified towards a working state.

proneon267

Modified to use the current working solution for the changelog.

proneon267 · 2023-03-22T16:22:13Z

Hello
So, the CHANGELOG file needs to be in the same directory where the PKGBUILD file is located. This is only possible if the CHANGELOG is either copied or a hardlink for the CHANGELOG file is created in the directory where the PKGBUILD is located.

This is because the CHANGELOG's file path self.base_path / "CHANGELOG" won't be visible inside the docker and we would get the previous error.

Also, we cannot use"$srcdir/CHANGELOG" to reference the file as the options in the PKGBUILD are evaluated first and the "$srcdir" variable will not be set until the source tarball is extracted.

Since, you are not too keen to copy the CHANGELOG file, we could create a hardlink(points to the same data on the disk) of the CHANGELOG file on the directory where PKGBUILD file is present.

Until you decide, I am keeping the copying solution, as it is the only working solution.

proneon267 · 2023-03-22T17:03:11Z

This is why I'm suggesting that shipping a pre-compiled binary in an Arch package isn't a viable option. At the very least, we would need to have specific (or very narrow) version pins on every dependency, which is a maintenance overhead for app developers - and for Briefcase as well - as we need to take on the burden of ensuring that the version pins in the templates we provide are up to date with what Arch is currently publishing.

Hello
Well arch packages ship with pre-compiled binaries and the other way around for compiling on the user machine would be to distribute both a source tarball(containing only the source code) and a PKGBUILD file, which would then be run by the user to generate a package and install it. This doesn't seem like a good user experience.

Yes, may be the "latest" will not be the same for everyone, but any sane person running a bleeding edge rolling distro should be keeping their system up to date( should be running pacman -Syu once a week or at the very least run it before installing a new package). We should narrow version pin dependencies like >= greater than or equal to.

If the developers want to hard pin the dependencies, then they can manually add them to the generated PKGBUILD file and generate the package (after all the PKGBUILD file would be there). Briefcase should not take the role of ensuring compatibility of hard pin version dependencies.

proneon267 · 2023-03-26T06:15:49Z

Now it shows, the packager name correctly:

    ~/b/helloworld  pacman -Qip ./dist/helloworld-0.0.1-1-x86_64.pkg.tar.zst
Name            : helloworld
Version         : 0.0.1-1
Description     : My first application
Architecture    : x86_64
URL             : https://example.com/helloworld
Licenses        : BSD license
Groups          : None
Provides        : None
Depends On      : glibc>=2.37  python3  gtk3  gobject-introspection-runtime
Optional Deps   : None
Conflicts With  : None
Replaces        : None
Compressed Size : 1232.21 KiB
Installed Size  : 3055.50 KiB
Packager        : Jane Developer <jane@example.com>
Build Date      : Sun 26 Mar 2023 01:05:38 AM
Install Script  : No
Validated By    : None
Signatures      : None

But the Validated By and Signatures field remain None ~~as the option validpgpkeys=('') in PKGBUILD file is not specified~~. This is because there is no mechanism to generate or specify a PGP key for the project in briefcase. This can become a major drawback as some systems are configured to not allow packages with unverified signatures to be installed. Moreover, some systems show a negative warning message before installing the package which will degrade user experience.

makepkg has all options to sign ~~and even include the public key with the package so that the public key will installed on the user's system, the first time they install the package~~. If in future pgp key option is introduced in briefcase, it would greatly improve user experience(Maybe ?).

freakboy3742 · 2023-03-26T07:04:23Z

Package signing is definitely something we're interested in adding; we have it already for macOS, and there's a PR in flight adding it for Windows.

I'd consider it a separate feature to this PR, though; we need to ensure that whatever approach to signing we take, it is as broadly applicable as possible.

proneon267 · 2023-03-26T12:33:26Z

Yes you are right, package signing should be a separate PR. Researching on the subject, linux package signing is a bit of mess. I mean the signing part is easy but the part where the user verifies and installs the app is a mess.

In Debian and derivatives, although package signing and verifying tools exists but by default Debian and derivatives verify the signature of the repository instead of the package. So, even if we would sign the generated package, it would not be verified, as this verifying tool and feature is disabled by default.

And in Arch and derivatives system, signed packages cannot be installed on the system, until the public key is imported into pacman's keyring. But it will let users install packages without any signatures without raising any warnings.

In RHEL, I am not sure but it seems like there would be some sort of such nuances.

All in all, signing is easy but verifying and installing the package on users' system is difficult.

MarkusH · 2023-03-26T14:31:26Z

Okay, here's the problem. If I run the commands as outlined in the PR description, this is the package content:

❯ pacman -Ql helloworld             
helloworld /usr/
helloworld /usr/bin/
helloworld /usr/bin/helloworld
helloworld /usr/lib/
helloworld /usr/lib/helloworld/
helloworld /usr/lib/helloworld/app/
helloworld /usr/lib/helloworld/app/README
helloworld /usr/lib/helloworld/app_packages/
helloworld /usr/lib/helloworld/app_packages/README
helloworld /usr/share/
helloworld /usr/share/applications/
helloworld /usr/share/applications/com.example.helloworld.desktop
helloworld /usr/share/doc/
helloworld /usr/share/doc/helloworld/
helloworld /usr/share/doc/helloworld/changelog.gz
helloworld /usr/share/doc/helloworld/copyright
helloworld /usr/share/icons/
helloworld /usr/share/icons/hicolor/
helloworld /usr/share/icons/hicolor/128x128/
helloworld /usr/share/icons/hicolor/128x128/apps/
helloworld /usr/share/icons/hicolor/128x128/apps/com.example.helloworld.png
helloworld /usr/share/icons/hicolor/16x16/
helloworld /usr/share/icons/hicolor/16x16/apps/
helloworld /usr/share/icons/hicolor/16x16/apps/com.example.helloworld.png
helloworld /usr/share/icons/hicolor/256x256/
helloworld /usr/share/icons/hicolor/256x256/apps/
helloworld /usr/share/icons/hicolor/256x256/apps/com.example.helloworld.png
helloworld /usr/share/icons/hicolor/32x32/
helloworld /usr/share/icons/hicolor/32x32/apps/
helloworld /usr/share/icons/hicolor/32x32/apps/com.example.helloworld.png
helloworld /usr/share/icons/hicolor/512x512/
helloworld /usr/share/icons/hicolor/512x512/apps/
helloworld /usr/share/icons/hicolor/512x512/apps/com.example.helloworld.png
helloworld /usr/share/icons/hicolor/64x64/
helloworld /usr/share/icons/hicolor/64x64/apps/
helloworld /usr/share/icons/hicolor/64x64/apps/com.example.helloworld.png
helloworld /usr/share/man/
helloworld /usr/share/man/man1/
helloworld /usr/share/man/man1/helloworld.1.gz

As you can see, all the Python files are missing.

proneon267 · 2023-03-26T14:38:18Z

Hello
Could you also check if the files are also missing from the generated folder /bee-test/helloworld/build/helloworld/manjaro/rolling/pkgbuild/src/helloworld-0.0.1/usr/lib/helloworld/app/helloworld/

MarkusH · 2023-03-26T14:38:46Z

I've just ran pip install and briefcase new from the first code block again and now I have a working app. Both, briefcase run and briefcase package yield the expected results, such that pacman -U worked as desired as well:

$ pacman -Qi helloworld 
Name            : helloworld
Version         : 0.0.1-1
Description     : My first application
Architecture    : x86_64
URL             : https://example.com/helloworld
Licenses        : BSD license
Groups          : None
Provides        : None
Depends On      : glibc>=2.37  python3  gtk3  gobject-introspection-runtime
Optional Deps   : None
Required By     : None
Optional For    : None
Conflicts With  : None
Replaces        : None
Installed Size  : 2.98 MiB
Packager        : Jane Developer <jane@example.com>
Build Date      : Sun 26 Mar 2023 04:34:35 PM CEST
Install Date    : Sun 26 Mar 2023 04:35:00 PM CEST
Install Reason  : Explicitly installed
Install Script  : No
Validated By    : None

proneon267 · 2023-03-26T14:41:19Z

Hello
what packages did you install with pip, or did you mean you re-installed this branch with pip?

MarkusH · 2023-03-26T14:47:38Z

Hello what packages did you install with pip?

I reran these commands:

~$ cd /tmp/bee-test
/tmp/bee-test$ source venv/bin/activate
(venv) /tmp/bee-test$ rm -rf helloworld
(venv) /tmp/bee-test$ pip install git+https://github.com/proneon267/briefcase.git@patch-1
(venv) /tmp/bee-test$ briefcase new --template https://github.com/beeware/briefcase-template --template-branch arch
(venv) /tmp/bee-test$ cd helloworld
(venv) /tmp/bee-test/helloworld$ briefcase run
(venv) /tmp/bee-test/helloworld$ briefcase package
(venv) /tmp/bee-test/helloworld$ sudo pacman -U dist/helloworld-0.0.1-1-x86_64.pkg.tar.zst

proneon267 · 2023-03-26T15:00:53Z

I am able to reproduce the error. On a fresh new installation of Manjaro, I got the same No module named helloworld error. Everything went smoothly until I ran briefcase run.

The error suddenly disappears when you exit the venv, delete the helloworld folder, again source venv and rerun the steps.

But, this doesn't occur when you run the steps without using venv

proneon267 · 2023-03-26T15:23:52Z

Here is the directory structure, of helloworld project when the error inside the venv occurs:

./helloworld
├── build
│   └── helloworld
│       └── manjaro
│           └── rolling
│               ├── bootstrap
│               │   ├── helloworld
│               │   ├── main.c
│               │   ├── Makefile
│               │   └── pyversion.h
│               ├── briefcase.toml
│               ├── Dockerfile
│               ├── helloworld-0.0.1
│               │   └── usr
│               │       ├── bin
│               │       │   └── helloworld
│               │       ├── lib
│               │       │   └── helloworld
│               │       │       ├── app
│               │       │       │   └── README
│               │       │       └── app_packages
│               │       │           └── README
│               │       └── share
│               │           ├── applications
│               │           │   └── com.example.helloworld.desktop
│               │           ├── doc
│               │           │   └── helloworld
│               │           │       ├── changelog.gz
│               │           │       └── copyright
│               │           ├── icons
│               │           │   └── hicolor
│               │           │       ├── 128x128
│               │           │       │   └── apps
│               │           │       │       └── com.example.helloworld.png
│               │           │       ├── 16x16
│               │           │       │   └── apps
│               │           │       │       └── com.example.helloworld.png
│               │           │       ├── 256x256
│               │           │       │   └── apps
│               │           │       │       └── com.example.helloworld.png
│               │           │       ├── 32x32
│               │           │       │   └── apps
│               │           │       │       └── com.example.helloworld.png
│               │           │       ├── 512x512
│               │           │       │   └── apps
│               │           │       │       └── com.example.helloworld.png
│               │           │       └── 64x64
│               │           │           └── apps
│               │           │               └── com.example.helloworld.png
│               │           └── man
│               │               └── man1
│               │                   └── helloworld.1.gz
│               └── helloworld.1
├── CHANGELOG
├── LICENSE
├── logs
│   ├── briefcase.2023_03_26-15_17_01.run.log
│   └── briefcase.2023_03_26-15_18_29.run.log
├── pyproject.toml
├── README.rst
├── src
│   └── helloworld
│       ├── app.py
│       ├── __init__.py
│       ├── __main__.py
│       └── resources
│           ├── helloworld.icns
│           ├── helloworld.ico
│           ├── helloworld.png
│           └── __init__.py
└── tests
    ├── helloworld.py
    ├── __init__.py
    └── test_app.py

38 directories, 36 files

proneon267 · 2023-03-26T16:28:09Z

Confirmed this error is caused when the requirements are not installed on a fresh system. For example, on a new system, the following error would be raised:

[helloworld] Installing support package...
No support package required.

Unable to build helloworld due to missing system dependencies. Run:

    sudo ['pacman'] ['-Syu'] base-devel gobject-introspection

to install the missing dependencies, and re-run Briefcase.

Log saved to /home/manjaro/io/helloworld/logs/briefcase.2023_03_26-15_43_19.run.log

Now install the missing dependencies with sudo pacman -Syu base-devel gobject-introspection. Re-run briefcase run. It won't raise any errors during the build process, but will show:


[helloworld] Finalizing application configuration...
Targeting manjaro:rolling (Vendor base arch)
Determining glibc version... done
Targeting glibc 2.37
Targeting Python3.10

[helloworld] Building application...
Build bootstrap binary...
make: Entering directory '/home/manjaro/io/helloworld/build/helloworld/manjaro/rolling/bootstrap'
python3.10 -c "import sys; print(f'#define PY_TAG \"{sys.version_info.major}.{sys.version_info.minor}\"')" > pyversion.h
gcc -o helloworld main.c pyversion.h -I/usr/include/python3.10 -I/usr/include/python3.10  -Wno-unused-result -Wsign-compare -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/python/src=/usr/src/debug/python -flto=auto -ffat-lto-objects -DNDEBUG -g -fwrapv -O3 -Wall  -L/usr/lib -lpython3.10 -lcrypt -ldl  -lm -lm  -fPIE
mkdir -p ../helloworld-0.0.1/usr/bin
cp helloworld ../helloworld-0.0.1/usr/bin
make: Leaving directory '/home/manjaro/io/helloworld/build/helloworld/manjaro/rolling/bootstrap'
Building bootstrap binary... done
Installing license... done
Installing changelog... done
Installing man page... done
Update file permissions...
Updating file permissions... done
Stripping binary... done

[helloworld] Built build/helloworld/manjaro/rolling/helloworld-0.0.1/usr/bin/helloworld

[helloworld] Starting app...
===========================================================================
Install path: /home/manjaro/io/helloworld/build/helloworld/manjaro/rolling/helloworld-0.0.1/usr
Pre-initializing Python runtime...
PYTHONPATH:
- /usr/lib/python3.10
- /usr/lib/python3.10/lib-dynload
- /home/manjaro/io/helloworld/build/helloworld/manjaro/rolling/helloworld-0.0.1/usr/lib/helloworld/app
- /home/manjaro/io/helloworld/build/helloworld/manjaro/rolling/helloworld-0.0.1/usr/lib/helloworld/app_packages
Configure argc/argv...
Initializing Python runtime...
Running app module: helloworld
---------------------------------------------------------------------------
Application quit abnormally (Exit code -1)!
/usr/bin/python3: No module named helloworld

Problem running app helloworld.

Log saved to /home/manjaro/io/helloworld/logs/briefcase.2023_03_26-15_45_47.run.log

Again running briefcase run will show the same error. Manually running briefcase build and then running briefcase run would still produce the error. However, if you exit this instance of venv and again source this instance of venv, this time briefcase run will run successfully.

Looks like something is getting cached in the venv, so briefcase run doesn't re-run again properly. I have attached the log files of both before and after installing the missing dependencies. This error cannot be reproduced if the same steps are followed but without using a venv instance.
[Before Installing Dependencies]briefcase.2023_03_26-15_43_19.run.log
[After installing Dependencies]briefcase.2023_03_26-15_45_47.run.log

rmartin16

This is great; glad we're able to round this out with arch packaging.

Toga worked fine for me including with webview and numpy.

PySide6 was a different story....because ofc it was 🙃

makepkg is eager to strip the binaries in the package which is likely to cause problems for libraries from PyPI wheels
- Adding options=('!strip') to PKGBUILD mitigates this
I needed qt6-base at runtime for the pyside6 helloworld app
- Interestingly, makepkg complained unless qt6-base was also installed at build time...

Other Notes

Adding libcanberra as a dependency avoids the typical canberra module error
I hit an error for missing gcc for briefcase package; installing base-devel fixed this. We should add this to the BeeWare tutorial as well like we did with build-essential.
- I presume I got the error from pip about gcc missing because of what's outlined in system_requires aren't verified when building system packages locally #1137.

src/briefcase/platforms/linux/system.py

rmartin16 · 2023-03-26T17:47:19Z

@proneon267: Related to the empty build directories and differences between venv and system python....

I believe this is stemming from the fact the app template is rolled out in the build directory during the create command but then the build command fails on the missing dependencies. Once you install the missing dependencies and re-run briefcase run, it skips the steps to install the app's source and requires packages because it "looks" as though they've already been installed. If you run briefcase run -ur, it will perform those steps and avoid the situation. Once #1137 is resolved, I think situations like this will be prevented.

proneon267 · 2023-03-26T17:59:33Z

If you run briefcase run -ur, it will perform those steps and avoid the situation. Once #1137 is resolved, I think situations like this will be prevented.

So, what do you suggest we should do with this issue for now?

rmartin16 · 2023-03-26T18:20:51Z

If you run briefcase run -ur, it will perform those steps and avoid the situation. Once #1137 is resolved, I think situations like this will be prevented.

So, what do you suggest we should do with this issue for now?

Well, unless you want to fix #1137 (which I don't think would be appropriate for this PR), we probably shouldn't do anything. Once Briefcase is updated to confirm all of the needed system requirements are installed before making changes to the filesystem, this issue shouldn't happen anymore then.

proneon267 · 2023-03-26T20:11:16Z

Created system_installer_with_install_flag and corrected a condition which was checking system_installer as None, instead of system_verify as None. Also, made the output copy/paste friendly.

src/briefcase/platforms/linux/system.py

…rectly formatted. Co-authored-by: Russell Martin <rmartin16@gmail.com>

freakboy3742 · 2023-03-27T02:23:14Z

@MarkusH Awesome - thanks for that testing. Looks like you might have been bitten by #1137 on your first pass. That's a failure mode that will be particularly common when building system packages, so it's something we're definitely going to need to look at, but it's not an inherent problem with this Arch packaging PR, so we probably won't consider it a blocker on this PR.

On a more general note: the package works, but is it... "right"? i.e., does the way we're packaging the app conform to your expectations as an Arch user? In particular, the fact that we're including a pre-compiled binary on a system that doesn't have a hard concept of dependency pinning (in the sense of Debian/Ubuntu/Fedora that provide binary ABI guarantees for "Bullseye"/"22.04"/"37" et al) seems odd to me, but I also don't have a good read on Arch user expectations.

@rmartin16 Unfortunately, I can't get a working Arch/Manjaro system in my test config (Parallels doesn't have full integration for either AFAICT); I can build packages in Docker, but I can't do a non-Docker build or test the results. It looks like you're able to do runtime tests - are you happy with where this patch is at?

rmartin16 · 2023-03-27T03:20:05Z

@rmartin16 Unfortunately, I can't get a working Arch/Manjaro system in my test config (Parallels doesn't have full integration for either AFAICT); I can build packages in Docker, but I can't do a non-Docker build or test the results. It looks like you're able to do runtime tests - are you happy with where this patch is at?

Yeah - I think so; the only thing that seems off is toga apparently isn't respecting dark mode in Gnome 43 in arch. On my primary machine with Gnome 42, it does respect dark mode. Everything else is working though 🦾

freakboy3742

@proneon267 Thanks for the contribution!

proneon267 · 2023-03-27T03:31:30Z

@freakboy3742
Hello
If you are having problems installing Manjaro/Arch on a virtual machine then check for the following:

Make sure the virtual hard disk is about 64 GB. I know this sounds arbitrary but the installation will fail with some rsync error code, even though it doesn't use anywhere near that much of space.
Also, the installation may at first succeed but it won't be bootable. In that case boot with the live iso, chroot and reinstall and configure grub.
Also there is a high chance that the installation may succeed without any problems on an EFI virtual machine. If you do install it in such a virtual machine, make sure that the efi partition is populated with the grub efi image.
The virtual machine is a 64-bit.
If Parallels isn't working then maybe try virtualbox, it has a developer preview for M1/M2 mac, and i have heard that it works really well and is more configurable.

Also, I am planning to add support for generating a package which could be used for distributing through AUR. I mean it won't be like a traditional package per se, just a tarball containing the PKGBUILD , the source tarball(uncompiled sources) and related things. So, the user can host the source tarball somewhere and add its address to the generated PKGBUILD. The developer would then distribute the PKGBUILD through the AUR. This would then download the source tarball, compile it and then install it. But, that's for a separate PR though.

freakboy3742 · 2023-03-27T03:55:17Z

@proneon267 Thanks for the tips - I am able to get Manjaro installed; the issue is with the Parallels integration with the native system (such as mounting the native system drive as a network drive) which is what allows me to efficiently test multiple platforms concurrently. Moving to virtualbox won't fix those problem, as virtualbox doesn't provide those features (to the best of my knowledge).

As for the AUR package - it sounds like what you're describing is what is called a "source distribution" on Debian/Redhat platforms. That definitely sounds like an option worth adding; I've opened #1152 to track the feature.

If you decide to work on this feature, can I please ask that you do your development work locally, rather than through the Github editing interface. The approach you took on this PR results in lots of individual commits without meaningful commit messages, and causes a lot of CI resources to be wasted.

proneon267 · 2023-03-27T04:06:57Z

Sure, I will do the development work locally this time. Thank you for helping. I couldn't have done it without you.

proneon267 added 2 commits March 20, 2023 14:09

Added code for supporting arch system package

c86418d

Update system.py

2a287fa

proneon267 commented Mar 20, 2023

View reviewed changes

Fixed syntax errors and core logic errors.

920907b

freakboy3742 requested changes Mar 20, 2023

View reviewed changes

proneon267 commented Mar 21, 2023

View reviewed changes

freakboy3742 reviewed Mar 21, 2023

View reviewed changes

proneon267 closed this Mar 21, 2023

proneon267 reopened this Mar 21, 2023

freakboy3742 closed this Mar 21, 2023

freakboy3742 reopened this Mar 21, 2023

freakboy3742 reviewed Mar 21, 2023

View reviewed changes

Update system.py

c206f06

proneon267 commented Mar 22, 2023

View reviewed changes

Update system.py

1e18f1f

Modified towards a working state.

proneon267 commented Mar 22, 2023

View reviewed changes

freakboy3742 added 3 commits March 23, 2023 07:45

Various cleanups and bug fixes.

41e205d

Add changenote.

4a34713

Correct existing test failures.

1eb6ec4

rmartin16 requested changes Mar 26, 2023

View reviewed changes

src/briefcase/platforms/linux/system.py Outdated Show resolved Hide resolved

src/briefcase/platforms/linux/system.py Outdated Show resolved Hide resolved

rmartin16 reviewed Mar 26, 2023

View reviewed changes

src/briefcase/platforms/linux/system.py Outdated Show resolved Hide resolved

rmartin16 reviewed Mar 26, 2023

View reviewed changes

src/briefcase/platforms/linux/system.py Outdated Show resolved Hide resolved

proneon267 and others added 2 commits March 27, 2023 07:01

Rework system installer handling to simplify and ensure output is cor…

dc033ec

…rectly formatted. Co-authored-by: Russell Martin <rmartin16@gmail.com>

Disable stripping on Arch packages.

968b0f0

rmartin16 approved these changes Mar 27, 2023

View reviewed changes

freakboy3742 approved these changes Mar 27, 2023

View reviewed changes

freakboy3742 merged commit 90b3699 into beeware:main Mar 27, 2023

freakboy3742 mentioned this pull request Mar 27, 2023

Add an Arch/Pacman backend for Linux #1064

Closed

proneon267 deleted the patch-1 branch March 27, 2023 19:32

Added code for supporting arch system package #1143

Added code for supporting arch system package #1143

Conversation

proneon267 commented Mar 20, 2023 • edited Loading

Testing instructions:

PR Checklist:

proneon267 left a comment

Choose a reason for hiding this comment

freakboy3742 left a comment

Choose a reason for hiding this comment

proneon267 commented Mar 21, 2023

freakboy3742 commented Mar 21, 2023

proneon267 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

proneon267 Mar 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

proneon267 Mar 21, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

proneon267 commented Mar 21, 2023 • edited Loading

freakboy3742 commented Mar 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

proneon267 left a comment • edited Loading

Choose a reason for hiding this comment

proneon267 commented Mar 22, 2023

freakboy3742 commented Mar 22, 2023

proneon267 left a comment • edited Loading

Choose a reason for hiding this comment

proneon267 commented Mar 22, 2023 • edited Loading

proneon267 commented Mar 22, 2023

proneon267 commented Mar 26, 2023 • edited Loading

freakboy3742 commented Mar 26, 2023

proneon267 commented Mar 26, 2023

MarkusH commented Mar 26, 2023

proneon267 commented Mar 26, 2023

MarkusH commented Mar 26, 2023

proneon267 commented Mar 26, 2023 • edited Loading

MarkusH commented Mar 26, 2023

proneon267 commented Mar 26, 2023 • edited Loading

proneon267 commented Mar 26, 2023

proneon267 commented Mar 26, 2023 • edited Loading

rmartin16 left a comment • edited Loading

Choose a reason for hiding this comment

rmartin16 commented Mar 26, 2023

proneon267 commented Mar 26, 2023

rmartin16 commented Mar 26, 2023

proneon267 commented Mar 26, 2023 • edited Loading

freakboy3742 commented Mar 27, 2023

rmartin16 commented Mar 27, 2023

freakboy3742 left a comment

Choose a reason for hiding this comment

proneon267 commented Mar 27, 2023

freakboy3742 commented Mar 27, 2023

proneon267 commented Mar 27, 2023

proneon267 commented Mar 20, 2023 •

edited

Loading

proneon267 Mar 22, 2023 •

edited

Loading

proneon267 Mar 21, 2023 •

edited

Loading

proneon267 commented Mar 21, 2023 •

edited

Loading

proneon267 left a comment •

edited

Loading

proneon267 left a comment •

edited

Loading

proneon267 commented Mar 22, 2023 •

edited

Loading

proneon267 commented Mar 26, 2023 •

edited

Loading

proneon267 commented Mar 26, 2023 •

edited

Loading

proneon267 commented Mar 26, 2023 •

edited

Loading

proneon267 commented Mar 26, 2023 •

edited

Loading

rmartin16 left a comment •

edited

Loading

proneon267 commented Mar 26, 2023 •

edited

Loading