Packages on GitHub: Point dist URL to release file

[lukasbestle]

I'm currently building a package that is also going to contain front-end code like CSS and JavaScript files. As those need to be built before the package can be used, I'd like to provide a distribution ZIP that is already ready to use and can simply be installed via Composer.

The package is going to be hosted on GitHub. GitHub has a release feature. Each Git tag can be converted to a release, which allows you to upload release files. In my case I will attach the distribution ZIP to the GitHub release.

URL format of the release files

The missing part is that Packagist needs to point to the URL of the release file as the dist of the package.

What Packagist currently uses for GitHub repos is the following URL format:

"dist": {
  "type": "zip",
  "url": "https:\/\/api.github.com\/repos\/my-org\/my-repo\/zipball\/a0ab4f5cdaced5bced734bdca9e9cc24f0e473a5",
  "reference": "a0ab4f5cdaced5bced734bdca9e9cc24f0e473a5",
  "shasum": ""
}

That URL points to the zipball of the commit that corresponds to the respective Git tag.

Instead, I need Packagist to point to the release file. That URL has the following format on GitHub:

https://github.com/my-org/my-repo/releases/download/<name of tag>/<filename>

Implementation in Packagist

Packagist could offer a configuration option on packagist.org for this. By switching it on, you tell Packagist to link to the release file instead of to the source zipball.

As the filename of the release file on GitHub isn't fixed, there would also need to be a field where package authors can set the filename pattern like this:

my-package-{version}.zip

Why this is useful for other packages as well

The currently used URL format (the one that points to the zipball of the commit directly) doesn't really point to the dist, but actually to a ZIP of the source.

I think that this is not what people expect. If they use Composer's --prefer-dist flag, they actually expect to get the dist version without the excluded files and so on. But what they are getting instead is the full source.

By supporting this feature, you can empower package authors to release optimized dist ZIPs of their packages that make installation a lot faster and also easier in case something needs to be built like in my package with the front-end code.

[stof]

They might not get the full source, as your can already configure git about what gets excluded when building an archive (this happens in .gitattributes and the github zip archive respects it (as it uses git under the hood).

I see several issues with this proposal:

• such package would not work when using source install
• such package would not work when using dev versions

IMO, frontend code are better handled using the dedicated ecosystem (i.e. the npm registry) rather than changing the architecture of composer to try to make it handle such package (I'm saying "try" because this requires breaking several features of composer for these packages).

[lukasbestle]

Thanks for the super quick response!

Do you have a recommendation how I can solve this? So far, I only have the following ideas:

• Set up my own Composer repository and point dist to the correct URLs. Source and dev installs are not an issue as I expect those users to build the assets themselves. At least it would work for the releases.
• Commit the dist files into the repo. Not so great.

npm wouldn't really work as my package is still a PHP project. It doesn't really make sense to have that installed in node_modules.

[stof]

@lukasbestle what I meant with npm is that you could provide the frontend code through npm (while still providing the PHP code through composer). The drawback would be that people could be installing separate versions though.

the 2 ideas you have are the 2 I'm thinking of as well.

[lukasbestle]

Splitting it up doesn't work for my use-case. What I'm publishing is not a library but a CMS that ships with its own assets for the admin panel.

I have tried the "custom Composer repo" approach and it worked great while testing, but it turned out to be too complex in the long run. So I have now decided to commit the dist files to the repo on each release.

I'm still leaving this issue open because I think that this feature would be useful to make the workflow for this kind of package a lot simpler.

[rasteiner]

I have exactly the same use case. My end users are not javascript developers, they don't necessarily have node or npm installed.

Keeping the transpiled front end assets in the dist makes maintaining the package harder: every change in the source code is bundled into 1 file, this creates unnecessary conflicts on that file for every merge. You also need to remember to always include the new builds.

It's basically the same request as this one #270

[stof]

@rasteiner see the drawbacks listed in #903 (comment)
Having packages working only for dist installs of a release, but not for dev versions or source installs would not be good for your users either.

[lukasbestle]

But isn't it better if a package works in release versions than if it doesn't work at all?

If someone installs a dev or source version, it can be expected that the user will be able to compile the dist files themselves.

The only other way would be to commit the dist files to the repo. But for that to work on source and dev versions (what you consider to be essential), the dist files would need to be rebuilt on every single commit. As stated multiple times here and in #270, that just won't work if pull requests and multiple branches are involved because of the resulting merge conflicts.

I think it makes sense to give people options in places where there is no "right or wrong". Both ways can make sense, but for some packages a custom package dist URL would make a lot more sense than any other option.

[stof]

@lukasbestle users will expect that if they switch versions, things will work the same for them. Adding minimum-stability: dev in your composer.json or @dev in the constraint is easy, and does not involve changing the way the package is used generally.

[lukasbestle]

That makes sense. :)

What would be your recommendation for such packages? As I wrote, committing dist files with every single commit isn't an option. But maybe there are other options.

[rasteiner]

Pardon my ignorance. I don't really see the connection between this issue and version constraints.

In my case, every tagged commit would have a build release. If users chose to require a -dev tag, they still could do so, it would simply download the build artifact I know would be better for them instead of the source code.

In other words, the artifact url would still need to be specified somewhere (wouldn't be easy to infer the correct one anyway). Probably this would be in the package's composer.json. Composer would download whatever artifact was specified at any version, branch, commit or tag.

It's the maintainers responsibility to keep that link up to date.

[lukasbestle]

But that doesn't solve the issue of source installs, which come directly from Git.

[rasteiner]

If "source installs" means people installing stuff with "--prefer-source": I would be happy to serve them the git repo without dist files.
At that point, since those installs are explicitly meant to be used when working on a package, users would be expected to be able to compile dist files themselves.

Even at that point, it wouldn't be any worse than the situation we're in now.

[lukasbestle]

Me too actually. I just wanted to clarify that to avoid confusion. :)

[rasteiner]

For @lukasbestle and everyone else in the same situation:

for me the best (only) solution is to have CI push the dist files of tagged commits to a separate repository that is used only for composer.

[Seldaek]

Closing this as it's not something we want to support. It breaks too many assumptions we currently have, and isn't really a widespread need in the PHP world.

[pfrenssen]

The composer-artifacts plugin allows to download custom tarballs.

[roborourke]

@pfrenssen I couldn't get that working unfortunately.

[nikeee]

I'm facing a similar issue: I'd like to ship natively compiled .dll/.dynlib/.so files with my composer package, so I can interface with them using FFI.
These libraries would be compiled for multiple architectures (arm/x86 etc). Since I cannot expect a user to install an entire native cross-compiling toolchain (this can get quite complex), I'd like to ship the binaries with the package.

Is it possible today (even if this means that I have to modify some entry in the composer.json)?
Are there alternatives?

[stof]

There is no support for that, which is partially related to the fact that you can install dev versions with composer as well, not only releases (btw, composer actually cares about tags, not about github releases).

An alternative is to have a separate repo in which you push the output of your build (and then you publish that repo for composer usage). For instance, phpstan does that: https://github.com/phpstan/phpstan-src contains the source code (where development happens) and the CI is building the phar file and pushing it to https://github.com/phpstan/phpstan (which is then the source of the phpstan/phpstan package on packagist.org)

[nikeee]

So it would be possible to have an orphan branch in the same repo, publish builds to that branch and put tags on the files in that branch?

[stof]

Well, having it in the main repo would likely create confusion because non-build branches would appear on packagist but would not have the files in them. So a separate repo would provide a better experience. But if you are OK with the fact that packagist will show non-working dev versions, you could indeed use an orphan branch in the same repo.

Side note: if you keep separate repos, you could also create tags in the original repo, allowing you to see easily which commit belongs to which release in the git history with which you will work as a contributor or maintainer.

[nikeee]

Yes, doing it in a separate rpository seems reasonable. I just wanted to know whether I understood the implications correctly.