Feature: expose repo as composer package.

[jeppech]

I've seen over at GitLab that someone is in the making of Composer Service, that makes it possible to expose a repository, as a Composer package for PHP.

That would be a pretty cool feature, if you'd like to host your own private packages. Atm, it's possible to have a static package repo, by using the static repository generator Satis.

Furthermore, support for npm packages or ruby gems, could also be possible.

Actually when I'm thinking about it, these features might not belong to the Gogs core, but maybe as some sort of plugins/add-ons, as mentioned in #2438

[unknwon]

Indeed, sounds like a good use case of #2438.

BTW, how does composer package of PHP work? Never used PHP... sorry.

[jeppech]

Heh, no need to be sorry :)

Basically every, project kan be a composer package. It just need to have composer.json file in the root folder structure. The composer file, describes the project and its dependencies. The very basics for requiring some dependencies, could be something like this:

{
    "repositories": [
        {
            "type": "composer",
            "url": "http://try.gogs.io"
        }
    ],
    "require": {
        "monolog/monolog": "~1.17.0"
    }
}

To install the above dependency, you'll type composer install, and the Composer CLI tool, will look for a package.json in the root of http://try.gogs.io, this file tells which packages are available and where they're located. A very basic version of this could be

{
    "packages": {
        "monolog/monolog": {
            "v1.17.0": {
                "name": "monolog/monolog",
                "version": "v1.17.0",
                "version_normalized": "1.17.0.0",
                "source": {
                    "type": "git",
                    "url": "http://try.gogs.io/monolog/monolog",
                    "reference": "d7c2fa6f5aa9ac73724c234c62f7f2d83035a7bd"
                },
                "dist": {
                    "type": "tar",
                    "url": "http://try.gogs.io/monolog/monolog/archive/v1.17.0.tar.gz"
                },
                "time": "2015-02-10 19:36:36",
                "type": "library"
            }
        }
    }
}

Actually at this point, I'm kinda uncertain how this file is generated, because the official repo, Packagist, has a search key with a value of /search.json?q=%query% in their packages.json which is undocumented.

What I've found so far, is that if you for instance search for monolog https://packagist.org/search.json?q=monolog it returns a list of packages i.e

{
    "results": [
        {
            "name": "monolog/monolog",
            "description": "Sends your logs to files, sockets, inboxes, databases and various web services",
            "url": "https://packagist.org/packages/monolog/monolog",
            "repository": "https://github.com/Seldaek/monolog",
            "downloads": 22744596,
            "favers": 4443
        }
    ]
}

This result set, contains a key url, which points to the location of the package in packagist. And I noticed if you prepend .json to that url like this https://packagist.org/packages/monolog/monolog.json it actually shows, what would correspond to an entry of the before mentioned packages.json

Further reading for the packages.json can be found here

[bkcsoft]

This would indeed be a valid use-case for plugins. The /search.json you mention is (at least) referenced here https://packagist.org/apidoc . Though it isn't much help 😒

[jeppech]

I think I figured out what it's used for, when executing composer -vvv search monolog/monolog you can see that the composer app actually looks up /search.json?q=monolog/monolog, to retrieve some info:

jc@mbp $ composer -vvv search monolog/monolog
You are running composer with xdebug enabled. This has a major impact on runtime performance. See https://getcomposer.org/xdebug
Loading config file /Users/jc/.composer/auth.json
No composer.json found in the current directory, showing packages from packagist
Downloading https://packagist.org/packages.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/packages.json into cache
Downloading https://packagist.org/search.json?q=monolog/monolog
monolog/monolog Sends your logs to files, sockets, inboxes, databases and various web services
monolog/monolog
kdyby/monolog Integration of Monolog into Nette Framework
stackify/monolog Stackify logs and errors for Monolog
thinframe/monolog Monolog Application
cakephp/monolog CakePHP Monolog Plugin
flowpack/monolog Monolog integration for Flow
symfony/monolog-bundle Symfony MonologBundle
symfony/monolog-bridge Symfony Monolog Bridge
kamisama/monolog-init Very basic and light Dependency Injector Container for Monolog
logentries/logentries-monolog-handler A handler for Monolog that sends messages to Logentries.com.
m6web/monolog-extra-bundle Provide extra features for Monolog
wazaari/monolog-mysql A handler for Monolog that sends messages to MySQL
enlitepro/enlite-monolog Monolog integration to Zend Framework 2
flynsarmy/slim-monolog Monolog logging support Slim Framework

But when requiring the actual package it does not use the search method

jc@mbp $ composer -vvv search monolog/monolog
Reading ./composer.json
Loading config file /Users/jc/.composer/auth.json
Loading config file ./composer.json
Executing command (/Users/jc): git describe --exact-match --tags
Executing command (/Users/jc): git branch --no-color --no-abbrev -v
Executing command (/Users/jc): hg branch
Executing command (/Users/jc): svn info --xml
Failed to initialize global composer: Composer could not find the config file: /Users/jc/.composer/composer.json
To initialize a project, please create a composer.json file as described in the https://getcomposer.org/ "Getting Started" section
Downloading https://packagist.org/packages.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/packages.json into cache
Downloading http://packagist.org/p/provider-2013%24ff90a5caee747457233c7927b1a81797850de6779661815ac4c7663b3b7d6eda.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/p-provider-2013.json into cache
Downloading http://packagist.org/p/provider-2014%248776bf44e4162545d15610ae7b6d085218b489d66f5192a8725a59a3bab9f13a.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/p-provider-2014.json into cache
Downloading http://packagist.org/p/provider-2015%24353f1f241024bc840a505bc1281aee94ae0d7953a8e8e2cab24154d06f996cf0.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/p-provider-2015.json into cache
Downloading http://packagist.org/p/provider-2015-04%24d238240864b8325773ac496b3daaf9bc4481376eb2d5d55715fdb9ee50b4c8f8.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/p-provider-2015-04.json into cache
Downloading http://packagist.org/p/provider-2015-07%24d295bda30c49ee368869484582a264579d1a7f33095651b999b5d599277967bd.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/p-provider-2015-07.json into cache
Downloading http://packagist.org/p/provider-2015-10%243cc3c0be7fab051132a873b60ffb687f3eb403a63058a893f087295dc3599552.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/p-provider-2015-10.json into cache
Downloading http://packagist.org/p/provider-2016-01%24fe5a870932496d05a6cf523b842f9497898ba9cad673d10d6e6a7330392cc9fd.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/p-provider-2016-01.json into cache
Downloading http://packagist.org/p/provider-archived%24a1dda04e44e120babb82f2449ab01bc68a2a2bc0b1483498b4959bf1305e2b4b.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/p-provider-archived.json into cache
Downloading http://packagist.org/p/provider-latest%24e32f2df453895917a4dc11b74275f18b62255230d0a20464b50f87dc44dfcd81.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/p-provider-latest.json into cache
Downloading http://packagist.org/p/monolog/monolog%24590f46ce86d4f72467a792cf3a4fadcad266eb527ac9aabf9cb2048737fd07b2.json
Writing /Users/jc/.composer/cache/repo/https---packagist.org/provider-monolog$monolog.json into cache
Using version ^1.17 for monolog/monolog
./composer.json has been updated

You'll see that it downloads all those references it can find in the packages.json, and probably parsing them until it finds a match.

[unknwon]

Thanks for the great details!

So... what exactly is missing from Gogs side?

[unknwon]

ping @jeppech

[jeppech]

Well, I think this should be implemented as a plugin. So this might, should be put on hold, until the plugin module for Gogs is ready?

[unknwon]

@jeppech yes. But hmm, from my zero knowledge of PHP, what is missing from Gogs side right now?

[jeppech]

Oh, like that. Well, you don't have to know any PHP at all, this is just a package manager for PHP. I don't see Gogs missing anything, when speaking of the composer integration, besides the plugin system of course.

As of now, I'm not very familiar with the composer internal system, so I'll have to read up on that. I'm unsure whether the composer system should be accessible to Gogs, in order to make the 'package.json', or if we can generate that our self.

So, for now, nothing further is needed in Gogs, besides the plugin module.

[unknwon]

I'm getting more confused....

[jeppech]

Uhm well, didn't we agree on that this was a good use case, for a plugin earlier?

[unknwon]

Yeah.... will look back again when we have plugin... 0.0

[jeppech]

Yeah, sorry for confusing you!

[X-Coder]

I like the idea of having composer support in gogs which would allow to use gogs as a private/public composer repository similar to packagist.

I have tested the composer wip implementation for gitlab, jeppech mentioned (https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/1921) which works well for me.
Additionally the gitlab implementation generates, if a repository has no composer support, because it has no composer.json, a virtual composer.json on the fly, so the repository can still be included and exposed as composer package, even if has no direct composer support. This would be a nice to have bonus too for gogs.

Basically, all gogs has to do, is parsing the root of each repository, read the contents of composer.json, and merge it into a global packages.json (don't forget the to handle all the versions). If a repository was updated and the version / composer.json has changed, the package.json needs to be updated too to include the new version.

Better method of implementing this, is using the documented provider-include-method which works better for large repositories like gogs. The packages.json and provider-include-format is well documented at https://getcomposer.org/doc/05-repositories.md#packages and https://getcomposer.org/doc/05-repositories.md#includes.

You can see on packagist.org how it was implemented:
https://packagist.org/packages.json

In gogs there should be a repository based setting, to allow the inclusion of the repository in the global packages.json, because maybe someone don't want to expose his repository in the global packages.json.

[unknwon]

@X-Coder thanks for the details!

[fengqi]

Like webhook,
When there is a push action, send the request to packagist,

The api is /api/update-package, default domain is https://packagist.org, and required parameter is:

• username: In packagist username
• apiToken: packagist API token, you can find on your profile page.
• payload: a string of json, example: {"repository":{"url":"https://try.gogs.io/fengqi/compsoertest.git"}}

when the packagist is processed it will be returned:

{
  "status": "success"
}

it's mean packagist update package complete, then https://packagist.org/packages/fengqi/compsoertest will be up to date: