"do not match its signature"

[Stef-33560]

Hi,

I've mirrored a packagist repo sucessfully

$php webysthe-packagist-mirror/bin/mirror create -vvv
Loading providers from https://packagist.hesse.im/
packages.json updated

My private NGINX server shows me on https://private.repo.lan/packages.json :
{ "packages": [], "notify-batch": "https:\/\/packagist.org\/downloads\/", "providers-url": "\/p\/%package%$%hash%.json", "search": "https:\/\/packagist.org\/search.json?q=%query%&type=%type%", "list": "https:\/\/packagist.org\/packages\/list.json", "providers-api": "https:\/\/packagist.org\/providers\/%package%.json", "warning": "You are using an outdated version of Composer. Composer 2 is now available and you should upgrade. See https:\/\/getcomposer.org\/2", "warning-versions": "<1.99", "provider-includes": { "p\/provider-2013$%hash%.json": { "sha256": "d0dc77fd03e5732f1bdb80037c61df42ed94764bddb5f24774761c59739791cc" }, "p\/provider-2014$%hash%.json": { "sha256": "ab43f689a426163c7bd97eee1359d66270cf969532a976b03d4d4e608395b25c" }, "p\/provider-2015$%hash%.json": { "sha256": "5dd99e47c377393321c91a2b7cede7c1dff20f23728c89dd551c20543750cf5e" }, "p\/provider-2016$%hash%.json": { "sha256": "7b8f1749c41d2c755e25992d763588fa527308b20f53fb15523c5a938d960bc4" }, "p\/provider-2017$%hash%.json": { "sha256": "62c94621ae4a052b1d7e00cb9398703ef09bc8f69fe8fce22e309a831b6011f9" }, "p\/provider-2018$%hash%.json": { "sha256": "df4e171e392f024639a924670d6c2a243a645277d43fb3b141cef9efe7b9df12" }, "p\/provider-2019$%hash%.json": { "sha256": "166db562306e9fd8967eb173a9dbf601c65ce4a2e67bf3d18c132730f1f1f6f9" }, "p\/provider-2020$%hash%.json": { "sha256": "ff7e5318ac974ec8ce64a1d0232f038f9fbac3d987a2146719ff31c6fc578d40" }, "p\/provider-2020-07$%hash%.json": { "sha256": "22517059027c992f913d337d6b79f67b1f3305c1d969e99bfd8ec859e54d9866" }, "p\/provider-2020-10$%hash%.json": { "sha256": "9d48effa58bf64e0764d894494902ffdee59924ad8f63fe1126b315601a2cb9b" }, "p\/provider-2021-01$%hash%.json": { "sha256": "0d37334fff077049a79655aa7f4ecd4ad7a4c01b66a4fe4b24c8aa6170ae5789" }, "p\/provider-2021-04$%hash%.json": { "sha256": "070fc16ec215d0422e7791c4fb0e1279d00704fecaac221863b56f0d68bb3514" }, "p\/provider-archived$%hash%.json": { "sha256": "60d5f896e444581ddd4baae5eaf7f04b63ec618d00f1d8ddf3af1fcf41264015" }, "p\/provider-latest$%hash%.json": { "sha256": "be8f9b5a0f6e46b14e976d6f56ae297509644bbd11916bd90d6894aea4946d27" } } }

But when requiring diffferent libs, I got each time an error ; example with drush

$ composer require drush/drush

Using version ^10.5 for drush/drush
./composer.json has been created
Running composer update drush/drush
Loading composer repositories with package information
Installation failed, deleting ./composer.json.
[Composer\Repository\RepositorySecurityException]
The contents of https://private.repo.lan/p/consolidation/site-process%24f510087614b0f459ed763ade5338c9bbade5e69b1b574df465e4e13105e11643.json do not match its signature. This could indicate a man-in-the-middle attack or e.g. antivirus software corrupting files. Try running composer again and report this if you think it is a mistake.
require [--dev] [--dry-run] [--prefer-source] [--prefer-dist] [--prefer-install PREFER-INSTALL] [--fixed] [--no-suggest] [--no-progress] [--no-update] [--no-install] [--no-scripts] [--update-no-dev] [-w|--update-with-dependencies] [-W|--update-with-all-dependencies] [--with-dependencies] [--with-all-dependencies] [--ignore-platform-req IGNORE-PLATFORM-REQ] [--ignore-platform-reqs] [--prefer-stable] [--prefer-lowest] [--sort-packages] [-o|--optimize-autoloader] [-a|--classmap-authoritative] [--apcu-autoloader] [--apcu-autoloader-prefix APCU-AUTOLOADER-PREFIX] [--] []...

What's wrong with my mirror ??
Thanks a lot :)

[peter279k]

It seems that signature is not valid for original Composer mirror.

You may sync local mirror from original mirror again.

If it's not worked, I think you should consider using the composer/mirror.

[Stef-33560]

Alas, it does not work :(

I've considered composer/mirror and even more your kickstarter script, but I do not have the same result :-/

With this script, I got this :

-rw-rw-r-- 1 me me 8307 juin 6 17:22 apple-touch-icon.png
-rw-rw-r-- 1 me me 8307 juin 6 17:22 apple-touch-icon-precomposed.png
-rw-rw-r-- 1 me me 20603 juin 6 17:22 favicon.ico
-rw-rw-r-- 1 me me 14 juin 5 22:26 .gitignore
-rw-rw-r-- 1 me me 9297 juin 7 22:33 index.html
-rw-r--r-- 1 root root 20 juin 5 22:49 info.php
-rw-rw-r-- 1 me me 14437 juin 6 17:22 logo.drawio
-rw-rw-r-- 1 me me 52764 juin 6 17:22 logo.svg
-rw-rw-r-- 1 me me 247878 juin 6 17:22 mirror-creation.gif
-rw-rw-r-- 1 me me 16178 juin 6 17:22 network.drawio
-rw-rw-r-- 1 me me 85986 juin 6 17:22 network.svg
drwxr-xr-x 1 me me 2871296 juin 7 22:17 p/
lrwxrwxrwx 1 me me 16 juin 7 22:17 packages.json -> packages.json.gz
-rw-rw-r-- 1 me me 973 juin 7 22:17 packages.json.gz
-rw-rw-r-- 1 me me 106 juin 6 17:22 robots.txt
-rw-rw-r-- 1 me me 8930 juin 6 17:22 touch-icon-192x192.png
-rw-rw-r-- 1 me me 1106464 juin 6 17:22 world_map.svg

And with composer/mirror only this :

drwxr-xr-x 4 www-data www-data 4096 juin 8 21:53 ./
drwxr-xr-x 13 root root 4096 juin 7 22:41 ../
drwxrwxr-x 4 www-data www-data 4096 juin 8 08:19 mirror/
drwxrwxr-x 1 www-data www-data 2654208 juin 8 22:11 p2/

How did you achieve to get the other files on root folder to make the mirror usable ??

[peter279k]

Do you want to have the p1 folder?

The p1 folder is for the Composer V1 and p2 folder is for Composer V2.

If you've the p1 folder is presented, please change has_v1_mirror to be true.

More details about mirror setting can be found here.

[Stef-33560]

Well, it seems that a packages.json on the root folder is mandatory to make it work throught a nginx vhost ?
In any case using

"local": {
            "type": "composer",
            "url": "https://mirror-composer.lan",
	    "options": {
		"ssl": {
                    "verify_peer": false,
                    "allow_self_signed": true,
                }
	    }
        }

misses me this file !

Edit : Using a composer.json with

"repositories": {
	"folder": {
	    "type": "path",
            "url": "/var/repos/mirror/p2"
	}
    }

works like a charm, but is quite inadapted for my needs (share an offlline repo for a drupal team offline from Internet, not on the same site)

[peter279k]

It seems that your issue is very similar with composer/mirror#10.

[peter279k]

Just notice that the Composer mirror is not synced correctly if you miss the pckages.json file on the mirror root directory.

Here is the normal/correct captured picture:

[bertploeger]

I have the same problem. I dig into it and i found that some json.gz files are double-gzipped.
gunzipping it twice produces the original json.

I am looking into this. Maybee it has something to do with Gzip.php and this code:

    public function isGzip(string $gzip):bool
    {
        if (mb_strpos($gzip, "\x1f"."\x8b"."\x08") === 0) {
            return true;
        }

        return false;
    }

When i find the issue i will report back.

[bertploeger]

I have the same problem. I dig into it and i found that some json.gz files are double-gzipped.
gunzipping it twice produces the original json.

I am looking into this. Maybee it has something to do with Gzip.php and this code:

    public function isGzip(string $gzip):bool
    {
        if (mb_strpos($gzip, "\x1f"."\x8b"."\x08") === 0) {
            return true;
        }

        return false;
    }

When i find the issue i will report back.

I tested it with the specific double gzipped package.json file, in my case polyfill-php54$f9df1f5a1df7b1d216477f306bbeb226f4afb19b380cac394cf8456fea97127f.json.gz.

I gunzipped it once, then ran this test script:

$gzip = file_get_contents("php54$f9df1f5a1df7b1d216477f306bbeb226f4afb19b380cac394cf8456fea97127f.json");
if (mb_strpos($gzip, "\x1f"."\x8b"."\x08") === 0) {
    echo "gzip";
}else {
    echo "no gzip";
}

The output is "gzip", so it correctly detects whether a file is downloaded as gzip from the source mirror.
I will dig deeper to see if there is any reason i can find in the code a file is double gzipped.

[bertploeger]

I have the same problem. I dig into it and i found that some json.gz files are double-gzipped.
gunzipping it twice produces the original json.
I am looking into this. Maybee it has something to do with Gzip.php and this code:

    public function isGzip(string $gzip):bool
    {
        if (mb_strpos($gzip, "\x1f"."\x8b"."\x08") === 0) {
            return true;
        }

        return false;
    }

When i find the issue i will report back.

I tested it with the specific double gzipped package.json file, in my case polyfill-php54$f9df1f5a1df7b1d216477f306bbeb226f4afb19b380cac394cf8456fea97127f.json.gz.

I gunzipped it once, then ran this test script:

$gzip = file_get_contents("php54$f9df1f5a1df7b1d216477f306bbeb226f4afb19b380cac394cf8456fea97127f.json");
if (mb_strpos($gzip, "\x1f"."\x8b"."\x08") === 0) {
    echo "gzip";
}else {
    echo "no gzip";
}

The output is "gzip", so it correctly detects whether a file is downloaded as gzip from the source mirror.
I will dig deeper to see if there is any reason i can find in the code a file is double gzipped.

Strange thing was that half an hour later php54$f9df1f5a1df7b1d216477f306bbeb226f4afb19b380cac394cf8456fea97127f.json.gz was nog longer available and replaced with another hash after the $ sign. Maybe files are double gzipped at the source?

[Stef-33560]

@bertploeger I did not experienced double-zipped files... but may you check if my PR composer/mirror #11 solves the problem for you too ?

[webysther]

Hey @Stef-33560 you found the problem here? I will test the drush and tell you soon.

[Stef-33560]

Hi @webysther

Still broken with your mirror script :(

git clone https://github.com/webysther/packagist-mirror
cat .env|grep -v '^$'|grep -v '^#'

APP_NAME='Packagist Mirror'
APP_COUNTRY_NAME='France'
APP_COUNTRY_CODE='fr'
PUBLIC_DIR=./public
SLEEP=300
MAINTAINER_MIRROR='Webysther'
MAINTAINER_PROFILE='https://github.com/Webysther'
MAINTAINER_REPO='https://github.com/Webysther/packagist-mirror'
MAINTAINER_LICENSE='MIT License'
MAIN_MIRROR=https://repo.packagist.org
URI_PATTERN='p/%s$%s.json'
TZ='Europe/Paris'
DATA_MIRROR=https://packagist.fr/,https://packagist.co.za,https://mirrors.aliyun.com/composer,https://packagist.mirrors.sjtug.sjtu.edu.cn,https://php.cnpkg.org,https://mirrors.cloud.tencent.com/composer,https://packagist.jp,https://packagist.kr,https://packagist.com.br,https://packagist.phpindonesia.id,https://repo-eu-uk-1.packagist.org
URL=packagist.lan
SINCE="Q2'21"
MAX_CONNECTIONS=15
GOOGLE_ANALYTICS_ID=
GOOGLE_ANALYTICS_MAIN_ID='UA-58875124-2'

php bin/mirror create -vvv

some 404 but it seems going well, serving public dir is OK

mkdir test
composer init
composer config -g repos.packagist composer https://packagist.lan
composer require phpunit/phpunit

Installation failed, reverting ./composer.json and ./composer.lock to their original content.
[Composer\Repository\RepositorySecurityException]
The contents of https://packagist.lan/p/doctrine/instantiator%2443f853a801f01284cb5b4bb83b105da84bc535029cad74afa4690b6b9719088c.json do not match its signature.
This could indicate a man-in-the-middle attack or e.g. antivirus software corrupting files.
Try running composer again and report this if you think it is a mistake.

[Stef-33560]

@bertploeger: RFC 1952 suggests testing only 2 first fields

These have the fixed values ID1 = 31 (0x1f, \037), ID2 = 139 (0x8b, \213), to identify the file as being in gzip format.

But that does not change a lot of things.

I've launched again a new sync at 11:30 PM UTC

composer require phpunit/phpunit

gives :(

The contents of https://packagist.lan/p/phpunit/php-timer%24ed59f9ff1b99c0f19b427ff838c7a1bb3f5935ceb34d33f04a72e5cda97d40f5.json