Composer lock file support

[rarkins]

No description provided.

[swashata]

Hi,

Is this feature ready and available?

[rarkins]

No. Only the composer.json file is updated currently. This issue will be updated/closed once lock file support is added.

[swashata]

Great. Thank you for letting me know.

[swashata]

I have a question. Right now, the app provides upgrading composer.json file. When this is done, and then if we run composer update then we do get an updated composer.lock file?

Can't it be done this way? If so, how can we enable the composer through app (self hosted). I would like to give it a try.

Also any hints/tips on which files I need to look, if I were to try implement this?

[rarkins]

Yes, the short version is that we need to run composer after updating the composer.json in order to get an updated composer.lock file. I'm not sure though how many "gotchas" there are. One potential gotcha is the composer version, however if you are running self-hosted then you could make sure to install the version you want anyway.

I'm not 100% sure if we should be running composer update or composer install. Typically we like to update the package file (composer.json manually ourselves) and then use the CLI tool for lock file only. So potentially we should be running composer install to generate a correct lock file for our composer.json instead of composer update x.

Another thing I'm not sure about is which files to write locally before running composer. e.g. should we write the old composer.lock and the new composer.json? Or is the old composer.lock ignored anyway?

Right now npm is the only package manager we support lock files for, and it's kind of hardcoded (here). But I can un-hardcode that pretty quickly.

Probably the best way to move this forward is if you can help me research/plan the best way to update the composer.lock files (e.g. what I discussed above). If there is a firm set of requirements then I am much more likely to be able to implement it soon.

[swashata]

Hi,

composer install behaves a little different than yarn install or npm install. When you run composer install and if

1. There is a composer.lock file, then it looks into the lock file to get the dependency and not at all to the composer.json file.
2. If there isn't, then it simply resolves the dependencies and creates a lock file.

So what it means is, if you already have a composer.lock file and you update the semver of composer.json file or change the range and then you run composer install, then it will NOT update the composer.lock file at all. Instead it will show a notice.

Warning: The lock file is not up to date with the latest changes in composer.json. You may be getting outdated dependencies. Run update to update them.

So, composer install can be compared with yarn install --frozen-lockfile, where it installs everything in the lock file and doesn't rely on composer.json file.

Now you have to run composer update which will look at composer.json file, compare with composer.lock file for semver and update as necessary.

Say I have "package": "~1.0.11". Through renovateapp, it gets updated to "package": "~1.0.12" (assuming 1.0.12 is the latest release). After that, to actually update the lock file, we have to run composer update` (without any other arguments).

If we had run composer update without letting renovateapp updating the composer.json file, then also it would have done the same.

But the advantage here is definitely, being able to update to a major version.

Here is how you can test it out.

Create an empty project with composer init and run them in the following. The package can be found here.

composer require symfony/console:~4.0.5

./composer.json has been updated
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 2 installs, 0 updates, 0 removals
  - Installing symfony/polyfill-mbstring (v1.8.0): Loading from cache
  - Installing symfony/console (v4.0.12): Loading from cache
symfony/console suggests installing psr/log-implementation (For using the console logger)
symfony/console suggests installing symfony/event-dispatcher ()
symfony/console suggests installing symfony/lock ()
symfony/console suggests installing symfony/process ()
Writing lock file
Generating autoload files

Notice that it actually installs 4.0.12 although, in the composer.json file, it leaves ~4.0.5, which should be the behavior of semver.

Now edit the composer.json file and replace ~4.0.5 with ~4.1.0.

Run

composer install

and you will get

Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Warning: The lock file is not up to date with the latest changes in composer.json. You may be getting outdated dependencies. Run update to update them.
Nothing to install or update
Generating autoload files

So you run

composer update

and you get

Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 0 installs, 1 update, 0 removals
  - Updating symfony/console (v4.0.12 => v4.1.1): Loading from cache
Writing lock file
Generating autoload files

Notice, here also composer correctly installed 4.1.1, since we had specified ~4.1.0.

If I am not mistaken, then I believe, this has always been the behavior of composer dependency resolution from the early age. So in order to update the composer.lock file, we do need to run the cli composer update after renovateapp has done its magic.

So here's a question. Right now, how do you update package-lock.json and yarn.lock files. Is this done within the renovateapp, or through npm install and/or yarn install?

And to answer your confusions

I'm not 100% sure if we should be running composer update or composer install. Typically we like to update the package file (composer.json manually ourselves) and then use the CLI tool for lock file only. So potentially we should be running composer install to generate a correct lock file for our composer.json instead of composer update x.

1. Yes, update the composer.json manually ourselves.
2. Then run composer update which will look into the composer.json file and update the lock file.

I think the confusing part is, the corresponding commands of yarn or npm. (I do not use npm much, so I can't help here).

To generate lock files of manually updated versions inside .json file.

yarn install | npm install -> composer update.

To install everything inside lock file and not update it

yarn install --frozen-lockfile -> composer install.

Another thing I'm not sure about is which files to write locally before running composer. e.g. should we write the old composer.lock and the new composer.json? Or is the old composer.lock ignored anyway?

1. Write new composer.json.
2. Write old composer.lock.
3. Run composer update.

Theoretically, we can ignore composer.lock file, but there isn't any harm copying it. It resembles the usual flow.

Also one caveat could be, if renovateapp didn't update version in composer.json, but during composer update, it finds a range update, then it will do something we didn't mean to do. For that we can approach like this:

1. During manual composer.json update procedure, take a list of ['vendor/package', 'vendor/package2'] and hold it somewhere.
2. While constructing the composer update command

const updatedPackages = someHowWeGetUpdatedPackages();
const composerCli = `composer update ${updatedPackages.join(' ')}`;

Although since renovateapp is meant to take care of this, I don't think the above overhead is necessary.

[rarkins]

@swashata this is exactly the type of knowledge I was after - thank you!

At the end you illustrate the last problem I was worrying about - that composer update might go ahead and update all packages even though we don't want it to.

Let's use a simple case:

1. We have two pinned dependencies dep-a and dep-b in composer.json, both are out of date by a patch release
2. We update dep-a's version in composer.json, but not dep-b's.

From your description I think the problem is that if we just run composer update then both dep-a and dep-b will get updated? i.e. composer will update not only the composer.lock for dep-b but also update both composer.json and composer.lock for dep-b too? But if we run composer update dep-a then it will only update dep-a in composer.lock as we'd like?

[swashata]

Hello,

If we have a pinned dependency in composer.json, then it would look something like this

"vendor/package": "1.0.0", no range sign here, i.e, not something "vendor/package": "~1.0.0".

In this case, composer update will not update that dependency, even if some patch release is there.

But is this what you are asking? If by pinned dependency you mean, something pinned by renovateapp config (I have just started using this morning, so not very good at this right now), then definitely composer update will update the dependency in composer.lock file.

One thing I would like to focus here is, composer update does not change anything in composer.json file, only which are necessary in composer.lock file.

Say you have symfony/console installed with ~4.0.0. If you do composer update symfony/console, the it will change the composer.lock file to 4.0.12, but will not update composer.json file to ~4.0.12. That's the way it works.

If theoratically, you would want to use composer to update a package in composer.json then you don't use composer update. You would simply do composer require vendor/package:~x.x.x.

Please check the output of the screenshot below and you will understand what I am saying.

[swashata]

And to quote your questions directly

From your description I think the problem is that if we just run composer update then both dep-a and dep-b will get updated? i.e. composer will update not only the composer.lock for dep-b but also update both composer.json and composer.lock for dep-b too?

Yes, both dep-a and dep-b will get updated, but only in composer.lock file. The composer.json file is never touched by composer update.

But if we run composer update dep-a then it will only update dep-a in composer.lock as we'd like?

And yes. It will only update the resolution for dep-a, and we would like it.

[swashata]

After using renovateapp a bit, I've found that it does update one package at a time creating MR. I was under the impression that it will send a single MR updating all the stuff.

In this case, it is definitely needed to run composer update dep-a as you've eloquently put it.

[swashata]

Now, if you help me understand how you are firing the lock file update of yarn and npm, then I can do the same for composer.

[rarkins]

Thanks again @swashata

In summary, you could say that my main concern was that composer would update dependencies that we don't intend to. e.g. Renovate tells you "Update dep-a to v1.1.0" but actually composer ends up updating everything else too. With your descriptions, it sounds like we can avoid that.

I need to think a little bit first about the most appropriate way to incorporate composer lock files. Right now the logic for npm is hardcoded but also the logic for npm is very complex so we don't necessarily need to do the same for composer. e.g. if Lerna or Yarn Workspaces are in use then we need to do a lot of work to correlate different package.json files together but in this case each composer.json can simply be updated independently. Stay tuned!

[swashata]

Great. Do let me know if I could be of help.

[rarkins]

@swashata see #2255

It's a little hacky but seems to work

[swashata]

Great. I will check this out locally.

[renovate-bot]

🎉 This issue has been resolved in version 13.23.0 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀