Locally developed packages are not taken into account when resolving http dependencies

[zah]

Consider the following scenario:

1. You check out a project having a nimble file specifying some of its dependencies as a HTTP URLs.

2. You want to fix an issue in one of the HTTP dependencies, so you clone the this package locally and you run nimble develop to make it available system-wide. You fix the issue and you delete any previously downloaded version from your ~/.nimble/pkgs folder.

3. You run nimble test in the original repo, cloned in step 1. At this point, Nimble doesn't recognize your local copy of the dependent package and instead downloads a fresh copy from the specified HTTP URL. Your local fixes are ignored in favor of the downloaded unmodified copy.

Expected outcome:

Nimble should be able to recognize that a locally linked version of a package (with nimble develop) satisfies a HTTP dependency for the same package.

Possible solutions:

1. The problem is that Nimble cannot figure out that a particular URL used as a dependency matches a locally installed package. One way to solve this is to introduce a local database for the installed packages which will keep a reverse "URL to package name" lookup table.

2. Nimble can download the HTTP dependency in a temporary location, then it can inspect the downloaded package name and figure out that it's already present as a nimble develop link.

3. The user may be required to specify the package name when introducing a HTTP dependency. The syntax may look like this:

requires "nim >= 0.18.1",
         "httputils@https://github.com/status-im/nim-http-utils >= 0.2.0"
         # here, `httputils` is the package name

The specified package name will allow Nimble to check for an existing installation of the package. A similar solution is used by other build tools such as Scala's SBT. Alternatively, we can introduce some new properly typed dependency-specification API:

uses("eth_p2p", git = "https://github.com/p2p", branch = "master")

---

I've pushed a very simple project demonstrating the issue here:

https://github.com/zah/nim-nimble-issue

To see the issue, you'll also need to clone the httputils package from https://github.com/status-im/nim-http-utils and then attempt to introduce changes in it after linking it with nimble develop.

[zah]

@dom96, can you comment on which of the proposed solutions would be considered most acceptable?

[dom96]

I think the package name can be reverse looked up in the package list. It wouldn't be perfect but I don't particularly like any of the other solutions.

To be honest, I consider this as more of a user issue. Don't specify dependencies using URLs.

[zah]

Using URLs is mandatory when your packages are not registered in the Nimble package list. This will be true for example if your packages are considered private. Also, every package starts as being private. During the initial most active development, most users are likely to experience the problems described here.

[brentp]

IIUC, this is also useful if, for example one builds a package that depends on a fork of the canonical package. This is a situation that I'm currently encountering.

[zah]

This issue can be solved by implementing a proper lock file (#127).

The lock file would include the resolved package name, the URL appearing in the nimble file and the revision hash/tag. The resolved package name would be enough for Nimble to look for a locally linked development version of the package.

When I create new commits in a locally linked library, the lock files of other projects that depend on it will be updated every time nimble test completes successfully. This will provide strong enough version pinning for libraries that haven't published tagged versions yet. It would be recommended to the users that lock files are stored in git.

[dom96]

Not high-pri as it will be fixed via lock file support.

[yyoncho]

I believe this one is already fixed (most likely when nimble.develop was introduced). I tested with the following steps:

1. Created a package with dependency https://github.com/GULPF/timezones
2. I did nimble develop timezones
3. I did a change in the locally cloned copy and then created a test depending on this change.
4. I deleted pkgs2 copy. I did nimble test and the test was still green.

I tested also with nimble develop --global and the results seem to be the same.