Support redirects

[RyanCavanaugh]

We should allow a folder structure which is

/some-lib
  redirect.json

The contents of redirect.json would be something that tells types-publisher how to enlist and parse the linked repo. We can probably start with just git URLs and expand as needed

[blakeembrey]

What do you think the specificity of this file should be? Per directory or per version or per commit? Directory seems to make sense for the most part. Does the redirect structure just need to follow the @types structure (E.g. versioned sub-folders, I think files must be the same name as the typed name?). Will typings be able to expand over multiple files and specify the main?

[blakeembrey]

Has the structure of DefinitelyTyped been finalised now? I think the best approach to redirects, and one that keeps things consistent, would be to use git submodules. The advantage here is GitHub has a nifty feature that'll redirect when you click on another GitHub submodule. The only issue here is who has onus on updating the submodule references.

[felixfbecker]

+1 for git submodules.

Currently we have a lot of high-quality typings in the registry that are well-maintained, have their own tests, issue tracker, PRs, docs, and CI. It is easy to contribute to them (no need to pull down all the other thousand typings), and the typings structure follow the library structure (instead of 10,000+ LOC in one file).
At the same time, the typings for these exact libraries are still maintained in DT, even though often outdated compared to types/. People are basically duplicating their efforts, because they might not even know that an equivalent in types/ exists. We really need some way to redirect them from DT, and I think submodules would be an excellent solution, because of the mentioned GitHub feature. They are generally frowned upon because the submodules don't get recursively cloned by default. But for DT, this would be exactly what we want - if you clone DT, you want to modify some typings in there. If you want to modify types/ typings, you can clone that repo directly.

We currently use subfolders for different versions of typings and could easily give them an individual package.json with a version field and proper @types/ name.

[felixfbecker]

@mhegazy #52 (comment)

We would be happy to take PRs for this feature if anyone is interested in contributing. most of the infrastructure is already in place.

It would be nice if one of the TypeScript members could comment on the submodules proposal before putting work into a PR :)

[mhegazy]

The open question for submodules, as noted by @blakeembrey, is the update strategy. is the publisher responsible for always updating to latest? or is the package author responsible for this whenever they are ready to push a new version out? if it is the publisher how often should it look for these package changes? currently we have a webhook that is listening on definitelytyped, and ideally we would like keep it this way.

[felixfbecker]

Currently, when we change on of the repos, we update the commit hash manually in the typings registry. That can be done really quickly with the GitHub online editor, but there were some ideas floating around for having a bot that does this or a typings publish command. For updating the git submodule, the typings maintainer should be responsible, just as he is for updating the registry entry. I guess you cannot do it online though, so it needs to happen with a PR. Updating happens afaik through git submodule update --remote when inside the submodule dir, but I dont know a lot about submodules. It may be worth to find out if this can be done programmatically through the GitHub API so typings publish can automatically open a PR at DT. @blakeembrey what do you think?

[blakeembrey]

I'm not really sure, there's advantages on both sides and I believe I have presented each side (though offline, not particularly here). Hand editing is definitely made easier with text files, but it's possible we can make a bot to update submodules and submit PRs also. I imagine we'll want to do that flow so that tests can be run on the PR and an external change doesn't break DefinitelyTyped. Since both methods can be automated, sticking to the more valuable one (likely submodules, since the GitHub interface is friendly towards it) would be better.

[felixfbecker]

@RyanCavanaugh So what is going on with this? Quoting yourself from microsoft/TypeScript#9184 (comment):

For large and popular libraries, I agree that having a separate repo is the way to go. We fully intend to support that scenario by allowing redirection from DefinitelyTyped to other repos so that well-maintained libraries can have their own repo.

We have 350 typings in the registry in total and of those are 200 typings in the @types org ranging from the types of node itself to the most-used libraries on NPM like express. These are well-maintained by a core team and can be improved with PRs aswell, and honestly from my own experience of much higher quality than the ones on DT. The issue is that with TS2 being stable now and the release notes actively telling people that "typings from the typings may not be compatible with TS2, so use npm", people will only use these huge >1000 lines per file typings from DT, that share an issue and PR tracker with 2000 other declarations. And as people use these, they will submit PRs and contribute to the typings on DT, that will eventually get replaced by a redirect. I feel you should really prioritize this feature much much higher, it feels like you are just putting this at the end of the backlog as a "nice to have" while people are duplicating their efforts everyday. You promised to have full support for this scenario and now TS2 is out and it doesn't seem like anyone has even started working on this... Some clear statement or ETA would be really nice 😐

[RyanCavanaugh]

I apologize that this hasn't happened sooner. We're the same engineers as on TypeScript and as you can see we just shipped 2.0, so things have been pretty busy. We're also spending a lot of time just reviewing DT PRs and this has been taking longer than expected during the transition period.

We'd totally accept a PR on this if you'd be willing to help implement it. Otherwise I think we're looking at 4-6 weeks out at the earliest.

[unional]

Meaning, there should be no "compile" step.

To give you an example why compile step is important and beneficial, see typed-typings/npm-text-buffer#3 (comment)

In github.com/types/* repositories, we add the source package as dev dependency and actually run the tests (i.e. require compiling).
With greenkeeper, this allows the typings maintainer to get up to date immediately with the latest version.

I have written about why no compile step is not sufficient and mentioned that is a previous comment. See: https://github.com/typings/generator-typings#about-writing-tests-for-typings

[unional]

Another approach is to just use github.com/types/*. Have a bot as an owner in the organization.

The bot will be notified by github that a new PR is available.
It can then get the PR, ensure the toolchain is there and run tests.
When the PR is merged, then the bot can publish a new version to npm.

Contributors will use a cli tool or web portal to create new repo under the organization.

This way we can utilize many features available to the npm ecosystem for free: dependency management, source linking, real test, green keeper, CI, PR, etc.

[unional]

Another example of how important the compile step is:
https://travis-ci.org/types/npm-react-router/builds/254269560?utm_source=github_status&utm_medium=notification
The underlying library removed the method call. Without an actual running test against the actual library, you will have to wait until someone files an issue to fix it. With actual tests, you catch it right away with the help of greenkeeper.

[demurgos]

This is a relatively old issue. I originally was in favor of it, but today I am no longer really sure it's needed.

Typescript reached a point where I see interest by Javascript projects to host their own types. I tend to present it as documentation: it reduces ambiguity compared to free-form text and lets some editors to better understand the lib (providing a better experience: code completion, quick doc, etc.). Gulp is gradually adding types to its dependencies, chai is in the process of bundling its types, chalk bundles them, sequelize too, etc.

Because of this, I think that it's simpler to convince maintainers to keep type definitions (and their tests) in the main repo and use DT as a fallback if the maintainer is inactive or not interested. Redirects may still help for these libs, but in practice I found that developers don't mind hosting their own types.

[unional]

Adding typings directly to the libs is always the recommended way.
Even though that has its own challenge (one challenge is correctly maintaining sem-ver).

This has always been about redirect from DT so that it is easier to maintain the typings when they are not directly distributed with the libs.

Personally, I have not been contributing to DT for a long time precisely because of this.
Now when I need some typings that do not come with the libs nor DT, I just define them locally.
It just doesn't make sense to me to get the whole DT repo and run tests just for adding one typings.

Another reason is the use of global script files vs module files.
I understand TS team like to have a tighter control of the typings, but as npm packages are self-regulated, I believe typings packages should be the same.

And as I pointed out above, there are ways to improve both the quality of the typings and the experience of writing them without the need to contributing to a single mono repo.

[daerogami]

Typings from @types are easily accessed globally in a project not using modules; however, I have not been able to find documentation around getting types from projects that include them for the aforementioned scenario.
I am using MSBuild, so changing typeRoots/paths is not supported w/o creating a tsconfig.json which I really don't want to do. Any idea how to resolve those types in this situation without using import?