Import river position data and make available via the MetaCPAN API

[neilb]

In the River discussion at the QAH 2016 we agreed one of the things that would help most is if a dist's river position were available via the MetaCPAN API, and also displayed on a dist's home page on MetaCPAN (I'll raise a separate ticket for that).

I'm calculating river position for all dists every week, and have agreed to put this data somewhere. After a chat with @oalders we agreed with that initially this will be simple JSON data like the following:

[
    {
        "dist":      "System-Command",
        "total":     92,
        "immediate": 4,
        "bucket",    2
    },
    {
        "d": "Text-Markdown",
        "t": 92,
        "i": 56,
        "b", 2
    }
]

Here's what those fields are:

• immediate is the number of immediate downstream dependents (considering required, non-developer prereqs) that the dist has
• total is the total number of downstream dependents
• bucket is a number between 0 and 5, the logarithmic binning of total. 0 means no downstream deps, and 5 is the head of the river.

JSON always seems stupidly verbose, so for internal things I tend to use shorter names, like the second example above. Or would you rather go with verbosity?

Once we've agreed on the above format, I'll decide where to publish it so you can grab a first version, then I'll set up something to ensure it's getting regularly updated.

[oalders]

I'd rather go with verbose, I think. We have to deal with a bunch of different web services and making keys obvious makes for less head scratching. If you're worried about file size, you could compress it.

[jberger]

#462 was merged

[neilb]

Here's a first version of the river data:

http://neilb.org/river-of-cpan.json.gz

Need to think where to put this ongoing. Maybe in github, then we'll have history data, but the repo (history) will end up getting very big.

[oalders]

@jberger did you want to look at tweaking your work to deal with the compressed file?

[jberger]

ah good point, yeah thanks

[jberger]

With Mojo::UserAgent if I munge the headers it will just transparently decode the gzip. I'm investigating if the similar behavior is possible with LWP. If it works then we should see if we can get @neilb to serve the file with Content-Encoding: gzip, Content-Type: application/json

[oalders]

At the very least, it should work with WWW::Mechanize, so feel free to switch to that.