add a script for finding popular, but unused projects #29

adamdecaf · 2018-09-12T18:17:45Z

popular: the more godoc importers the more popular
unused: long time since last commit? few issue updates/comments

Issue: #14

adamdecaf · 2018-09-12T18:23:20Z

Here's an example run:

name                                           stars last commit (days) importers
github.com/go-martini/martini                  10264 478                2096
github.com/docker/libcontainer                 1037  664                1363
github.com/cihub/seelog                        1183  356                1118
github.com/bmizerany/pat                       1133  398                434
github.com/justinas/alice                      1630  329                328
github.com/google/gxui                         4138  1010               264
github.com/hoisie/web                          3122  454                166
github.com/braintree/manners                   828   461                140
github.com/bradfitz/http2                      1674  974                82
github.com/goraft/raft                         1883  1227               74
github.com/yhat/scrape                         1444  658                44
github.com/rcrowley/goagain                    1791  389                41
github.com/docker/libchan                      2329  326                30
github.com/rsms/gotalk                         884   295                15
github.com/gtank/cryptopasta                   1346  472                13
github.com/JoelOtter/termloop                  902   315                10
github.com/olebedev/when                       879   319                8
github.com/tleyden/open-ocr                    908   493                6
github.com/haxpax/gosms                        1157  437                6
github.com/pravj/geopattern                    961   495                6
github.com/alexflint/gallium                   3591  542                3
github.com/rakyll/coop                         1212  1076               2
github.com/codahale/sneaker                    784   610                1
github.com/kshvmdn/fsql                        3493  264                1
github.com/tobyhede/go-underscore              977   1175               1
github.com/mop-tracker/mop                     933   314                1

adamdecaf · 2018-09-12T18:26:05Z

Is this useful so far? I can easily add some sorting.

acln0

Looks nice. I'd like to request a few minor adjustments before we merge, though.

Besides the comments in the review, please squash the PR to one commit too.

Thanks.

acln0 · 2018-09-12T18:54:03Z

main.go

+	if err != nil {
+		return -1, fmt.Errorf("problem loading %s: %v", req.URL, err)
+	}
+	if resp.Body != nil {


The net/http documentation says "The http Client and Transport guarantee that Body is always non-nil, even on responses without a body or responses with a zero-length body."

I don't think we need this if statement.

acln0 · 2018-09-12T18:54:39Z

main.go

+
+	doc, err := html.Parse(resp.Body)
+	if err != nil {
+		panic(err)


Let's just return the error here. No need to panic.

acln0 · 2018-09-12T18:55:04Z

main.go

+func scrapeGodocImports(importPath string) (int, error) {
+	req, err := http.NewRequest("GET", "https://godoc.org/"+importPath, nil)
+	if err != nil {
+		panic(err)


Return -1, err please, instead of panicking.

acln0 · 2018-09-12T18:56:26Z

main.go

+		},
+	})
+	if err != nil {
+		panic(err)


Let's log err to standard error and bail out more gracefully than this. Avoid panic please.

acln0 · 2018-09-12T18:58:44Z

main.go

+	}
+}
+
+func createGithubClient(ctx context.Context) *github.Client {


I think this function should return an error. Currently, it prints a message to standard error if the github token isn't available, but then it continues creating the client regardless.

acln0 · 2018-09-12T18:59:33Z

main.go

+var (
+	ctx = context.Background()
+
+	ghClient = createGithubClient(ctx)


Move this inside of main, please, and let's handle the error it will return more gracefully, by bailing out, instead of continuing.

acln0 · 2018-09-12T19:00:48Z

main.go

+)
+
+var (
+	ctx = context.Background()


I think we should avoid this. For now, let's pass context.Background() to all things that need it, explicitly.

If the context we use changes in the future, it'll be easier to spot at a glance and adjust.

acln0 · 2018-09-12T19:04:19Z

Oh, something else too. I only just now noticed that this vendors dependencies, but it is using modules as well. Isn't that strange? I thought modules and vendoring were incompatible.

adamdecaf · 2018-09-12T21:19:07Z

@acln0 Thanks for the review. I dropped vendor/, but it's still used by go build. This script requires the internet already.

adamdecaf · 2018-09-12T21:47:59Z

There are a lot of -1's returned still, so I should probably look into that. I hope we can just make that refresh call and retry.

acln0 · 2018-09-13T08:45:18Z

Thanks. This looks pretty good now.

I guess the next question is if we want to have a main.go in the repository root, or if we should move it to cmd/something.

Ping @theckman for review and further thoughts.

theckman

If the option is between using modules and vendoring, I'd prefer we do what we need to have a functioning vendor directory. I'd like to make sure the binary is buildable no matter the state of any of its dependencies.

Can you please update this PR to add a vendor directory that ensures the script will build?

theckman · 2018-09-15T21:50:00Z

main.go

@@ -0,0 +1,115 @@
+package main


Let's make this cmd/finder/main.go or something similar.

Maybe add a // import "github.com/gofrs/help-requests/cmd/finder" comment here too?

Perhaps a package-level comment would be nice too, like here https://github.com/golang/tools/blob/master/cmd/stringer/stringer.go

Issue: gofrs#14

adamdecaf · 2018-09-17T15:43:29Z

Added vendor/ back. Updated the script a bit. There are still a lot of -1 godoc counts coming back. I think we should try a manual refresh on godoc.org to grab their count. Thoughts?

Edit: I updated the example: #29 (comment)

adamdecaf mentioned this pull request Sep 12, 2018

Review GitHub Go repositories with stars but old last commit #14

Closed

acln0 suggested changes Sep 12, 2018

View reviewed changes

adamdecaf force-pushed the init-script branch 2 times, most recently from 614b967 to 08a6a8c Compare September 12, 2018 21:07

theckman suggested changes Sep 15, 2018

View reviewed changes

adamdecaf force-pushed the init-script branch from 08a6a8c to 7220b86 Compare September 17, 2018 15:37

adding script to find popular, but unused projects

35923ca

Issue: gofrs#14

adamdecaf force-pushed the init-script branch from 7220b86 to 35923ca Compare September 17, 2018 15:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add a script for finding popular, but unused projects #29

add a script for finding popular, but unused projects #29

adamdecaf commented Sep 12, 2018

adamdecaf commented Sep 12, 2018 •

edited

Loading

adamdecaf commented Sep 12, 2018

acln0 left a comment

acln0 Sep 12, 2018

acln0 Sep 12, 2018

acln0 Sep 12, 2018

acln0 Sep 12, 2018

acln0 Sep 12, 2018

acln0 Sep 12, 2018

acln0 Sep 12, 2018

acln0 commented Sep 12, 2018

adamdecaf commented Sep 12, 2018

adamdecaf commented Sep 12, 2018 via email •

edited

Loading

acln0 commented Sep 13, 2018

theckman left a comment

theckman Sep 15, 2018

acln0 Sep 15, 2018

adamdecaf commented Sep 17, 2018 •

edited

Loading

add a script for finding popular, but unused projects #29

Are you sure you want to change the base?

add a script for finding popular, but unused projects #29

Conversation

adamdecaf commented Sep 12, 2018

adamdecaf commented Sep 12, 2018 • edited Loading

adamdecaf commented Sep 12, 2018

acln0 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

acln0 commented Sep 12, 2018

adamdecaf commented Sep 12, 2018

adamdecaf commented Sep 12, 2018 via email • edited Loading

acln0 commented Sep 13, 2018

theckman left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamdecaf commented Sep 17, 2018 • edited Loading

adamdecaf commented Sep 12, 2018 •

edited

Loading

adamdecaf commented Sep 12, 2018 via email •

edited

Loading

adamdecaf commented Sep 17, 2018 •

edited

Loading