new badge: f-droid #1965

niccokunzmann · 2018-08-23T20:38:57Z

I document the progress here as an imperfect PR and ask for review once I am confident it works.

edit server.js
first untested version
- get the website
- regex query as string
- get version info

fixes #1963

shields-ci · 2018-08-23T20:42:54Z

	Messages
📖	✨ Thanks for your contribution to Shields, @niccokunzmann!

Generated by 🚫 dangerJS

niccokunzmann · 2018-08-23T22:54:18Z

services/f-droid/f-droid.tester.js

+  .get('/f-droid/version/org.pacien.tincapp.json')
+  .intercept(nock =>
+    nock('https://f-droid.org')
+      .get('/en/packages/org.pacien.tincapp/')


The tests are red although and I do not know why.

$ npm run test:services -- --grep F-Droid > gh-badges@1.3.0 test:services /home/user/shields > HANDLE_INTERNAL_ERRORS=false mocha --delay lib/service-test-runner/cli.js "--grep" "F-Droid" Running all service tests. No secret data found at /home/user/shields/private/secret.json (see lib/server-secrets.js) raven@2.4.2 alert: no DSN provided, error reporting disabled 0823224940 FsTokenPersistence configured with /home/user/shields/private/github-user-tokens.json 0823224940 Server is starting up: http://lib/service-test-runner/cli.js:80/ No fallback font set. Unable to load fallback font. Using Helvetica-Bold instead. F-Droid Package is found GET /f-droid/version/org.pacien.tincapp.json has FAILED with the following response: 1) [ GET /f-droid/version/org.pacien.tincapp.json ] Package is not found GET /f-droid/version/org.pacien.tincapp.json has FAILED with the following response: 2) [ GET /f-droid/version/org.pacien.tincapp.json ] 0 passing (827ms) 2 failing 1) F-Droid Package is found [ GET /f-droid/version/org.pacien.tincapp.json ]: AssertionError: expected { Object (name, value) } to deeply equal { Object (name, value) } + expected - actual { - "name": "404" - "value": "badge not found" + "name": "F-Droid" + "value": "v1.8asdsadsa" } at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38) at _expect (node_modules/icedfrisby/lib/icedfrisby.js:583:10) at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1261:26) at start (node_modules/icedfrisby/lib/icedfrisby.js:1244:12) at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1131:16) at Request.self.callback (node_modules/request/request.js:185:22) at Request.<anonymous> (node_modules/request/request.js:1161:10) at IncomingMessage.<anonymous> (node_modules/request/request.js:1083:12) at endReadableNT (_stream_readable.js:1056:12) at _combinedTickCallback (internal/process/next_tick.js:138:11) at process._tickDomainCallback (internal/process/next_tick.js:218:9) 2) F-Droid Package is not found [ GET /f-droid/version/org.pacien.tincapp.json ]: AssertionError: expected { Object (name, value) } to deeply equal { Object (name, value) } + expected - actual { - "name": "404" - "value": "badge not found" + "name": "F-Droid" + "value": "app not found" } at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38) at _expect (node_modules/icedfrisby/lib/icedfrisby.js:583:10) at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1261:26) at start (node_modules/icedfrisby/lib/icedfrisby.js:1244:12) at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1131:16) at Request.self.callback (node_modules/request/request.js:185:22) at Request.<anonymous> (node_modules/request/request.js:1161:10) at IncomingMessage.<anonymous> (node_modules/request/request.js:1083:12) at endReadableNT (_stream_readable.js:1056:12) at _combinedTickCallback (internal/process/next_tick.js:138:11) at process._tickDomainCallback (internal/process/next_tick.js:218:9)

See https://f-droid.org/en/packages/org.pacien.tincapp/
and https://github.com/badges/shields/pull/1965/files#diff-63441b8c1c2f41111e85c4e116ac3804R10

Resolved: The path is /version and not /f-droid/version

niccokunzmann · 2018-08-23T23:14:44Z

Now, I have the problem, that the nock data is not delivered as website. The HTTP get always returns an empty website.

See yourself: search for website and see 23a97a0

$ npm run test:services -- --grep F-Droid

> gh-badges@1.3.0 test:services /home/user/shields
> HANDLE_INTERNAL_ERRORS=false mocha --delay lib/service-test-runner/cli.js "--grep" "F-Droid"

Running all service tests.


No secret data found at /home/user/shields/private/secret.json (see lib/server-secrets.js)
raven@2.4.2 alert: no DSN provided, error reporting disabled
0823231217 FsTokenPersistence configured with /home/user/shields/private/github-user-tokens.json
0823231217 Server is starting up: http://lib/service-test-runner/cli.js:80/
No fallback font set.
Unable to load fallback font. Using Helvetica-Bold instead.
  F-Droid
    Package is found
website

GET /version/org.pacien.tincapp.json has FAILED with the following response:
      1) 
	[ GET /version/org.pacien.tincapp.json ]
    Package is not found
website

GET /version/org.pacien.tincapp.json has FAILED with the following response:
      2) 
	[ GET /version/org.pacien.tincapp.json ]


  0 passing (810ms)
  2 failing

  1) F-Droid
       Package is found
         
	[ GET /version/org.pacien.tincapp.json ]:

      AssertionError: expected { Object (name, value) } to deeply equal { name: 'F-Droid', value: 'v0.19' }
      + expected - actual

       {
         "name": "F-Droid"
      -  "value": "fix this badge"
      +  "value": "v0.19"
       }
      
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38)
      at _expect (node_modules/icedfrisby/lib/icedfrisby.js:583:10)
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1261:26)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1244:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1131:16)
      at Request.self.callback (node_modules/request/request.js:185:22)
      at Request.<anonymous> (node_modules/request/request.js:1161:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1083:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

  2) F-Droid
       Package is not found
         
	[ GET /version/org.pacien.tincapp.json ]:

      AssertionError: expected { Object (name, value) } to deeply equal { Object (name, value) }
      + expected - actual

       {
         "name": "F-Droid"
      -  "value": "fix this badge"
      +  "value": "app not found"
       }
      
      at Object.pathMatch.matchJSON (node_modules/icedfrisby/lib/pathMatch.js:138:38)
      at _expect (node_modules/icedfrisby/lib/icedfrisby.js:583:10)
      at IcedFrisbyNock._invokeExpects (node_modules/icedfrisby/lib/icedfrisby.js:1261:26)
      at start (node_modules/icedfrisby/lib/icedfrisby.js:1244:12)
      at Request.runCallback [as _callback] (node_modules/icedfrisby/lib/icedfrisby.js:1131:16)
      at Request.self.callback (node_modules/request/request.js:185:22)
      at Request.<anonymous> (node_modules/request/request.js:1161:10)
      at IncomingMessage.<anonymous> (node_modules/request/request.js:1083:12)
      at endReadableNT (_stream_readable.js:1056:12)
      at _combinedTickCallback (internal/process/next_tick.js:138:11)
      at process._tickDomainCallback (internal/process/next_tick.js:218:9)

niccokunzmann · 2018-08-23T23:20:10Z

Status:

the badge is working
the tests are failing because I can not mock the API. I copied the code from https://github.com/badges/shields/blob/master/services/apm/apm.tester.js#L38 Am I missing something?

paulmelnikow · 2018-08-24T03:04:34Z

In terms of the mock, I'm not sure what is up with those APM tests passing an array to .reply(), though the nock API doc is here: https://github.com/nock/nock#specifying-replies

I got your tests to pass by changing:

.reply([200, manyVersions])

to:

.reply(200, manyVersions)

- credits: badges#1965 (comment)

niccokunzmann · 2018-08-24T08:12:53Z

@paulmelnikow Thanks for the reply. I was assuming something else because of
https://github.com/badges/shields/blob/master/services/apm/apm.tester.js#L41 Maybe, this means it returns JSON.

- credits: badges#1965 (comment)

niccokunzmann · 2018-08-24T08:15:46Z

@PyvesB @chris48s @paulmelnikow Thanks for your replies and the help. Please review as I think, this is ready to merge.

RedSparr0w · 2018-08-24T08:17:29Z

Looks like you need to run prettier for the test to pass.
(On mobile sorry for the short reply)

niccokunzmann · 2018-08-24T08:43:19Z

I forgot: The examples do not work. The tests pass but I can not use the front page.
bc418ab

http://[::]:8080/examples/build shows

I have nothing to offer for this request
… but blood, toil, tears and sweat.
Fortunately, the main page can offer rainbows instead!
(You probably landed on this page because a link was broken, because someone mistyped its URL, or because of an irrelevant mistake. Don't worry, though: there is all of the rest of the Web to explore!)

And I see this in the main page console:

Loading failed for the <script> with source “http://[::]:8080/_next/102fbd59-c519-4278-ba64-757fb07c3e37/page/index.js”. [::]:8080:16
Loading failed for the <script> with source “http://[::]:8080/_next/102fbd59-c519-4278-ba64-757fb07c3e37/page/_error.js”. [::]:8080:16
Loading failed for the <script> with source “http://[::]:8080/_next/373e62b9ccd276ee0427a69f4f991d68/app.js”. [::]:8080:16

shields/build$ tree
.
├── index.html
├── _next
│   ├── 085d9632aa638b42a1aef20e59bfcbef
│   │   └── app.js
│   └── 120c30d5-2ce6-4314-8234-bd0e2594cb8a
│       └── page
│           ├── _error.js
│           ├── _error.js.map
│           ├── index.js
│           └── index.js.map
└── static
    ├── logo.svg
    └── main.css

5 directories, 8 files

I cannot find this string anywhere:

shields$ grep -r 102fbd59-c519-4278-ba64-757fb07c3e37

It looks like the next build is is not updated.

niccokunzmann · 2018-08-24T09:08:02Z

The service works on http://localhost:3000/#/ but not on http://[::]:8080/

niccokunzmann · 2018-08-24T09:12:25Z

Problem now: I do not see the example badge.

niccokunzmann · 2018-08-24T09:16:00Z

Problem ideintified:

<img class="badge-img clickable" src="http://[::]:8080/badge/F-Droid-v1.0-blue.svg" alt="">

is a 404 badge not found

niccokunzmann · 2018-08-24T09:17:19Z

Solution:

http://[::]:8080/badge/F--Droid-test-blue.svg

The - needs to be escaped when in title.

Problem: When a - was in the title, this lead to an invalid preview badge badges#1965 (comment) Solution: Escape the title of the badge.

this is the first untested version - get the website - parse as xml - get version info

Some services only require HTTP requests and not JSON requests. This splits up the two services.

- related: badges#1963 (comment)

This introduces a new error because the nock results in an empty website.

- addresses badges#1965 (comment)

niccokunzmann · 2018-08-30T12:55:33Z

F-Droid allows downloading a 1.360.393 bytes big jar file with meta data of all the different apps. https://f-droid.org/repo/index-v1.jar This is a jar file which contains JSON.

Right now, there is a single index file that is updated a few times a week. So the easiest way to implement this would be to have something on shields.io poll for the index-v1.jar, and then when it updates, it would generate all of the badges.

If there were many people using the f-droid badges, caching this big file would reduce communication to external servers. It also depends on how often this file is updated. I do not think I will use this as a base for the badge.

- improved performance - addresses badges#1965 (comment)

niccokunzmann · 2018-08-30T14:24:03Z

I think, I addressed the changes marked in the roadmap #1965 (comment).

paulmelnikow · 2018-08-30T15:19:26Z

Thank you for addressing these! I will leave this to @PyvesB to wrap up.

If there were many people using the f-droid badges, caching this big file would reduce communication to external servers. It also depends on how often this file is updated. I do not think I will use this as a base for the badge.

I agree with you. It will be lighter to just snag the text files we need. Down the line, we could revisit this if we find many people are using these badges.

PyvesB

I'm happy with the current state of things. Let's get this merged! Thanks for all your work @niccokunzmann, we've all definitely learned at lot here and hopefully things will be smoother next time. 😉

Here's a summary of the follow up tasks to this PR:

Move request method to super class and delete BaseHttp. I'm happy to tackle this one in the coming days as it's related to another piece of work I'm doing.
Make sure all badges consistently use lower casing for labels. See Lowercase badge labels #1997.
Make sure all services use the renderVersionBadge function consistently. See Ensure consistent usage of renderVersionBadge #2026.
Consolidate our usage of URL encoding functions. See Consolidate URL encoding functions #2027.
Keep an eye on GitLab and any progress on a more appropriate F-Droid API.

shields-deployment · 2018-08-30T22:02:12Z

This pull request was merged to master branch. This change is now waiting for deployment, which will usually happen within a few days. Stay tuned by joining our #ops channel on Discord!

After deployment, changes are copied to gh-pages branch:

niccokunzmann · 2018-08-31T07:24:05Z

@PyvesB @paulmelnikow @RedSparr0w and of cause @shields-ci Thanks for the help during the pull request!

paulmelnikow added the service-badge New or updated service badge label Aug 23, 2018

niccokunzmann force-pushed the fdroid branch from 1b557bc to 79ab58b Compare August 23, 2018 21:03

niccokunzmann mentioned this pull request Aug 23, 2018

Pull request message about tests should mention the documentation #1968

Closed

niccokunzmann commented Aug 23, 2018

View reviewed changes

niccokunzmann added a commit to niccokunzmann/shields that referenced this pull request Aug 24, 2018

remove debug output and make tests work

a1ebef3

- credits: badges#1965 (comment)

niccokunzmann force-pushed the fdroid branch from a1ebef3 to 0d0f65d Compare August 24, 2018 08:13

niccokunzmann added a commit to niccokunzmann/shields that referenced this pull request Aug 24, 2018

remove debug output and make tests work

0d0f65d

- credits: badges#1965 (comment)

niccokunzmann added a commit to niccokunzmann/shields that referenced this pull request Aug 24, 2018

escape title for preview badge

e9a4a47

Problem: When a - was in the title, this lead to an invalid preview badge badges#1965 (comment) Solution: Escape the title of the badge.

niccokunzmann changed the title ~~[WIP] new badge: f-droid~~ new badge: f-droid Aug 24, 2018

niccokunzmann added 9 commits August 25, 2018 00:27

code frame

ec9bfaf

this is the first untested version - get the website - parse as xml - get version info

query f-droid website for version code

fc8eb13

refactor: split JSON and HTTP

4ee1c12

Some services only require HTTP requests and not JSON requests. This splits up the two services.

remove F-Drouid from server

5a41b83

add new style F-Droid service

7cfce52

- related: badges#1963 (comment)

shorter error message

d34d2d6

add tests for f-droid

22c9be9

fix badge not found in tests

ba5e0b0

This introduces a new error because the nock results in an empty website.

debug print website

51d5b2c

invalid response instead of fix this badge

91891bb

- addresses badges#1965 (comment)

niccokunzmann dismissed PyvesB’s stale review via 91891bb August 30, 2018 12:37

paulmelnikow temporarily deployed to shields-staging-pr-1965 August 30, 2018 12:37 Inactive

use gitlab instead of website

55e1c13

- improved performance - addresses badges#1965 (comment)

paulmelnikow temporarily deployed to shields-staging-pr-1965 August 30, 2018 13:49 Inactive

npm run prettier

7926f17

paulmelnikow temporarily deployed to shields-staging-pr-1965 August 30, 2018 13:53 Inactive

Merge branch 'master' into fdroid

5fb036f

paulmelnikow temporarily deployed to shields-staging-pr-1965 August 30, 2018 14:23 Inactive

niccokunzmann added 3 commits August 30, 2018 22:52

remove debug output

9d2579a

Merge branch 'fdroid' of github.com:niccokunzmann/shields into fdroid

fdf6010

Merge branch 'master' of github.com:badges/shields into fdroid

610008c

paulmelnikow temporarily deployed to shields-staging-pr-1965 August 30, 2018 20:53 Inactive

Merge branch 'master' into fdroid

12ada71

paulmelnikow temporarily deployed to shields-staging-pr-1965 August 30, 2018 21:37 Inactive

This was referenced Aug 30, 2018

Ensure consistent usage of renderVersionBadge #2026

Closed

Consolidate URL encoding functions #2027

Closed

PyvesB approved these changes Aug 30, 2018

View reviewed changes

Merge branch 'master' into fdroid

af6c638

paulmelnikow deployed to shields-staging-pr-1965 August 30, 2018 21:59 View deployment

PyvesB merged commit 11fa611 into badges:master Aug 30, 2018

paulmelnikow mentioned this pull request Aug 30, 2018

Consistent use of async/await #2028

Closed

chris48s mentioned this pull request Aug 31, 2018

Uptimerobot unparseable json format #2034

Closed

PyvesB mentioned this pull request Sep 1, 2018

Delete BaseHTTPService and implement new BaseXmlService (affects [eclipse-marketplace f-droid]; also testing on [uptimerobot circleci]) #2037

Merged

snyk-bot mentioned this pull request Jun 22, 2020

[Snyk] Upgrade simple-icons from 1.9.15 to 1.20.0 Richienb/shields#5

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

new badge: f-droid #1965

new badge: f-droid #1965

niccokunzmann commented Aug 23, 2018 •

edited

Loading

shields-ci commented Aug 23, 2018 •

edited

Loading

niccokunzmann Aug 23, 2018

niccokunzmann Aug 23, 2018

niccokunzmann commented Aug 23, 2018 •

edited

Loading

niccokunzmann commented Aug 23, 2018 •

edited

Loading

paulmelnikow commented Aug 24, 2018

niccokunzmann commented Aug 24, 2018

niccokunzmann commented Aug 24, 2018

RedSparr0w commented Aug 24, 2018

niccokunzmann commented Aug 24, 2018 •

edited

Loading

niccokunzmann commented Aug 24, 2018

niccokunzmann commented Aug 24, 2018 •

edited

Loading

niccokunzmann commented Aug 24, 2018

niccokunzmann commented Aug 24, 2018

niccokunzmann commented Aug 30, 2018 •

edited

Loading

niccokunzmann commented Aug 30, 2018

paulmelnikow commented Aug 30, 2018

PyvesB left a comment

shields-deployment bot commented Aug 30, 2018

niccokunzmann commented Aug 31, 2018

new badge: f-droid #1965

new badge: f-droid #1965

Conversation

niccokunzmann commented Aug 23, 2018 • edited Loading

shields-ci commented Aug 23, 2018 • edited Loading

niccokunzmann Aug 23, 2018

Choose a reason for hiding this comment

niccokunzmann Aug 23, 2018

Choose a reason for hiding this comment

niccokunzmann commented Aug 23, 2018 • edited Loading

niccokunzmann commented Aug 23, 2018 • edited Loading

paulmelnikow commented Aug 24, 2018

niccokunzmann commented Aug 24, 2018

niccokunzmann commented Aug 24, 2018

RedSparr0w commented Aug 24, 2018

niccokunzmann commented Aug 24, 2018 • edited Loading

niccokunzmann commented Aug 24, 2018

niccokunzmann commented Aug 24, 2018 • edited Loading

niccokunzmann commented Aug 24, 2018

niccokunzmann commented Aug 24, 2018

niccokunzmann commented Aug 30, 2018 • edited Loading

niccokunzmann commented Aug 30, 2018

paulmelnikow commented Aug 30, 2018

PyvesB left a comment

Choose a reason for hiding this comment

shields-deployment bot commented Aug 30, 2018

niccokunzmann commented Aug 31, 2018

niccokunzmann commented Aug 23, 2018 •

edited

Loading

shields-ci commented Aug 23, 2018 •

edited

Loading

niccokunzmann commented Aug 23, 2018 •

edited

Loading

niccokunzmann commented Aug 23, 2018 •

edited

Loading

niccokunzmann commented Aug 24, 2018 •

edited

Loading

niccokunzmann commented Aug 24, 2018 •

edited

Loading

niccokunzmann commented Aug 30, 2018 •

edited

Loading