Performance is inconsistent

[dandn]

On my Windows PC with Chrome v 84.0.4147.105, my 8 gitlab projects display correctly some of the times.
The dashboard also seems to update pipeline information for a brief time, and then stop.

Other times, the page doesn't completely load at all; one or two projects are fetched successfully, but the remainder of the projects fail to load.

I've tried using both the https://gitlab-monitor.timoschwarzer.com/ site as well as running a server from the docker locally using docker run -d -p 9000:80 timoschwarzer/gitlab-monitor.

My projects are hosted at gitlab.com.

My configuration is as follows:

gitlabApi: https://gitlab.com/api/v4
privateToken: <token with read_api scope>
maxAge: 336
fetchCount: 100
pipelinesOnly: true
autoZoom: true
showPipelineIds: true
showUsers: true
pollingIntervalMultiplier: 0.5
projectVisibility: private
projectScope: groups
projectScopeId:
  - <group id 1>
  - <group id 2>
  - <group id 3>
  - <group id 4>

Could you make any recommendations?
How to debug? A preferred browser with version to use?
Is it recommended to run from the source code with additional configuration settings?

Your input is much appreciated.

Thanks.

[timoschwarzer]

Hi!

The server is unlikely to be the problem, as all the requests happen on the client side. You can take a look at your browser's developer console (press F12 in Chrome on Linux/Windows) and look for errors there. :)

[dandn]

Thanks for the quick reply. It seems I am getting many CORS policy-related errors. Is this an expected error?
Any advice on how to fix?

[timoschwarzer]

Hmm, it seems like GitLab.com is having issues...

According to this GitLab should allow cross origin requests for the API. I don't use GitLab.com and I have no problems with CORS errors on my own GitLab instance.

Can you try again later or can anyone reproduce this behavior with GitLab.com?

[dandn]

I have retried several time with no success.

Similar console errors can also be seen with the following config:

gitlabApi: https://gitlab.com/api/v4
privateToken: <some read_api token>
maxAge: 12
fetchCount: 10
pipelinesOnly: true
autoZoom: true
showPipelineIds: true
pollingIntervalMultiplier: 1
projectVisibility: public

I believe the above is expected to grab 10 public projects with pipelines?

The gitlab-monitor does manage to display some pipelines to the screen but I'm not certain the behaviour is how you intended.
E.g. a single pipeline is displayed, then nothing, then another pipeline, then nothing...

Here is the resulting log:

[timoschwarzer]

Can you switch to the network tab in the DevTools, click on one of the failed requests and send back the result? (Make sure to redact any tokens from the screenshot if there are any)

[dandn]

Here are the two failure types I could see.
Please let me know if I should be get more info related to them.

[timoschwarzer]

HTTP status 429 is "Too many requests". You may need to further decrease the pollingIntervalMultiplier.

Are you (or other people) making API requests from the same IP address using other applications? Maybe GitLab is rate-limiting your public IP.

[dandn]

I don't think anyone else is making that many API requests against our GitLab repos.
Though you are obviously correct and we have many HTTP status 429s coming if the polling interval is high.

I have been playing with certain configurations.
I'm now using the following settings (I restrict the scope to only certain group ids as I show above):

maxAge: 168
fetchCount: 100
pipelinesOnly: true
pollingIntervalMultiplier: 5

Previously I had pollingIntervalMultiplier set to 1 and I definitely see an improvement now with 5 set.

With my current setup, I expect to see 12 projects on the screen.
I managed to get all 12 projects to display on my MacBook in Chrome, but it only seems to be reliable with the 'Network' tab open; otherwise, only a subset of the projects is loaded at times.

On my Windows system in Firefox and Chrome I didn't manage to ever load all projects.

My observation:

• Once the projects are loaded, they are being updated; the program can handle the occasional error
• It seems we can never recover if all the projects are not found during an initial discovery phase.

Is the actual time between API calls affected by pollingIntervalMultiplier? Or is there some other delay that could be inserted?
Might the initial project discovery phase benefit from retries?

Here is another screencap of an error that (I believe) is from the project discovery phase.

Thanks for your continued help.

[timoschwarzer]

Is the actual time between API calls affected by pollingIntervalMultiplier?

Yes. But only between full refreshes. I think there needs to be another mechanic that handles 429s without queueing up indefinite amounts of requests. Unfortunately, as far as I can see, GitLab does not provide information on how long the rate limiting is applied.

Another, way more elegant solution, would be a push-based solution. (e.g. using WebPush and VAPID)
Drawbacks:

1. You need a separate server, although I could provide a global one
2. GitLab currently does not allow global webhooks for Job/Pipeline events. But GitLab-Monitor could set up these webhooks by itself through the API for each project.

[dandn]

This seems to be the relevant documentation: https://docs.gitlab.com/ee/user/gitlab_com/#haproxy-api-throttle
"GitLab.com responds with HTTP status code 429 to API requests that exceed 10 requests per second per IP address."

Perhaps the easiest approach is to throttle the requests?

[erzz]

I'm not much of a node/vue expert - but I think part of the problem is that even when filtering (e.g. maxAge, project filters etc) what is shown and using a group scope the app still queries all branches and tags etc for all projects right? The filter only changes what is displayed and not what is queried?

This can cause a huge deluge of requests (for all 40 projects in a subgroup for example) even though configuring for 5 projects to be shown.

Gitlab.com rightly smashes down with its rate control under these circumstances.

For me a simple explicit list of projectId's would be the nicest way to control both what is queried and displayed