indexer in 1.7.x still consumes more CPU than in 1.5

[vladak]

I was hoping that this issue was resolved in the past year, unfortunately it is still the case:

I guess we'll stay forever on the last 1.5 release as it works just fine. Not sure if we need to configure or tune anything, I was hoping for an easy out-of-the-box solution. I even tried to disable the suggester by disabling it in the read-only config, but either it didn't work or it didn't help.

Originally posted by @der-eismann in #3585 (comment)

[vladak]

@der-eismann Looks like it is necessary to properly root cause the change. Could you share the details about the indexer run and the overall environment ? Also, what exact 1.7.x version are you using at the moment ?

[chenchuanliang]

I also encountered the same problem. 1.7 indexing is too slow.

[GCSimba]

Hi, do you know how to increase the number of threads in opengrok? I think it takes a lot of time to build an index

[vladak]

Hi, do you know how to increase the number of threads in opengrok? I think it takes a lot of time to build an index

Please create separate discussion for that question.

[der-eismann]

@vladak sorry for the late reply. We continued running on 1.5 since it was just working™. I tried to investigate a bit again with 1.12 as the usage is still high, however it seems to be related to the whole per-project management that was introduced in 1.6.

Our setup consists of 240 Git repositories that are externally updated (automatic removal of archived repos, adding new ones...). So back in 1.5 we only ran it with NOMIRROR=true and it was fine. We had a project overview on the default page with the dates of the last commits, everything was good.
And it's still fine if I run 1.12 with AVOID_PROJECTS=true, CPU usage is similar to how it was in 1.5 and indexing is done in ~3 mins. However it is not possible to select specific projects anymore on the main page, since projects are avoided, and there's no Git History or Annotation, which sucks a bit.

Now when I remove AVOID_PROJECTS and replace it with WORKERS=1, I can see in the startup logs that it takes ~4 minutes to add all projects and then it's just Sync starting without any further output. Even after 20 minutes the CPU usage is consistently high (about 2-3 CPU cores), and I still don't have a project overview on the main page. Is there any misconfiguration?

All I want is to index our 240 Git repos, re-index them every 10 minutes without updating or syncing them and ideally have an overview on the main page. So basically the 1.5 behavior. Any way to achieve that?