fix: check artifacts in DB to reduce number of sha1 queries

[DmitriyLewen]

Description

Recently, we’ve been encountering too many requests error more frequently (https://github.com/aquasecurity/trivy-java-db/actions/runs/12458901714).
So we need to check the version in the DB.
If the version for this artifact is already added - do not request sha1 for this version.

test builds:

• https://github.com/DmitriyLewen/trivy-java-db/actions/runs/12738972546
• https://github.com/DmitriyLewen/trivy-java-db/actions/runs/12729266847

[DmitriyLewen]

This is weird...
even with these changes I get too many requests... - https://github.com/DmitriyLewen/trivy-java-db/actions/runs/12502776627/job/34882197681

But an hour ago I ran the action with the wrong DB path (so the scanner checked and saved all versions) and there were no errors -https://github.com/DmitriyLewen/trivy-java-db/actions/runs/12502425532/job/34881350678

[knqyf263]

This is weird... even with these changes I get too many requests... - https://github.com/DmitriyLewen/trivy-java-db/actions/runs/12502776627/job/34882197681

But an hour ago I ran the action with the wrong DB path (so the scanner checked and saved all versions) and there were no errors -https://github.com/DmitriyLewen/trivy-java-db/actions/runs/12502425532/job/34881350678

Does it mean Maven Central Repository counts the overall number of requests, not by IP address?

[DmitriyLewen]

Does it mean Maven Central Repository counts the overall number of requests, not by IP address?

I'm not sure about that.
Is it possible that some runners use the same IP?

I saw that before and rechecked now:
maven central currently returns 429 error - https://github.com/DmitriyLewen/trivy-java-db/actions/runs/12683666619/job/35351108235

But locally it works for me without errors.

[knqyf263]

OK, anyway, it's better to reduce HTTP requests. Did you confirm the database with this change is identical to the existing one?

[DmitriyLewen]

I still can't build new DB - https://github.com/DmitriyLewen/trivy-java-db/actions/runs/12683666619/job/35351108235

I will let you know when build and compare DBs.

[DmitriyLewen]

@knqyf263
I tested DBs from this PR.

I can confirm that new DBs contain all artifacts from actual DB (with new artifacts of course).
Also new DBs built faster (1.30 - 2.30 hours) https://github.com/DmitriyLewen/trivy-java-db/actions

[knqyf263]

I'm looking into the central index, but unpacking never ended. I again kicked unpacking on the server and am waiting for it to complete now.

$ java -jar indexer-cli.jar --unpack nexus-maven-repository-index.gz   --destination ./data/central-lucene-index --type full

[DmitriyLewen]

IICR we already thought about that when started working on trivy-java-db.
I even started writing code on Java (https://github.com/DmitriyLewen/maven-indexes-saver).

But IIRC there were problem with Luke and converting this into json/saving into sql db.
And for Go there was no normal adapter for working with Luke

[knqyf263]

Since the purpose is different this time, I thought it would be great if we could simply retrieve the updated artifact with the Luke CLI. In any case, it's been two years since we started trivy-java-db, so I'm willing to give it another try.

[DmitriyLewen]

Okay 👍
Let me know if you need help.
Maybe I can even remember something about my work on this 😄

[knqyf263]

My server has 40GB of space, but it failed due to out of disk...

Exception in thread "Lucene Merge Thread #150" org.apache.lucene.index.MergePolicy$MergeException: java.io.IOException: No space left on device
        at org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:509)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)
Caused by: java.io.IOException: No space left on device
        at java.base/java.io.RandomAccessFile.writeBytes(Native Method)
        at java.base/java.io.RandomAccessFile.write(RandomAccessFile.java:559)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:448)
        at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:99)
        at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:88)
        at org.apache.lucene.store.BufferedIndexOutput.close(BufferedIndexOutput.java:113)
        at org.apache.lucene.store.FSDirectory$FSIndexOutput.close(FSDirectory.java:458)
        at org.apache.lucene.index.TermInfosWriter.close(TermInfosWriter.java:248)
        at org.apache.lucene.index.TermInfosWriter.close(TermInfosWriter.java:251)
        at org.apache.lucene.util.IOUtils.close(IOUtils.java:141)
        at org.apache.lucene.index.FormatPostingsFieldsWriter.finish(FormatPostingsFieldsWriter.java:70)
        at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:432)
        at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:108)
        at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4263)
        at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908)
        at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
        at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
        Suppressed: java.io.IOException: No space left on device
                at java.base/java.io.RandomAccessFile.writeBytes(Native Method)
                at java.base/java.io.RandomAccessFile.write(RandomAccessFile.java:559)
                at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:448)
                at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:99)
                at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:88)
                at org.apache.lucene.store.BufferedIndexOutput.close(BufferedIndexOutput.java:113)
                at org.apache.lucene.store.FSDirectory$FSIndexOutput.close(FSDirectory.java:458)
                at org.apache.lucene.util.IOUtils.close(IOUtils.java:141)
                at org.apache.lucene.index.FormatPostingsDocsWriter.close(FormatPostingsDocsWriter.java:134)
                at org.apache.lucene.index.FormatPostingsTermsWriter.close(FormatPostingsTermsWriter.java:71)
                ... 8 more
                Suppressed: java.io.IOException: No space left on device
                        at java.base/java.io.RandomAccessFile.writeBytes(Native Method)
                        at java.base/java.io.RandomAccessFile.write(RandomAccessFile.java:559)
                        at org.apache.lucene.store.FSDirectory$FSIndexOutput.flushBuffer(FSDirectory.java:448)
                        at org.apache.lucene.store.BufferedIndexOutput.flushBuffer(BufferedIndexOutput.java:99)
                        at org.apache.lucene.store.BufferedIndexOutput.flush(BufferedIndexOutput.java:88)
                        at org.apache.lucene.store.BufferedIndexOutput.close(BufferedIndexOutput.java:113)
                        at org.apache.lucene.store.FSDirectory$FSIndexOutput.close(FSDirectory.java:458)
                        at org.apache.lucene.util.IOUtils.close(IOUtils.java:141)
                        at org.apache.lucene.index.FormatPostingsPositionsWriter.close(FormatPostingsPositionsWriter.java:87)
                        ... 11 more

[DmitriyLewen]

I downloaded nexus-maven-repository-index.866.gz
archive size is 9.7 MB. After extraction - 139 MB.

It looks like all indexes need at least 35 GB
But this is just an estimate. It is possible that you will need more space (possibly much more)

PS
Maven central says about 50TB (https://mvnrepository.com/repos/central)
Also goneall said about 45 TB (aquasecurity/trivy#8118 (reply in thread))

[knqyf263]

I gave up the maven index again in 2025 😄 Feel free to merge this PR.