Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

download nvd data only with squid proxy #7270

Open
venerari opened this issue Dec 23, 2024 · 0 comments
Open

download nvd data only with squid proxy #7270

venerari opened this issue Dec 23, 2024 · 0 comments
Labels

Comments

@venerari
Copy link

venerari commented Dec 23, 2024

Running with no proxy is fine but running it against Squid proxy,

token=<omitted>
podman --log-level=debug run --rm -t -e 'http_proxy=http://192.168.99.90:3129 https_proxy=http://192.168.99.90:3129 no_proxy="localhost,127.0.0.1,localaddress,localdomain.com"' -v /home/ric/tmp4:/dependency-check-data:z docker.io/owasp/dependency-check:10.0.2 --updateonly --data /dependency-check-data --nvdApiKey $token

Will have this errors,

[INFO] Checking for updates
[WARN] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=0 : 2 time
[WARN] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=0 : 3 time
[WARN] Retrying request /rest/json/cves/2.0?resultsPerPage=2000&startIndex=0 : 4 time
[WARN] NVD API request failures are occurring; retrying request for the 5 time
[WARN] NVD API request failures are occurring; retrying request for the 6 time

Repeatedly and doesn't end.
On the /var/log/messages of the gateway,

Dec 23 23:05:57 centos9 named[5230]: client @0x7f0bf16419e8 192.168.99.91#59812 (services.nvd.nist.gov): query (cache) 'services.nvd.nist.gov/A/IN' denied
Dec 23 23:05:57 centos9 named[5230]: client @0x7f0bf163dbc8 192.168.99.91#59812 (services.nvd.nist.gov): query (cache) 'services.nvd.nist.gov/A/IN' denied
Dec 23 23:05:57 centos9 named[5230]: client @0x7f0bf1636af8 192.168.99.91#59812 (services.nvd.nist.gov): query (cache) 'services.nvd.nist.gov/AAAA/IN' denied
Dec 23 23:05:57 centos9 named[5230]: client @0x7f0bf1639d98 192.168.99.91#59812 (services.nvd.nist.gov): query (cache) 'services.nvd.nist.gov/AAAA/IN' denied

I have no problem on other tasks like wget download or curl nist.gov.

[ric@client1 ~]$ curl -I https://services.nvd.nist.gov
HTTP/1.1 200 Connection established

HTTP/1.1 200 OK
content-length: 0
content-type: text/html
x-frame-options: SAMEORIGIN
access-control-allow-origin: *
access-control-allow-headers: accept, apiKey, content-type, origin, x-requested-with
access-control-allow-methods: GET, HEAD, OPTIONS
access-control-allow-credentials: false
date: Tue, 24 Dec 2024 04:04:37 GMT
apikey: No
strict-transport-security: max-age=31536000

If I used the lastest 11.1.1, there will be new errors and it will end,

[WARN] NVD API request failures are occurring; retrying request for the 9 time
[WARN] NVD API request failures are occurring; retrying request for the 10 time
[WARN] NVD API request failures are occurring; retrying request for the 11 time
[ERROR] Error updating the NVD Data
org.owasp.dependencycheck.data.update.exception.UpdateException: Error updating the NVD Data
        at org.owasp.dependencycheck.data.update.NvdApiDataSource.processApi(NvdApiDataSource.java:397)
        at org.owasp.dependencycheck.data.update.NvdApiDataSource.update(NvdApiDataSource.java:117)
        at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:906)
        at org.owasp.dependencycheck.Engine.doUpdates(Engine.java:878)
        at org.owasp.dependencycheck.App.runUpdateOnly(App.java:431)
        at org.owasp.dependencycheck.App.run(App.java:175)
        at org.owasp.dependencycheck.App.main(App.java:90)

Here are the env of my proxy,

export http_proxy=http://192.168.99.90:3129
export https_proxy=http://192.168.99.90:3129
export no_proxy="localhost,127.0.0.1,localaddress,localdomain.com"
export HTTP_PROXY=${http_proxy}
export HTTPS_PROXY=${https_proxy}
export NO_PROXY=${no_proxy}

My /etc/named.conf,

options {
        listen-on port 53 { 127.0.0.1; any; };
        listen-on-v6 port 53 { ::1; any; };
        directory       "/var/named";
        dump-file       "/var/named/data/cache_dump.db";
        statistics-file "/var/named/data/named_stats.txt";
        memstatistics-file "/var/named/data/named_mem_stats.txt";
        secroots-file   "/var/named/data/named.secroots";
        recursing-file  "/var/named/data/named.recursing";
        allow-query     { localhost; };

        /*
         - If you are building an AUTHORITATIVE DNS server, do NOT enable recursion.
         - If you are building a RECURSIVE (caching) DNS server, you need to enable
           recursion.
         - If your recursive DNS server has a public IP address, you MUST enable access
           control to limit queries to your legitimate users. Failing to do so will
           cause your server to become part of large scale DNS amplification
           attacks. Implementing BCP38 within your network would greatly
           reduce such attack surface
        */
        recursion yes;

        dnssec-validation yes;

        managed-keys-directory "/var/named/dynamic";
        geoip-directory "/usr/share/GeoIP";

        pid-file "/run/named/named.pid";
        session-keyfile "/run/named/session.key";

        /* https://fedoraproject.org/wiki/Changes/CryptoPolicy */
        include "/etc/crypto-policies/back-ends/bind.config";

};

My squid,

acl mynetwork src 192.168.99.0/24
acl myurl dstdom_regex salesforce azure edgekey awsdns-22 akam cisa nist ntp
http_access allow mynetwork myurl

My gateway have two networks (external-192.168.133.0/24 and internal-192.168.99.0/24). I'm running from the client of internal.
The squid log have no errors, so even if I turned off squid, still the same error, meaning the error is only on the bind/dns.


[root@centos9 ~]# tail -f /var/log/squid/
access.log           access.log-20241223  cache.log            cache.log-20241223
[root@centos9 ~]# tail -f /var/log/squid/access.log
1734991799.552    355 192.168.99.91 TCP_MISS/200 468 POST https://eastus-1.in.applicationinsights.azure.com/v2.1/track - HIER_DIRECT/52.179.73.44 application/json
1734991799.557    360 192.168.99.91 TCP_MISS/200 468 POST https://eastus-1.in.applicationinsights.azure.com/v2.1/track - HIER_DIRECT/52.179.73.44 application/json
1734991802.343    659 192.168.99.91 TCP_MISS/200 468 POST https://westus-0.in.applicationinsights.azure.com/v2.1/track - HIER_DIRECT/20.189.172.33 application/json
1734991802.356    672 192.168.99.91 TCP_MISS/200 468 POST https://westus-0.in.applicationinsights.azure.com/v2.1/track - HIER_DIRECT/20.189.172.33 application/json
1735009581.138    553 192.168.99.91 TCP_MISS/301 449 GET http://salesforce.com/ - HIER_DIRECT/104.109.11.129 -
1735010154.549     71 192.168.99.91 TCP_MISS/301 278 GET http://nist.gov/ - HIER_DIRECT/129.6.13.49 -
1735010167.952     66 192.168.99.91 TCP_MISS/301 278 GET http://nist.gov/ - HIER_DIRECT/129.6.13.49 -
1735011402.145    636 192.168.99.91 TCP_MISS/301 278 GET http://nist.gov/ - HIER_DIRECT/129.6.13.49 -
1735011429.835     79 192.168.99.91 TCP_MISS/301 278 GET http://nist.gov/ - HIER_DIRECT/129.6.13.49 -
1735013077.883    647 192.168.99.91 TCP_TUNNEL/200 3549 CONNECT services.nvd.nist.gov:443 - HIER_DIRECT/18.235.227.114 -
1735014284.781    373 192.168.99.91 TCP_TUNNEL/200 3549 CONNECT services.nvd.nist.gov:443 - HIER_DIRECT/54.85.30.225 -

[root@centos9 ~]# tail -f /var/log/squid/cache.log
2024/12/23 23:22:58 kid1| Max Mem  size: 262144 KB
2024/12/23 23:22:58 kid1| Max Swap size: 0 KB
2024/12/23 23:22:58 kid1| Using Least Load store dir selection
2024/12/23 23:22:58 kid1| Set Current Directory to /var/spool/squid
2024/12/23 23:22:58 kid1| Finished loading MIME types and icons.
2024/12/23 23:22:58 kid1| HTCP Disabled.
2024/12/23 23:22:58 kid1| Squid plugin modules loaded: 0
2024/12/23 23:22:58 kid1| Adaptation support is off.
2024/12/23 23:22:58 kid1| Accepting HTTP Socket connections at conn3 local=[::]:3129 remote=[::] FD 11 flags=9
2024/12/23 23:22:59 kid1| storeLateRelease: released 0 objects

It didn't help putting the proxy as parameter, when I run it inside the docker,

[ric@client1 ~]$ ./dependency-check.sh --updateonly --data /tmp --nvdApiKey $token --proxyserver 192.16
8.99.90 --proxyport 3129 --nonProxyHosts "localhost,127.0.0.1,localaddress,localdomain.com" --log logs

Got this logs,


DEBUG - Setting: analyzer.nuspec.enabled='true'
2024-12-24 05:11:28,606 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.nugetconf.enabled='true'
2024-12-24 05:11:28,606 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.assembly.enabled='true'
2024-12-24 05:11:28,606 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.bundle.audit.enabled='true'
2024-12-24 05:11:28,606 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.filename.enabled='true'
2024-12-24 05:11:28,606 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.mix.audit.enabled='true'
2024-12-24 05:11:28,606 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.openssl.enabled='true'
2024-12-24 05:11:28,607 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.composer.lock.enabled='true'
2024-12-24 05:11:28,607 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.cpanfile.enabled='true'
2024-12-24 05:11:28,607 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.golang.dep.enabled='true'
2024-12-24 05:11:28,607 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.golang.mod.enabled='true'
2024-12-24 05:11:28,607 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.dart.enabled='true'
2024-12-24 05:11:28,607 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.node.package.enabled='true'
2024-12-24 05:11:28,607 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.node.audit.enabled='true'
2024-12-24 05:11:28,607 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.yarn.audit.enabled='true'
2024-12-24 05:11:28,608 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.pnpm.audit.enabled='true'
2024-12-24 05:11:28,608 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.node.audit.use.cache='true'
2024-12-24 05:11:28,608 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.retirejs.enabled='true'
2024-12-24 05:11:28,608 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.swift.package.manager.enabled='true'
2024-12-24 05:11:28,608 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.swift.package.resolved.enabled='true'
2024-12-24 05:11:28,608 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.cocoapods.enabled='true'
2024-12-24 05:11:28,608 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.carthage.enabled='true'
2024-12-24 05:11:28,609 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.ruby.gemspec.enabled='true'
2024-12-24 05:11:28,609 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.central.enabled='true'
2024-12-24 05:11:28,609 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.central.use.cache='true'
2024-12-24 05:11:28,609 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.ossindex.enabled='true'
2024-12-24 05:11:28,609 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.ossindex.use.cache='true'
2024-12-24 05:11:28,609 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: junit.fail.on.cvss='0.0'
2024-12-24 05:11:28,610 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: analyzer.nexus.proxy='true'
2024-12-24 05:11:28,610 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: nvd.api.key='********'
2024-12-24 05:11:28,610 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: hosted.suppressions.enabled='true'
2024-12-24 05:11:28,610 org.owasp.dependencycheck.utils.Settings:973
DEBUG - Setting: odc.autoupdate='true'
2024-12-24 05:11:28,650 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer MSBuild Project Analyzer
2024-12-24 05:11:28,651 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer Assembly Analyzer
2024-12-24 05:11:28,657 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer OpenSSL Source Analyzer
2024-12-24 05:11:28,659 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer Node.js Package Analyzer
2024-12-24 05:11:28,660 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer Node Audit Analyzer
2024-12-24 05:11:28,661 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer Yarn Audit Analyzer
2024-12-24 05:11:28,662 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer Pnpm Audit Analyzer
2024-12-24 05:11:28,664 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer RetireJS Analyzer
2024-12-24 05:11:28,667 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer Ruby Bundle Audit Analyzer
2024-12-24 05:11:28,671 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer Version Filter Analyzer
2024-12-24 05:11:28,672 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer Sonatype OSS Index Analyzer
2024-12-24 05:11:28,711 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer Known Exploited Vulnerability Analyzer
2024-12-24 05:11:28,712 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer Unused Suppression Rule Analyzer
2024-12-24 05:11:28,713 org.owasp.dependencycheck.analyzer.AnalyzerService:113
DEBUG - Loaded Analyzer Libman Analyzer
2024-12-24 05:11:28,714 org.owasp.dependencycheck.utils.Settings:1174
DEBUG - Settings.getDataFile() - file: '/tmp'
2024-12-24 05:11:28,881 org.owasp.dependencycheck.utils.Settings:1174
DEBUG - Settings.getDataFile() - file: '/tmp'
2024-12-24 05:11:28,881 org.owasp.dependencycheck.utils.Settings:1431
DEBUG - Data directory: /tmp
2024-12-24 05:11:28,881 org.owasp.dependencycheck.utils.Settings:1447
DEBUG - Connection String: 'jdbc:h2:file:/tmp/odc;AUTOCOMMIT=ON;CACHE_SIZE=65536;RETENTION_TIME=1000;MAX_COMPACT_TIME=10000;'
2024-12-24 05:11:28,882 org.owasp.dependencycheck.utils.Settings:1174
DEBUG - Settings.getDataFile() - file: '/tmp'
2024-12-24 05:11:28,908 org.owasp.dependencycheck.utils.WriteLock:168
DEBUG - Lock file created (main) ec7ba6813fe43af33e379c8b014ae2fe @ 2024-12-24 05:11:28.908
2024-12-24 05:11:28,916 org.owasp.dependencycheck.data.nvdcve.DatabaseManager:143
DEBUG - Loading driver 'org.h2.Driver'
2024-12-24 05:11:28,919 org.owasp.dependencycheck.utils.Settings:1174
DEBUG - Settings.getDataFile() - file: '/tmp'
2024-12-24 05:11:28,920 org.owasp.dependencycheck.utils.Settings:1431
DEBUG - Data directory: /tmp
2024-12-24 05:11:28,920 org.owasp.dependencycheck.utils.Settings:1447
DEBUG - Connection String: 'jdbc:h2:file:/tmp/odc;AUTOCOMMIT=ON;CACHE_SIZE=65536;RETENTION_TIME=1000;MAX_COMPACT_TIME=10000;'
2024-12-24 05:11:28,920 org.owasp.dependencycheck.utils.Settings:1174
DEBUG - Settings.getDataFile() - file: '/tmp'
2024-12-24 05:11:28,920 org.owasp.dependencycheck.data.nvdcve.DatabaseManager:172
DEBUG - Need to create DB Structure: true
2024-12-24 05:11:28,921 org.owasp.dependencycheck.data.nvdcve.DatabaseManager:178
DEBUG - Loading database connection
2024-12-24 05:11:28,921 org.owasp.dependencycheck.data.nvdcve.DatabaseManager:179
DEBUG - Connection String: jdbc:h2:file:/tmp/odc;AUTOCOMMIT=ON;CACHE_SIZE=65536;RETENTION_TIME=1000;MAX_COMPACT_TIME=10000;
2024-12-24 05:11:28,921 org.owasp.dependencycheck.data.nvdcve.DatabaseManager:180
DEBUG - Database User: dcuser
2024-12-24 05:11:29,078 org.owasp.dependencycheck.data.nvdcve.DatabaseManager:244
DEBUG - Database product: h2
2024-12-24 05:11:29,078 org.owasp.dependencycheck.data.nvdcve.DatabaseManager:371
DEBUG - Creating database structure
2024-12-24 05:11:29,137 org.owasp.dependencycheck.data.nvdcve.DatabaseManager:505
DEBUG - DC Schema: 5.5
2024-12-24 05:11:29,146 org.owasp.dependencycheck.data.nvdcve.DatabaseManager:506
DEBUG - DB Schema: 5.5
2024-12-24 05:11:29,209 org.owasp.dependencycheck.Engine:897
INFO  - Checking for updates
2024-12-24 05:11:29,226 io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient:127
DEBUG - rate limited call delay: 5000
2024-12-24 05:11:29,383 io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient:127
DEBUG - rate limited call delay: 5000
2024-12-24 05:11:29,384 io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient:127
DEBUG - rate limited call delay: 5000
2024-12-24 05:11:29,386 io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient:127
DEBUG - rate limited call delay: 5000
2024-12-24 05:11:29,432 io.github.jeremylong.openvulnerability.client.nvd.NvdCveClient:274
DEBUG - requesting URI: https://services.nvd.nist.gov/rest/json/cves/2.0?resultsPerPage=2000&startIndex=0
2024-12-24 05:11:29,437 io.github.jeremylong.openvulnerability.client.nvd.RateMeter:83
DEBUG - Ticket taken At: 05:11:29; count: 1; by 47
2024-12-24 05:11:29,437 io.github.jeremylong.openvulnerability.client.nvd.RateLimitedClient:198
DEBUG - Requested At: 05:11:29; URI: /rest/json/cves/2.0?resultsPerPage=2000&startIndex=0
2024-12-24 05:11:49,925 org.owasp.dependencycheck.utils.WriteLock:240
DEBUG - Lock released (Thread-4) ec7ba6813fe43af33e379c8b014ae2fe @ 2024-12-24 05:11:49.925

Please help me!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant