Debian 8.6 (jessie) scan does not show vulnerable packages

[jrw171819]

Environment

Debian 8.6 (Jessie)

Vuls

vuls v0.1.6 6f012fc (we patched the current /scan/debian.go a little (mostly Infof's) to understand the problem)

OS

• Target Server: Debian 8.6
• Vuls Server: Debian 8.6

Go

• Go version: go1.6 linux/amd64

Current Output

[Oct 29 17:05:04] DEBUG [SERVER-Name] Ensure changelog cache: SERVER-Name
[Oct 29 17:05:04] DEBUG [SERVER-Name] Reuse meta: SERVER-Name
[Oct 29 17:05:04] DEBUG [localhost] key:SERVER-Name, value:{"Name":"SERVER-Name","Distro":{"Family":"debian","Release":"8.6"},"Packs":[{"Name ":"libdbd-mysql-perl","Version":"4.028-2+deb8u1","Release":"","NewVersion":"4.028-2+deb8u2","NewRelease":""}]}
[Oct 29 17:05:04] DEBUG [localhost] key:ghostscript, len: 8635, E: Unable to replace /home/inv...
[Oct 29 17:05:04] DEBUG [localhost] key:libapache2-mod-php5, len: 26300, E: Unable to replace /home/inv...
[Oct 29 17:05:04] DEBUG [localhost] key:libdbd-mysql-perl, len: 4248, E: Changelog download failed: ...
[Oct 29 17:05:04] DEBUG [localhost] key:libgd3, len: 6410, E: Changelog download failed: ...
[Oct 29 17:05:04] DEBUG [localhost] key:libgs9, len: 8500, E: Changelog download failed: ...
[Oct 29 17:05:04] DEBUG [localhost] key:libgs9-common, len: 8500, E: Changelog download failed: ...
[Oct 29 17:05:04] DEBUG [localhost] key:linux-image-3.16.0-4-amd64, len: 54537, E: Unable to replace /home/inv...
[Oct 29 17:05:04] DEBUG [localhost] key:linux-libc-dev, len: 54402, E: Changelog download failed: ...
[Oct 29 17:05:04] DEBUG [localhost] key:php5-cli, len: 26300, E: Unable to replace /home/inv...
[Oct 29 17:05:04] DEBUG [localhost] key:php5-common, len: 26300, E: Unable to replace /home/inv...
[Oct 29 17:05:04] DEBUG [localhost] key:php5-curl, len: 26165, E: Changelog download failed: ...
[Oct 29 17:05:04] DEBUG [localhost] key:php5-gd, len: 26165, E: Changelog download failed: ...
[Oct 29 17:05:04] DEBUG [localhost] key:php5-mcrypt, len: 26165, E: Changelog download failed: ...
[Oct 29 17:05:04] DEBUG [localhost] key:php5-mysql, len: 26165, E: Changelog download failed: ...
[Oct 29 17:05:04] DEBUG [localhost] key:php5-readline, len: 26165, E: Changelog download failed: ...
[Oct 29 17:05:04] DEBUG [localhost] key:tzdata, len: 7257, tzdata (2016h-0+deb8u1) stable...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 8635, E: Unable to replace /home/inv...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: ghostscript (9.06dfsg-2+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 26165, E: Changelog download failed: ...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: php5 (5.6.24+dfsg-0+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 26165, E: Changelog download failed: ...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: php5 (5.6.24+dfsg-0+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 54402, E: Changelog download failed: ...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: linux (3.16.36-1+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 54537, E: Unable to replace /home/inv...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: linux (3.16.36-1+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 4248, E: Changelog download failed: ...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: libdbd-mysql-perl (4.028-2+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 26300, E: Unable to replace /home/inv...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: php5 (5.6.24+dfsg-0+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 26165, E: Changelog download failed: ...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: php5 (5.6.24+dfsg-0+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 26300, E: Unable to replace /home/inv...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: php5 (5.6.24+dfsg-0+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 6410, E: Changelog download failed: ...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: libgd2 (2.1.0-5+deb8u6) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 7257, tzdata (2016h-0+deb8u1) stable...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: tzdata (2016f-0+deb8u1) stable; urgency=medium
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 26165, E: Changelog download failed: ...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: php5 (5.6.24+dfsg-0+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 26165, E: Changelog download failed: ...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: php5 (5.6.24+dfsg-0+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 26300, E: Unable to replace /home/inv...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: php5 (5.6.24+dfsg-0+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 8500, E: Changelog download failed: ...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: ghostscript (9.06dfsg-2+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] DEBUG [SERVER-Name] Cache hit: SERVER-Name, len: 8500, E: Changelog download failed: ...
[Oct 29 17:05:05] DEBUG [SERVER-Name] Found the stop line. line: ghostscript (9.06dfsg-2+deb8u1) jessie-security; urgency=high
[Oct 29 17:05:05] INFO SERVER-Name Scanned ghostscript-9.06dfsg-2+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned php5-mysql-5.6.24+dfsg-0+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned php5-gd-5.6.24+dfsg-0+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned linux-libc-dev-3.16.36-1+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned linux-image-3.16.0-4-amd64-3.16.36-1+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned libdbd-mysql-perl-4.028-2+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned libapache2-mod-php5-5.6.24+dfsg-0+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned php5-curl-5.6.24+dfsg-0+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned php5-common-5.6.24+dfsg-0+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned libgd3-2.1.0-5+deb8u6 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned tzdata-2016f-0+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned php5-readline-5.6.24+dfsg-0+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned php5-mcrypt-5.6.24+dfsg-0+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned php5-cli-5.6.24+dfsg-0+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned libgs9-common-9.06dfsg-2+deb8u1 : []
[Oct 29 17:05:05] INFO SERVER-Name Scanned libgs9-9.06dfsg-2+deb8u1 : []
[Oct 29 17:05:05] DEBUG [SERVER-Name] 0 Cves are found. cves: []
[Oct 29 17:05:05] INFO [SERVER-Name] Fetching CVE details...
[Oct 29 17:05:05] INFO [SERVER-Name] Done

Scan Result:
SERVER-Name (debian8.6)
No unsecure packages.

Actual Behavior

We are scanning four nearly identical servers (all Debian 8.6) since some weeks, two of them did recently show up unsecure packages, the other two didn't. All Servers were taken out of our weekly upgrade circle, so they were all in the same unpatched state since a while. While two servers were shown corretly as vulnerable, the other two, like the example above, report status healthy, which is clearly wrong.

After some research we found out some problems that in sum can lead to this behavior in the end:

1.) It seems, that vuls caches Error-Messages from apt-get/aptitude like (Cache hit: SERVER-Name, len: 26300, E: Unable to replace /home/scanuser/.aptitude/config, file does'nt exist or Cache hit: SERVER-Name, len: 26165, E: Changelog download failed: ...) like "real" changelogs
2.) The cache does not seem to expire, which leads into the situation, that error messages from apt-get/aptitude once cached are always handled like real changelog entries
3.) These "broken" Changlog entries lead into no CVE-Hits for the package, even if that package was recently upgraded and is very likely unsecure
4.) The lack of CVE hits for the package obv. leads to a wrongly healthy package

(This is at least our impression, but we are not very familiar with go)

Expected Behavior

1.) Error messages from apt-get/aptitude should not be cached - it should be made sure that the cache entry is a valid changelog
2.) If this is not possible, the cache should maybe expire, or the usermanual should mention that the cache db should be deleted frequently (which works perfectly for us)
3.) Even if a changelog is unavailable or doesnt say anything useful for an updated package, the chance is pretty high that the server isn't healthy anymore. Missing changelogs should maybe not lead to a healthy server, but to a server with unspecifyable problems.

Steps to reproduce the behaviour

1.) Fill your cache.db with some trash, for example by running an older vuls version, that still uses apt-get changelog for debian systems - these requests lead mostly into a 404 error - OR - use a debian system that is not configured for aptitude (and aptitude sudo likewise)
2.) update your vuls to current version - but do not delete the cache
3.) Scan - and there you go :)

Steps to solve the issue

1.) Aptitude should be added as required package for debian/Ubuntu Systems (and in the sudo Nopassword section), also, aptitude must be manually executed at least once by the scanuser (command aptitude, then close again)
2.) delete your cache.db (most important)

One more thing

Your approach is great, we really appreciate your idea and work! Thank you a lot!

[kotakanbe]

Hi, @jrw171819
Thanks for reporting.

I understand the problem.
I am going to fix it.

[kotakanbe]

Fixed.

Now, Vuls doesn't cache a changelog If the grepped length is zero.

        cmd = fmt.Sprintf(`aptitude changelog %s | grep '\(urgency\|CVE\)'`, pack.Name)

7681b27#diff-44e83f60130e99fae5610fad77771526L535

Please test on your environment if you have time.
Thanks for reporting !

[jrw171819]

Wow - Now that was really fast! Thank you for looking into it, we will try to reproduce the old behavior (and test the new one) until beginning of next week.

Thank you a lot!