Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[GSOC23] - B - Enable the downloading and synchronization of OVAL data #7509

Merged
merged 64 commits into from
Sep 5, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
64 commits
Select commit Hold shift + click to select a range
bae562e
Read OVAL data sources from a configuration file
HoussemNasri Sep 14, 2023
01aeecb
Implement the downloading of OVALs based on the config file
HoussemNasri Sep 14, 2023
44ec0e4
Add taskomatic task for syncing OVAL data
HoussemNasri Sep 14, 2023
d2bd4d9
Indicate in UI when vulnerability scan was based on OVAL or Channels …
HoussemNasri Aug 22, 2023
6c3ea0e
Add Leap 15.2 OVAL sources
HoussemNasri Aug 22, 2023
ddbf4d5
Deploy oval.config.json to /usr/share/susemanager/scc/
HoussemNasri Sep 5, 2023
c298db7
Add support for SLE Micro OVAL-based CVE auditing
HoussemNasri Sep 5, 2023
a180ead
Add support for openSUSE Micro OVAL-based CVE auditing
HoussemNasri Sep 5, 2023
f97857a
Add SLE and Leap Micro OVAL sources
HoussemNasri Sep 7, 2023
914fb48
Synchronize OVAL data only for products used by registered clients
HoussemNasri Sep 9, 2023
4160040
Log the products eligible for OVAL synchronization
HoussemNasri Sep 14, 2023
03c6bb1
Translate oval sync task related strings
HoussemNasri Oct 2, 2023
7c2f376
Test OVAL sources
HoussemNasri Oct 3, 2023
51a3a47
Use Leap in oval.config.json
HoussemNasri Oct 26, 2023
529ddd7
Use proper logging when syncing OVAL data
HoussemNasri Feb 18, 2024
c53980f
Rename 'openSUSE_LEAP_MICRO' to 'LEAP_MICRO'
HoussemNasri Feb 18, 2024
d9102ab
Document the OVAL downloader component classes
HoussemNasri Feb 18, 2024
4e7c6da
Refactor and fix checkstyle
HoussemNasri Feb 18, 2024
ae819ba
Use format specifiers instead of concatenation when logging
HoussemNasri Feb 24, 2024
170d3af
Remove unused methods
HoussemNasri Feb 25, 2024
7a9ac7f
Refactor Server#toOVALProduct
HoussemNasri Feb 25, 2024
e783046
Add changelogs
HoussemNasri Feb 29, 2024
4298480
Revert "Use channel data until full OVAL implementation is ready"
HoussemNasri Feb 29, 2024
e96be51
Copyright (c) 2024
HoussemNasri Mar 9, 2024
0f13988
Prevent synchronizing OVAL data for the same product twice
HoussemNasri Mar 9, 2024
baed757
Avoid fetching all servers to detect the set of operating systems the…
HoussemNasri Mar 16, 2024
ddcfda1
Log the starting and ending of the OVAL syncing task
HoussemNasri Mar 16, 2024
89dc45a
Rephrase comment in OVALCleaner
HoussemNasri Mar 16, 2024
00ae7c0
Fix typo in oval.config.json
HoussemNasri Mar 16, 2024
374bb8e
Refactor OVALDownloader
HoussemNasri Mar 17, 2024
8e61f4a
Refactor to avoid code duplication
HoussemNasri Mar 17, 2024
518a859
Clarify comments
HoussemNasri Mar 17, 2024
17d3379
Increase connection and read timeouts while downloading OVAL data
HoussemNasri Apr 11, 2024
e57ce14
Store downloaded OVAL files in a cache directory
HoussemNasri Apr 11, 2024
4f42473
Move the toOVALProduct method to OsReleasePair
HoussemNasri Mar 16, 2024
c3c46e4
Undo OVAL support for ubuntu
HoussemNasri Apr 11, 2024
954c7dd
Extract the cve from <cve> tag in OVAL files
HoussemNasri May 5, 2024
a9aa82f
Display the list of data sources used to scan each client server
HoussemNasri May 5, 2024
1332260
Change regex to not be vulnerable to polynomial runtime due to backtr…
HoussemNasri May 5, 2024
c4103f0
Introduce UI warning for potential inaccuracies in CVE audit results
HoussemNasri May 11, 2024
8f7e4cd
Implement the check for existence of erratas in server CVE channels
HoussemNasri May 12, 2024
21c56b8
Delete cveaudit.less
HoussemNasri May 12, 2024
998916e
Remove the "Scan Data" column from the CVE audit page
HoussemNasri May 14, 2024
8f66bc9
Translate strings
HoussemNasri May 16, 2024
f666d1b
Log error on invalid CVE scan data sources
HoussemNasri May 16, 2024
3a5dc43
Insert missing data related to the sync OVAL taskomatic task
HoussemNasri May 16, 2024
d9e36a4
Rephrase CVE auditing related UI messages
HoussemNasri May 18, 2024
1615168
Fix bug where the system name in the CVE auditing table results not d…
HoussemNasri May 18, 2024
44ebb6e
Introduce an "unknown" patch status to be returned when OVAL and Chan…
HoussemNasri May 19, 2024
1cc5f08
Introduce a shell script that synchronizes the oval.config.json file …
HoussemNasri May 19, 2024
f96ddc0
Add missing OVAL data sources
HoussemNasri May 19, 2024
01567d4
Move OVAL config file to a more appropriate directory
HoussemNasri May 21, 2024
27d86c7
Configure OVAL for SLE/Leap Micro 6.0 and Leap 15.6
HoussemNasri May 21, 2024
18c8852
Fix linter
HoussemNasri May 21, 2024
c838766
Frontend lint
HoussemNasri May 28, 2024
0f128b7
Clear previous OVAL metadata for platform before inserting the newer …
HoussemNasri May 28, 2024
b0d1d34
Add an configuration option to enable/disable OVAL metadata usage in …
HoussemNasri Jun 6, 2024
54372ea
Correct the location of the OVAL config file in RPM spec
HoussemNasri Jun 29, 2024
9a64863
Fix sles and sled OVAL product matching
HoussemNasri Jun 29, 2024
88e06ef
Fix OVAL's release package regex
HoussemNasri Jun 29, 2024
3c7637d
Relocate database migration script
HoussemNasri Jul 1, 2024
34229c5
Continue syncing other products if OVAL data syncing fails for one
HoussemNasri Jul 7, 2024
81026c5
Change string#replace by string#replaceFirst
HoussemNasri Jul 11, 2024
5855da9
Move schema migration to current dir
parlt91 Sep 5, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions java/code/src/com/redhat/rhn/common/conf/ConfigDefaults.java
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,8 @@ public class ConfigDefaults {

public static final String MESSAGE_QUEUE_THREAD_POOL_SIZE = "java.message_queue_thread_pool_size";

public static final String CVE_AUDIT_ENABLE_OVAL_METADATA = "java.cve_audit.enable_oval_metadata";

/**
* Token lifetime in seconds
*/
Expand Down Expand Up @@ -1188,4 +1190,13 @@ public int getRebootDelay() {

return rebootDelay;
}

/**
* Check if the usage of OVAL metadata is permitted in scanning systems for CVE vulnerabilities.
*
* @return {@code true} if OVAL usage is permitted and {@code false} otherwise.
* */
public boolean isOvalEnabledForCveAudit() {
return Config.get().getBoolean(CVE_AUDIT_ENABLE_OVAL_METADATA, false);
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -29,4 +29,27 @@
AND cve.name = :cve_name;
</query>
</mode>

<mode name="check_oval_availability">
<query params="cpe">
SELECT 1 FROM suseOVALPlatform plat WHERE starts_with(:cpe, plat.cpe);
HoussemNasri marked this conversation as resolved.
Show resolved Hide resolved
</query>
</mode>

<mode name="check_errata_availability">
<query params="server_id">
SELECT 1
FROM suseCVEServerChannel,
rhnChannelErrata
WHERE suseCVEServerChannel.channel_id = rhnChannelErrata.channel_id
AND server_id = :server_id
</query>
</mode>

<write-mode name="clear_oval_metadata_by_platform">
<query params="cpe">
DELETE FROM suseOVALPlatformVulnerablePackage pvp WHERE pvp.platform_id = (SELECT id FROM suseOVALPlatform plat WHERE plat.cpe = :cpe);
DELETE FROM suseOVALPlatform plat where plat.cpe = :cpe;
</query>
</write-mode>
</datasource_modes>
23 changes: 23 additions & 0 deletions java/code/src/com/redhat/rhn/domain/server/Server.java
Original file line number Diff line number Diff line change
Expand Up @@ -2431,6 +2431,13 @@ public boolean doesOsSupportPtf() {
return ServerConstants.SLES.equals(getOs());
}

boolean isSLES() {
return ServerConstants.SLES.equalsIgnoreCase(getOs());
}
boolean isSLED() {
return ServerConstants.SLED.equalsIgnoreCase(getOs());
}

/**
* Return <code>true</code> if OS supports Confidential Computing Attestation
*
Expand Down Expand Up @@ -2476,6 +2483,10 @@ boolean isSLES15() {
return ServerConstants.SLES.equals(getOs()) && getRelease().startsWith("15");
}

boolean isLeap() {
return ServerConstants.LEAP.equalsIgnoreCase(getOs());
}

boolean isLeap15() {
return ServerConstants.LEAP.equalsIgnoreCase(getOs()) && getRelease().startsWith("15");
}
Expand All @@ -2494,6 +2505,10 @@ boolean isopenSUSEMicroOS() {
return ServerConstants.OPENSUSEMICROOS.equals(getOs());
}

boolean isUbuntu() {
return ServerConstants.UBUNTU.equalsIgnoreCase(getOs());
}

boolean isUbuntu1804() {
return ServerConstants.UBUNTU.equals(getOs()) && getRelease().equals("18.04");
}
Expand All @@ -2506,6 +2521,10 @@ boolean isUbuntu2204() {
return ServerConstants.UBUNTU.equals(getOs()) && getRelease().equals("22.04");
}

boolean isDebian() {
return ServerConstants.DEBIAN.equalsIgnoreCase(getOs());
}

boolean isDebian12() {
return ServerConstants.DEBIAN.equals(getOs()) && getRelease().equals("12");
}
Expand All @@ -2518,6 +2537,10 @@ boolean isDebian10() {
return ServerConstants.DEBIAN.equals(getOs()) && getRelease().equals("10");
}

boolean isRHEL() {
return ServerConstants.RHEL.equals(getOs());
rjmateus marked this conversation as resolved.
Show resolved Hide resolved
}

/**
* This is supposed to cover all RedHat flavors (incl. RHEL, RES and CentOS Linux)
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,8 @@ public class ServerConstants {
public static final String ALMA = "AlmaLinux";
public static final String AMAZON = "Amazon Linux";
public static final String ROCKY = "Rocky";
public static final String SLED = "SLED";
public static final String RHEL = "Red Hat Enterprise Linux";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we re-use the existing variable for REDHAT?


private ServerConstants() {

Expand Down
15 changes: 15 additions & 0 deletions java/code/src/com/redhat/rhn/domain/server/ServerFactory.java
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@
import com.redhat.rhn.frontend.dto.SystemOverview;
import com.redhat.rhn.frontend.xmlrpc.ChannelSubscriptionException;
import com.redhat.rhn.frontend.xmlrpc.ServerNotInGroupException;
import com.redhat.rhn.manager.audit.OsReleasePair;
import com.redhat.rhn.manager.entitlement.EntitlementManager;
import com.redhat.rhn.manager.rhnset.RhnSetDecl;
import com.redhat.rhn.manager.system.SystemManager;
Expand All @@ -66,6 +67,7 @@
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.Date;
import java.util.HashMap;
import java.util.HashSet;
Expand Down Expand Up @@ -249,6 +251,19 @@ public static Optional<Server> lookupProxyServer(String name) {
}
}

/**
* List the <b>unique</b> set of pairs of os and release versions used by servers
*
* @return the set of unique pairs of os and release version used by servers
* */
public static Set<OsReleasePair> listAllServersOsAndRelease() {
List<Object[]> result = SINGLETON.listObjectsByNamedQuery("Server.listAllServersOsAndRelease",
Collections.emptyMap());

return result.stream().map(row -> new OsReleasePair((String) row[0], (String) row[1]))
.collect(Collectors.toSet());
}

/**
* Return a map from Salt minion IDs to System IDs.
* Map entries are limited to systems that are visible by the specified user.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -361,6 +361,12 @@ PUBLIC "-//Hibernate/Hibernate Mapping DTD 3.0//EN"
WHERE USP.user_id = :user_id AND USP.server_id = s.id)
]]>
</sql-query>

<sql-query name="Server.listAllServersOsAndRelease">
<![CDATA[
SELECT DISTINCT s.os, s.release FROM rhnServer s
]]>
</sql-query>

<sql-query name="Server.findSimpleMinionsByServerIds">
<![CDATA[
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8791,6 +8791,9 @@ Alternatively, you will want to download &lt;strong&gt;Incremental Channel Conte
<trans-unit id="task.status.cve-server-channels" xml:space="preserve">
<source>CVE Server Channels</source>
</trans-unit>
<trans-unit id="task.status.oval-data-sync" xml:space="preserve">
<source>Sync OVAL Data</source>
</trans-unit>
<trans-unit id="task.status.mgr-sync-refresh" xml:space="preserve">
<source>Refresh mgr-sync data</source>
</trans-unit>
Expand Down Expand Up @@ -8833,6 +8836,9 @@ Alternatively, you will want to download &lt;strong&gt;Incremental Channel Conte
<trans-unit id="bunch.jsp.description.cve-server-channels-bunch" xml:space="preserve">
<source>Generates data required for performing CVE audit queries</source>
</trans-unit>
<trans-unit id="bunch.jsp.description.oval-data-sync-bunch" xml:space="preserve">
<source>Generate OVAL data required to increase the accuracy of CVE audit queries</source>
</trans-unit>
<trans-unit id="bunch.jsp.description.mgr-sync-refresh-bunch" xml:space="preserve">
<source>Refreshes data about channels, products and subscriptions</source>
</trans-unit>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
import com.redhat.rhn.frontend.xmlrpc.MethodInvalidParamException;
import com.redhat.rhn.frontend.xmlrpc.UnknownCVEIdentifierFaultException;
import com.redhat.rhn.manager.audit.CVEAuditImage;
import com.redhat.rhn.manager.audit.CVEAuditManager;
import com.redhat.rhn.manager.audit.CVEAuditManagerOVAL;
import com.redhat.rhn.manager.audit.CVEAuditServer;
import com.redhat.rhn.manager.audit.PatchStatus;
import com.redhat.rhn.manager.audit.UnknownCVEIdentifierException;
Expand Down Expand Up @@ -117,8 +117,7 @@ public List<CVEAuditServer> listSystemsByPatchStatus(User loggedInUser,
}

try {
// TODO: Use CVEAuditManagerOVAL once it's ready
List<CVEAuditServer> result = CVEAuditManager.listSystemsByPatchStatus(
List<CVEAuditServer> result = CVEAuditManagerOVAL.listSystemsByPatchStatus(
loggedInUser, cveIdentifier, patchStatuses);

result.sort(Comparator.comparingInt(s -> s.getPatchStatus().getRank()));
Expand Down Expand Up @@ -210,8 +209,7 @@ public List<CVEAuditImage> listImagesByPatchStatus(User loggedInUser,
}

try {
// TODO: Use CVEAuditManagerOVAL once it's ready
List<CVEAuditImage> result = CVEAuditManager.listImagesByPatchStatus(
List<CVEAuditImage> result = CVEAuditManagerOVAL.listImagesByPatchStatus(
loggedInUser, cveIdentifier, patchStatuses);

result.sort(Comparator.comparingInt(i -> i.getPatchStatus().getRank()));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -94,5 +94,9 @@ public Set<ErrataIdAdvisoryPair> getErratas() {
return erratas;
}

@Override
public Set<ScanDataSource> getScanDataSources() {
return Set.of();
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -877,7 +877,8 @@ public static List<CVEAuditServer> listSystemsByPatchStatus(User user,
system.getSystemName(),
system.getPatchStatus(),
system.getChannels(),
system.getErratas()
system.getErratas(),
Set.of(ScanDataSource.CHANNELS)
)).collect(Collectors.toList());
}

Expand Down
Loading
Loading