ZGC Generational Support #414

j-bahr · 2025-02-20T17:13:26Z

This patch adds support for generational ZGC and maintains support for legacy (non generational ZGC). There are a number of potential API breaking changes for consumers of GC events. List of changes and modifications are listed below.

Introduce new GCEvent types, FullZGCCycle (legacy), MajorZGCCycle and MinorZGCCycle representing the respective types.
GCEvents Full/Major/Minor GC Cycles hold reference to internal ZGCCycle, a delegate in the case of Full (legacy) cycle, young and for Major cycle and simply a young for Minor cycle. All cycle types full/young/old, have mostly similar overlapping fields, which allows us to have a single data class for all cycle types. Cycles themselves have a few fields that are specific to entire cycle and are held in the top level object.

┌─────────────┐
│    Full     │
│             │      ┌────────┐
│  delegate ──┼─────▶│ZGCCycle│
│             │      └────────┘
└─────────────┘
┌─────────────┐
│    Major    │
│             │      ┌────────┐
│    young ───┼─────▶│ZGCCycle│
│             │      ├────────┤
│     old  ───┼─────▶│ZGCCycle│
└─────────────┘      └────────┘
┌─────────────┐
│    Minor    │
│             │     ┌────────┐
│    young ───┼────▶│ZGCCycle│
│             │     └────────┘
└─────────────┘

Add stronger typing on the heap summary report. Live, Garbage, Allocated, Reclaimed, Promoted and Compacted carry information for a different stages of the GC. Each holds a different number of fields. Having a single summary with primitive double doesn't make sense as you can't differentiate between 0 and not present. We could use optional or boxed type but it seemed more direct to declare a type. In future java versions these could simply be Record classes.
Added concept of phase to the ZGC Parsers. The phase represents whether the GC cycle is in Major young/old, Minor young or Full phase (legacy). This maps to the GC output of Y, y, O prefix on the log line. We key off this to make a determination about which heap statistics we are writing to. Phase is determined by an optional capture group at the head of all log lines.
Introduced new hierarchical forward reference type to allow it to cary more contextual information about either young or old generation. Major obviously has two references, one for old and one for young. Minor and full GC's cary a single forward ref corresponding to their cycle.
Added new ZGC generational test for both Minor and Major cycle types ensuring that both are parsed correctly.

This patch adds support for generational ZGC and maintains support for legacy (non generational ZGC). There are a number of potential API breaking changes for consumers of GC events. List of changes and modifications are listed below. 1. Introduce new GCEvent types, FullZGCCycle (legacy), MajorZGCCycle and MinorZGCCycle representing the respective types. 2. GCEvents Full|Major|Minor GC Cycles hold reference to internal ZGCCycle, a delegate in the case of Full (legacy) cycle, young and for Major cycle and simply a young for Minor cycle. All cycle types full/young/old, have mostly similar overlapping fields, which allows us to have a single data class for all cycle types. Cycles themselves have a few fields that are specific to entire cycle and are held in the top level object. ``` ┌─────────────┐ │ Full │ │ │ ┌────────┐ │ delegate ──┼─────▶│ZGCCycle│ │ │ └────────┘ └─────────────┘ ┌─────────────┐ │ Major │ │ │ ┌────────┐ │ young ───┼─────▶│ZGCCycle│ │ │ ├────────┤ │ old ───┼─────▶│ZGCCycle│ └─────────────┘ └────────┘ ┌─────────────┐ │ Minor │ │ │ ┌────────┐ │ young ───┼────▶│ZGCCycle│ │ │ └────────┘ └─────────────┘ ``` 3. Add stronger typing on the heap summary report. Live, Garbage, Allocated, Reclaimed, Promoted and Compacted carry information for a different stages of the GC. Each holds a different number of events. Having a single summary with primitive `double` doesn't make sense as you can't differentiate between 0 and non present. We could use optional or boxed type but it seemed more direct to declare a type. In future java versions these could simply be Record classes. 4. Added concept of phase to the ZGC Parsers. The phase represents whether the GC cycle is in Major young/old, Minor young or Full phase (legacy). This maps to the GC output of 'Y', 'y', 'O' prefix on the log line. We key off this to make a determination about which heap statistics we are writing to. Phase is determined by an optional capture group at the head of all log lines. 5. Introduced new hierarchical forward reference type to allow it to cary more contextual information about either young or old generation. Major obviously has two references, one for old and one for young. Minor and full GC's cary a single forward ref corresponding to their cycle. 6. Added new ZGC generational test for both Minor and Major cycle types ensuring that both are parsed correctly.

Adds missing GC cause, based on enum toString defined in `jdk/src/hotspot/share/gc/shared/gcCause.cpp`

adolfojunior · 2025-02-21T17:16:31Z

👍

karianna · 2025-02-22T03:33:33Z

Thanks for this PR! It may take us a day or two to pick through it, but it's appreciated.

karianna · 2025-02-22T03:33:36Z

On a side note: @kcpeppe & @dsgrieve - Not sure if our LICENSE file covers everything or whether we need to do a separate PR to add the minimal license header to all of the files that are missing it.

kcpeppe

Thank you for the very high quality patch. We will release this ASAP, likely tomorrow (Feb 26). Please create a PR for any comments you would like to address.

kcpeppe · 2025-02-25T21:30:12Z

api/src/main/java/com/microsoft/gctoolkit/event/zgc/ZGCMemorySummary.java

sorry if this is pedantic but occupancyBefore and occupancyAfter are more meaningful than start and end. I believe this is consistent with the other MemoryPool. Also, the other MemoryPool abstractions contain the size of the pool. In most cases this is the current size of Java heap.

I think this makes sense. The memory summary was meant to capture the closing line on any of the cycles eg

GC(40) Garbage Collection (Proactive) 698M(2%)->262M(1%)

where the occupancy was 698M before and 262M after.

There are other other log lines that convey the capacity of the heap which could be combined to form a complete summary. I'm happy to attempt to implement this.

GC(40) Min Capacity: 8M(0%) GC(40) Max Capacity: 30718M(100%) GC(40) Soft Max Capacity: 30718M(100%)

kcpeppe · 2025-02-25T22:01:04Z

parser/src/main/java/com/microsoft/gctoolkit/parser/GCLogTrace.java

@@ -289,4 +302,13 @@ public void notYetImplemented() {
        System.out.println("-----------------------------------------");
        //}
    }
+
+    public ZGCPhase getZCollectionPhase() {


A GCLogTrace is a convenience wrapper over the Matcher. It contains convenience methods to extract data from the matcher. It is expected that the parser has the context to know which methods to call and which groups need to be extracted. Thus ZGCPhase is either known and extracted by the parser at that time or ZGCPhase isn't present in the log line (matcher) and the parser, again, should know this and not call this method. Either way, ZGCPhase should be consistent and there should be no need for downstream exception handling. This is something that was noted in ZGCPhase that an illegal argument was being thrown which likely is unnecessary.

kcpeppe · 2025-02-25T22:01:44Z

parser/src/main/java/com/microsoft/gctoolkit/parser/GCLogTrace.java

@@ -85,6 +89,15 @@ public double getMilliseconds(int index) {
        return getDoubleGroup(index);
    }

+    /**


The parser should know this.. and it should be picked up in testing

kcpeppe · 2025-02-25T22:03:26Z

api/src/main/java/com/microsoft/gctoolkit/event/zgc/ZGCCollectionType.java

+                return status;
+            }
+        }
+        throw new IllegalArgumentException("No matching ZGCCollectionType found for: " + label);


We should never get here.

kcpeppe · 2025-02-25T22:05:29Z

api/src/main/java/com/microsoft/gctoolkit/event/zgc/ZGCGarbageSummary.java

+// Licensed under the MIT License.
+package com.microsoft.gctoolkit.event.zgc;
+
+public class ZGCGarbageSummary {


are these values heap occupancy before or after the collection?

kcpeppe · 2025-02-25T22:07:33Z

api/src/main/java/com/microsoft/gctoolkit/event/zgc/ZGCMemorySummary.java

+
+    public ZGCMemorySummary(long start, long end) {
+        this.start = start;
+        this.end = end;


start and end are ambiguous, can these names be more descriptive?

kcpeppe · 2025-02-25T22:08:13Z

api/src/main/java/com/microsoft/gctoolkit/event/zgc/ZGCPhase.java

+                return zgcPhase;
+            }
+        }
+        throw new IllegalArgumentException("No matching ZGCPhase found for: " + label);


should never get here

kcpeppe · 2025-02-25T22:10:56Z

parser/src/main/java/com/microsoft/gctoolkit/parser/GenericTokens.java

@@ -20,6 +20,7 @@ public interface GenericTokens {
    //Time
    String TIME = "(-?" + REAL_NUMBER + ")";
    String DURATION_MS = TIME + "\\s?ms";
+    String DURATION_S = TIME + "\\s?s";
    String INT_DURATION_MS = INTEGER + "ms";
    //0.0700188
    String PAUSE_TIME = TIME + "\\s?(?:secs?|ms)";


I would have added the s to the PAUSE_TIME pattern but that is just me.

kcpeppe · 2025-02-25T22:13:51Z

parser/src/test/java/com/microsoft/gctoolkit/parser/ShenandoahParserTest.java

@@ -4,6 +4,9 @@

 import com.microsoft.gctoolkit.event.jvm.JVMEvent;
 import com.microsoft.gctoolkit.event.shenandoah.ShenandoahCycle;
+import com.microsoft.gctoolkit.event.zgc.OccupancySummary;


are these changes intentional or should they be reverted? The added methods are never called?

I think this was a rebase error. I'll revert.

kcpeppe · 2025-02-25T22:18:11Z

parser/src/main/java/com/microsoft/gctoolkit/parser/ZGCParser.java

+        GCEvent getGCEVent(DateTimeStamp endTime);
+    }
+
+    private static class ZGCMinorForwardReference implements ZFwdRef {


would prefer if this was ZForwardReference instead of ZFwdRef

dsgrieve · 2025-02-27T12:49:31Z

@j-bahr Thank you for your contribution which is now in gctoolkit 3.5.0 on Maven central.

j-bahr · 2025-02-27T20:01:44Z

@kcpeppe, Thanks for the review I will incorporate feedback into another PR.

Minor fixes from feedback in microsoft#414 - Fix enum types to avoid having to throw IllegalArgumentException - Fix Memory summary to have more descriptive variable names - ZGCPhase now handles the regex null case correctly for Full (legacy) phase types - Use pause time to collect optional GC duration - Rename ZFwdRef -> ZForwardReference - Remove unused methods in ShenandoahParserTest.java

Minor fixes from feedback in #414 - Fix enum types to avoid having to throw IllegalArgumentException - Fix Memory summary to have more descriptive variable names - ZGCPhase now handles the regex null case correctly for Full (legacy) phase types - Use pause time to collect optional GC duration - Rename ZFwdRef -> ZForwardReference - Remove unused methods in ShenandoahParserTest.java

j-bahr added 2 commits February 20, 2025 10:08

ZGC Add Missing GCCause

871dae4

Adds missing GC cause, based on enum toString defined in `jdk/src/hotspot/share/gc/shared/gcCause.cpp`

j-bahr requested review from kcpeppe and karianna February 21, 2025 04:33

karianna requested a review from dsgrieve February 22, 2025 03:33

kcpeppe approved these changes Feb 25, 2025

View reviewed changes

kcpeppe merged commit e00e597 into microsoft:main Feb 25, 2025
8 checks passed

j-bahr mentioned this pull request Feb 28, 2025

ZGC Fixes from #414 #417

Merged

j-bahr mentioned this pull request Feb 28, 2025

Generational ZGC parsing logic assumes only one collection is happening at any given time #395

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZGC Generational Support #414

ZGC Generational Support #414

j-bahr commented Feb 20, 2025

adolfojunior commented Feb 21, 2025

karianna commented Feb 22, 2025

karianna commented Feb 22, 2025

kcpeppe left a comment

kcpeppe Feb 25, 2025

j-bahr Feb 27, 2025

kcpeppe Feb 25, 2025

kcpeppe Feb 25, 2025

kcpeppe Feb 25, 2025

kcpeppe Feb 25, 2025

kcpeppe Feb 25, 2025

kcpeppe Feb 25, 2025

kcpeppe Feb 25, 2025

kcpeppe Feb 25, 2025

j-bahr Feb 27, 2025

kcpeppe Feb 25, 2025

dsgrieve commented Feb 27, 2025

j-bahr commented Feb 27, 2025

ZGC Generational Support #414

ZGC Generational Support #414

Conversation

j-bahr commented Feb 20, 2025

adolfojunior commented Feb 21, 2025

karianna commented Feb 22, 2025

karianna commented Feb 22, 2025

kcpeppe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsgrieve commented Feb 27, 2025

j-bahr commented Feb 27, 2025