Skip to content

Conversation

udesou
Copy link
Contributor

@udesou udesou commented May 30, 2025

This PR removes all the old copies of julia and mmtk-julia, leaving only the latest mmtk-julia (with a change that prints out the utilization after each GC, necessary for the logs.
Instead of having the fixed versions, it should download the latest version of mmtk/julia using the mmtk-support-moving-upstream branch as default. It should clone different versions of julia:

  • julia-immix-max-moving (which copies every object that can be copied at every GC)
  • julia-immix-always-moving (which makes every GC a defrag GC)
  • julia-immix (the default version of immix)
  • julia-immix-non-moving (the non-moving version of immix)
  • julia-stock (the version that uses the stock GC)

It should then run the benchmarks as usual considering all these variants.

@d-netto
Copy link
Owner

d-netto commented May 30, 2025

Cool, thanks. Could you share what output you get after running these scripts? I'm also curious about what the fragmentation numbers look like for these GCs.

@udesou
Copy link
Contributor Author

udesou commented May 30, 2025

The problem is that @qinsoon recently changed the mmtk-julia repo to add address-based hashing, but there's a bug somewhere we're trying to figure out. I've got some plots with a fixed heap size, but I'm not sure how reliable they are, and if they're even comparable to the stock GC without a fixed heap. Note that the changes above are not setting a fixed heap for the stock GC.

With max copying:
inference_benchmark_utilization_immix
inference_benchmark_fragmentation_immix

With non-moving:
inference_benchmark_utilization_immix
inference_benchmark_fragmentation_immix

@d-netto
Copy link
Owner

d-netto commented May 30, 2025

What benchmark is this for?

@udesou
Copy link
Contributor Author

udesou commented May 30, 2025

I have also collected some data about the block histogram:

Block Histogram (Pinned objects per block)

Pinned objects Block count Percent Top Types Pinned
0 28417 42.29%
1 11009 16.38% InferenceBenchmarkerCache (100.0%)
2 8848 13.17% InferenceBenchmarkerCache (99.9%), Union (0.0%), string (0.0%)
3 5991 8.92% InferenceBenchmarkerCache (99.6%), string (0.3%), Union (0.0%)
4 3697 5.50% InferenceBenchmarkerCache (96.0%), string (3.8%), Union (0.1%)
5 2154 3.21% InferenceBenchmarkerCache (82.4%), string (17.4%), Union (0.1%)
6 1392 2.07% InferenceBenchmarkerCache (57.1%), string (42.8%), Tuple (0.1%)
7 1164 1.73% string (63.0%), InferenceBenchmarkerCache (36.9%), LineNumberNode (0.1%)
8 996 1.48% string (74.0%), InferenceBenchmarkerCache (26.0%), Union (0.0%)
9 980 1.46% string (81.0%), InferenceBenchmarkerCache (19.0%), Union (0.0%)
10 879 1.31% string (84.0%), InferenceBenchmarkerCache (15.9%), Union (0.0%)
11 674 1.00% string (85.8%), InferenceBenchmarkerCache (14.2%), Union (0.0%)
12 469 0.70% string (86.0%), InferenceBenchmarkerCache (13.9%), LineNumberNode (0.0%)
13 288 0.43% string (85.8%), InferenceBenchmarkerCache (13.8%), PhiNode (0.3%)
14 147 0.22% string (85.8%), InferenceBenchmarkerCache (13.6%), LineNumberNode (0.6%)
15 47 0.07% string (85.4%), InferenceBenchmarkerCache (14.3%), Union (0.3%)
16 18 0.03% string (84.4%), InferenceBenchmarkerCache (14.6%), PipeEndpoint (0.7%)
17 6 0.01% string (91.2%), InferenceBenchmarkerCache (8.8%)
18 2 0.00% string (36.1%), LineNumberNode (33.3%), InferenceBenchmarkerCache (30.6%)
19 1 0.00% string (100.0%)
20 2 0.00% QuoteNode (35.0%), LineNumberNode (32.5%), Expr (20.0%)
21 2 0.00% LineNumberNode (50.0%), QuoteNode (23.8%), string (16.7%)
29 1 0.00% QuoteNode (65.5%), LineNumberNode (31.0%), InferenceBenchmarkerCache (3.4%)
35 1 0.00% QuoteNode (54.3%), LineNumberNode (34.3%), Int64 (5.7%)
38 1 0.00% LineNumberNode (97.4%), InferenceBenchmarkerCache (2.6%)
42 1 0.00% LineNumberNode (90.5%), InferenceBenchmarkerCache (9.5%)
46 1 0.00% LineNumberNode (100.0%)
48 1 0.00% LineNumberNode (100.0%)
53 1 0.00% LineNumberNode (54.7%), QuoteNode (45.3%)

Object Type Pinning Stats

Type Total Total (size) Pinned Pinned (size) Pinned %
RawFD 1 16 1 16 100.00%
AsyncCondition 1 48 1 48 100.00%
GenericCondition 2 64 2 64 100.00%
TTY 1 80 1 80 100.00%
PipeEndpoint 2 160 2 160 100.00%
InferenceBenchmarkerCache 89636 1434176 85394 1366304 95.27%
LineNumberNode 309 9888 287 9184 92.88%
string 258270 35020288 48796 15799400 18.89%
InferenceParams 6 384 1 64 16.67%
Pairs 45 1296 2 64 4.44%
EnterNode 94 3008 1 32 1.06%
Binding 212 10208 2 96 0.94%
Int64 401 6416 3 48 0.75%
GlobalRef 269 8608 2 64 0.74%
Pair 194 5264 1 16 0.52%
Union 10917 349344 53 1696 0.49%
UnionAll 2275 72800 10 320 0.44%
QuoteNode 31863 509808 90 1440 0.28%
Tuple 6324 201376 7 192 0.11%
PhiNode 24646 788672 10 320 0.04%
Expr 292446 9358272 13 416 0.00%
CodeInfo 24147 3477168 1 144 0.00%
array 1494786 47835424 1 32 0.00%
IntrusiveLinkedListSynchronized 1 32 0 0 0.00%
GenericMemory 1244351 100643008 0 0 0.00%
TypeMapEntry 28554 2284320 0 0 0.00%
IdDict 35 1120 0 0 0.00%
Set 1 16 0 0 0.00%
Dict 287 22960 0 0 0.00%
IntrusiveLinkedList 10 320 0 0 0.00%
SpinLock 10 160 0 0 0.00%
simplevector 143603 7907504 0 0 0.00%
MethodMatch 42129 2022192 0 0 0.00%
ConstPropResult 9441 151056 0 0 0.00%
SemiConcreteResult 6907 1436656 0 0 0.00%
ConcreteResult 6315 303120 0 0 0.00%
ConstCallInfo 22627 724064 0 0 0.00%
MethodMatchInfo 41874 3349920 0 0 0.00%
ReturnNode 54536 872576 0 0 0.00%
Const 207949 3327184 0 0 0.00%
GotoNode 48431 774896 0 0 0.00%
GotoIfNot 44640 1428480 0 0 0.00%
Argument 37644 602304 0 0 0.00%
PartialStruct 8151 260832 0 0 0.00%
DebugInfo 107616 5165568 0 0 0.00%
UnitRange 145 4640 0 0 0.00%
GlobalAccessInfo 52 832 0 0 0.00%
PiNode 1746 55872 0 0 0.00%
NamedTuple 430 7936 0 0 0.00%
DebugInfoStream 21888 1050624 0 0 0.00%
UInt64 2085 33360 0 0 0.00%
UInt32 237 3792 0 0 0.00%
UnionSplitInfo 102 1632 0 0 0.00%
IntrinsicFunction 90 1440 0 0 0.00%
TypeVar 658 24240 0 0 0.00%
Char 624 9984 0 0 0.00%
UnionSplitApplyCallInfo 1144 18304 0 0 0.00%
StepRange 22 704 0 0 0.00%
InferenceResult 24164 3092992 0 0 0.00%
KeySet 8 128 0 0 0.00%
Float64 849 13584 0 0 0.00%
KeyError 107 1712 0 0 0.00%
InvokeCallInfo 168 10752 0 0 0.00%
ModifyOpInfo 6 96 0 0 0.00%
Ptr 353 5648 0 0 0.00%
Fix 62 992 0 0 0.00%
ConcurrencyViolationError 31 496 0 0 0.00%
OneTo 28 448 0 0 0.00%
Broadcasted 584 26192 0 0 0.00%
NewvarNode 2079 33264 0 0 0.00%
MethodResultPure 24 384 0 0 0.00%
UInt16 57 912 0 0 0.00%
#copy##0#copy##1 3 672 0 0 0.00%
TypeMapLevel 621 39744 0 0 0.00%
InferenceState 14 5152 0 0 0.00%
UUID 26 832 0 0 0.00%
NewSSAValue 10 160 0 0 0.00%
BitSet 18393 588576 0 0 0.00%
IdSet 46 2208 0 0 0.00%
InferenceBenchmarker 14 2016 0 0 0.00%
Conditional 718 34464 0 0 0.00%
AnalysisResults 1876 60032 0 0 0.00%
OptimizationResult 14802 2605152 0 0 0.00%
ArgEscapeCache 1876 60032 0 0 0.00%
ArgumentError 21 336 0 0 0.00%
InterConditional 609 19488 0 0 0.00%
InvalidStateException 10 320 0 0 0.00%
BitArray 24098 771136 0 0 0.00%
OptimizationState 14803 3789568 0 0 0.00%
Int128 66 2112 0 0 0.00%
UInt128 156 4992 0 0 0.00%
TypeofVararg 322 10304 0 0 0.00%
Method 168 37632 0 0 0.00%
BindingPartition 575 27600 0 0 0.00%
MethodTable 35 3360 0 0 0.00%
PkgEntry 12245 1567360 0 0 0.00%
Int32 27 432 0 0 0.00%
#start_profile_listener##0#start_profile_listener##1 1 16 0 0 0.00%
#errormonitor##0#errormonitor##1 1 16 0 0 0.00%
IncrementalCompact 2 864 0 0 0.00%
CFGInliningState 1 80 0 0 0.00%
PkgId 8 384 0 0 0.00%
AssertionError 14 224 0 0 0.00%
GenericIOBuffer 3 192 0 0 0.00%
ReentrantLock 3 192 0 0 0.00%
ComposedFunction 85 1488 0 0 0.00%
SSAValue 387 6192 0 0 0.00%
VolatileInferenceResult 52 832 0 0 0.00%
InliningTodo 178 39872 0 0 0.00%
Future 8 256 0 0 0.00%
LimitedAccuracy 18 576 0 0 0.00%
Returns 11 176 0 0 0.00%
ImmutableDict 19 608 0 0 0.00%
#start_profile_listener##2#start_profile_listener##3 1 32 0 0 0.00%
ReturnTypeCallInfo 24 384 0 0 0.00%
#temp_cleanup_purge_prelocked##0#temp_cleanup_purge_prelocked##1 6 96 0 0 0.00%
UpsilonNode 129 2064 0 0 0.00%
PhiCNode 124 1984 0 0 0.00%
#new#replace_pairs!##0 4 128 0 0 0.00%
#create_expr_cache##0#create_expr_cache##1 2 32 0 0 0.00%
#_replace!##0#_replace!##1 4 128 0 0 0.00%
Effects 8 256 0 0 0.00%
#ntuple##0#ntuple##1 1 16 0 0 0.00%
PartialTypeVar 16 512 0 0 0.00%
Generator 5 208 0 0 0.00%
Box 1 16 0 0 0.00%
IncludeInto 1 16 0 0 0.00%
EvalInto 1 16 0 0 0.00%
TermInfo 1 64 0 0 0.00%
RefValue 9 400 0 0 0.00%
ExtensionId 1 96 0 0 0.00%
ConsoleLogger 1 64 0 0 0.00%
#invoke_fixedworld#fix_world_age##2 1 16 0 0 0.00%
IOContext 3 96 0 0 0.00%
LogState 1 32 0 0 0.00%
VersionNumber 61 2928 0 0 0.00%
CachedTOMLDict 54 3456 0 0 0.00%
TextDisplay 1 16 0 0 0.00%
PkgOrigin 39 1248 0 0 0.00%
CurrentState 9 432 0 0 0.00%
WorldRange 3 96 0 0 0.00%

Nonmoving Object Type Reachability Stats

Type Total Total (size) Reachable Reachable %
task 4 960 4 100.00%
DataType 43305 2771520 43305 100.00%
CodeInstance 91581 13187664 91581 100.00%
MethodInstance 26947 1724608 26947 100.00%
TypeName 37 4144 37 100.00%
module 1 400 1 100.00%
buff 1 464 1 100.00%

Pinned Object Types in Single-Pinned Blocks

Type Count
InferenceBenchmarkerCache 11009

@udesou
Copy link
Contributor Author

udesou commented May 30, 2025

What benchmark is this for?

The Inference benchmarks.

@qinsoon
Copy link
Collaborator

qinsoon commented May 31, 2025

This branch builds Julia with using a copy of mmtk-julia in this repo, which does not have the recent changes in mmtk-support-moving-upstream. Can you fix this? @udesou We should check out from the branch that we are developing from.

EDIT: I am not sure if there are other differences between the copy of mmtk-julia and our developing branch. One difference that I noticed is that dump_immix_block_stats is called in resume_mutator in this repo, but that line is commented out for our developing branch.

@d-netto
Copy link
Owner

d-netto commented Jun 3, 2025

Agree with @qinsoon: given that we're now fetching the Julia repos, it would make sense to also fetch the mmtk-julia repo.

Also, IIUC, these scripts will run the benchmarks with the HEAD of each branch. This is fine for now, but it would be nice to have a way of specifying a commit in the future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants