Feature: Extraction LightMerger usage in post-ready step. #4502

DarkWeird · 2021-02-09T15:09:43Z

Contains

LightMerger runs after chunk is ready now.
Simplified ChunkProcessingPipeline.
Fixed tests
Faster chunk delivery to ready.. especially for tests! (should not stuck at chunk generation)

How to test

Run tests
Run Singleplayer game
2.1. fly on hspeed, check how fast chunk loadings.
Run Multiplayer
3.1. fly on hspeed, check how fast chunk loadings.

skaldarnar · 2021-02-09T19:23:36Z

When I try to start a new game with just Core gameplay it crashes with:

20:22:09.333 [main] ERROR o.t.e.event.internal.EventSystemImpl - Failed to invoke event
java.lang.IllegalArgumentException: Cannot find field java.util.concurrent.CompletableFuture$AsyncSupply.task
	at org.terasology.utilities.ReflectionUtil.readField(ReflectionUtil.java:474)
	at org.terasology.world.chunks.pipeline.ChunkProcessingPipeline.lambda$unwrappingComporator$0(ChunkProcessingPipeline.java:70)
	at java.base/java.util.concurrent.PriorityBlockingQueue.siftUpUsingComparator(PriorityBlockingQueue.java:377)
	at java.base/java.util.concurrent.PriorityBlockingQueue.offer(PriorityBlockingQueue.java:488)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1347)
	at java.base/java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1714)
	at java.base/java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:1931)
	at org.terasology.world.chunks.pipeline.ChunkProcessingPipeline.invokeGeneratorTask(ChunkProcessingPipeline.java:107)
	at org.terasology.world.chunks.localChunkProvider.LocalChunkProvider.createOrLoadChunk(LocalChunkProvider.java:122)
	at org.terasology.world.chunks.localChunkProvider.RelevanceSystem.lambda$addRelevanceEntity$0(RelevanceSystem.java:184)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:183)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
	at java.base/java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:395)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
	at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
	at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497)
	at org.terasology.world.chunks.localChunkProvider.RelevanceSystem.addRelevanceEntity(RelevanceSystem.java:178)
	at org.terasology.logic.players.PlayerSystem.addRelevanceEntity(PlayerSystem.java:197)
	at org.terasology.logic.players.PlayerSystem.onConnect(PlayerSystem.java:146)
	at org.terasology.logic.players.PlayerSystemMethodAccess.invoke(Unknown Source)
	at org.terasology.entitySystem.event.internal.EventSystemImpl$ByteCodeEventHandlerInfo.invoke(EventSystemImpl.java:536)
	at org.terasology.entitySystem.event.internal.EventSystemImpl.sendStandardEvent(EventSystemImpl.java:292)
	at org.terasology.entitySystem.event.internal.EventSystemImpl.send(EventSystemImpl.java:283)
	at org.terasology.entitySystem.entity.internal.BaseEntityRef.send(BaseEntityRef.java:205)
	at org.terasology.network.internal.NetworkSystemImpl.connectClient(NetworkSystemImpl.java:882)
	at org.terasology.network.internal.NetworkSystemImpl.joinLocal(NetworkSystemImpl.java:326)
	at org.terasology.engine.modes.loadProcesses.SetupLocalPlayer.step(SetupLocalPlayer.java:45)
	at org.terasology.engine.modes.StateLoading.update(StateLoading.java:237)
	at org.terasology.engine.TerasologyEngine.tick(TerasologyEngine.java:496)
	at org.terasology.engine.TerasologyEngine.mainLoop(TerasologyEngine.java:459)
	at org.terasology.engine.TerasologyEngine.runMain(TerasologyEngine.java:435)
	at org.terasology.engine.TerasologyEngine.run(TerasologyEngine.java:401)
	at org.terasology.engine.Terasology.main(Terasology.java:174)
20:22:09.353 [main] ERROR o.t.engine.modes.StateLoading - Error while loading org.terasology.engine.modes.loadProcesses.AwaitCharacterSpawn@11c40a65
java.lang.IllegalArgumentException: Cannot find field java.util.concurrent.CompletableFuture$AsyncSupply.task
	at org.terasology.utilities.ReflectionUtil.readField(ReflectionUtil.java:474)
	at org.terasology.world.chunks.pipeline.ChunkProcessingPipeline.lambda$unwrappingComporator$0(ChunkProcessingPipeline.java:70)
	at java.base/java.util.concurrent.PriorityBlockingQueue.siftUpUsingComparator(PriorityBlockingQueue.java:377)
	at java.base/java.util.concurrent.PriorityBlockingQueue.offer(PriorityBlockingQueue.java:488)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1347)
	at java.base/java.util.concurrent.CompletableFuture.asyncSupplyStage(CompletableFuture.java:1714)
	at java.base/java.util.concurrent.CompletableFuture.supplyAsync(CompletableFuture.java:1931)
	at org.terasology.world.chunks.pipeline.ChunkProcessingPipeline.invokeGeneratorTask(ChunkProcessingPipeline.java:107)
	at org.terasology.world.chunks.localChunkProvider.LocalChunkProvider.createOrLoadChunk(LocalChunkProvider.java:122)
	at org.terasology.world.chunks.localChunkProvider.RelevanceSystem.updateRelevance(RelevanceSystem.java:134)
	at org.terasology.world.chunks.localChunkProvider.RelevanceSystem.update(RelevanceSystem.java:219)
	at org.terasology.engine.modes.loadProcesses.AwaitCharacterSpawn.step(AwaitCharacterSpawn.java:50)
	at org.terasology.engine.modes.StateLoading.update(StateLoading.java:237)
	at org.terasology.engine.TerasologyEngine.tick(TerasologyEngine.java:496)
	at org.terasology.engine.TerasologyEngine.mainLoop(TerasologyEngine.java:459)
	at org.terasology.engine.TerasologyEngine.runMain(TerasologyEngine.java:435)
	at org.terasology.engine.TerasologyEngine.run(TerasologyEngine.java:401)
	at org.terasology.engine.Terasology.main(Terasology.java:174)

skaldarnar · 2021-02-10T22:25:51Z

🚀 The chunks indeed load faster, but ...

... it feels like too much is loaded at once, causing a short freeze/frame drop 🙈
... there is a "shadow" border at the edge of the region of loaded chunks that does not go away. It retracts farther away if I move in that direction, but then just the next line of chunks is just shadowy...

DarkWeird · 2021-02-11T05:11:22Z

Freeze happened,
because final part processing at main thread with event sending takes many time,
because processed in place, because it is main thread.

Shadowy chunks.
Seems this is border at non-lod chunks.
Yeah. Light merging cannot process this chunks, because not enough data for it.
But strange that this chunks not received light data from internal light grnerator...

DarkWeird · 2021-02-11T07:07:36Z

Oh.
Voxel*WorldSystem takes many time for almost useless work at ChunkLoading...

when chunk loads - VoxelSystem getAll blocks from chunk..
but used only Id all time.. and tryToRegister block for voxelinfo(liquid/solid/penetrable/etc) this part should use only 1 time on every ID.

But VoxelSystems requests block every block position...
Both VoxelSystems!

Suggestion:

Provide method Chunk#getBlockId(x,y,z)
variants:
2.1. use BlockRegistrationListener
2.2 wait for Disable dynamic block allocation. #4036 and get block data in game start, where this info is available.

especially now, when chunks delivering faster.

skaldarnar

I'm not really the best reviewer for this, I guess (peeking at @4Denthusiast 🙄 ), but I found at least one little bug with iterating over BlockRegion.
Please have a look at the comments 🤓

skaldarnar · 2021-02-12T19:02:52Z

engine/src/main/java/org/terasology/world/chunks/pipeline/ChunkTask.java

-/**
- * @deprecated Use {@link org.terasology.world.chunks.pipeline.stages.ChunkTask} instead
- */
-@Deprecated


So we reverted the deprecation? Was the other ChunkTask recently introduced, or has it been around for some time but was never put to "special" use (justifying this deprecation)?

skaldarnar · 2021-02-12T19:07:45Z

engine/src/main/java/org/terasology/world/chunks/ChunkLightMergingSystem.java

+ * System which will process light merging after chunk is ready.
+ */
+@RegisterSystem
+public class ChunkLightMergingSystem extends BaseComponentSystem {


Oh, we don't even hook that up to the magic chunk generation off-thread process, but it is just a regular system 🤓 neat!

engine/src/main/java/org/terasology/world/chunks/ChunkLightMergingSystem.java

engine/src/main/java/org/terasology/world/chunks/CoreChunk.java

engine/src/main/java/org/terasology/world/chunks/LodChunk.java

engine/src/main/java/org/terasology/world/chunks/internal/ChunkImpl.java

* Creates ChunkLightMergingSystem which process light merging after chunk is ready for light merging * Remove LightMerging from ChunkProcessingPipeline

* ChunkProcessingPipeline have only one-to-one steps.

* RelativeSystem should sort chunks to loads itself.

…osition, if processing for ChunkProcessingPipeline.

…eline's methods. * ChunkProcessingPipeline use Function and Consumer for stage now.

…EGA ViewDistance's volume

…unk generation now.

…ocks to `boolean[]`

Co-authored-by: Tobias Nett <skaldarnar@googlemail.com>

…kImpl.java Co-authored-by: Tobias Nett <skaldarnar@googlemail.com>

Co-authored-by: Tobias Nett <skaldarnar@googlemail.com>

…gingSystem.java Co-authored-by: Tobias Nett <skaldarnar@googlemail.com>

jdrueckert · 2021-02-19T21:10:33Z

Tried this out and the chunks seemed to load very slowly... it took quite a while and then a bunch of them popped up... usually it felt like they pop up one by one 🤔

DarkWeird · 2021-02-20T09:36:37Z

Tried this out and the chunks seemed to load very slowly... it took quite a while and then a bunch of them popped up... usually it felt like they pop up one by one thinking

every chunk ready is faster, because lightmerger(a huge time eater) take a lot of time. (also internal chunkprocessing sorting was eliminated and compilcated wait logic was removed too)
then it is only feels. :)
and yeah it looks like load bunch... because RelativesSystem request only bunches of chunks in one tick.
then this chunks loads in executor concurrently..
then bunch of chunks(all or part of chunks) ready onetime and became ready on main thread.

other profit in tests. tests with chunks works faster(and take less memory) now

pollend · 2021-02-22T05:34:26Z

@DarkWeird testing this a bit but sometimes restarting the world makes the whole world black.

DarkWeird · 2021-02-22T05:46:05Z

@pollend
Restarting - world purge?

pollend · 2021-02-24T05:16:05Z

something isn't quite working. I'm also seeing some very strange lag time between chunks start showing up.

DarkWeird · 2021-02-24T05:45:00Z

something isn't quite working. I'm also seeing some very strange lag time between chunks start showing up.

Lag is physics's onChunkLoading event handler.

Very strange about black.. maybe you needs to update your reflections cache?

pollend · 2021-02-24T05:49:28Z

Some of the chunks are loading in this way and the other half are fine.

DarkWeird · 2021-02-24T06:10:03Z

Some of the chunks are loading in this way and the other half are fine.

No. It is not about some of the chunks.
It bunch of them at once. (Iterate 100+ chunks iterate every chunk per block)
I try load them with limits.. it was worster

* strange, hashset not determinate that two Vector3i has same hashcode, if this vectors not same...

…postready' into feature/extract-light-merger-to-postready

…d world.

…mpletableFuture with simple function Composition. * function composition seems work faster.

skaldarnar · 2021-02-25T22:42:29Z

The lag spikes don't seem to be that big anymore (at least I did not notice them that much) but you can still end up with black/unlit areas.

This happened after I've waited for the LOD chunks to generate, and then moved into some direction...

DarkWeird · 2021-02-26T06:50:59Z

The lag spikes don't seem to be that big anymore (at least I did not notice them that much) but you can still end up with black/unlit areas.

This happened after I've waited for the LOD chunks to generate, and then moved into some direction...

Just wait. Idk why this works so bad.

ChunkLightMergingSystem checks around chunks on every chunk loading for possible to start light merging...
Maybe because working threads have the Lowest priority....

4Denthusiast · 2021-03-13T23:27:56Z

engine/src/main/java/org/terasology/world/chunks/ChunkLightMergingSystem.java

+            }
+            chunks[index++] = chunkProvider.getChunk(position);
+        }
+        threadManager.submitTask("Chunk Light Merging", () -> new LightMerger().merge(chunks));


This doesn't look thread safe. It's changing the light values for a chunk that's already accessible elsewhere, so another system could be updating the lighting at the same time. Also, it looks like you don't mark the chunk dirty anywhere after light merging happens, so it may never be re-meshed (depending on the order of operations).

Yeah it not thread safe, but it costs low price if error happens there.
About dirting chunks. It marks implicitly.(somewhere during merging)

4Denthusiast

The changes to the world collision systems seems pretty irrelevant, and it's annoying to have that mixed into the same PR. I can't find any obvious problems that would cause the mostly unlit chunks shown in the screenshots. Internal light merging is clearly still happening, otherwise there wouldn't be that light at the bottom of the chunks, but somehow it's starting off dark, rather than starting off lit at the top like it should be.

I haven't had a look at everything yet. In particular, I haven't read through all of the changes to ChunkProcessingPipeline. @DarkWeird Could you summarise what those changes are meant to do (except directly removing the light merging step)? It looks like a lot has been removed. Does that reduce the functionality of it? If it removes the ability to have steps that require multiple chunks at once, I would expect it could be simplified much more than this.

DarkWeird · 2021-03-14T05:44:31Z

@4Denthusiast

Light merging as separate system which works at chunks's onLoaded event
Simplify pipeline with saves interface..(almost)
Changes to physics system because chunks began incoming to it too fast (freezes main thread, because needs to process many chunks at main thread):
3.1. Direct blockID getter for chunks
3.2. merge physics voxels systems to one (removes second full chunk scan for block id)
You can load only one chunk if you require instead 27(3x3x3) chunks when needs only one.

DarkWeird · 2021-03-14T06:00:38Z

@4Denthusiast

About unlit.
I don't know how internalLightProcessor works exactly.
I don't change it.
Mostly lightning data at early stage receives from world generator or internalLightProcessor.

Mostly lightning data at all - LightMerger.

Unlit border happens because lightmerger is not processed those chunks. But it cannot process them, because not enough chunks (next chunks is lod chunk already)

pollend · 2021-03-14T06:08:00Z

@4Denthusiast

About unlit.
I don't know how internalLightProcessor works exactly.
I don't change it.
Mostly lightning data at early stage receives from world generator or internalLightProcessor.

Mostly lightning data at all - LightMerger.

Unlit border happens because lightmerger is not processed those chunks. But it cannot process them, because not enough chunks (next chunks is lod chunk already)

What are the main issues that make lighting inefficient. just shuffling things around does not resolve the problem does it :?. I see large spikes now with the lighting being moved to the main thread. this should be a parallelizable problem since lighting is local to its chunk. the only thing that would matter is the locations of the active light source and a border similar to how facets work. I also don't see why we can't cache out the lighting to the filesystem to avoid having to recalculate it every time.

DarkWeird · 2021-03-14T06:42:18Z

@pollend
Seems i saw old field in protobuf, which used for light data..

Hmm. I suspect that internalLightProcesdor works incorrect.

Also.. provide facet data to game can helps make lightning better... Quality change

pollend · 2021-03-14T06:43:48Z

@pollend
Seems i saw old field in protobuf, which used for light data..

Hmm. I suspect that internalLightProcesdor works incorrect.

Also.. provide facet data to game can helps make lightning better... Quality change

one thing I would like is to have a dynamic light facet for sunlight and just move the light column up the chain until it hits an unloaded chunk and then move the up the next visible block. so we don't have areas underground that are lit because a chunk was momentarily exposed to the sky.

4Denthusiast · 2021-03-15T03:36:45Z

What are the main issues that make lighting inefficient. just shuffling things around does not resolve the problem does it :?.

The issue that this PR fixes is that you need to load all the chunks around a chunk before it can become visible. That's sort of an efficiency thing in that it reduces the amount of woth necessary before a given amount of chunks become visible.

4Denthusiast · 2021-03-20T18:44:12Z

I'm not happy with just merging this with the concurrency issues left in. Although I agree that the issues would be mostly minor, and probably rare, they would also likely be really confusing. Others who have actually tested this PR have commented that it causes the game to have noticable lag spikes, and fixing the concurrency issue would make that even worse. Possibly that could be improved by improving the efficiency of the lighting calculations (which Michael thinks is very possible), but I think overall this change may just not be worth making, so I suggest just closing the PR unmerged.

DarkWeird requested review from pollend and 4Denthusiast February 9, 2021 15:09

DarkWeird added Category: Performance Requests, Issues and Changes targeting performance Type: Improvement Request for or addition/enhancement of a feature Topic: Stabilization Requests, Issues and Changes related to improving stablity and reducing flakyness labels Feb 9, 2021

DarkWeird requested a review from skaldarnar February 9, 2021 15:10

skaldarnar requested changes Feb 12, 2021

View reviewed changes

DarkWeird added 9 commits February 15, 2021 15:10

feat(light-merging): move lightmerging to post-ready async job.

1f5e882

* Creates ChunkLightMergingSystem which process light merging after chunk is ready for light merging * Remove LightMerging from ChunkProcessingPipeline

feat(light-merging): Simplify chunkProcessingPipeline.

10c7abf

* ChunkProcessingPipeline have only one-to-one steps.

fix(light-merging): remove task comparing for ChunkTasks.

f0b61b4

* RelativeSystem should sort chunks to loads itself.

feat(light-merging): return possible to return same future for same p…

2b02e15

…osition, if processing for ChunkProcessingPipeline.

chore(light-merging): remove unused code, simplify ChunkProcessingPip…

eed4e72

…eline's methods. * ChunkProcessingPipeline use Function and Consumer for stage now.

chore(light-merging): set queue size for ChunkProcessingPipeline to M…

53aefec

…EGA ViewDistance's volume

perf(light-merging): Physic's Voxel*WorldSystems take less time on ch…

f3b1dcb

…unk generation now.

perf(light-merging, Physics): Change TShortHashSet for registred bl…

ab8277d

…ocks to `boolean[]`

pref(light-merge,physics): merge VoxelLiquid and Voxel Systems

72c4129

DarkWeird force-pushed the feature/extract-light-merger-to-postready branch from cb6bbf2 to 72c4129 Compare February 15, 2021 14:49

DarkWeird and others added 5 commits February 15, 2021 17:49

Update engine/src/main/java/org/terasology/world/chunks/LodChunk.java

e9cb4fb

Co-authored-by: Tobias Nett <skaldarnar@googlemail.com>

Update engine/src/main/java/org/terasology/world/chunks/internal/Chun…

ef5c574

…kImpl.java Co-authored-by: Tobias Nett <skaldarnar@googlemail.com>

Update engine/src/main/java/org/terasology/world/chunks/CoreChunk.java

58895d7

Co-authored-by: Tobias Nett <skaldarnar@googlemail.com>

Update engine/src/main/java/org/terasology/world/chunks/ChunkLightMer…

8dad8c2

…gingSystem.java Co-authored-by: Tobias Nett <skaldarnar@googlemail.com>

Update engine/src/main/java/org/terasology/world/chunks/ChunkLightMer…

b270cf0

…gingSystem.java Co-authored-by: Tobias Nett <skaldarnar@googlemail.com>

DarkWeird requested a review from skaldarnar February 16, 2021 06:08

skaldarnar added this to the v4.2.0 milestone Feb 18, 2021

Merge branch 'develop' into feature/extract-light-merger-to-postready

a34abd7

DarkWeird mentioned this pull request Feb 21, 2021

Improve performance with GraalVM. #4530

Closed

DarkWeird added 3 commits February 24, 2021 13:25

Merge branch 'develop' into feature/extract-light-merger-to-postready

357e435

fix(extract-light-merger): fixing light merge for load positions..

ce2294b

* strange, hashset not determinate that two Vector3i has same hashcode, if this vectors not same...

Merge remote-tracking branch 'origin/feature/extract-light-merger-to-…

ccf1544

…postready' into feature/extract-light-merger-to-postready

DarkWeird force-pushed the feature/extract-light-merger-to-postready branch from 9c99f00 to 8c837a1 Compare February 24, 2021 14:42

DarkWeird added 2 commits February 24, 2021 17:44

fix(light-merger): copying blockRegion's iterator and fix black loade…

8c837a1

…d world.

perf(light-merger): Add thread activity for generatorTask. replace Co…

0be0704

…mpletableFuture with simple function Composition. * function composition seems work faster.

DarkWeird and others added 2 commits March 3, 2021 13:13

Merge branch 'develop' into feature/extract-light-merger-to-postready

c6ce864

Merge branch 'develop' into feature/extract-light-merger-to-postready

f9421f2

4Denthusiast reviewed Mar 13, 2021

View reviewed changes

keturn modified the milestones: v4.3.0, v4.4.0 Mar 14, 2021

DarkWeird closed this Mar 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feature: Extraction LightMerger usage in post-ready step. #4502

Feature: Extraction LightMerger usage in post-ready step. #4502

DarkWeird commented Feb 9, 2021 •

edited

Loading

skaldarnar commented Feb 9, 2021

skaldarnar commented Feb 10, 2021

DarkWeird commented Feb 11, 2021

DarkWeird commented Feb 11, 2021 •

edited

Loading

skaldarnar left a comment

skaldarnar Feb 12, 2021

skaldarnar Feb 12, 2021

jdrueckert commented Feb 19, 2021

DarkWeird commented Feb 20, 2021 •

edited

Loading

pollend commented Feb 22, 2021 •

edited

Loading

DarkWeird commented Feb 22, 2021 •

edited

Loading

pollend commented Feb 24, 2021 •

edited

Loading

DarkWeird commented Feb 24, 2021

pollend commented Feb 24, 2021

DarkWeird commented Feb 24, 2021

skaldarnar commented Feb 25, 2021

DarkWeird commented Feb 26, 2021

4Denthusiast Mar 13, 2021

DarkWeird Mar 14, 2021

4Denthusiast left a comment

DarkWeird commented Mar 14, 2021 •

edited

Loading

DarkWeird commented Mar 14, 2021

pollend commented Mar 14, 2021 •

edited

Loading

DarkWeird commented Mar 14, 2021

pollend commented Mar 14, 2021

4Denthusiast commented Mar 15, 2021

4Denthusiast commented Mar 20, 2021

Feature: Extraction LightMerger usage in post-ready step. #4502

Feature: Extraction LightMerger usage in post-ready step. #4502

Conversation

DarkWeird commented Feb 9, 2021 • edited Loading

Contains

How to test

skaldarnar commented Feb 9, 2021

skaldarnar commented Feb 10, 2021

DarkWeird commented Feb 11, 2021

DarkWeird commented Feb 11, 2021 • edited Loading

skaldarnar left a comment

Choose a reason for hiding this comment

skaldarnar Feb 12, 2021

Choose a reason for hiding this comment

skaldarnar Feb 12, 2021

Choose a reason for hiding this comment

jdrueckert commented Feb 19, 2021

DarkWeird commented Feb 20, 2021 • edited Loading

pollend commented Feb 22, 2021 • edited Loading

DarkWeird commented Feb 22, 2021 • edited Loading

pollend commented Feb 24, 2021 • edited Loading

DarkWeird commented Feb 24, 2021

pollend commented Feb 24, 2021

DarkWeird commented Feb 24, 2021

skaldarnar commented Feb 25, 2021

DarkWeird commented Feb 26, 2021

4Denthusiast Mar 13, 2021

Choose a reason for hiding this comment

DarkWeird Mar 14, 2021

Choose a reason for hiding this comment

4Denthusiast left a comment

Choose a reason for hiding this comment

DarkWeird commented Mar 14, 2021 • edited Loading

DarkWeird commented Mar 14, 2021

pollend commented Mar 14, 2021 • edited Loading

DarkWeird commented Mar 14, 2021

pollend commented Mar 14, 2021

4Denthusiast commented Mar 15, 2021

4Denthusiast commented Mar 20, 2021

DarkWeird commented Feb 9, 2021 •

edited

Loading

DarkWeird commented Feb 11, 2021 •

edited

Loading

DarkWeird commented Feb 20, 2021 •

edited

Loading

pollend commented Feb 22, 2021 •

edited

Loading

DarkWeird commented Feb 22, 2021 •

edited

Loading

pollend commented Feb 24, 2021 •

edited

Loading

DarkWeird commented Mar 14, 2021 •

edited

Loading

pollend commented Mar 14, 2021 •

edited

Loading