Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

!!! FEATURE: Publishing Version 3 #5301

Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
c0d6478
WIP task/schnappsidee-rebase-and-partial-publish-in-memory-event-stream
mhsdesign Oct 19, 2024
7190f83
WIP: Commit for command handling in inSimulation
mhsdesign Oct 21, 2024
f47a5e6
WIP: handlePublishIndividualNodesFromWorkspace with remaining events …
mhsdesign Oct 21, 2024
3f9d534
WIP: move handleRebaseWorkspace to run inSimulation
mhsdesign Oct 21, 2024
5f72ead
WIP: refactor handleDiscardIndividualNodesFromWorkspace to run inSimu…
mhsdesign Oct 21, 2024
686d522
WIP todos
mhsdesign Oct 21, 2024
40cebe8
Merge remote-tracking branch 'origin/9.0' into task/schnappsidee-reba…
mhsdesign Oct 22, 2024
b99a498
TASK: Adjustments after merge of cr first level projection
mhsdesign Oct 22, 2024
6847e5c
Merge branch 'task/schnappsidee-zwo-yield-events-to-publish-in-worksp…
mhsdesign Oct 23, 2024
13de53c
TASK: Introduce CommandSimulator and refactor publish partially and r…
mhsdesign Oct 23, 2024
3f0db1d
TASK: Remove now obsolete `ForkContentStream` command
mhsdesign Oct 23, 2024
c766f0b
TASK: Adjust todo comments
mhsdesign Oct 23, 2024
f1c8af6
BUGFIX: Partial publish breaks uri and change Projection
kitsunet Oct 16, 2024
605715b
TASK: Adjust test 02-RebasingWithAutoCreatedNodes.feature
mhsdesign Oct 23, 2024
3b85cf8
TASK: Adjust exceptions to be a noop if PublishIndividualNodesFromWor…
mhsdesign Oct 23, 2024
20c3827
TASK: Introduce test for quick path if rebased commands emitted 0 events
mhsdesign Oct 23, 2024
6343694
TASK: Add quick paths for partial discarding with tests
mhsdesign Oct 23, 2024
e0815bc
Add todos to optimise partial publish to full publish and discard all
mhsdesign Oct 23, 2024
824ca76
TASK: Make publish all a no-op if there are no changes and also ensur…
mhsdesign Oct 23, 2024
96e58ea
TASK: Remove PublishIndividualNodesFromWorkspace::contentStreamIdForM…
mhsdesign Oct 23, 2024
31cfcc5
Workspace aware simulation
kitsunet Oct 23, 2024
aeee478
TASK: Remove outdated comments and adjust variable namings
mhsdesign Oct 23, 2024
77719c9
Cosmetics & Nagellack
mhsdesign Oct 23, 2024
a54a081
TASK: Remove obsolete check to skip events
mhsdesign Oct 24, 2024
653bb8b
Merge remote-tracking branch 'origin/9.0' into task/schnappsidee-reba…
mhsdesign Oct 24, 2024
a0dd931
TASK: Remove extra interface for workspace command handler and combin…
mhsdesign Oct 24, 2024
40233dc
TASK: Use `currentSequenceNumber` instead of calculating it ourselves
mhsdesign Oct 24, 2024
e345f7c
BUGFIX: Move workspace pointer BEFORE remaining events are emitted
mhsdesign Oct 24, 2024
0d0da5c
TASK: Remove old content-stream after rebase
mhsdesign Oct 24, 2024
c63b019
TASK: Ensure that dangling content streams are removed after publish …
mhsdesign Oct 24, 2024
9c06afc
TASK: Simplify CommandSimulator by exposing handle function only duri…
mhsdesign Oct 24, 2024
25d79d3
TASK: Adjust content stream prune test as rebase is a noop without ch…
mhsdesign Oct 25, 2024
29a27de
TASK: Always rebase during publication and do not copy events directly
mhsdesign Oct 25, 2024
e2b5652
DimensionSpacePoint events enriched
kitsunet Oct 25, 2024
0167efd
Just some linter happiness
kitsunet Oct 25, 2024
359a74e
TASK: Consistently use workspace instead of content graph in workspac…
mhsdesign Oct 25, 2024
4ee595e
TASK: Introduce `InitiatingEventMetadata` utility to centralise acces…
mhsdesign Oct 26, 2024
56fd230
TASK: Introduce ExtractedCommand vo to collect original commands AND …
mhsdesign Oct 26, 2024
f7c39ec
TASK: Combine `NodeAggregateEventPublisher` and `ExtractedCommand` to…
mhsdesign Oct 26, 2024
7430bbc
TASK: Add test that rebase keeps the originalCreated intact
mhsdesign Oct 26, 2024
4e26f33
TASK: Move `CommandHandlingDependencies` to CommandHandler namespace
mhsdesign Oct 26, 2024
f0ba01a
TASK: Dont pass `CommandHandlingDependencies` through half of the world
mhsdesign Oct 26, 2024
c62e6db
TASK: Update todos and require base content stream to be not closed d…
mhsdesign Oct 26, 2024
f6742de
TASK: Remove obsolete `ContentGraphReadModelInterface::buildContentGr…
mhsdesign Oct 26, 2024
d2d9faa
TASK: Turn partial publish into full publish if all nodes are about t…
mhsdesign Oct 26, 2024
936f39d
TASK: Update documentation
mhsdesign Oct 26, 2024
3679157
TASK: Revert currently unused error handling strategy from workspace …
mhsdesign Oct 26, 2024
8ad0a3e
FEATURE: Throw `WorkspaceRebaseFailed` during publication or partial …
mhsdesign Oct 26, 2024
49d5870
BUGFIX: Dont discard remaining user changes if $matchingCommands is e…
mhsdesign Oct 26, 2024
34395d8
TASK: Minor cosmetic adjustments and documentation
mhsdesign Oct 27, 2024
5339f97
TASK: Use `setRollbackOnly` to ensure nothing is commited during the …
mhsdesign Oct 28, 2024
f0e088e
TASK: Make publish a no-op if there are no changes instead of a rebase
mhsdesign Oct 28, 2024
5e12a48
TASK: Turn full discard also into a no-op if there are no changes
mhsdesign Oct 28, 2024
38835ca
TASK: Adjust comments to not reference renamed `NodeAggregateEventPub…
mhsdesign Oct 28, 2024
d442544
TASK: Make `CommandsThatFailed::sequenceNumber` internal only for tes…
mhsdesign Oct 28, 2024
08a0d67
Apply suggestions from code review
mhsdesign Oct 28, 2024
3761fc2
TASK: Declare `$handlers` of `CommandBus` private
mhsdesign Oct 28, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,19 @@ public function apply(EventInterface $event, EventEnvelope $eventEnvelope): void
}
}

public function inSimulation(\Closure $fn): mixed
{
if ($this->dbal->isTransactionActive()) {
kitsunet marked this conversation as resolved.
Show resolved Hide resolved
throw new \RuntimeException(sprintf('Invoking %s is not allowed to be invoked recursively. Current transaction nesting %d.', __FUNCTION__, $this->dbal->getTransactionNestingLevel()));
}
$this->dbal->beginTransaction();
try {
return $fn();
} finally {
$this->dbal->rollBack();
}
}

private function whenContentStreamWasClosed(ContentStreamWasClosed $event): void
{
$this->updateContentStreamStatus($event->contentStreamId, ContentStreamStatus::CLOSED);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -213,6 +213,19 @@ public function apply(EventInterface $event, EventEnvelope $eventEnvelope): void
};
}

public function inSimulation(\Closure $fn): mixed
{
if ($this->dbal->isTransactionActive()) {
throw new \RuntimeException(sprintf('Invoking %s is not allowed to be invoked recursively. Current transaction nesting %d.', __FUNCTION__, $this->dbal->getTransactionNestingLevel()));
}
$this->dbal->beginTransaction();
try {
return $fn();
} finally {
$this->dbal->rollBack();
}
}

public function getCheckpointStorage(): DbalCheckpointStorage
{
return $this->checkpointStorage;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,13 @@ Feature: Rebasing auto-created nodes works
| propertyValues | {"text": {"value":"Modified","type":"string"}} |
| propertiesToUnset | {} |

# user ws must be outdated!!! otherwise cheesy

# rebase of SetSerializedNodeProperties
# More than one node anchor point for content stream: b2a1d336-38a7-4183-815c-c49b2c7eba8c, node aggregate id: 7ef3c166-16d5-4aed-b8cb-5a44670607b7 and origin dimension space point: [] – this should not happen and might be a conceptual problem!
mhsdesign marked this conversation as resolved.
Show resolved Hide resolved
When the command RebaseWorkspace is executed with payload:
| Key | Value |
| workspaceName | "user-test" |
| rebasedContentStreamId | "user-cs-rebased" |
# This should properly work; no error.

Original file line number Diff line number Diff line change
Expand Up @@ -134,40 +134,40 @@ Feature: Publishing individual nodes (basics)
| Key | Value |
| image | "Modified image" |

Scenario: It is possible to publish no node
When the command PublishIndividualNodesFromWorkspace is executed with payload:
| Key | Value |
| workspaceName | "user-test" |
| nodesToPublish | [] |
| contentStreamIdForRemainingPart | "user-cs-identifier-remaining" |

When I am in workspace "live" and dimension space point {}
Then I expect node aggregate identifier "sir-david-nodenborough" to lead to node cs-identifier;sir-david-nodenborough;{}
And I expect this node to have the following properties:
| Key | Value |
| text | "Initial t1" |
And I expect node aggregate identifier "nody-mc-nodeface" to lead to node cs-identifier;nody-mc-nodeface;{}
And I expect this node to have the following properties:
| Key | Value |
| text | "Initial t2" |
And I expect node aggregate identifier "sir-nodeward-nodington-iii" to lead to node cs-identifier;sir-nodeward-nodington-iii;{}
And I expect this node to have the following properties:
| Key | Value |
| image | "Initial image" |

When I am in workspace "user-test" and dimension space point {}
Then I expect node aggregate identifier "sir-david-nodenborough" to lead to node user-cs-identifier-remaining;sir-david-nodenborough;{}
And I expect this node to have the following properties:
| Key | Value |
| text | "Modified t1" |
Then I expect node aggregate identifier "nody-mc-nodeface" to lead to node user-cs-identifier-remaining;nody-mc-nodeface;{}
And I expect this node to have the following properties:
| Key | Value |
| text | "Modified t2" |
Then I expect node aggregate identifier "sir-nodeward-nodington-iii" to lead to node user-cs-identifier-remaining;sir-nodeward-nodington-iii;{}
And I expect this node to have the following properties:
| Key | Value |
| image | "Modified image" |
# Scenario: It is possible to publish no node
mhsdesign marked this conversation as resolved.
Show resolved Hide resolved
# When the command PublishIndividualNodesFromWorkspace is executed with payload:
# | Key | Value |
# | workspaceName | "user-test" |
# | nodesToPublish | [] |
# | contentStreamIdForRemainingPart | "user-cs-identifier-remaining" |
#
# When I am in workspace "live" and dimension space point {}
# Then I expect node aggregate identifier "sir-david-nodenborough" to lead to node cs-identifier;sir-david-nodenborough;{}
# And I expect this node to have the following properties:
# | Key | Value |
# | text | "Initial t1" |
# And I expect node aggregate identifier "nody-mc-nodeface" to lead to node cs-identifier;nody-mc-nodeface;{}
# And I expect this node to have the following properties:
# | Key | Value |
# | text | "Initial t2" |
# And I expect node aggregate identifier "sir-nodeward-nodington-iii" to lead to node cs-identifier;sir-nodeward-nodington-iii;{}
# And I expect this node to have the following properties:
# | Key | Value |
# | image | "Initial image" |
#
# When I am in workspace "user-test" and dimension space point {}
# Then I expect node aggregate identifier "sir-david-nodenborough" to lead to node user-cs-identifier-remaining;sir-david-nodenborough;{}
# And I expect this node to have the following properties:
# | Key | Value |
# | text | "Modified t1" |
# Then I expect node aggregate identifier "nody-mc-nodeface" to lead to node user-cs-identifier-remaining;nody-mc-nodeface;{}
# And I expect this node to have the following properties:
# | Key | Value |
# | text | "Modified t2" |
# Then I expect node aggregate identifier "sir-nodeward-nodington-iii" to lead to node user-cs-identifier-remaining;sir-nodeward-nodington-iii;{}
# And I expect this node to have the following properties:
# | Key | Value |
# | image | "Modified image" |

Scenario: It is possible to publish all nodes
When the command PublishIndividualNodesFromWorkspace is executed with payload:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,11 @@
final class CommandBus
{
/**
* @var CommandHandlerInterface[]
* @var (CommandHandlerInterface|ControlFlowAwareCommandHandlerInterface)[]
mhsdesign marked this conversation as resolved.
Show resolved Hide resolved
*/
private array $handlers;

public function __construct(CommandHandlerInterface ...$handlers)
public function __construct(CommandHandlerInterface|ControlFlowAwareCommandHandlerInterface ...$handlers)
{
$this->handlers = $handlers;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@

use Neos\ContentRepository\Core\CommandHandlingDependencies;
use Neos\ContentRepository\Core\EventStore\EventsToPublish;
use Neos\ContentRepository\Core\EventStore\EventsToPublishFailed;

/**
* Common interface for all Content Repository command handlers
Expand All @@ -19,9 +18,5 @@
interface CommandHandlerInterface
{
public function canHandle(CommandInterface $command): bool;

/**
* @return EventsToPublish|\Generator<int, EventsToPublish, ?EventsToPublishFailed, void>
*/
public function handle(CommandInterface $command, CommandHandlingDependencies $commandHandlingDependencies): EventsToPublish|\Generator;
public function handle(CommandInterface $command, CommandHandlingDependencies $commandHandlingDependencies): EventsToPublish;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
<?php

declare(strict_types=1);

namespace Neos\ContentRepository\Core\CommandHandler;

use Neos\ContentRepository\Core\CommandHandlingDependencies;
use Neos\ContentRepository\Core\ContentRepository;
use Neos\ContentRepository\Core\EventStore\EventNormalizer;
use Neos\ContentRepository\Core\EventStore\EventsToPublish;
use Neos\ContentRepository\Core\Projection\ContentGraph\ContentGraphProjectionInterface;
use Neos\EventStore\Helper\InMemoryEventStore;
use Neos\EventStore\Model\Event\SequenceNumber;
use Neos\EventStore\Model\Events;
use Neos\EventStore\Model\EventStream\EventStreamInterface;
use Neos\EventStore\Model\EventStream\ExpectedVersion;
use Neos\EventStore\Model\EventStream\VirtualStreamName;

/**
* Implementation detail of {@see ContentRepository::handle}, when rebasing or partially publishing
mhsdesign marked this conversation as resolved.
Show resolved Hide resolved
*
* @internal
*/
final class CommandSimulator
{
private bool $inSimulation = false;

/**
* @param ContentGraphProjectionInterface $contentRepositoryProjection
* @param EventNormalizer $eventNormalizer
* @param array<CommandHandlerInterface> $handlers
*/
public function __construct(
private readonly CommandHandlingDependencies $commandHandlingDependencies,
private readonly ContentGraphProjectionInterface $contentRepositoryProjection,
private readonly EventNormalizer $eventNormalizer,
private readonly array $handlers,
private readonly InMemoryEventStore $inMemoryEventStore
) {
}

/**
* @template T
* @param \Closure(): T $fn
* @return T
*/
public function run(\Closure $fn): mixed
{
$this->inSimulation = true;
try {
return $this->contentRepositoryProjection->inSimulation($fn);
kitsunet marked this conversation as resolved.
Show resolved Hide resolved
} finally {
$this->inSimulation = false;
}
}

public function handle(CommandInterface $command): void
{
if ($this->inSimulation === false) {
throw new \RuntimeException('Simulation is not running');
}
kitsunet marked this conversation as resolved.
Show resolved Hide resolved

$eventsToPublish = $this->handleCommand($command, $this->commandHandlingDependencies);

if ($eventsToPublish->events->isEmpty()) {
return;
}

// the following logic could also be done in an AppEventStore::commit method (being called
// directly from the individual Command Handlers).
$normalizedEvents = Events::fromArray(
$eventsToPublish->events->map($this->eventNormalizer->normalize(...))
);

$commitResult = $this->inMemoryEventStore->commit(
$eventsToPublish->streamName,
$normalizedEvents,
ExpectedVersion::ANY() // The version of the stream in the IN MEMORY event store does not matter to us,
// because this is only used in memory during the partial publish or rebase operation; so it cannot be written to
// concurrently.
// HINT: We cannot use $eventsToPublish->expectedVersion, because this is based on the PERSISTENT event stream (having different numbers)
);


$eventStream = $this->inMemoryEventStore->load(VirtualStreamName::all())->withMinimumSequenceNumber(
// fetch all events that were now committed. Plus one because the first sequence number is one too otherwise we get one event to many.
// (all elephants shall be placed shamefully placed on my head)
SequenceNumber::fromInteger($commitResult->highestCommittedSequenceNumber->value - $eventsToPublish->events->count() + 1)
);

foreach ($eventStream as $eventEnvelope) {
$event = $this->eventNormalizer->denormalize($eventEnvelope->event);

if (!$this->contentRepositoryProjection->canHandle($event)) {
continue;
}

$this->contentRepositoryProjection->apply($event, $eventEnvelope);
}
}

public function currentSequenceNumber(): SequenceNumber
{
foreach ($this->inMemoryEventStore->load(VirtualStreamName::all())->backwards()->limit(1) as $eventEnvelope) {
mhsdesign marked this conversation as resolved.
Show resolved Hide resolved
return $eventEnvelope->sequenceNumber;
}
return SequenceNumber::none();
}

public function eventStream(): EventStreamInterface
{
return $this->inMemoryEventStore->load(VirtualStreamName::all());
}

private function handleCommand(CommandInterface $command, CommandHandlingDependencies $commandHandlingDependencies): EventsToPublish
{
// TODO fail if multiple handlers can handle the same command
foreach ($this->handlers as $handler) {
mhsdesign marked this conversation as resolved.
Show resolved Hide resolved
if ($handler->canHandle($command)) {
return $handler->handle($command, $commandHandlingDependencies);
}
}
throw new \RuntimeException(sprintf('No handler found for Command "%s"', get_debug_type($command)), 1649582778);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
<?php

declare(strict_types=1);

namespace Neos\ContentRepository\Core\CommandHandler;

use Neos\ContentRepository\Core\CommandHandlingDependencies;
use Neos\ContentRepository\Core\EventStore\EventNormalizer;
use Neos\ContentRepository\Core\Projection\ContentGraph\ContentGraphProjectionInterface;
use Neos\EventStore\Helper\InMemoryEventStore;

/**
* @internal
*/
final class CommandSimulatorFactory
{
/**
* @param array<CommandHandlerInterface> $handlers
*/
public function __construct(
private readonly CommandHandlingDependencies $commandHandlingDependencies,
private readonly ContentGraphProjectionInterface $contentRepositoryProjection,
private readonly EventNormalizer $eventNormalizer,
private readonly array $handlers
) {
}

public function createSimulator(): CommandSimulator
{
return new CommandSimulator(
$this->commandHandlingDependencies,
$this->contentRepositoryProjection,
$this->eventNormalizer,
$this->handlers,
new InMemoryEventStore()
);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
<?php

declare(strict_types=1);

namespace Neos\ContentRepository\Core\CommandHandler;

use Neos\ContentRepository\Core\CommandHandlingDependencies;
use Neos\ContentRepository\Core\EventStore\EventsToPublish;
use Neos\ContentRepository\Core\EventStore\EventsToPublishFailed;

/**
* Common interface for all Content Repository command handlers
mhsdesign marked this conversation as resolved.
Show resolved Hide resolved
*
* Note: The Content Repository instance is passed to the handle() method for it to do soft-constraint checks or
* trigger "sub commands"
*
* @internal no public API, because commands are no extension points of the CR
*/
interface ControlFlowAwareCommandHandlerInterface
{
public function canHandle(CommandInterface $command): bool;

/**
* @return \Generator<int, EventsToPublish, ?EventsToPublishFailed, void>
*/
public function handle(CommandInterface $command, CommandHandlingDependencies $commandHandlingDependencies): \Generator;
}
Loading
Loading