PHPORM-99 Add optimized cache and lock drivers

[GromNaN]

Fix fo PHPORM-99

In theory, we can use DatabaseStore and DatabaseLock. But various issues prove the changing nature of their implementation, based on new features in the query builder, make this feature unstable for MongoDB users. fix #2718, fix #2609

By introducing dedicated drivers, we can optimize the implementation to use the mongodb library directly instead of the subset of features provided by laravel query builder.

Usage:

# config/cache.php
return [

    'stores' => [

        'mongodb' => [
            'driver' => 'mongodb',
            'connection' => 'mongodb',
            'collection' => 'cache',
            'lock_connection' => 'mongodb',
            'lock_collection' => 'cache_locks',
            'lock_lottery' => [2, 100],
            'lock_timeout' => '86400',
        ]
    ]

]

Cache:

// Store any value into the cache. The value is serialized in MongoDB
Cache::set('foo', [1, 2, 3]);

// Read the value
dump(Cache::get('foo'));

// Clear the cache
Cache::flush();

Lock:

// Get an unique lock. It's very important to keep this object in memory
// so that the lock can be released.
$lock = Cache::lock('foo');
$lock->block(10); // Wait 10 seconds before throwing an exception if the lock isn't released

// Any time-consuming task
sleep(5);

// Release the lock
$lock->release();

Checklist

• Add tests and ensure they pass
• Add an entry to the CHANGELOG.md file
• Update documentation for new features

[GromNaN]

Here I'm using findOneAndUpdate and a comparison of the value to return true when the lock is acquired twice by the same instance during the same second. If I use updateOne and check $result->getUpsertedCount() > 0 || $result->getModifiedCount() > 0;, the result is false.

[jmikola]

when the lock is acquired twice by the same instance during the same second

What is the significance of "same instance during the same second"?

If updateOne reports that no document was upserted or modified, wouldn't that imply that the lock could not be acquired (i.e. has another owner and has not expired)?

I was testing this locally:

$coll = $client->selectCollection('test', 'coll');
$coll->drop();

$key = 'foo';
$owner = 'bar';
$currentTime = time();
$expiresAt = $currentTime + 10;

$isExpiredOrAlreadyOwned = ['$or' => [
    ['$lte' => ['$expiration', $currentTime]],
    ['$eq' => ['$owner', $owner]],
]];

$result = $coll->updateOne(
    ['key' => $key],
    [
        ['$set' => [
            'owner' => ['$cond' => [$isExpiredOrAlreadyOwned, $owner, '$owner']],
            'expiration' => ['$cond' => [$isExpiredOrAlreadyOwned, $expiresAt, '$expiration']],
        ]],
    ],
    ['upsert' => true],
);

printf("\n\nupdateOne: matched(%d) modified(%d) upserted(%d)\n\n",
    $result->getMatchedCount(),
    $result->getModifiedCount(),
    $result->getUpsertedCount(),
);

if ($result->getMatchedCount()) {
    echo "lock '$key' already existed\n";
}

if ($result->getModifiedCount()) {
    echo "took over an expired lock for '$owner', which will now expire at $expiresAt\n";
}

if ($result->getUpsertedCount()) {
    echo "created the lock for '$owner', which will now expire at $expiresAt\n";
}

[GromNaN]

I'm talking about this test:

laravel-mongodb/tests/Cache/MongoLockTest.php

Lines 30 to 34 in 3fa18f3

	public function testLockCanBeAcquired()
	{
	$lock = $this->getCache()->lock('foo');
	$this->assertTrue($lock->get());
	$this->assertTrue($lock->get());

updateOne reports no document modified if the values in $set are not different from the values already stored.

In your example, if you run updateOne 2 times, the 2nd time you get "lock 'foo' already existed" and nothing else. In Laravel, true is expected because the lock is acquired and owned by the current instance.

[jmikola]

Now I understand the significance of "same second" in your first comment. The timestamp field wouldn't change on subsequent updates within the same clock second, and you rely on checking the owner field (not possible with updateOne) to determine if the lock is acquired.

[GromNaN]

I reverted to updateOne in this additional PR, because UTCDateTime is more precise and acquiring a lock with the same owner in the same microsecond will not happen.

[GromNaN]

I'm not sure if I should ignore only "duplicate key errors". Laravel docs

The insertOrIgnore method will ignore errors while inserting records into the database. When using this method, you should be aware that duplicate record errors will be ignored and other types of errors may also be ignored depending on the database engine.

[alcaeus]

I dislike the ambiguity that is built into the documentation - you can't really be sure what's ignored. One option is to ignore any errors that purely come from the data (e.g. unique key violations, document constraint violations) but still report more serious problems (like server selection errors). IMO, users should be more aware of what could go wrong and explicitly ignore those errors if they don't care for them.

For the purpose of the cache/lock work, this seems sufficient so I'm fine with only ignoring unique key errors and returning to this if people have more errors they think should be ignored.

[GromNaN]

This method is no longer used by the cache. I'm reverting this change, and creating a dedicated ticket PHPORM-170

[GromNaN]

This can be improved later in a new PR.

[GromNaN]

Required to access Cache::isOwnedCurrentProcess
laravel/framework@234444e

[jmikola]

I realize this is beyond the PR, but is there a particular reason that [2, 100] is used instead of 0.02? This just seems like an odd API.

[GromNaN]

This comes from Laravel's Lottery: https://laravel.com/api/11.x/Illuminate/Support/Lottery.html

Used in DatabaseLock config: https://github.com/laravel/framework/blob/9b3bb15ab4d8e7a80bda82ffbe92613f15f69fee/src/Illuminate/Cache/DatabaseLock.php#L49

[jmikola]

I see see the same structure used in DatabaseLock, but not the Lottery class itself. Consistency is probably more important here. It may be worth elaborating on the structure of that two-element array in the doc block comment, though. As is, it's not entirely clear how the two values relate to one another.

[jmikola]

when the lock is acquired twice by the same instance during the same second

What is the significance of "same instance during the same second"?

If updateOne reports that no document was upserted or modified, wouldn't that imply that the lock could not be acquired (i.e. has another owner and has not expired)?

I was testing this locally:

$coll = $client->selectCollection('test', 'coll');
$coll->drop();

$key = 'foo';
$owner = 'bar';
$currentTime = time();
$expiresAt = $currentTime + 10;

$isExpiredOrAlreadyOwned = ['$or' => [
    ['$lte' => ['$expiration', $currentTime]],
    ['$eq' => ['$owner', $owner]],
]];

$result = $coll->updateOne(
    ['key' => $key],
    [
        ['$set' => [
            'owner' => ['$cond' => [$isExpiredOrAlreadyOwned, $owner, '$owner']],
            'expiration' => ['$cond' => [$isExpiredOrAlreadyOwned, $expiresAt, '$expiration']],
        ]],
    ],
    ['upsert' => true],
);

printf("\n\nupdateOne: matched(%d) modified(%d) upserted(%d)\n\n",
    $result->getMatchedCount(),
    $result->getModifiedCount(),
    $result->getUpsertedCount(),
);

if ($result->getMatchedCount()) {
    echo "lock '$key' already existed\n";
}

if ($result->getModifiedCount()) {
    echo "took over an expired lock for '$owner', which will now expire at $expiresAt\n";
}

if ($result->getUpsertedCount()) {
    echo "created the lock for '$owner', which will now expire at $expiresAt\n";
}

[jmikola]

Now I understand the significance of "same second" in your first comment. The timestamp field wouldn't change on subsequent updates within the same clock second, and you rely on checking the owner field (not possible with updateOne) to determine if the lock is acquired.

[jmikola]

Did you consider dropping the collection? My storage engine knowledge is outdated, but I still reckon that'd be a faster operation.

And if there's no concern about preserving indexes or other metadata (e.g. validation rules), there isn't much benefit to keeping the original collection in place.

[GromNaN]

Why not, if the engine supports it well. What is the risk of race-conditioning if there is an insert at the same time?

[jmikola]

Dropping locks the collection, and thus I expect it'd avoid any potential race conditions. In contrast, I expect deleteMany() would allow other operations to interleave.

To clarify: I don't consider this a real risk for the PR. The drop just seemed like a more straightforward operation.

[GromNaN]

I don't want to drop the collection. You could legitimately want to create a TTL index on the expiration field, and dropping the collection to purge the cache would have the side-effect of deleting the index.

[GromNaN]

See additional PR that enable TTL index: #2891

[jmikola]

I've said my peace, so I'll defer to you to incorporate any feedback you like.

[GromNaN]

Reverted to use array-access to please static analysis tools that doesn't understand that MongoDB\MongoDB\BSONDocument has magic property-access.

https://phpstan.org/r/1e7b5d92-8b98-4e53-863f-b0c8652820c1
https://psalm.dev/r/ab76084ecf

$doc = new ArrayObject(['foo' => 'bar'], ArrayObject::ARRAY_AS_PROPS);
$doc->bar;

It's also more robust in case array is used as default typemap.

[GromNaN]

I don't want to drop the collection. You could legitimately want to create a TTL index on the expiration field, and dropping the collection to purge the cache would have the side-effect of deleting the index.