Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

PHPORM-215 Implement Schema::getColumns and getIndexes #3045

Merged
merged 4 commits into from
Jul 16, 2024
Merged

PHPORM-215 Implement Schema::getColumns and getIndexes #3045

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
72be9c2
PHPORM-215 Implement Schema::getColumns and getIndexes
GromNaN Jul 15, 2024
79b429a
Fix compatibility with MongoDB 4.4
GromNaN Jul 15, 2024
cf3eb71
Use $sample to extract field names
GromNaN Jul 16, 2024
cda02a9
Reduce $sample size to fit with Atlas M0
GromNaN Jul 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,8 @@ All notable changes to this project will be documented in this file.
## [4.9.0] - coming soon

* Add `Connection::getServerVersion()` by @GromNaN in [#3043](https://github.com/mongodb/laravel-mongodb/pull/3043)
* Add `Schema\Builder::getTables()` and `getTableListing` by @GromNaN in [#3044](https://github.com/mongodb/laravel-mongodb/pull/3044)
* Add `Schema\Builder::getTables()` and `getTableListing()` by @GromNaN in [#3044](https://github.com/mongodb/laravel-mongodb/pull/3044)
* Add `Schema\Builder::getColumns()` and `getIndexes()` by @GromNaN in [#3045](https://github.com/mongodb/laravel-mongodb/pull/3045)

## [4.6.0] - 2024-07-09

Expand Down
77 changes: 77 additions & 0 deletions src/Schema/Builder.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,16 @@

use Closure;
use MongoDB\Model\CollectionInfo;
use MongoDB\Model\IndexInfo;

use function array_keys;
use function assert;
use function count;
use function current;
use function implode;
use function iterator_to_array;
use function sort;
use function sprintf;
use function usort;

class Builder extends \Illuminate\Database\Schema\Builder
Expand Down Expand Up @@ -146,6 +151,78 @@ public function getTableListing()
return $collections;
}

public function getColumns($table)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method could get slow for larger collections, so it may be worth adding a $sample pipeline stage down the road.

Depending on how much the feature is used, we can also consider expanding its functionality in a separate project to be more similar to Compass' schema analysis feature. For now, the current solution provides a good starting point.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should try to replicate what is done by Compass. I added $sample and $limit stages.

Copy link
Member Author

@GromNaN GromNaN Jul 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried on atlas sample dataset sample_airbnb.listingsAndReviews. With $sample, I get this error:

  PlanExecutor error during aggregation :: caused by :: Sort exceeded memory limit of 33554432 bytes, but did not opt in to external sorting. Aborting operation. Pass allowDiskUse:true to opt in.

It works without it. There might be an optimization on the server for this use-case.

{
$stats = $this->connection->getMongoDB()->selectCollection($table)->aggregate([
// Sample 10,000 documents to get a representative sample of the collection
['$sample' => ['size' => 10_000]],
// Convert each document to an array of fields
['$project' => ['fields' => ['$objectToArray' => '$$ROOT']]],
// Unwind to get one document per field
['$unwind' => '$fields'],
// Group by field name, count the number of occurrences and get the types
[
'$group' => [
'_id' => '$fields.k',
'total' => ['$sum' => 1],
'types' => ['$addToSet' => ['$type' => '$fields.v']],
],
],
// Sort by field name
['$sort' => ['_id' => 1]],
// Limit to 1,000 fields
['$limit' => 1000],
], ['typeMap' => ['array' => 'array']])->toArray();

$columns = [];
foreach ($stats as $stat) {
sort($stat->types);
$type = implode(', ', $stat->types);
$columns[] = [
'name' => $stat->_id,
'type_name' => $type,
'type' => $type,
'collation' => null,
'nullable' => $stat->_id !== '_id',
'default' => null,
'auto_increment' => false,
'comment' => sprintf('%d occurrences', $stat->total),
'generation' => $stat->_id === '_id' ? ['type' => 'objectId', 'expression' => null] : null,
];
}

return $columns;
}

public function getIndexes($table)
{
$indexes = $this->connection->getMongoDB()->selectCollection($table)->listIndexes();

$indexList = [];
foreach ($indexes as $index) {
assert($index instanceof IndexInfo);
$indexList[] = [
'name' => $index->getName(),
'columns' => array_keys($index->getKey()),
'primary' => $index->getKey() === ['_id' => 1],
'type' => match (true) {
$index->isText() => 'text',
$index->is2dSphere() => '2dsphere',
$index->isTtl() => 'ttl',
default => 'default',
},
'unique' => $index->isUnique(),
];
}

return $indexList;
}

public function getForeignKeys($table)
{
return [];
}

/** @inheritdoc */
protected function createBlueprint($table, ?Closure $callback = null)
{
Expand Down
67 changes: 67 additions & 0 deletions tests/SchemaTest.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,11 @@

use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Schema;
use MongoDB\BSON\Binary;
use MongoDB\BSON\UTCDateTime;
use MongoDB\Laravel\Schema\Blueprint;

use function collect;
use function count;

class SchemaTest extends TestCase
Expand Down Expand Up @@ -416,6 +419,70 @@ public function testGetTableListing()
$this->assertContains('newcollection_two', $tables);
}

public function testGetColumns()
{
$collection = DB::connection('mongodb')->collection('newcollection');
$collection->insert(['text' => 'value', 'mixed' => ['key' => 'value']]);
$collection->insert(['date' => new UTCDateTime(), 'binary' => new Binary('binary'), 'mixed' => true]);

$columns = Schema::getColumns('newcollection');
$this->assertIsArray($columns);
$this->assertCount(5, $columns);

$columns = collect($columns)->keyBy('name');

$columns->each(function ($column) {
$this->assertIsString($column['name']);
$this->assertEquals($column['type'], $column['type_name']);
$this->assertNull($column['collation']);
$this->assertIsBool($column['nullable']);
$this->assertNull($column['default']);
$this->assertFalse($column['auto_increment']);
$this->assertIsString($column['comment']);
});

$this->assertEquals('objectId', $columns->get('_id')['type']);
$this->assertEquals('objectId', $columns->get('_id')['generation']['type']);
$this->assertNull($columns->get('text')['generation']);
$this->assertEquals('string', $columns->get('text')['type']);
$this->assertEquals('date', $columns->get('date')['type']);
$this->assertEquals('binData', $columns->get('binary')['type']);
$this->assertEquals('bool, object', $columns->get('mixed')['type']);
$this->assertEquals('2 occurrences', $columns->get('mixed')['comment']);

// Non-existent collection
$columns = Schema::getColumns('missing');
$this->assertSame([], $columns);
}

public function testGetIndexes()
{
Schema::create('newcollection', function (Blueprint $collection) {
$collection->index('mykey1');
$collection->string('mykey2')->unique('unique_index');
$collection->string('mykey3')->index();
});
$indexes = Schema::getIndexes('newcollection');
$this->assertIsArray($indexes);
$this->assertCount(4, $indexes);

$indexes = collect($indexes)->keyBy('name');

$indexes->each(function ($index) {
$this->assertIsString($index['name']);
$this->assertIsString($index['type']);
$this->assertIsArray($index['columns']);
$this->assertIsBool($index['unique']);
$this->assertIsBool($index['primary']);
});
$this->assertTrue($indexes->get('_id_')['primary']);
$this->assertTrue($indexes->get('unique_index_1')['unique']);

// Non-existent collection
$indexes = Schema::getIndexes('missing');
$this->assertSame([], $indexes);
}

protected function getIndex(string $collection, string $name)
{
$collection = DB::getCollection($collection);
Expand Down
Loading