Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/nex 136/configure spanner #636

Closed
Show file tree
Hide file tree
Changes from 63 commits
Commits
Show all changes
65 commits
Select commit Hold shift + click to select a range
e1fdcd5
Use DBAL now expression
jbout Jun 6, 2019
8052082
Move to a uuid statements primary key
jbout Jun 6, 2019
517bb8d
Fixed statements table creation.
julien-sebire Jun 6, 2019
c55d854
Fixed schema and typos...
julien-sebire Jun 6, 2019
bc4d9d3
Changes to make it work a little bit more.
Jun 6, 2019
4f95439
Merge branch 'feature/autoincrement-less-statements' of https://githu…
Jun 6, 2019
7bc6e55
Fixed wrong call to uuid generator.
julien-sebire Jun 6, 2019
e363a7a
Changed hard-coded value for writable model id with a key generated w…
julien-sebire Jun 6, 2019
6d11a56
Some codestyle.
julien-sebire Jun 6, 2019
0432cf5
Got rid of gcp Spanner reference in comments.
julien-sebire Jun 6, 2019
fe81ec1
PR fixes.
julien-sebire Jun 6, 2019
b43204a
Fixed insert return value testing.
julien-sebire Jun 6, 2019
6f1d3a3
Use the configured writable model id instead of hardcoded value.
julien-sebire Jun 6, 2019
d424b0b
Ensure that the searched model uri has a trailing #.
julien-sebire Jun 6, 2019
5e3b903
Merge branch 'develop' into feature/autoincrement-less-statements
Jun 7, 2019
33477d3
Upgraded dbal to version 2.9 and required doctrine/annotations.
julien-sebire Jun 11, 2019
c80a0b7
Added spanner dbal driver.
julien-sebire Jun 12, 2019
909b164
Added minimum stability dev to get lib-dbal-spanner:dev-develop.
julien-sebire Jun 12, 2019
f55f38b
Required branch feature/NEX-84/spanner-query-refactoring of lib-dbal-…
julien-sebire Jun 12, 2019
c1c8b98
Adding minimum stability:dev.
julien-sebire Jun 12, 2019
05078e2
Merge remote-tracking branch 'origin/feature/NEX-121/upgrade-php71' i…
julien-sebire Jun 13, 2019
2f7ec12
Merge remote-tracking branch 'origin/fix/now-expression' into feature…
julien-sebire Jun 13, 2019
7ee367c
Merge remote-tracking branch 'origin/feature/autoincrement-less-state…
julien-sebire Jun 13, 2019
6b559fb
requiring lib-search added quoting of modelId.
julien-sebire Jun 17, 2019
e79ab52
Merge branch 'feature/NEX-136/configure-spanner' of github.com:oat-sa…
julien-sebire Jun 17, 2019
710c1c9
Moved Uuid primary key generation to oat\generis\Helper\UuidPrimaryKe…
julien-sebire Jun 20, 2019
5cb5df3
Merge remote-tracking branch 'origin/feature/NEX-136/configure-spanne…
julien-sebire Jun 20, 2019
b21434f
Changed Uuid storing size to 23 bytes.
julien-sebire Jun 21, 2019
bd16e4a
Added enclosing quotes for model id.
julien-sebire Jul 16, 2019
90dc968
Merge remote-tracking branch 'origin/feature/NEX-136/configure-spanne…
julien-sebire Jul 16, 2019
99317e3
Merged develop.
julien-sebire Jul 16, 2019
03e832b
Opened statement type hinting to SpannerStatement also.
julien-sebire Jul 18, 2019
a768a64
Casted object property to string (Spanner is strict on field types).
julien-sebire Jul 18, 2019
76b32d1
Added temporary LIKE comparator for Spanner.
julien-sebire Jul 23, 2019
d64e869
Merge remote-tracking branch 'origin/develop' into feature/NEX-136/co…
julien-sebire Jul 31, 2019
2cf8572
Merge branch 'develop' into feature/NEX-136/configure-spanner
jbout Aug 7, 2019
4900c30
Merge branch 'develop' into feature/NEX-136/configure-spanner
julien-sebire Aug 20, 2019
08f3286
Removed Spanner dependency for AWS benchmark.
julien-sebire Aug 27, 2019
e90251b
Removed Spanner class in type hint for aws benchmark.
julien-sebire Sep 2, 2019
6c1886d
Casted boolean values to integer in prepared statement for postgres d…
julien-sebire Sep 5, 2019
fd45ee8
Set bin2hex as default uri provider.
julien-sebire Sep 6, 2019
18052a8
Added types to all queries.
julien-sebire Sep 9, 2019
9c5a791
Removed wrong fix for postgres boolean issue.
julien-sebire Sep 16, 2019
261a59d
Merged develop back into NEX-136
julien-sebire Sep 19, 2019
eb9cc5c
Added abstract changes to enable schema changes.
julien-sebire Sep 23, 2019
9dd2ba3
Merge branch 'feature/NEX-136/configure-spanner' of https://github.co…
julien-sebire Sep 23, 2019
e0b556e
Merge branch 'develop' into feature/NEX-136/configure-spanner
julien-sebire Sep 23, 2019
634889d
Removed forgotten TODOs.
julien-sebire Sep 23, 2019
7940538
Merge branch 'feature/NEX-136/configure-spanner' of https://github.co…
julien-sebire Sep 23, 2019
92163d3
Added an abstraction level for ModelFactory.
julien-sebire Oct 2, 2019
9032ebe
Revert CRLF change.
julien-sebire Oct 2, 2019
949e004
Merge branch 'develop' into feature/NEX-136/configure-spanner
julien-sebire Oct 2, 2019
de849cf
Merge branch 'develop' into feature/NEX-136/configure-spanner
julien-sebire Oct 3, 2019
c8d7097
Updated composer.json to require changes in lib-generis-search.
julien-sebire Oct 3, 2019
f397a74
Removed unrelated commits.
julien-sebire Oct 15, 2019
ee36b3b
Reverted CRLF changes in Namespace.
julien-sebire Oct 15, 2019
1c614d9
PR fixes.
julien-sebire Oct 17, 2019
dfcd1fa
Moved smooth tests to integration.
julien-sebire Oct 17, 2019
eee9d6a
FQCN fixes + code style.
julien-sebire Oct 17, 2019
6e020dc
Independent tests.
julien-sebire Oct 17, 2019
472b511
Removed getUniquePrimaryKey from persistence, provided by helper.
julien-sebire Oct 17, 2019
d109921
Merge branch 'feature/NEX-136/configure-spanner' of https://github.co…
julien-sebire Oct 17, 2019
6417cdb
Model ids as md5 of namespaces.
julien-sebire Oct 17, 2019
e513237
Removed GenerisInstaller, already in other PR.
julien-sebire Oct 17, 2019
d3f9565
Replaced dbWrapper with Persistence.
julien-sebire Oct 17, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion common/class.Collection.php
Original file line number Diff line number Diff line change
Expand Up @@ -347,4 +347,4 @@ public function toArray()
return (array) $returnValue;
}

}
}
6 changes: 3 additions & 3 deletions common/class.Utils.php
Original file line number Diff line number Diff line change
@@ -1,7 +1,4 @@
<?php
use oat\oatbox\PhpSerializable;
use oat\oatbox\service\ServiceManager;
use oat\generis\model\kernel\uri\UriProvider;
/**
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
Expand All @@ -22,6 +19,9 @@
* 2009-2012 (update and modification) Public Research Centre Henri Tudor (under the project TAO-SUSTAIN & TAO-DEV);
*
*/
use oat\oatbox\PhpSerializable;
use oat\oatbox\service\ServiceManager;
use oat\generis\model\kernel\uri\UriProvider;

/**
*
Expand Down
31 changes: 14 additions & 17 deletions common/ext/class.GenerisInstaller.php
Original file line number Diff line number Diff line change
Expand Up @@ -16,30 +16,22 @@
*
* Copyright (c) 2008-2010 (original work) Deutsche Institut für Internationale Pädagogische Forschung (under the project TAO-TRANSFER);
* 2009-2012 (update and modification) Public Research Centre Henri Tudor (under the project TAO-SUSTAIN & TAO-DEV);
* 2013-2014 (update and modification) Open Assessment Technologies SA;
*
* 2013-2019 (update and modification) Open Assessment Technologies SA;
*/

use core_kernel_api_ModelFactory as ModelFactory;
use oat\generis\model\data\ModelManager;
use oat\generis\model\kernel\persistence\smoothsql\search\ComplexSearchService;
use oat\oatbox\service\ServiceManager;

/**
* Custom extension installer for generis
*
* @access public
* @author Joel Bout, <joel.bout@tudor.lu>
* @package generis
*
*/
class common_ext_GenerisInstaller extends common_ext_ExtensionInstaller
{

/**
* Setup the ontology configuration
*
* @access public
* @author Joel Bout, <joel.bout@tudor.lu>
* @return mixed
* @throws common_ext_ExtensionException
*/
public function install()
{
Expand All @@ -48,16 +40,21 @@ public function install()
}

$this->installLoadDefaultConfig();


// Id of the writable model.
/** @var ModelFactory $modelFactory */
$modelFactory = $this->getServiceManager()->get(ModelFactory::SERVICE_ID);
$writableModelId = $modelFactory->getModelId(LOCAL_NAMESPACE);

$model = new \core_kernel_persistence_smoothsql_SmoothModel(array(
\core_kernel_persistence_smoothsql_SmoothModel::OPTION_PERSISTENCE => 'default',
\core_kernel_persistence_smoothsql_SmoothModel::OPTION_READABLE_MODELS => array('1'),
\core_kernel_persistence_smoothsql_SmoothModel::OPTION_WRITEABLE_MODELS => array('1'),
\core_kernel_persistence_smoothsql_SmoothModel::OPTION_NEW_TRIPLE_MODEL => '1',
\core_kernel_persistence_smoothsql_SmoothModel::OPTION_READABLE_MODELS => [$writableModelId],
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

\core_kernel_persistence_smoothsql_SmoothModel::OPTION_WRITEABLE_MODELS => [$writableModelId],
\core_kernel_persistence_smoothsql_SmoothModel::OPTION_NEW_TRIPLE_MODEL => $writableModelId,
\core_kernel_persistence_smoothsql_SmoothModel::OPTION_SEARCH_SERVICE => ComplexSearchService::SERVICE_ID,
\core_kernel_persistence_smoothsql_SmoothModel::OPTION_CACHE_SERVICE => common_cache_Cache::SERVICE_ID
));
$model->setServiceLocator(ServiceManager::getServiceManager());
$model->setServiceLocator($this->getServiceManager());
ModelManager::setModel($model);

$this->installOntology();
Expand Down
11 changes: 1 addition & 10 deletions common/persistence/class.Persistence.php
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@
* @package generis
*
*/

abstract class common_persistence_Persistence
{
/**
Expand Down Expand Up @@ -102,15 +103,5 @@ protected function getParams(){
protected function setParams($params){
$this->params = $params;
}

/**
* Generates a unique, not auto-increment based, primary key.
*
* @return string
*/
public function getUniquePrimaryKey()
{
return strrev(uniqid('', true));
}
}

30 changes: 13 additions & 17 deletions common/persistence/sql/SetupDb.php
Original file line number Diff line number Diff line change
Expand Up @@ -20,16 +20,19 @@
* @license GPLv2
* @package tao
*/

namespace oat\generis\persistence\sql;

use Doctrine\DBAL\Schema\AbstractSchemaManager;
use oat\oatbox\log\LoggerAwareTrait;
use oat\oatbox\service\ConfigurableService;
use Psr\Log\LoggerAwareInterface;
use Doctrine\DBAL\Schema\Schema;
use oat\generis\model\kernel\persistence\smoothsql\install\SmoothRdsModel;
use Doctrine\DBAL\Exception\ConnectionException;
use common_persistence_SqlPersistence as Persistence;

class SetupDb implements LoggerAwareInterface
class SetupDb extends ConfigurableService implements LoggerAwareInterface
{
use LoggerAwareTrait;

Expand All @@ -38,7 +41,7 @@ class SetupDb implements LoggerAwareInterface
* @param \common_persistence_SqlPersistence $p
* @throws \common_exception_InconsistentData
*/
public function setupDatabase(\common_persistence_SqlPersistence $p)
public function setupDatabase(Persistence $p)
{
$dbalDriver = $p->getDriver();
if (!$dbalDriver instanceof \common_persistence_sql_dbal_Driver) {
Expand All @@ -50,21 +53,15 @@ public function setupDatabase(\common_persistence_SqlPersistence $p)
$this->setupTables($p);
}

/**
* @author "Lionel Lecaque, <lionel@taotesting.com>"
*/
private function verifyDatabase(\common_persistence_SqlPersistence $p, $dbName)
private function verifyDatabase(Persistence $p, $dbName)
{
$schemaManager = $p->getSchemaManager()->getDbalSchemaManager();
if (!$this->dbExists($schemaManager, $dbName)) {
throw new \tao_install_utils_Exception('Unable to find the database, make sure that the db exists and that the db user has the rights to use it.');
}
}

/**
* @author "Lionel Lecaque, <lionel@taotesting.com>"
*/
private function setupTables(\common_persistence_SqlPersistence $p)
private function setupTables(Persistence $p)
{
$queries = $p->getPlatForm()->schemaToSql($this->getSchema($p));
foreach ($queries as $query){
Expand All @@ -76,10 +73,11 @@ private function setupTables(\common_persistence_SqlPersistence $p)
* Generate databse schema
* @return Schema
*/
public function getSchema(\common_persistence_SqlPersistence $p)
public function getSchema(Persistence $p)
{
$schema = $p->getSchemaManager()->createSchema();
SmoothRdsModel::addSmoothTables($schema);
$smoothRdsModel = $this->getServiceLocator()->get(SmoothRdsModel::class);
$smoothRdsModel->addSmoothTables($schema);
$this->addKeyValueStoreTable($schema);
return $schema;
}
Expand All @@ -99,8 +97,9 @@ protected function addKeyValueStoreTable(Schema $schema)
}

/**
* @author "Lionel Lecaque, <lionel@taotesting.com>"
* @param AbstractSchemaManager $schemaManager
* @param string $dbName
* @return bool
*/
private function dbExists(AbstractSchemaManager $schemaManager, $dbName)
{
Expand All @@ -112,10 +111,7 @@ private function dbExists(AbstractSchemaManager $schemaManager, $dbName)
}
}

/**
* @author "Lionel Lecaque, <lionel@taotesting.com>"
*/
private function cleanDb(\common_persistence_SqlPersistence $p)
private function cleanDb(Persistence $p)
{
$schema = $p->getSchemaManager()->createSchema();
$queries = $p->getPlatForm()->toDropSql($schema);
Expand Down
5 changes: 3 additions & 2 deletions composer.json
Original file line number Diff line number Diff line change
Expand Up @@ -63,11 +63,12 @@
"oat-sa/oatbox-extension-installer": "~1.1||dev-master",
"php": "^7.1",
ferenckiss89 marked this conversation as resolved.
Show resolved Hide resolved
"psr/log": "~1.0",
"oat-sa/lib-generis-search": "^2.0.1",
"oat-sa/lib-generis-search": "^2.1.0",
"monolog/monolog": "^1.23.0",
"fluent/logger": "^1.0.1",
"symfony/lock": "^3.4",
"psr/container": "^1.0.0"
"psr/container": "^1.0.0",
"ramsey/uuid": "3.8.0"
},
"require-dev": {
"mikey179/vfsstream": "1.4.0",
Expand Down
119 changes: 119 additions & 0 deletions core/kernel/api/NewSqlModelFactory.php
Original file line number Diff line number Diff line change
@@ -0,0 +1,119 @@
<?php
/**
* This program is free software; you can redistribute it and/or
* modify it under the terms of the GNU General Public License
* as published by the Free Software Foundation; under version 2
* of the License (non-upgradable).
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU General Public License for more details.
*
* You should have received a copy of the GNU General Public License
* along with this program; if not, write to the Free Software
* Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.
*
* Copyright (c) 2019 (original work) Open Assessment Technologies SA
*/

namespace oat\generis\model\kernel\api;

use core_kernel_api_ModelFactory as ModelFactory;
use Doctrine\DBAL\Schema\Schema;
use oat\generis\Helper\UuidPrimaryKeyTrait;
use RuntimeException;

class NewSqlModelFactory extends ModelFactory
{
use UuidPrimaryKeyTrait;

/**
* @inheritdoc
*/
public function addNewModel($namespace)
{
$modelId = md5($namespace);

if ($this->getPersistence()->insert('models', ['modelid' => $modelId, 'modeluri' => $namespace]) === 0) {
throw new RuntimeException('A problem occurred while creating a new model.');
}
return $modelId;
}

/**
* @inheritdoc
*/
public function prepareStatement($modelId, $subject, $predicate, $object, $lang, $author)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Method prepareStatement has 6 arguments (exceeds 4 allowed). Consider refactoring.

{
$date = $this->getPersistence()->getPlatForm()->getNowExpression();

return [
'id' => $this->getUniquePrimaryKey(),
'modelid' => $modelId,
'subject' => $subject,
'predicate' => $predicate,
'object' => $object,
'l_language' => $lang,
'author' => $author ?? '',
'epoch' => $date,
];
}

/**
* @inheritdoc
*/
public function buildModelSqlCondition(array $models)
{
$models = array_map(
function ($a) {
return "'" . $a . "'";

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we use cast (string) instead of the concatenation

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where adding the quotes, in fact. $a is supposed to be a string already.

},
$models
);
return parent::buildModelSqlCondition($models);
}

/**
* @inheritdoc
*/
public function getPropertySortingField()
{
return 'epoch';
}

/**
* @inheritdoc
*/
public function createModelsTable(Schema $schema)
{
$table = $schema->createTable('models');
$table->addColumn('modelid', 'string', ['length' => 36, 'notnull' => true]);
$table->addColumn('modeluri', 'string', ['length' => 255]);
$table->setPrimaryKey(['modelid']);

return $table;
}

/**
* @inheritdoc
*/
public function createStatementsTable(Schema $schema)
{
$table = $schema->createTable('statements');
$table->addColumn('id', 'string', ['length' => 36, 'notnull' => true]);
$table->addColumn('modelid', 'string', ['length' => 23, 'notnull' => true]);
$table->addColumn('subject', 'string', ['length' => 255]);
$table->addColumn('predicate', 'string', ['length' => 255]);
$table->addColumn('object', 'text', []);
$table->addColumn('l_language', 'string', ['length' => 255]);
$table->addColumn('author', 'string', ['length' => 255]);
$table->addColumn('epoch', 'string', ['notnull' => true]);

$table->setPrimaryKey(['id']);
$table->addIndex(['subject', 'predicate'], 'k_sp');
$table->addIndex(['predicate', 'object'], 'k_po');

return $table;
}
}
Loading