Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor Accounts Realm #50

Merged
merged 5 commits into from
Feb 11, 2017
Merged

Refactor Accounts Realm #50

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
8ab116f
Minor improvements to ETLv2 logging
smgallo Feb 7, 2017
a1017d4
Remove unused min_activity_time column from peopleonaccount table
smgallo Feb 8, 2017
691273c
Merge branch 'xdmod6.6' of github.com:ubccr/xdmod into accounts-realm
smgallo Feb 8, 2017
182fb9f
Add schema migration file
smgallo Feb 8, 2017
a31e294
Updated schema versions and removed unused file
smgallo Feb 10, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions classes/ETL/Aggregator/pdoAggregator.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1205,7 +1205,7 @@ protected function processAggregationPeriods(
"unit" => $aggregationUnit,
"num_records" => $numRecords
);
$this->logger->debug(array_merge($msg, $date_result));
$this->logger->debug(array_merge($msg, $aggregationPeriodInfo));

// Insert the new rows.

Expand All @@ -1228,7 +1228,7 @@ protected function processAggregationPeriods(
. " of $totalNumAggregationPeriods) $periodId records = $numRecords, time = " .
round((microtime(true) - $dateIdStartTime), 2) . "s");

} // foreach ($aggregationPeriodList as $date_result)
} // foreach ($aggregationPeriodList as $aggregationPeriodInfo)

return $numPeriodsProcessed;

Expand Down
1 change: 1 addition & 0 deletions classes/ETL/Ingestor/pdoIngestor.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -720,6 +720,7 @@ function ($s) {
} // foreach ( $this->etlDestinationTableList as $etlTableKey => $etlTable )

if ( $this->etlOverseerOptions->isDryrun() ) {
$this->logger->debug("Source query " . $this->sourceEndpoint . ":\n" . $this->sourceQueryString);
// If this is DRYRUN mode clean up the files that tempnam() created
foreach ( $infileList as $etlTableKey => $infileName ) {
@unlink($infileName);
Expand Down
2 changes: 1 addition & 1 deletion classes/ETL/aRdbmsDestinationAction.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,8 +254,8 @@ protected function performTruncateDestinationTasks()
$sql = "TRUNCATE TABLE $tableName";

try {
$this->logger->debug("Truncate destination task " . $this->destinationEndpoint . ":\n$sql");
if ( ! $this->etlOverseerOptions->isDryrun() ) {
$this->logger->debug("Truncate destination task " . $this->destinationEndpoint . ": $sql");
$this->destinationHandle->execute($sql);
}
} catch (PDOException $e) {
Expand Down
8 changes: 0 additions & 8 deletions classes/OpenXdmod/Ingestor/Hpcdb/PeopleOnAccounts.php
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@ function __construct($dest_db, $src_db, $start_date = '1997-01-01', $end_date =
poah.resource_id,
poah.person_id,
poah.state_id AS allocationstate_id,
poah4.min_activity_time,
poah.activity_time AS start_time,
(
SELECT MIN(poah2.activity_time) AS min
Expand All @@ -32,11 +31,6 @@ function __construct($dest_db, $src_db, $start_date = '1997-01-01', $end_date =
) AS end_time,
comments
FROM hpcdb_people_on_accounts_history poah
LEFT OUTER JOIN (
SELECT poah3.person_id, MIN(poah3.activity_time) AS min_activity_time
FROM hpcdb_people_on_accounts_history poah3
GROUP BY poah3.person_id
) AS poah4 ON poah4.person_id = poah.person_id
',
'peopleonaccount',
array(
Expand All @@ -45,12 +39,10 @@ function __construct($dest_db, $src_db, $start_date = '1997-01-01', $end_date =
'resource_id',
'person_id',
'allocationstate_id',
'min_activity_time',
'start_time',
'end_time',
'comments',
)
);
}
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this file being used by ETLv2? It doesn't appear to be used anywhere in Open XDMoD. I tried adding it to the list of ingestors (in the old-style ETL), but get this error:

SQLSTATE[42S22]: Column not found: 1054 Unknown column 'poah.state_id' in 'field list'

... which isn't even related to your changes here. This looks like one of those classes that was copy/pasted from the TGcDB ingestors, but we aren't using it since we do not yet have the corresponding data in Open XDMoD.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This isn't used in ETLv2 at all. If we're not using it in Open XDMoD then I'll remove it from the codebase and we can deal with it when we make modifications to support allocations/projects.


12 changes: 12 additions & 0 deletions db/migrations/6.5.0-6.6.0/modw.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
use modw;

LOCK TABLES `peopleonaccount` WRITE;
ALTER TABLE `peopleonaccount` DISABLE KEYS;

ALTER TABLE `peopleonaccount` DROP COLUMN `min_activity_time`;
ALTER TABLE `peopleonaccount` DROP COLUMN `min_activity_time_ts`;
ALTER TABLE `peopleonaccount` DROP KEY `aggregation_index`;
ALTER TABLE `peopleonaccount` ADD KEY `aggregation_index` (`resource_id`,`account_id`,`person_id`,`start_time_ts`,`end_time_ts`,`allocationstate_id`);

ALTER TABLE `peopleonaccount` ENABLE KEYS;
UNLOCK TABLES;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add INSERT INTO schema_version_history VALUES ('modw', '6.6.0', NOW(), 'upgraded', 'N/A'); at the end of this file. Also, please update the schema version in db/data/modw.sql.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the schema version to 6.6.0 in db/data/modw.sql:
INSERT INTO schema_version_history VALUES ('modw', '6.6.0', NOW(), 'created', 'N/A');

4 changes: 1 addition & 3 deletions db/schema/modw.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -534,15 +534,13 @@ CREATE TABLE `peopleonaccount` (
`resource_id` int(11) NOT NULL,
`person_id` int(11) NOT NULL,
`allocationstate_id` int(11) NOT NULL,
`min_activity_time` datetime DEFAULT NULL,
`min_activity_time_ts` int(11) DEFAULT NULL,
`start_time` datetime NOT NULL,
`start_time_ts` int(11) DEFAULT NULL,
`end_time` datetime DEFAULT NULL,
`end_time_ts` int(11) DEFAULT NULL,
`comments` varchar(200) DEFAULT NULL,
PRIMARY KEY (`id`),
KEY `aggregation_index` (`resource_id`,`account_id`,`person_id`,`start_time_ts`,`end_time_ts`,`allocationstate_id`,`min_activity_time_ts`)
KEY `aggregation_index` (`resource_id`,`account_id`,`person_id`,`start_time_ts`,`end_time_ts`,`allocationstate_id`)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
/*!40101 SET character_set_client = @saved_cs_client */;
DROP TABLE IF EXISTS `peopleunderpi`;
Expand Down