Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix newlines for ORDER BY and GROUP BY in ETL DbModel #151

Merged
merged 2 commits into from
Jun 6, 2017

Conversation

smgallo
Copy link
Contributor

@smgallo smgallo commented Jun 6, 2017

Fix newlines for ORDER BY and GROUP BY in ETL DbModel

Description

A new line was missing when both an ORDER BY and GROUP BY was specified in a constructed ETLv2 query.

Motivation and Context

Bugfix

Tests performed

Tested query generation with only GROUP BY, only ORDER BY, and both by modifying the OSG job reporting verification action.

php etl_overseer.php -c ../../../etc/etl/etl.json -a VerifyOsgJobReporting -t -v debug -n 5

Generated queries:

SELECT
r.id AS id,
r.name AS name,
r.code AS code,
max(maxts.max_ts) AS last_entry_date,
5 AS number_of_days
FROM `federated_osg`.`resourcefact` AS r
JOIN `federated_osg`.`raw_jobs_test` AS j ON r.code = 'OSG'
JOIN ( SELECT 2799 as resource_id, max(j.ts) as max_ts FROM federated_osg.raw_jobs_test j ) AS maxts ON maxts.resource_id = r.id
WHERE maxts.max_ts < (NOW() - INTERVAL 5 DAY)
AND r.id IN (2799)
GROUP BY r.id

SELECT
r.id AS id,
r.name AS name,
r.code AS code,
max(maxts.max_ts) AS last_entry_date,
5 AS number_of_days
FROM `federated_osg`.`resourcefact` AS r
JOIN `federated_osg`.`raw_jobs_test` AS j ON r.code = 'OSG'
JOIN ( SELECT 2799 as resource_id, max(j.ts) as max_ts FROM federated_osg.raw_jobs_test j ) AS maxts ON maxts.resource_id = r.id
WHERE maxts.max_ts < (NOW() - INTERVAL 5 DAY)
AND r.id IN (2799)
ORDER BY name asc

SELECT
r.id AS id,
r.name AS name,
r.code AS code,
max(maxts.max_ts) AS last_entry_date,
5 AS number_of_days
FROM `federated_osg`.`resourcefact` AS r
JOIN `federated_osg`.`raw_jobs_test` AS j ON r.code = 'OSG'
JOIN ( SELECT 2799 as resource_id, max(j.ts) as max_ts FROM federated_osg.raw_jobs_test j ) AS maxts ON maxts.resource_id = r.id
WHERE maxts.max_ts < (NOW() - INTERVAL 5 DAY)
AND r.id IN (2799)
GROUP BY r.id
ORDER BY name asc

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project as found in the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@smgallo smgallo added bug Bugfixes Category:ETL Extract Transform Load labels Jun 6, 2017
@smgallo smgallo added this to the v6.7.0 milestone Jun 6, 2017
@smgallo smgallo requested a review from plessbd June 6, 2017 13:53
Copy link
Contributor

@plessbd plessbd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update unit tests (I would probably just trim the results) and all good.

@tyearke tyearke changed the base branch from xdmod6.7 to xdmod7.0 June 6, 2017 16:54
@tyearke tyearke modified the milestones: v7.0.0, v6.7.0 Jun 6, 2017
@smgallo smgallo merged commit aa0a13e into ubccr:xdmod7.0 Jun 6, 2017
@smgallo smgallo deleted the etl/fix-order-by branch June 6, 2017 17:09
ryanrath pushed a commit to ryanrath/xdmod that referenced this pull request Jul 24, 2017
chakrabortyr pushed a commit to chakrabortyr/xdmod that referenced this pull request Oct 17, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Bugfixes Category:ETL Extract Transform Load
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants