Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filter Realms processed by acl-config #1000

Merged
merged 7 commits into from
Aug 6, 2019

Conversation

ryanrath
Copy link
Contributor

@ryanrath ryanrath commented Aug 6, 2019

Co-authored-by: @plessbd

Description

  • acl-config: Utilizes the [resources|resource_types].json files to determine
    which realms are enabled. A realm is considered enabled if there is at
    least one resource w/ a resource_type that has a matching realm record.
  • ConfigFilesMigration: Makes sure that we update the existing
    resource_types.json file from resource_type_id -> resource_type.
  • etl.d/resource_types.json: We're manually inserting the 'unknown' resource
    now as opposed to including it in resources.json. This allows us to utilize
    -1 as the id. To account for that, a new action has been added to
    the ingest-resource-types pipeline.
  • xdmod-migration-8_1_2-8_5_0.json: We need to make sure that the foreign
    key constraint on resources is updated.
  • etl_action_defs.d/common/hpcdb/resources.json: Just updating the default
    value to be in line with the changes made in etl.d/resource_types.json.
  • unknown_resource_type.sql: The manual insertion of the unknown resource
    type.
  • etl_tables.d/.../resource-types.json: Adding the type_abbr column so that
    we can translate from type_abbr -> type_id and vice versa.
  • etl_tables.d/...resources.json: Updated the fk_resource_res_type fk to
    include on_delete|on_update clauses. This change allows the DB to take care
    of keeping resource_types -> resources in sync.
  • configuration/resource_types.json: Removing the Unknown resource_type as
    it's now manually inserted.
  • roles.json: The query_descripters / summary_charts for default were
    previously removed. We are adding them back in here.

Motivation and Context

We're moving towards allowing XDMoD to be installed with any combination of Realms as opposed to assuming that the Jobs realm is present. Specifically, this PR updates acl-config to utilize resource_types.json and resources.json to determine which realms are actually valid / enabled for the current installation. It will then only process the configuration data for those realms.

Tests performed

  • All automated tests were run in both a fresh install & upgrade scenario.
  • Manual spot testing was also conducted to ensure there were no knock on effects from the changes.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project as found in the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

ryanrath and others added 7 commits July 23, 2019 13:36
Co-authored-by: Benjamin D. Plessinger  <ben@plessinger.us>
- `acl-config` now takes into account which Realms have been added to the XDMoD
installation as opposed to those which have records in `datawarehouse.json`.

- Updated `ResourcesSetup.php` so that `acl-config` is run after the newly
  updated resources data is saved to `resources.json`.
We want to ensure that these realms are processed in a particular order so we've
pre-pended a number indicating when it will be processed min->max.
It was decided that acl-config should only be run if an installation adds at
least one resource. That being the case, these bits of code are no longer needed.
Just silencing warnings at a few places in the code.
- `acl-config`: Utilizes the [resources|resource_types].json files to determine
  which realms are `enabled`. A realm is considered `enabled` if there is at
  least one resource w/ a resource_type that has a matching `realm` record.
- `ConfigFilesMigration`: Makes sure that we update the existing
  `resource_types.json` file from `resource_type_id` -> `resource_type`.
- `etl.d/resource_types.json`: We're manually inserting the 'unknown' resource
  now as opposed to including it in `resources.json`. This allows us to utilize
  -1 as the id. To account for that, a new action has been added to
  the `ingest-resource-types` pipeline.
- `xdmod-migration-8_1_2-8_5_0.json`: We need to make sure that the foreign
  key constraint on `resources` is updated.
- `etl_action_defs.d/common/hpcdb/resources.json`: Just updating the default
  value to be in line with the changes made in `etl.d/resource_types.json`.
- `unknown_resource_type.sql`: The manual insertion of the `unknown` resource
  type.
- `etl_tables.d/.../resource-types.json`: Adding the `type_abbr` column so that
  we can translate from `type_abbr` -> `type_id` and vice versa.
- `etl_tables.d/...resources.json`: Updated the `fk_resource_res_type` fk to
  include `on_delete|on_update` clauses. This change allows the DB to take care
  of keeping resource_types -> resources in sync.
- `configuration/resource_types.json`: Removing the `Unknown` resource_type as
  it's now manually inserted.
- `roles.json`: The query_descripters / summary_charts for `default` were
  previously removed. We are adding them back in here.
Turns out this data has been moved to roles.d/jobs.json
@plessbd
Copy link
Contributor

plessbd commented Aug 6, 2019

as an addendum to the motivation.
this also allows an administrator to change the configuration files and it be handled by rpm when a change is made so that we do not have to migrate things like the realms files.

@ryanrath ryanrath merged commit a4ee1a1 into ubccr:xdmod8.5 Aug 6, 2019
@plessbd plessbd added Category:General General enhancement Enhancement of the functionality of an existing feature labels Aug 8, 2019
@plessbd plessbd added Category:Infrastructure Internal infrastructure updates/changes and removed Category:General General labels Oct 16, 2019
@plessbd plessbd added this to the 8.5.0 milestone Oct 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category:Infrastructure Internal infrastructure updates/changes enhancement Enhancement of the functionality of an existing feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants