Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support placeholders for TransformStep #157

Merged
merged 13 commits into from
Oct 18, 2021

Conversation

ca-nguyen
Copy link
Contributor

@ca-nguyen ca-nguyen commented Sep 2, 2021

Description

Please include a summary of the change being made.

Fixes #117 and #171

Why is the change necessary?

Currently, it is not possible to use placeholders for Sagemaker Transform properties. The properties cannot be defined dynamically, as they need to be defined in the Transformer which does not accept placeholders.
This change makes it possible to use placeholders for Transform properties by using the parameters field that are passed down from the TransformStep.

Solution

Use the parameters field that is compatible with placeholders to define TrainingStep properties.
Merge the parameters that were generated from the Transformer with the input parameters:

  • The input parameters will overwrite the parameters generated from the Transformer if the properties were defined in both
  • All TransformStep properties will be placeholder compatible
  • The input parameters should follow the schema described in CreateTransformJob API doc

Testing

Added integration and unit tests


Pull Request Checklist

Please check all boxes (including N/A items)

Testing

  • Unit tests added
  • integration test added

Documentation

  • docs: All relevant docs updated
  • docstrings: All public APIs documented

Title and description

  • Change type: Title is prefixed with change type: and follows conventional commits
  • References: Indicate issues fixed via: Fixes #xxx

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license.

@ca-nguyen
Copy link
Contributor Author

The tests pass locally - will need more investigation for throttling issues during build

Copy link

@evscott evscott left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Besides the small comment typo, lgtm 🚀

Copy link
Contributor

@shivlaks shivlaks left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getting close. some minor nits but otherwise looks great!

tests/integ/test_sagemaker_steps.py Outdated Show resolved Hide resolved
tests/integ/test_sagemaker_steps.py Outdated Show resolved Hide resolved
tests/unit/test_sagemaker_steps.py Outdated Show resolved Hide resolved
ca-nguyen and others added 3 commits October 15, 2021 10:26
Co-authored-by: Eliot V Scott <eliotvscott@gmail.com>
Co-authored-by: Shiv Lakshminarayan <shivlaks@amazon.com>
@ca-nguyen ca-nguyen requested a review from shivlaks October 15, 2021 17:44
@ca-nguyen ca-nguyen dismissed shivlaks’s stale review October 15, 2021 17:45

Updated PR per suggestions and addressed comments

@ca-nguyen ca-nguyen requested a review from evscott October 15, 2021 17:45
@StepFunctions-Bot
Copy link
Contributor

AWS CodeBuild CI Report

  • CodeBuild project: AutoBuildProject6AEA49D1-sEHrOdk7acJc
  • Commit ID: 1422243
  • Result: SUCCEEDED
  • Build Logs (available for 30 days)

Powered by github-codebuild-logs, available on the AWS Serverless Application Repository

@ca-nguyen ca-nguyen merged commit 8b6d0eb into aws:main Oct 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

add placeholders support for all properties
4 participants