-
Notifications
You must be signed in to change notification settings - Fork 9.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Remove readonly properties from DMS ReplicationTaskSettings #13476
Conversation
aws/diff_suppress_funcs.go
Outdated
@@ -8,7 +8,7 @@ import ( | |||
"strings" | |||
|
|||
"github.com/hashicorp/terraform-plugin-sdk/helper/schema" | |||
"github.com/jen20/awspolicyequivalence" | |||
awspolicy "github.com/jen20/awspolicyequivalence" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure why this appeared here, but the tests pass and it seems like it should be this way... Happy to remove it if it's not needed. (FWIW, I'm relatively new to golang)
delete(logging, "CloudWatchLogGroup") | ||
delete(logging, "CloudWatchLogStream") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you define these in your task settings, DMS API will return an error that they are read only.
|
||
controlTablesSettings, ok := settingsData["ControlTablesSettings"].(map[string]interface{}) | ||
if ok { | ||
delete(controlTablesSettings, "historyTimeslotInMinutes") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There is another property HistoryTimeslotInMinutes
(with a capital H) that is the real property. I don't know why this one exists, but I'm cleaning it out since leaving it in when we set state would cause a difference to show.
{ | ||
Config: dmsReplicationTaskConfig(randId), | ||
PlanOnly: true, | ||
ExpectNonEmptyPlan: false, | ||
}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Now we run plan after create to assert that there are no changes. Without the fix, this will fail.
data "aws_iam_policy_document" "dms_assume_role_policy_document" { | ||
statement { | ||
actions = ["sts:AssumeRole"] | ||
|
||
principals { | ||
identifiers = ["dms.amazonaws.com"] | ||
type = "Service" | ||
} | ||
} | ||
} | ||
|
||
resource "aws_iam_role" "dms_vpc_role" { | ||
assume_role_policy = data.aws_iam_policy_document.dms_assume_role_policy_document.json | ||
name = "dms-vpc-role" | ||
} | ||
|
||
resource "aws_iam_role" "dms_cloudwatch_logs_role" { | ||
assume_role_policy = data.aws_iam_policy_document.dms_assume_role_policy_document.json | ||
name = "dms-cloudwatch-logs-role" | ||
} | ||
|
||
resource "aws_iam_role_policy_attachment" "dms_vpc_access_policy" { | ||
role = aws_iam_role.dms_vpc_role.name | ||
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole" | ||
} | ||
|
||
resource "aws_iam_role_policy_attachment" "dms_cloudwatch_logs_access_policy" { | ||
role = aws_iam_role.dms_cloudwatch_logs_role.name | ||
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonDMSCloudWatchLogsRole" | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Configuring DMS via the API requires two access policies, without them you'll get an error:
Error: AccessDeniedFault: The IAM Role arn:aws:iam::<accountid>:role/dms-vpc-role is not configured properly.
I'm assuming that people run these tests against an empty AWS account, but if that's not the case, then this may cause failures because the role names are fixed.
@@ -178,7 +219,7 @@ resource "aws_dms_replication_task" "dms_replication_task" { | |||
migration_type = "full-load" | |||
replication_instance_arn = "${aws_dms_replication_instance.dms_replication_instance.replication_instance_arn}" | |||
replication_task_id = "tf-test-dms-replication-task-%[1]s" | |||
replication_task_settings = "{\"TargetMetadata\":{\"TargetSchema\":\"\",\"SupportLobs\":true,\"FullLobMode\":false,\"LobChunkSize\":0,\"LimitedSizeLobMode\":true,\"LobMaxSize\":32,\"LoadMaxFileSize\":0,\"ParallelLoadThreads\":0,\"BatchApplyEnabled\":false},\"FullLoadSettings\":{\"FullLoadEnabled\":true,\"ApplyChangesEnabled\":false,\"TargetTablePrepMode\":\"DROP_AND_CREATE\",\"CreatePkAfterFullLoad\":false,\"StopTaskCachedChangesApplied\":false,\"StopTaskCachedChangesNotApplied\":false,\"ResumeEnabled\":false,\"ResumeMinTableSize\":100000,\"ResumeOnlyClusteredPKTables\":true,\"MaxFullLoadSubTasks\":8,\"TransactionConsistencyTimeout\":600,\"CommitRate\":10000},\"Logging\":{\"EnableLogging\":false,\"LogComponents\":[{\"Id\":\"SOURCE_UNLOAD\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"TARGET_LOAD\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"SOURCE_CAPTURE\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"TARGET_APPLY\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"TASK_MANAGER\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"}],\"CloudWatchLogGroup\":null,\"CloudWatchLogStream\":null},\"ControlTablesSettings\":{\"historyTimeslotInMinutes\":5,\"ControlSchema\":\"\",\"HistoryTimeslotInMinutes\":5,\"HistoryTableEnabled\":false,\"SuspendedTablesTableEnabled\":false,\"StatusTableEnabled\":false},\"StreamBufferSettings\":{\"StreamBufferCount\":3,\"StreamBufferSizeInMB\":8,\"CtrlStreamBufferSizeInMB\":5},\"ChangeProcessingDdlHandlingPolicy\":{\"HandleSourceTableDropped\":true,\"HandleSourceTableTruncated\":true,\"HandleSourceTableAltered\":true},\"ErrorBehavior\":{\"DataErrorPolicy\":\"LOG_ERROR\",\"DataTruncationErrorPolicy\":\"LOG_ERROR\",\"DataErrorEscalationPolicy\":\"SUSPEND_TABLE\",\"DataErrorEscalationCount\":0,\"TableErrorPolicy\":\"SUSPEND_TABLE\",\"TableErrorEscalationPolicy\":\"STOP_TASK\",\"TableErrorEscalationCount\":0,\"RecoverableErrorCount\":-1,\"RecoverableErrorInterval\":5,\"RecoverableErrorThrottling\":true,\"RecoverableErrorThrottlingMax\":1800,\"ApplyErrorDeletePolicy\":\"IGNORE_RECORD\",\"ApplyErrorInsertPolicy\":\"LOG_ERROR\",\"ApplyErrorUpdatePolicy\":\"LOG_ERROR\",\"ApplyErrorEscalationPolicy\":\"LOG_ERROR\",\"ApplyErrorEscalationCount\":0,\"FullLoadIgnoreConflicts\":true},\"ChangeProcessingTuning\":{\"BatchApplyPreserveTransaction\":true,\"BatchApplyTimeoutMin\":1,\"BatchApplyTimeoutMax\":30,\"BatchApplyMemoryLimit\":500,\"BatchSplitSize\":0,\"MinTransactionSize\":1000,\"CommitTimeout\":1,\"MemoryLimitTotal\":1024,\"MemoryKeepTime\":60,\"StatementCacheSize\":50}}" | |||
replication_task_settings = "{\"BeforeImageSettings\":null,\"ChangeProcessingDdlHandlingPolicy\":{\"HandleSourceTableAltered\":true,\"HandleSourceTableDropped\":true,\"HandleSourceTableTruncated\":true},\"ChangeProcessingTuning\":{\"BatchApplyMemoryLimit\":500,\"BatchApplyPreserveTransaction\":true,\"BatchApplyTimeoutMax\":30,\"BatchApplyTimeoutMin\":1,\"BatchSplitSize\":0,\"CommitTimeout\":1,\"MemoryKeepTime\":60,\"MemoryLimitTotal\":1024,\"MinTransactionSize\":1000,\"StatementCacheSize\":50},\"CharacterSetSettings\":null,\"ControlTablesSettings\":{\"ControlSchema\":\"\",\"HistoryTableEnabled\":false,\"HistoryTimeslotInMinutes\":5,\"StatusTableEnabled\":false,\"SuspendedTablesTableEnabled\":false},\"ErrorBehavior\":{\"ApplyErrorDeletePolicy\":\"IGNORE_RECORD\",\"ApplyErrorEscalationCount\":0,\"ApplyErrorEscalationPolicy\":\"LOG_ERROR\",\"ApplyErrorFailOnTruncationDdl\":false,\"ApplyErrorInsertPolicy\":\"LOG_ERROR\",\"ApplyErrorUpdatePolicy\":\"LOG_ERROR\",\"DataErrorEscalationCount\":0,\"DataErrorEscalationPolicy\":\"SUSPEND_TABLE\",\"DataErrorPolicy\":\"LOG_ERROR\",\"DataTruncationErrorPolicy\":\"LOG_ERROR\",\"FailOnNoTablesCaptured\":false,\"FailOnTransactionConsistencyBreached\":false,\"FullLoadIgnoreConflicts\":true,\"RecoverableErrorCount\":-1,\"RecoverableErrorInterval\":5,\"RecoverableErrorThrottling\":true,\"RecoverableErrorThrottlingMax\":1800,\"TableErrorEscalationCount\":0,\"TableErrorEscalationPolicy\":\"STOP_TASK\",\"TableErrorPolicy\":\"SUSPEND_TABLE\"},\"FullLoadSettings\":{\"CommitRate\":10000,\"CreatePkAfterFullLoad\":false,\"MaxFullLoadSubTasks\":8,\"StopTaskCachedChangesApplied\":false,\"StopTaskCachedChangesNotApplied\":false,\"TargetTablePrepMode\":\"DROP_AND_CREATE\",\"TransactionConsistencyTimeout\":600},\"Logging\":{\"EnableLogging\":false,\"LogComponents\":[{\"Id\":\"TRANSFORMATION\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"SOURCE_UNLOAD\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"IO\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"TARGET_LOAD\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"PERFORMANCE\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"SOURCE_CAPTURE\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"SORTER\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"REST_SERVER\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"VALIDATOR_EXT\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"TARGET_APPLY\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"TASK_MANAGER\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"TABLES_MANAGER\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"METADATA_MANAGER\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"FILE_FACTORY\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"COMMON\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"ADDONS\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"DATA_STRUCTURE\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"COMMUNICATION\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"},{\"Id\":\"FILE_TRANSFER\",\"Severity\":\"LOGGER_SEVERITY_DEFAULT\"}]},\"LoopbackPreventionSettings\":null,\"PostProcessingRules\":null,\"StreamBufferSettings\":{\"CtrlStreamBufferSizeInMB\":5,\"StreamBufferCount\":3,\"StreamBufferSizeInMB\":8},\"TargetMetadata\":{\"BatchApplyEnabled\":false,\"FullLobMode\":false,\"InlineLobMaxSize\":0,\"LimitedSizeLobMode\":true,\"LoadMaxFileSize\":0,\"LobChunkSize\":0,\"LobMaxSize\":32,\"ParallelApplyBufferSize\":0,\"ParallelApplyQueuesPerThread\":0,\"ParallelApplyThreads\":0,\"ParallelLoadBufferSize\":0,\"ParallelLoadQueuesPerThread\":0,\"ParallelLoadThreads\":0,\"SupportLobs\":true,\"TargetSchema\":\"\",\"TaskRecoveryTableEnabled\":false}}" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm updating the task settings to include all the writeable properties that the DescribeReplicationTasks
call returns. Even with my changes, this is the only way to get a clean plan since the endpoint returns all properties, even ones that are set to defaults or unused. Perhaps there is some future work in the same vein that would intelligently handle defaults or disabled properties.
As for what actually changed on this line, I added things like BeforeImageSettings
and more LogComponent
items, and I removed CloudWatchLogGroup
, CloudWatchLogStream
, and historyTimeslotInMinutes
.
@anGie44 @bflad any chance you could give this PR an eye? It would be great to have these read only DMS tasks settings removed, right now creating a task with logging enabled fails with: Error: InvalidParameterValueException: Task Settings CloudWatchLogGroup or CloudWatchLogStream cannot be modified.
status code: 400, request id: <REMOVED> As a workaround we're creating tasks with |
Since this is an active issue, and has many people reporting they are effected. Can someone please get this committed to master? Just 1 more test to pass it looks like we just need a linting test bug squashed. |
It's a new year; the tests are passing, and this isn't merged yet? Is there anything I can do to help, or should I wait another 6 months? |
I would appreciate it a lot if this gets finally merged |
Would be really great to get this merged, closing in on a year and it looks like the community wants this based on 👍 . |
@aburgel - are you able to resolve the merge conflicts on |
just resolved the merge conflicts |
@ewbankkit please can be looked at to be in the next release? |
@aburgel Thank you for this PR! I will be looking at it in the near future. I may need to make some minor changes. Coordinate with me if you intend to push to this branch and make sure to pull any changes I've made. Please note that you will still receive all credit for the PR and your code. Thanks again for your interest! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good! 🎉
Acceptance tests in commercial AWS:
--- PASS: TestAccAWSDmsReplicationTask_basic (453.75s)
Acceptance tests in GovCloud:
--- PASS: TestAccAWSDmsReplicationTask_basic (447.48s)
@aburgel Thanks for your work on this! I apologize it took so long to merge. It looks like AWS regularly adds new settings since between opening the PR and now 3 more were added (see below). That means we'll probably need to keep checking this.
|
Thank you for getting it merged @YakDriver !
Now that we have this pattern in place, it will be a lot easier to keep on top of new settings. Any tips for getting a faster turn around on PRs? Or is there some other mechanism we can introduce to make these kind of issues easier to manage? |
This has been released in version 3.36.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks! |
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
Community Note
Closes #1513
The DMS API
DescribeReplicationTasks
includes read only properties inReplicationTaskSettings
. This causesterraform plan
to show that there are changes to apply since your local state will aways be different from remote state. This PR removes these read only properties when setting state so that we only compare properties that are under our control.Release note for CHANGELOG:
Output from acceptance testing: