S3 bucket force_destroy error: MalformedXML #4168
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
AWS provider was not checking whether DeleteMarkers are left in S3 bucket causing s3.DeleteObjectsInput to send empty XML which resulted in 400 error and MalformedXML message.
This was referenced Dec 4, 2015
mtekel
added a commit
to alphagov/paas-cf
that referenced
this pull request
Dec 7, 2015
Add a pipeline to destroy environments. It currently consists of 2 jobs - one to destroy VPC which then triggers a job to destroy S3 bucket. We used default/recommended pipeline/task/script distribution with this pipeline. We include taks from file, because this task repeats. Never- theless, in this case, the pipeline needs to contain quite chunky config definitions (params), because they can't be included in the task itself, due to fly only expanding params in the pipeline. In comparison with deploy pipeline, we don't use terraform's ability to pull remote state file, due to bug encountered with `bucket.tfstate`: hashicorp/terraform#4154 The remote state file worked with VPC, but we wanted to unify both tasks on same code to demonstrate default concourse pipeline file organization. We also use fake/dummy `trigger-bucket-destroy` resource here to trigger bucket destroy job after vpc destroy job. The terraform bucket destroy job currently doesnt't work, due to bugs: hashicorp/terraform#3981 and missing functionality - the old file versions are actually not removed. @combor has risen PR to add this functionality here: hashicorp/terraform#4168
combor
pushed a commit
to alphagov/paas-docker-terraform
that referenced
this pull request
Dec 8, 2015
Due to bug in Terraform (
hashicorp/terraform#4168 ) we have to
use our patched fork to be able to delete S3 buckets. Also, we need to
reduce size of this contaier so it is using Alipne linux now instead
Ubuntu.
|
Checks out, thanks! |
|
whoops, didn't mean to close.. |
catsby
added a commit
that referenced
this pull request
Dec 8, 2015
S3 bucket force_destroy error: MalformedXML
|
Just curious, is there maybe an ETA for the next release containing this fix? Thanks! And happy new year! |
|
Sorry @aviflax , we don't give ETAs really. I can say that we had meant to release before the holidays but hid a blocker, so we had to delay. I expect a release soon, hopefully this week 😄 |
|
Thanks, no worries! Happy new year!
|
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further. |
ghost
locked and limited conversation to collaborators
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While doing forced destroy and if versioning is enabled for a bucket, AWS provider is not checking whether DeleteMarkers and versions of objects are left resulting in following error:
s3conn.ListObjectsis checking only wheter there are objects (not versions) in the bucket causings3.DeleteObjectsto send empty XML which results in 400 error and MalformedXML message.In this PR I'm using
ListObjectVersionsto get Versions and DeleteMarkers slice and createobjectsToDeleteslice that is passed tos3conn.DeleteObjectsAfter this change AWS provider is able to delete all versions and delete markers for versioned S3 bucket and finally properly delete versioned bucket.