Support AWS Data Pipeline

[petterl]

It would be nice to be able to use AWS Data Pipeline with Terraform.
We are using it for dumping data from RDS to S3 every night.

http://docs.aws.amazon.com/datapipeline/latest/APIReference/Welcome.html

I need ability to create:

• S3DataNode
  • CSVDataFormat
  • TSVDataFormat
• SqlDataNode
  • RdsDatabase
• CopyActivity
  • Ec2Resource

Similar issue in core before split: hashicorp/terraform#4077

[cep21]

Supporting the nodes may be too much at first. What about starting with JSON import/export of the pipeline definition.

[armanshan12]

Any updates on this?

[Constantin07]

Any status updates on this ? I need this too in TF, don't want to mix TF and CF.

[scraly]

Hi,
we terraformed EMR and EMR steps but now we need to terraform aws data pipeline which should schedule emr.
How can we do that?
When it will be possible to terraform data pipeline resources?
Thanks.

[djk29a]

You can deploy data pipelines via Terraform by using CloudFormation stacks to create data pipelines. Note that data pipelines in use (especially in "deactivating" state) can be very unstable in their provisioning states and can oftentimes fail to delete after several minutes of no feedback. This will result in a CloudFormation rollback or a stack creation that may require you to clean up the CF stack.

[dvopsway]

+1 Any updates on this?

[nolancash]

It looks like this is pretty close to being done. Do we know how long till this will be released?

[scottpazelt]

Is there any update on when this will be available?
Thanks.

[dave2535]

following

[scottpazelt]

@nolancash are you aware of any timing on this?

[nolancash]

@scottpazelt I have no connection to hashicorp and thus have no info.

[Szasza]

@bflad now that TF 0.12 is released, could the PR on this get some attention please?

Thank you in advance for your time.

[bflad]

The first piece of functionality here, the aws_datapipeline_pipeline resource, has been released in version 2.19.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this new resource, please create a new GitHub issue following the template for triage. Thanks!

[tbugfinder]

Current resource implementation seems to be very basic. Would any options be added to it soon ish?

[kevintrannz]

Any update on the release time frame support please?

[dotnetooo]

same question here. is the resource going to be available any time soon?

[breathingdust]

Closed via #22597

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.