feat: Added support for VPC Flow Logs in Parquet format

[drewmullen]

Description

VPC Flow Logs now support a new output format parameter destination_options. Users can now enable their flow logs to be plain-text or parquet format.

Motivation and Context

New Feature Support. MUST WAIT FOR Provider Version 3.63 which will release 10/14/2021

Breaking Changes

None

How Has This Been Tested?

• I have tested and validated these changes using one or more of the provided examples/* projects

module "flowlogs" {
  source = "../terraform-aws-vpc"
  name = "vpc-flow-logs-s3-bucket"
  cidr = "10.30.0.0/16"

  azs            = ["us-east-1"]

  enable_flow_log           = true
  flow_log_destination_type = "s3"
  flow_log_destination_arn  = aws_s3_bucket.example.arn
  log_file_format           = "parquet"

  vpc_flow_log_tags = {
    Name = "vpc-flow-logs-s3-bucket"
  }
}

Note: can you please LMK if you want me to run the pre-commit which runs fmt and terraform-docs or if you prefer to have this done in CI. I ask because I did run it locally and several files were fmt'd so i chose not to include in the PR out of concern that it would clog the changes i made. A Contributors note in the README.md would be 🚀

[antonbabenko]

The PR looks good. Minor comments about naming.

We currently have CI which only verifies it but does not fix the documentation or code, so developers have to run pre-commit run -a and push all the changes as part of PR (there should not be a lot of changes if terraform-docs version is correct, see GH actions config yaml file).

We planned and discussed some improvements to do for the CI process but none of us have enough time for that yet, so that's why we have what we have :)

[antonbabenko]

Almost perfect.

[virgofx]

Awesome work @drewmullen ! I'll be using this as soon as it's released. Can't wait 💯

[drewmullen]

$ pre-commit run -a
Terraform fmt............................................................Passed
Terraform validate.......................................................Passed
Terraform docs...........................................................Passed
Terraform validate with tflint...........................................Passed
Check for merge conflicts................................................Passed

We may want to actually include some more instructions around pre-commit. I version bumped the provider which caused errors with the lockfile constraint in each example and I kept getting errors. took me 15m to figure out that i had to go delete the lockfiles in each module manually 😅

lmk what you think if i should add that or if theres an easier workaround

everything else should be ready to go! Thanks for your help with this PR and for maintaining this (and many other) great repos!

[antonbabenko]

An upgrade to Terraform 0.13.1 was due for some time in this module. We can safely drop Terraform 0.12 support, so I have updated versions.tf accordingly.

[antonbabenko]

v3.8.0 has been just released.

[virgofx]

@drewmullen It's not really clear on the documentation in regards to flow_log_hive_compatible_partitions and flow_log_per_hour_partition. Are these meant to be used exclusively or can they both be enabled since they reflect partitioning changes.

[drewmullen]

@drewmullen It's not really clear on the documentation in regards to flow_log_hive_compatible_partitions and flow_log_per_hour_partition. Are these meant to be used exclusively or can they both be enabled since they reflect partitioning changes.

I can tell you that both can be enabled at the same time

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.