Support for creating Azure Data Explorer tables and ingestion mappings.

[ManojRaheja]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Support for creating Azure Data Explorer tables and ingestion mappings. This is required to enable e2e automated deployment of Azure Data Explorer solution that includes creating cluster, creating database, setting database permissions, creating tables, creating ingestion mappings, and creating data connections.

New or Affected Resource(s)

• azurerm_kusto

References

Related Issue: #5553
Documentation: Create table, Create data mappings

[jrauschenbusch]

I've created an initial draft for a resource azurerm_kusto_database_tablewith the following schema to show that it's possible to provide this feature:

resource "azurerm_kusto_database_table" "test" {
  name                = "table1"
  resource_group_name = azurerm_resource_group.test.name
  cluster_name        = azurerm_kusto_cluster.test.name
  database_name       = azurerm_kusto_database.test.name
  doc_string          = "Optional Docstring"
  folder              = "Optional Folder"

  column {
    name = "col1"
    type = "guid"
  }
  column {
    name = "col2"
    type = "dynamic"
  }
}

see master...jrauschenbusch:r-kusto-database-table for a Diff to the current master state.

I already tested it a bit and it seems like it works as expected.

Some notes:

• I know that the authentication stuff is a little bit hacky, but i just wanted to get it work without the need to add client authentication attributes to the resource schema.
• Unfortunately, there is currently no data plan functionality for Kusto within the azure-sdk-for-go dependency. So I had to add and use the azure-kusto-go library. (see Kusto Data Support Azure/azure-sdk-for-go#7916)
• I would also like to use prepared statements instead of these "unsafe" Kusto query methods, but it is currently not possible to use them with Mgmt statements. One also have to provide constant query strings when not using the unsafe methods, which is not a feasible approach when implementing a resource with dynamic attribute values.

@tombuildsstuff @mbfrahry Please let me know what you think about it. This would be (together with another resource azurerm_kusto_database_table_ingestion_mapping) the last required piece to get all kinds of Kusto Data Connections deployable via Terraform.

[jrauschenbusch]

Unfortunately there is no progress here due to the lack of a PR review from Hashicorp.

Maybe the used azure_kusto_go library is a problem, which brings a lot of dependencies. Also the lack of support for Prepared Statements for Management calls could be the reason:

• Can't use predefined string as Stmt in Client.Mgmt Azure/azure-kusto-go#33
• https://feedback.azure.com/forums/915733-azure-data-explorer/suggestions/41665129-migration-of-mgmt-calls-to-v2-backend-to-support-n)

Please vote up the related PR to increase the visibility and importance of this new resource as some related resources (Event Grid and IoT Hub Data connections) will only become available if this one and the mapping table resource are ready to use.

[favoretti]

FWIW - here's a separate provider implementation: https://github.com/favoretti/terraform-provider-adx

[jrauschenbusch]

@ManojRaheja I think this ticket can be closed as there is now a separate provider implementation. Or is there something missing?

[favoretti]

I'll close this one, since there's a separate provider now. We could potentially reconsider this in the future, but I suppose a separate provider solves the problem for now. Thank you.

[ManojRaheja]

Okay to close this PR, my team is looking any alternate option to build a generic module to run any ADX control command.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.