Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

overwrite an extent by DropByTag #215

Open
MinKPark opened this issue Dec 22, 2021 · 2 comments
Open

overwrite an extent by DropByTag #215

MinKPark opened this issue Dec 22, 2021 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@MinKPark
Copy link

Is your feature request related to a problem? Please describe.
Currently with Kusto connector, there is a way to set a DropByTag to an extent. however, there is no way to overwrite an extent with a new data.

Describe the solution you'd like
Either having a method to call to drop an extend or I would like to allow "overwrite" mode to drop an extent if there is a matched dropbytag.

Describe alternatives you've considered
One alternative to do it is using Kusto API directly. but it is a hassle.

Additional context
If there is any known way to do this via KustoConnector. please share the documentation.

@yogilad
Copy link
Contributor

yogilad commented Dec 28, 2021

You can drop extents using this command: https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/drop-extents
Does that meet your requirement?

@MinKPark
Copy link
Author

thanks for the reply, @yogilad.
I already use ".drop" with azure.kusto.data.KustoClinet.execut_mgmt().
since I'm using Synapse, for loading(querying) and saving, I use the belows.

# load
SparkSession.Builder().getOrCreate().read \
            .format("com.microsoft.kusto.spark.synapse.datasource") \
            .option("spark.synapse.linkedService", self.linkService) \
            .option("kustoDatabase", self.database) \
            .option("kustoQuery", query) \
            .load()

# save
        obj = dataset.dataFrame.write \
        .format("com.microsoft.kusto.spark.synapse.datasource") \
        .option("spark.synapse.linkedService", self.linkService) \
        .option("kustoDatabase", self.database) \
        .option("kustoTable", table) \
        .option("tableCreateOptions","CreateIfNotExist") \
        .mode('append') \
        .save()

All these can be done via a link service. However, for executing a management command, since I'm using the azure.kusto.data.KustoClinet, I need an AppId other than the same link service. it is a double work since I need to maintain a App secret.
I wonder whether there is a way to call '.drop' via a link service or using "overwrite" mode in the above approach.

@yogilad yogilad added the enhancement New feature or request label Feb 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

3 participants