Implement Azure Service Bus Topic (Create, Delete) Operators

[bharanidharan14]

• Added AzureServiceBusTopicCreateOperator, AzureServiceBusTopicDeleteOperator to create and delete topic in azure service bus
• Added example to create and delete topic operator in Example DAG
• Added Unit Test case for operators

[ashb]

LGTM, one small change plesae

[ashb]

What is the behaviour if the topic you try to create already exists? I'm assuming the underlying hook would fail with an exception, but I think the operator should catch that error so that the operator can be used idempotently: i.e. we should think of the job of this operator to ensure that the specific topic exists. (Similarly for delete operator)

[bharanidharan14]

What is the behaviour if the topic you try to create already exists? I'm assuming the underlying hook would fail with an exception, but I think the operator should catch that error so that the operator can be used idempotently: i.e. we should think of the job of this operator to ensure that the specific topic exists. (Similarly for delete operator)

yes the existing flow will through an error if the topic name exists, Now I have add try catch so before creating the topic just checking if the topic exists, if it exists logging it with message and returning the topic name, similarly for delete operator

[josh-fell]

yes the existing flow will through an error if the topic name exists, Now I have add try catch so before creating the topic just checking if the topic exists, if it exists logging it with message and returning the topic name, similarly for delete operator

I wonder if there should be a parameter for users to have the option of overwriting a topic that might exist. WDYT?

Also, for deleting a topic, if the topic doesn't exist should the task fail? Would it be misleading for a task to succeed if the operation is should perform does not occur and the user didn't explicitly choose to be indifferent about the operation? I could imagine a scenario where users expect to have topics deleted when they aren't even though "Airflow says" the deletion task succeeded. Just some thoughts.

[ashb]

Would it be misleading for a task to succeed if the operation is should perform does not occur and the user didn't explicitly choose to be indifferent about the operation

Not to my mind. Good tasks are idempotent, so deleting a non-existent topic has achieved the goal: to ensure that the topic does not exist.

[josh-fell]

Not to my mind. Good tasks are idempotent, so deleting a non-existent topic has achieved the goal: to ensure that the topic does not exist.

Fair point.

[bharanidharan14]

yes the existing flow will through an error if the topic name exists, Now I have add try catch so before creating the topic just checking if the topic exists, if it exists logging it with message and returning the topic name, similarly for delete operator

I wonder if there should be a parameter for users to have the option of overwriting a topic that might exist. WDYT?

Just thinking, overwriting the existing topic is kind of update operation right ?

[josh-fell]

Just thinking, overwriting the existing topic is kind of update operation right ?

Sure, depending on how Azure Service Bus management operates and/or if someone wanted to patch a particular attribute (if that's even possible). What if you wanted to completely rebuild the topic? Yes, you could check to see if it exists, delete it, and then rebuild it, but I was thinking some analogous feature to file operations.

I'm not saying it should or even can be done. I haven't been used ASB in a long time. Just tossing spaghetti to see if it sticks. You can tell me to go back to my corner and it could be something for the future (or never) maybe.

[bharanidharan14]

Just thinking, overwriting the existing topic is kind of update operation right ?

Sure, depending on how Azure Service Bus management operates and/or if someone wanted to patch a particular attribute (if that's even possible). What if you wanted to completely rebuild the topic? Yes, you could check to see if it exists, delete it, and then rebuild it, but I was thinking some analogous feature to file operations.

I'm not saying it should or even can be done. I haven't been used ASB in a long time. Just tossing spaghetti to see if it sticks. You can tell me to go back to my corner and it could be something for the future (or never) maybe.

Yes, there is an Option to patch/update the Topic, I was planning to add update topic operator as next future operator. on that the user can update whatever attribute they want(which is allowed to update).

Even I need suggestion this create topic operator should be.

1. Only create and if it exists just return it ?
2. Create the topic, if it exists delete it and create with new properties?

[bharanidharan14]

Just thinking, overwriting the existing topic is kind of update operation right ?

Sure, depending on how Azure Service Bus management operates and/or if someone wanted to patch a particular attribute (if that's even possible). What if you wanted to completely rebuild the topic? Yes, you could check to see if it exists, delete it, and then rebuild it, but I was thinking some analogous feature to file operations.
I'm not saying it should or even can be done. I haven't been used ASB in a long time. Just tossing spaghetti to see if it sticks. You can tell me to go back to my corner and it could be something for the future (or never) maybe.

Yes, there is an Option to patch/update the Topic, I was planning to add update topic operator as next future operator. on that the user can update whatever attribute they want(which is allowed to update).

Even I need suggestion this create topic operator should be.

1. Only create and if it exists just return it ?
2. Create the topic, if it exists delete it and create with new properties?

@josh-fell

[josh-fell]

Just thinking, overwriting the existing topic is kind of update operation right ?

Sure, depending on how Azure Service Bus management operates and/or if someone wanted to patch a particular attribute (if that's even possible). What if you wanted to completely rebuild the topic? Yes, you could check to see if it exists, delete it, and then rebuild it, but I was thinking some analogous feature to file operations.
I'm not saying it should or even can be done. I haven't been used ASB in a long time. Just tossing spaghetti to see if it sticks. You can tell me to go back to my corner and it could be something for the future (or never) maybe.

Yes, there is an Option to patch/update the Topic, I was planning to add update topic operator as next future operator. on that the user can update whatever attribute they want(which is allowed to update).
Even I need suggestion this create topic operator should be.

1. Only create and if it exists just return it ?
2. Create the topic, if it exists delete it and create with new properties?

@josh-fell

I think option 1 is fine for now. If users find that indeed having option 2 would be helpful, they are more than welcome to log a feature or submit a PR adding the functionality. Thinking about my suggestion, it was a little much and out of scope for the initial creation of these operators.

[bharanidharan14]

@josh-fell @ashb Can you review this PR, I have addressed the review comments