Investigate why some shares did not go to failed state, but remained stuck or in-progress

[anushka-singh]

Describe the bug

Investigate why the shares stayed in progress even if they actually must have failed because of missing permissions or any other reason.

How to Reproduce

I am able to replicate this in local dev env. 
I have created an error in grant_role_bucket_policy by doing this: y = 1/0 print("yyyyyyy", y)
Then, I caused another error in handle_share_failure alarm by doing this: x = 1/0 print(x)
At this point, the share remains stuck or SHARE_IN_PROGRESS.

Expected behavior

No response

Your project

No response

Screenshots

No response

OS

Mac

Python version

3.8

AWS data.all version

2.2

Additional context

There's a bug when you share 2 things.. but 1 fails and 1 succeeds.. then you end up with 1 being in_progress forever
Ideally it should mark the success one as share_succeeded and failed one as share_failed and overall status as Processed

But instead its doing :
mark the success one as share_succeeded and failed one as share_in_progress or share_approved and overall status as in progress

[anushka-singh]

The "in-progress" problem exists in 2 places. Share approval and revoke approval too.

I have found a resolution for the Revoke in Progress Bug.
Looking into share in progress bug now.

[anushka-singh]

Error in actual prod:

Sharing task failed due to: An error occurred (InvalidParameter) when calling the Publish operation: Invalid parameter: Subject

[anushka-singh]

Figured out the error and solution for Share In Progress.

The error is occurring in the publish method of the sns client, and it is related to the "Subject" parameter being considered invalid. The maximum length for the subject is 100 characters for AWS SNS.

I checked a few failed shares and their Subject exceeded length of 100 chars.
We were constructing Subject like this:

subject = ( f'ALARM: DATAALL S3 Bucket {bucket.S3BucketName} {alarm_type} Failure Notification' )

Example share:

• The Subject became: ALARM: DATAALL S3 Bucket xx-xxxx-xxxxx-xxxxxxxx-xxxxxxxxxxxx-xxxxx Sharing Failure Notification which is 106 chars.

When it exceeded the 100 char limit, the alarm creation failed share remained in STUCK state.

[anushka-singh]

Resolution:

I have changed the Subject to be constructed like this:
subject = f'ALARM: DATAALL S3 Bucket {bucket.S3BucketName} Sharing Failure'[:100]

This will ensure any unnecessary words from Subject are removed and that we always guarantee a 100 character limit. The message body is further created properly and with a lot of details in case the Subject line gets truncated.
I will repeat this pattern for revoke shares too and for folder and table shares

[dlpzx]

The discussion continues in the mentioned PR. Thanks @anushka-singh for the investigation and fix!!