Poison blob queue message in multi storage account scenario

[Fabian-Schmidt]

Hi,
I have setup two storage accounts in the connectionStrings. First one is setup as AzureWebJobsStorage and a second as AzureWebJobsMyToken.

In the second account a blob trigger is configured via an attribute on the class [StorageAccount("MyToken")].
In case this blob trigger fails multiple times a poison blob message is generated in the queue webjobs-blobtrigger-poison of the first storage account.
My expectation was that the same storage account, as used for the trigger, is used for the poison message.

Thanks,
Fabian

[mathewc]

Thanks for reporting. What version of the SDK are you using?

[Fabian-Schmidt]

Microsoft.Azure.WebJobs Version 1.1.2.0
Microsoft.Azure.WebJobs.Extensions Version 1.0.1.0
Microsoft.Azure.WebJobs.Host Version 1.1.2.0

[mamaso]

@christopheranderson @mathewc Unfortunately this scenario and #797 are basically incompatible (if we add the poison queue to the secondary storage account it cannot be a blob-only storage account).

Which issue do you guys want to prioritize?

I think that supporting blob-only storage unlocks more options. We could do a more complex fix where we put the poison queue on the secondary storage account unless if it is blob-only, but I'm not sure it's worth the complexity & inconsistent behavior.

[lindydonna]

@mamaso We were thinking of excluding blob only storage accounts for RC, since it requires perf testing as well. Does that simplify the decision?

[mathewc]

@mamaso My thought was that the queue we use for poison blobs would live in the primary storage account. That's the same place the trigger queue (our internal implementation detail) we create for blob triggers lives. We scan for blobs and add items to the trigger queue, which the shared blob listener is monitoring to trigger functions. Can't the poison queue live in the same place?

[mamaso]

Yep, I think it's fine for the primary storage account to have all the queues. Perhaps the answer is to document the poison queue behavior better since this didn't match the user's expectation of where it would be created.

[mathewc]

What is our behavior for poison queues for QueueTrigger functions in the context of secondary storage accounts? We want to ensure that we have a consistent story.

[mamaso]

Just tested, QueueTrigger will create the poison queue in the secondary account, so there's some inconsistency here.

[mathewc]

Yeah, that's what I thought. We'll need to drive towards a consistent model for all of these.

[mathewc]

The start of a fix is in a branch I just pushed blobfix. I will resume that work when I have time - the fix needs more improvements.

[mathewc]

Fixed in 56b5302. By default the poison queue will no be colocated in the account where the blob container lives. This is configurable via JobJostConfiguration.Blobs.CentralizedPoisonQueue. Preview build with these changes can be gotten on our myget feed.