History is lost when using BlobsStorage

[nemesv]

Describe the bug

I'm extending the 04.ai.a.teamsChefBot sample, and I've replaced the MemoryStorage with the BlobsStorage implementation from the botbuilder-azure-blobs package. Which is working as expected, and I can save the converstation history to blob storage most of the cases.

However when the user does not wait enough and types in two messages in quick succession, it throws an error:

[onTurnError] unhandled error: RestError: The condition specified using HTTP conditional header(s) is not met.

I think this is happening because the DefaultTrunStateManger tries to save the two histories using the same ETAG which fails on the blob storage side.

e.g., the starting history is look like this:
ETAG: 1
Human: first message
Bot: response to first message

in teams the user types in two messages, in this case the history is not updated in between and I end up with two histories where non of them containing all the messages, but having the same etag

ETAG: 2
Human: first message
Bot: response to first message
Human: second message
Bot: response to second message

ETAG: 2
Human: first message
Bot: response to first message
Human: third message
Bot: response to third message

I've tried to use the LastWriterWinsStore implementation from the example: 04.q.questBot which resolves this error, however in this case part history is lost because it loses some of the messages, because of the list writer semantic. However in this case because there is only writer it should somehow keep track the whole history and should not lose any of the messages.

Is there any solution this problem? To use BlobsStorage and keep all the history even when the user sends multiple messages quickly?

[corinagum]

Hi @nemesv thank you for the bug report. Could you share some code snippets for the bug? What parts of quest bot in particular is your code based off of? I will reproduce on my side.

Could you also give a little more detail on the ETAGS and how you're seeing 2 of the same tag? I haven't been able to dive into the latest version of quest bot yet.

Could you also share your Planner option settings? (With keys/ids removed)

This sounds like a larger problem to solve rather than a quick fix but we'll start looking into it.

[nemesv]

@corinagum thank you for looking into this, I'm afraid that this might be a larger problem to fix.

I've forked the pushed the repro here: https://github.com/nemesv/teams-ai/tree/history_repro
I've only modified the 04.ai.a.teamsChefBot. To run the code you need to setup and configure the blob storage in the .env.local.user file

SECRET_STORAGE_CONNECTION_STRING=
STORAGE_CONTAINER=

This is how it looks in the teams when using the BlobStorage:

I've logged the messages before sending to the blobstorage:

{
  "msteams/28:09446dad-bb7b-42f4-ab24-97f9c0410280/conversations/a:1hz5yXATMKzoEyZ675s29Fi2vxzW9T8yWUZKeZ4niHa3e9DU6nrkseqekztS_t2snH8o_HutlmU4esr9cmHlcickwADl2rV8MvJn6jTHsvmu7IGwNIc7AH26qU_B4Y1et": {
    "__history__": [
      "User: 1",
      "Assistant: Hello there! Welcome to Teams Chef, your expert guide in Microsoft Teams app development. May I know your name, please?",
      "User: test1",
      "Assistant: Great to meet you, test1! As a junior developer learning Microsoft Teams development for the first time, I'm here to help you build new apps for Microsoft Teams. Are you ready to get started?"
    ],
    "eTag": "\"0x8DB6BD2FE0EBA2B\""
  }
}

{
  "msteams/28:09446dad-bb7b-42f4-ab24-97f9c0410280/conversations/a:1hz5yXATMKzoEyZ675s29Fi2vxzW9T8yWUZKeZ4niHa3e9DU6nrkseqekztS_t2snH8o_HutlmU4esr9cmHlcickwADl2rV8MvJn6jTHsvmu7IGwNIc7AH26qU_B4Y1et": {
    "__history__": [
      "User: 1",
      "Assistant: Hello there! Welcome to Teams Chef, your expert guide in Microsoft Teams app development. May I know your name, please?",
      "User: test2",
      "Assistant: Great to meet you, test2! As a junior developer learning Microsoft Teams development for the first time, I'm here to guide you in your journey to build new apps for Microsoft Teams. Are you ready to get started?"
    ],
    "eTag": "\"0x8DB6BD2FE0EBA2B\""
  }
}

 [onTurnError] unhandled error: RestError: The condition specified using HTTP conditional header(s) is not met.
RequestId:61900150-f01e-0096-06bd-9d94e7000000
Time:2023-06-13T06:09:05.5851489Z

If I replace it with the LastWriterWinsStore there is no more RestError but the history does not contain both messages, only the second one

{
  "msteams/28:09446dad-bb7b-42f4-ab24-97f9c0410280/conversations/a:1hz5yXATMKzoEyZ675s29Fi2vxzW9T8yWUZKeZ4niHa3e9DU6nrkseqekztS_t2snH8o_HutlmU4esr9cmHlcickwADl2rV8MvJn6jTHsvmu7IGwNIc7AH26qU_B4Y1et": {
    "__history__": [
      "User: test",
      "Assistant: Hello there! My name is Teams Chef. What's your name? And how can I assist you in your journey to build new apps for Microsoft Teams?",
      "User: first",
      "Assistant: Nice to meet you, First! As a junior developer learning Microsoft Teams development for the first time, I'm here to guide you in your journey to build new apps for Teams. Do you have any specific questions or areas you'd like to focus on?"      
    ]
  }
}
{
  "msteams/28:09446dad-bb7b-42f4-ab24-97f9c0410280/conversations/a:1hz5yXATMKzoEyZ675s29Fi2vxzW9T8yWUZKeZ4niHa3e9DU6nrkseqekztS_t2snH8o_HutlmU4esr9cmHlcickwADl2rV8MvJn6jTHsvmu7IGwNIc7AH26qU_B4Y1et": {
    "__history__": [
      "User: test",
      "Assistant: Hello there! My name is Teams Chef. What's your name? And how can I assist you in your journey to build new apps for Microsoft Teams?",
      "User: second",
      "Assistant: Nice to meet you, Second! As a junior developer learning Microsoft Teams development for the first time, I'm here to guide you in your journey to build new apps for Teams. Do you have any specific questions or areas you'd like to focus on?"     
    ]
  }
}

Let me know if you need any further info to reproduce this issue!

[corinagum]

AH I see! Thanks so much for the detailed repro steps -- I will be looking into this today and report back asap.

[corinagum]

@nemesv Thanks for your patience in me getting back to you. I was looking into possible solutions for this problem, particularly around comparing etags for history, but input from Steve was different feedback.

from @Stevenic:

the last writer wins is typically the best approach. We normally tell developers to treat the bot's state memory as cookie storage and don’t use it to store anything transactional. If you need transactions you’ll need to write directly to a database.

The idea I was looking into (as a custom solution to deal with the rest error) would be to put a delay on writing to blob storage in case two of the same etag come in, resolving the discrepancy, then writing to storage.

---

I hope you don't mind that I've moved this to Discussions, in case other developers come across the same issue. This is a wontfix within the repo, but valuable knowledge nonetheless.

[corinagum]

@nemesv in case you might delete your fork branch, it might be helpful to add code blocks to this discussion with the changes you added as well!