Quickstart Playground: Add mitigation for scenarios where the model can exhaust the token limit when creating the Dockerfile #234
Conversation
…g install commands in Dockerfile
| RUN apt-get update && apt-get upgrade -y |
There was a problem hiding this comment.
The git installation here is likely unnecessary, so I removed it.
| @@ -3,5 +3,5 @@ FROM mcr.microsoft.com/devcontainers/base:ubuntu | |||
| ENV DEBIAN_FRONTEND noninteractive | |||
|
|
|||
| RUN apt update && \ | |||
| apt install -y curl && build-essentials \ | |||
| apt install -y curl && build-essential \ | |||
There was a problem hiding this comment.
I noticed this problem in the Rust sample (there's no build-essentials package), so I fixed that while I was in here.
| "readme": "\n\n=== ./README.md ===\n# C# Hello World App\n\nThis repository contains a C# hello world app.\n\n## Setup\n\nRun `dotnet restore`\n\n## Running the App\n\nRun `dotnet run`", | ||
| "code": "\n\n=== ./HelloWorld.csproj ===\n<Project Sdk=\"Microsoft.NET.Sdk\">\n <PropertyGroup>\n <OutputType>Exe</OutputType>\n <TargetFramework>net8.0</TargetFramework>\n </PropertyGroup>\n</Project>\n\n=== ./Program.cs ===\nusing System;\n\nnamespace HelloWorld\n{\n class Program\n {\n static void Main(string[] args)\n {\n Console.WriteLine(\"Hello World!\");\n Console.Write(\"Input: \");\n string data = Console.ReadLine();\n Console.WriteLine(\"Output: \" + data);\n }\n }\n}\n", | ||
| "code": "\n\n=== ./HelloWorld.csproj ===\n<Project Sdk=\"Microsoft.NET.Sdk\">\n <PropertyGroup>\n <OutputType>Exe</OutputType>\n <TargetFramework>net8.0</TargetFramework>\n </PropertyGroup>\n</Project>\n\n=== ./Program.cs ===\nnamespace HelloWorld;\n\npublic class Program\n{\n public static void Main(string[] args)\n {\n Console.WriteLine(\"Hello World!\");\n Console.Write(\"Input: \");\n var data = Console.ReadLine();\n Console.WriteLine(\"Output: \" + data);\n }\n}\n", |
There was a problem hiding this comment.
The embeddings file was missing some changes to the code files (mostly warnings that Visual Studio's static analysis had caught at one point before //BUILD), so I reincorporated those here.
| -0.037260186, | ||
| -0.014457619, | ||
| -0.01217319 | ||
| -0.027098652, |
There was a problem hiding this comment.
Noting here that the embeddings vector changed in spite of the prompt remaining the same. The API can return slight variations (e.g., if OpenAI adds some fine-tuning or other changes to the underlying model); this doesn't materially change the overall semantic meaning of the embedding, and I verified that the 'before' and 'after' vectors have a miniscule distance (on the order of thousandths).
(Incidentally, this comment highlights that the embeddings, strictly speaking, did not need to be revised as part of this change- I mistakenly concluded it was necessary due to the editing of the setup commands; but those are not included in the embeddings vector. Given that this part of the change doesn't impact behavior and was part of what I validated, I'll plan on leaving this as-is.)
| -0.01217319 | ||
| -0.027098652, | ||
| -0.014112577, | ||
| 0.065670945, |
There was a problem hiding this comment.
Theoretically, this same change should be made to the Azure OpenAI embeddings file. For now, I'm going to file a backlog issue for getting that back in sync with the OpenAI version.
There was a problem hiding this comment.
I added this as a task under #216
adrastogi
requested review from
manodasanW,
craigloewen-msft,
krschau,
EricJohnson327 and
jasmilimsft
Summary of the pull request
This change adds instructions and sample tweaks to help ensure that the model does not get stuck in a loop when generating Dockerfiles. This problem would appear to the end user as an error indicating that the token limit has been reached when generating the code files for the project.
References and relevant issues
#223
Detailed description of the pull request / Additional comments
We've identified a handful of scenarios (mostly in C# / dotnet) where the model can get stuck repeatedly printing out the same set of packages to install in the Dockerfile. It's not clear what exactly is causing this problem, but when it occurs, the model will continue until the token limit has been reached. The user receives back an error at this point instead of a project.
To mitigate this problem, we add more explicit instructions for this case, and also adjust the sample Dockerfile that's provided for C# projects (specifically to rearrange the package installation commands). Taken together, these changes seem to help the model make forward progress on the faulting scenarios. Because it's nondeterministic, I am not sure if we can completely eliminate the problem. So for now, I'd like to proceed with this set of changes and see if this improves success rate. In the longer term, I think we will likely need a solution that imposes more structure on the output.
Validation steps performed
I tested this change on a few different C# projects that were known to repro the problem. After applying the changes in this PR, I was able to generate the projects successfully. I also tested a Rust prompt which had reports of this behavior, as well as some basic regression testing across other scenarios (e.g., Python).
PR checklist