AI Returns text as the language - Cell Won't Be Executable In RunMe

[jlewi]

I'm seeing examples where the AI returns a code cell with language text.
RunMe won't know how to execute this.

This is related to #114. That issue surfaced an issue with nested blocks. It looks like there are also problems with the AI returning text. We should be able to verify this by looking at the Agent trace.

In my case one of the generated block ids where I observed this was 01HYZWP2288PJY0CV53WAK6632

[jlewi]

Can we enforce sometype of guardrail in the code?

[jlewi]

Actually I'm not sure where the bug is.

Here's the logs from Foyle

2024-05-28T14:04:45.469-0700	INFO	agent/agent.go:112	Agent.Generate returning response	{"traceId": "f70cb775a59113d02f1288e50d8d9b3e", "evalMode": false, "response": {"blocks": [{"kind": "MARKUP", "language": "", "contents": "To accomplish the task of verifying learning with Foyle, you have gone through the process of logging and checking logs. The last step is to try training Foyle. Here's the command to train Foyle:", "outputs": [], "trace_ids": [], "id": "01HZ0GA3WXJD028PNFW961ERXK"}, {"kind": "CODE", "language": "bash", "contents": "foyle train", "outputs": [], "trace_ids": [], "id": "01HZ0GA3WXJD028PNFWAXFQR4V"}, {"kind": "MARKUP", "language": "", "contents": "Please execute the command above to train Foyle and verify the learning process.", "outputs": [], "trace_ids": [], "id": "01HZ0GA3WXJD028PNFWDPGSD1Q"}]}}

We can clearly see the Agent.Generate has a block with the language set to bash.

{"kind": "CODE", "language": "bash", "contents": "foyle train", "outputs": [], "trace_ids": [], "id": "01HZ0GA3WXJD028PNFWAXFQR4V"}

However when I look at the Raw markdown that gets saved in RunMe I see

```text {"id":"01HZ0GA3WXJD028PNFWAXFQR4V"}
foyle train
```

So the language got changed somewhere.
It looks like the proxy and conversion server should be passing along the language field here

foyle/app/pkg/runme/converters/blocks_to_cells.go

Line 40 in 225d30f

LanguageId: block.Language,

[jlewi]

I added some logging in commit 51341f4

Here are the responses

2024-05-28T14:19:42.462-0700	INFO	agent/agent.go:105	Agent.Generate returning blocks	{"traceId": "478a5e525dfea695344a7bea88b37eda", "evalMode": false, "blockIds": ["01HZ0H5FVY07KGKVWKHH2T20BJ", "01HZ0H5FVY07KGKVWKHMYYV2VX", "01HZ0H5FVY07KGKVWKHPP53DVX", "01HZ0H5FVY07KGKVWKHPWNA8G4", "01HZ0H5FVY07KGKVWKHQZG4ZTD", "01HZ0H5FVY07KGKVWKHRGYDK8M"]}
2024-05-28T14:19:42.462-0700	INFO	agent/agent.go:112	Agent.Generate returning response	{"traceId": "478a5e525dfea695344a7bea88b37eda", "evalMode": false, "response": {"blocks": [{"kind": "MARKUP", "language": "", "contents": "It looks like you have been trying to verify learning with Foyle by fetching logs and checking RunMe configurations. Let's analyze the situation based on the output you provided:", "outputs": [], "trace_ids": [], "id": "01HZ0H5FVY07KGKVWKHH2T20BJ"}, {"kind": "MARKUP", "language": "", "contents": "1. The initial attempt to fetch logs for the image `vscode-ext` did not yield any results. This could be due to RunMe not recording the necessary logs.\n2. You checked if RunMe is running with the logs flag, and it seems that RunMe was not initially running with the `ai-logs=true` option.\n3. After bumping the package extension version and reinstalling it, you confirmed that RunMe is now running with `ai-logs=true`.\n4. You retried fetching the logs for the `vscode-ext` image and then checked the logs directory for any recent logs.\n5. You found a log associated with the ID `01HYZXS2Q5XYX7P3PT1KH5Q881`, which appears to be related to a terminal session configuration.", "outputs": [], "trace_ids": [], "id": "01HZ0H5FVY07KGKVWKHMYYV2VX"}, {"kind": "MARKUP", "language": "", "contents": "Given the information above, it seems that the system is now configured correctly to enable logging for the RunMe extension. Since you have the necessary logs and RunMe is running with the correct configuration, you can proceed with training Foyle.", "outputs": [], "trace_ids": [], "id": "01HZ0H5FVY07KGKVWKHPP53DVX"}, {"kind": "MARKUP", "language": "", "contents": "To train Foyle, you can follow the next steps:", "outputs": [], "trace_ids": [], "id": "01HZ0H5FVY07KGKVWKHPWNA8G4"}, {"kind": "CODE", "language": "bash", "contents": "# Execute the command to train Foyle\ntrain_foyle_command_here", "outputs": [], "trace_ids": [], "id": "01HZ0H5FVY07KGKVWKHQZG4ZTD"}, {"kind": "MARKUP", "language": "", "contents": "Please replace `train_foyle_command_here` with the specific command or steps needed to train Foyle in your environment based on your project's requirements. Let me know if you need any further assistance with this process!", "outputs": [], "trace_ids": [], "id": "01HZ0H5FVY07KGKVWKHRGYDK8M"}]}}
2024-05-28T14:19:42.462-0700	INFO	runme/proxy.go:67	Runme.Generate completed	{"traceId": "478a5e525dfea695344a7bea88b37eda", "response": "cells:{kind:CELL_KIND_MARKUP  value:\"It looks like you have been trying to verify learning with Foyle by fetching logs and checking RunMe configurations. Let's analyze the situation based on the output you provided:\"  metadata:{key:\"id\"  value:\"01HZ0H5FVY07KGKVWKHH2T20BJ\"}}  cells:{kind:CELL_KIND_MARKUP  value:\"1. The initial attempt to fetch logs for the image `vscode-ext` did not yield any results. This could be due to RunMe not recording the necessary logs.\\n2. You checked if RunMe is running with the logs flag, and it seems that RunMe was not initially running with the `ai-logs=true` option.\\n3. After bumping the package extension version and reinstalling it, you confirmed that RunMe is now running with `ai-logs=true`.\\n4. You retried fetching the logs for the `vscode-ext` image and then checked the logs directory for any recent logs.\\n5. You found a log associated with the ID `01HYZXS2Q5XYX7P3PT1KH5Q881`, which appears to be related to a terminal session configuration.\"  metadata:{key:\"id\"  value:\"01HZ0H5FVY07KGKVWKHMYYV2VX\"}}  cells:{kind:CELL_KIND_MARKUP  value:\"Given the information above, it seems that the system is now configured correctly to enable logging for the RunMe extension. Since you have the necessary logs and RunMe is running with the correct configuration, you can proceed with training Foyle.\"  metadata:{key:\"id\"  value:\"01HZ0H5FVY07KGKVWKHPP53DVX\"}}  cells:{kind:CELL_KIND_MARKUP  value:\"To train Foyle, you can follow the next steps:\"  metadata:{key:\"id\"  value:\"01HZ0H5FVY07KGKVWKHPWNA8G4\"}}  cells:{kind:CELL_KIND_CODE  value:\"# Execute the command to train Foyle\\ntrain_foyle_command_here\"  language_id:\"bash\"  metadata:{key:\"id\"  value:\"01HZ0H5FVY07KGKVWKHQZG4ZTD\"}}  cells:{kind:CELL_KIND_MARKUP  value:\"Please replace `train_foyle_command_here` with the specific command or steps needed to train Foyle in your environment based on your project's requirements. Let me know if you need any further assistance with this process!\"  metadata:{key:\"id\"  value:\"01HZ0H5FVY07KGKVWKHRGYDK8M\"}}"}

So it looks like language is set to bash

cells:{kind:CELL_KIND_CODE  value:\"# Execute the command to train Foyle\\ntrain_foyle_command_here\"  language_id:\"bash\"  metadata:{key:\"id\"  value:\"01HZ0H5FVY07KGKVWKHQZG4ZTD\"}} 

But in the markdown this ends up being

```text {"id":"01HZ0H5FVY07KGKVWKHQZG4ZTD"}
# Execute the command to train Foyle
train_foyle_command_here
```

[jlewi]

I think the bug is here
https://github.com/stateful/vscode-runme/blob/4420562e04d3c8eb2b0d4ef8e77d92015359c7ab/src/extension/ai/generate.ts#L92

It doesn't look like we are setting languageId when inserting the cell.