feat(sandbox): Add Jupyter Kernel for Interactive Python Interpreter for Sandbox #1215
Conversation
opendevin/sandbox/ssh_box.py
Outdated
| @@ -206,6 +211,28 @@ def execute(self, cmd: str) -> Tuple[int, str]: | |||
| exit_code = int(exit_code.lstrip('echo $?').strip()) | |||
| return exit_code, command_output | |||
|
|
|||
| def setup_jupyter(self): | |||
There was a problem hiding this comment.
Instead of putting this into SSHBox, could we have a JupyterBox that extends SSHBox?
There was a problem hiding this comment.
I am also debating about this: Ideally, the Python execution capability should be parallel to bash execution so that every agent can resort to one of them when needed. In that case, I think it probably makes sense for us to extend the existing SSHBox to have a separate Python execution capability via execute_python as it won't interfere with the existing actions going through the SSHBox and other existing agents can also have python execution natively supported easily without specifying a different SSHJupyterBox?
Or we can just do the sub-class and set the SSHJupyterBox to the default just to make the code look better :) - anyway, I can create a sub-class first, and then we can decide whether we want this Python execution capability to be a default behavior.
| @@ -371,6 +402,12 @@ def close(self): | |||
| ssh_box.kill_background(bg_cmd.id) | |||
| logger.info('Background process killed') | |||
| continue | |||
| if user_input.startswith('py:'): | |||
There was a problem hiding this comment.
Maybe we should have a JupyterAgent or something to demonstrate this functionality?
There was a problem hiding this comment.
For this PR, I'm just trying to get the sandbox component ready :) I'll submit follow-up PRs to update CodeActAgent using the sandbox with instructions on how to use the fine-tuned open-source models (CodeAgentAgent-Mistral-7b - it can run on a laptop!).
|
This is very cool! The demo is neat |
| self.base_url = f'http://{url_suffix}' | ||
| self.base_ws_url = f'ws://{url_suffix}' |
There was a problem hiding this comment.
| self.base_url = f'http://{url_suffix}' | |
| self.base_ws_url = f'ws://{url_suffix}' | |
| self.base_url = f'http://{url_suffix}' | |
| self.base_ws_url = f'ws://{url_suffix}' | |
| self.ws = None |
There was a problem hiding this comment.
My original implement was like this but mypy was not super happy about it: it assume self.ws is None and does not have methods like 'send_messages' etc, hence causing issue for its typing system :(
There was a problem hiding this comment.
Fine, let's keep the original code.
| if not hasattr(self, 'ws') or not self.ws: | ||
| return |
There was a problem hiding this comment.
| if not hasattr(self, 'ws') or not self.ws: | |
| return | |
| if not self.ws: | |
| return |
| ) | ||
|
|
||
| async def _connect(self): | ||
| if hasattr(self, 'ws') and self.ws: |
There was a problem hiding this comment.
| if hasattr(self, 'ws') and self.ws: | |
| if self.ws: |
| RUN pip install jupyterlab notebook jupyter_kernel_gateway | ||
| # Add common data science utils | ||
| RUN pip install transformers[torch] | ||
| RUN pip install torch --index-url https://download.pytorch.org/whl/cpu |
There was a problem hiding this comment.
Do you expect agent would run torch to solve user's problems? An example would be cool.
There was a problem hiding this comment.
Yes! I did try to use CodeActAgent-Mistral 7B and let it write Pytorch code for some simple tasks (e.g., regression), and it did actually solve the problem by 80% - I suspect by switching towards a stronger base model (e.g., 70B), the agent will be able to tackle such problems, so i'd like to keep these as in the container.
alternatively in the future if we are concerned about the size of the container, we can make these optional and/or allow user to use their customized executor image.
There was a problem hiding this comment.
I don't really have a concrete qualitative example at hand using PyTorch, the closest example I have is having the agent/LLM using SKLearn for machine learning tasks: https://twitter.com/xingyaow_/status/1754556862949994917
xingyaoww
changed the title
|
@rbren, I will merge this so that I can get the new sandbox image built for further development -- but feel free to revert if you think something is wrong! ;) |
…for Sandbox (All-Hands-AI#1215) * add initial version of py interpreter * fix bug * fix async issue * remove debugging print statement * initialize kernel & update printing * fix port mapping * uncomment debug lines * fix poetry lock * make jupyter py interpreter into a subclass
…rpreter for Sandbox (All-Hands-AI#1215)" This reverts commit 492feec.
This PR adds Python code execution capability from CodeAct based on IPython to augment the bash command line interface for current agents.
Eventual expected outcome (see video below): Agents can automatically solve user's requests by writing and executing Python code (interactively) and running bash commands (based on our existing sandbox). This interactive Python interpreter supports displaying figures/charts in markdown format via
base64, which we can later integrate into the front-end.codeact-demo.mp4
Demo of the expected capability - work-in-progress.
Right now, it only supports
SSHBox,and I have tested and confirmed it is working inubuntuwith bothRUN_AS_DEVIN='True'andRUN_AS_DEVIN='False'.You can test it locally by first building the image via
./containers/build.sh sandbox(comment out the build architecture line if error), then runpython3 opendevin/sandbox/ssh_box.py. Any command you type that starts withpy:will be forwarded to the Python Interpreter for execution.