🤖 AskUI Vision Agent

⚡ Automate computer tasks in Python ⚡

Join the AskUI Discord.

🔧 Setup

1. Install AskUI Agent OS

Agent OS is a device controller that allows agents to take screenshots, move the mouse, click, and type on the keyboard across any operating system.

Windows

AMD64

AskUI Installer for AMD64

ARM64

AskUI Installer for ARM64

Linux

⚠️ Warning: Agent OS currently does not work on Wayland. Switch to XOrg to use it.

AMD64

curl -o /tmp/AskUI-Suite-Latest-User-Installer-Linux-x64-Full.run https://files.askui.com/releases/Installer/Latest/AskUI-Suite-Latest-User-Installer-Linux-x64-Full.run

bash /tmp/AskUI-Suite-Latest-User-Installer-Linux-x64-Full.run

ARM64

curl -o /tmp/AskUI-Suite-Latest-User-Installer-Linux-ARM64-Full.run https://files.askui.com/releases/Installer/Latest/AskUI-Suite-Latest-User-Installer-Linux-ARM64-Full.run

bash /tmp/AskUI-Suite-Latest-User-Installer-Linux-ARM64-Full.run

MacOS

curl -o /tmp/AskUI-Suite-Latest-User-Installer-MacOS-ARM64-Full.run https://files.askui.com/releases/Installer/Latest/AskUI-Suite-Latest-User-Installer-MacOS-ARM64-Full.run

bash /tmp/AskUI-Suite-Latest-User-Installer-MacOS-ARM64-Full.run

2. Install vision-agent in your Python environment

pip install askui

Note: Requires Python version >=3.10.

3a. Authenticate with an Automation Model Provider

	AskUI INFO	Anthropic INFO
ENV Variables	ASKUI_WORKSPACE_ID, ASKUI_TOKEN	ANTHROPIC_API_KEY
Supported Commands	click()	click(), get(), act()
Description	Faster Inference, European Server, Enterprise Ready	Supports complex actions

To get started, set the environment variables required to authenticate with your chosen model provider.

How to set an environment variable?

Linux & MacOS

Use export to set an evironment variable:

export ANTHROPIC_API_KEY=<your-api-key-here>

Windows PowerShell

Set an environment variable with $env:

$env:ANTHROPIC_API_KEY="<your-api-key-here>"

3b. Test with 🤗 Hugging Face Model Demos

You can test the Vision Agent with Hugging Face models via their Spaces API. Please note that the API is rate-limited so for production use cases, it is recommended to choose step 3a.

Note: Hugging Face Spaces host model demos provided by individuals not associated with Hugging Face or AskUI. Don't use these models on screens with sensible information.

Supported Models:

• AskUI/PTA-1
• OS-Copilot/OS-Atlas-Base-7B
• showlab/ShowUI-2B
• Qwen/Qwen2-VL-2B-Instruct
• Qwen/Qwen2-VL-7B-Instruct

Example Code:

agent.click("search field", model_name="OS-Copilot/OS-Atlas-Base-7B")

▶️ Start Building

from askui import VisionAgent

# Initialize your agent context manager
with VisionAgent() as agent:
    # Use the webbrowser tool to start browsing
    agent.tools.webbrowser.open_new("http://www.google.com")

    # Start to automate individual steps
    agent.click("url bar")
    agent.type("http://www.google.com")
    agent.keyboard("enter")

    # Extract information from the screen
    datetime = agent.get("What is the datetime at the top of the screen?")
    print(datetime)

    # Or let the agent work on its own
    agent.act("search for a flight from Berlin to Paris in January")

🎛️ Model Selection

Instead of relying on the default model for the entire automation script, you can specify a model for each click command using the model_name parameter.

	AskUI	Anthropic
click()	askui-combo, askui-pta, askui-ocr	anthropic-claude-3-5-sonnet-20241022

Example: agent.click("Preview", model_name="askui-combo")

🛠️ Direct Tool Use

Under the hood agents are using a set of tools. You can also directly access these tools.

Agent OS

The controller for the operating system.

agent.tools.os.click("left", 2) # clicking
agent.tools.os.mouse(100, 100) # mouse movement
agent.tools.os.keyboard_tap("v", modifier_keys=["control"]) # Paste
# and many more

Web browser

The webbrowser tool powered by webbrowser allows you to directly access webbrowsers in your environment.

agent.tools.webbrowser.open_new("http://www.google.com")
# also check out open and open_new_tab

Clipboard

The clipboard tool powered by pyperclip allows you to interact with the clipboard.

agent.tools.clipboard.copy("...")
result = agent.tools.clipboard.paste()

📜 Logging & Reporting

You want a better understanding of what you agent is doing? Set the log_level to DEBUG. You can also generate a report of the automation run by setting enable_report to True.

import logging

with VisionAgent(log_level=logging.DEBUG, enable_report=True) as agent:
    agent...

🖥️ Multi-Monitor Support

You have multiple monitors? Choose which one to automate by setting display to 1 or 2.

with VisionAgent(display=1) as agent:
    agent...

What is AskUI Vision Agent?

AskUI Vision Agent is a versatile AI powered framework that enables you to automate computer tasks in Python.

It connects Agent OS with powerful computer use models like Anthropic's Claude Sonnet 3.5 v2 and the AskUI Prompt-to-Action series. It is your entry point for building complex automation scenarios with detailed instructions or let the agent explore new challenges on its own.

Agent OS is a custom-built OS controller designed to enhance your automation experience.

It offers powerful features like

• multi-screen support,
• support for all major operating systems (incl. Windows, MacOS and Linux),
• process visualizations,
• real Unicode character typing

and more exciting features like application selection, in background automation and video streaming are to be released soon.