ITB: The Interactive Text Browser
ITB aims to create a set of Python scripts to enable a text-based AI agent to interact with web pages. The system is designed to start a detached browser process and interact with it through subsequent scripts. Each script performs a specific action, such as starting the browser, taking a virtual screenshot, scrolling, clicking, and inputting text. The output of these scripts is optimized to be evaluated by a Large Language Model (LLM). So it can understand and interact with the webpage in a text-only manner.
The Scripts
itb_start
Starts a detached browser session and outputs the necessary information to interact with the browser process.
Usage
itb_start <url>
Output
- Process ID (PID) of the browser session.
- WebDriver session ID.
Implementation Strategies
Use Selenium to start a browser session. Ensure the browser runs in a detached process to allow other scripts to interact with it. Output the PID and WebDriver session ID for subsequent scripts to use.
itb_screenshot
Starts a virtual screenshot of the current browser view and outputs all visible information in text form.
Usage
itb_screenshot <session_id>
Output
- Scroll offset, visible range and limits.
- Text representation of all currently visible elements.
- IDs, size and offset for interactive elements (e.g., buttons, links, input fields).
Implementation Strategies
Get the identifiers for all visible elements. Identify elements with text content and interaction capabilities. Extract text content without markup. Create simplified XML structure.
itb_scroll
Scrolls the browser window to a specified position.
Usage
itb_scroll <session_id> <x> <y>
itb_click
Clicks on a specified viewport coordinates
Usage
itb_click <session_id> -x <x> -y <y> [--right] [--double]
Implementation Strategies
Use a random short time to slowly move the cursor to the target position. Click on a random position within the target element.
itb_input
Inputs text or keystrokes into the focussed element.
Usage
itb_input <session_id> -t <text>
itb_input <session_id> -k <keystroke> ...
Implementation Strategies
Use a random short delay between adding each character of text, to simulate human-like typing. Keystrokes are additive. e.g. for CTRL SHIFT a, this emulates the user pressing CTRL, keeping it pressed, then pressing SHIFT, keeping both CTRL and SHIFT pressed, then pressing 'a', keeping all three pressed, then releasing all three.
itb_links
Extracts all links from the full webpage.
Usage
itb_links <session_id>
Output
- List of all visible links with their text, URLs and position on the page.
itb_forms
Extracts all visible forms and their input fields from the current browser view.
Usage
itb_forms <session_id>
Output
List of all visible forms with their
- input fields
- connected labels
- identifier
- position on the page.
itb_navigate
Navigates to a specified URL within the existing browser session.
Usage
itb_navigate <session_id> <url>
itb_refresh
Refreshes the current browser page.
Usage
itb_refresh <session_id>
Implementation Strategies
Wait till the page is fully loaded and the DOM is stable.