diff --git a/tools/itb/README.md b/tools/itb/README.md index 394cb69..a399c60 100644 --- a/tools/itb/README.md +++ b/tools/itb/README.md @@ -1,151 +1,782 @@ # ITB: The Interactive Text Browser -ITB aims to create a set of Python scripts to enable a text-based AI agent to interact with web pages. -The system is designed to start a detached browser process and interact with it through subsequent scripts. -Each script performs a specific action, such as starting the browser, taking a virtual screenshot, scrolling, clicking, and inputting text. -The output of these scripts is optimized to be evaluated by a Large Language Model (LLM). -So it can understand and interact with the webpage in a text-only manner. +## Understanding the Challenge -# The Scripts +Modern websites are designed for human interaction. They rely on visual cues, spatial relationships, and complex interactions through mouse and keyboard. A human can instantly understand that a blue underlined piece of text is clickable, that a grid of product cards represents a catalog, or that a hamburger icon will reveal a navigation menu. -## itb_start +AI agents, however, process information differently. They need structured data that explicitly describes both content and interaction possibilities. Traditional web scraping tools can extract content but struggle with interaction. Screen readers come closer but are optimized for linear reading rather than complex interaction patterns. -Starts a detached browser session and outputs the necessary information to interact with the browser process. +ITB (Interactive Text Browser) bridges this gap. It transforms web pages into a format that AI agents can process effectively while preserving all the interaction capabilities a human would have. This transformation focuses on what matters for interaction - the content that's visible and the ways it can be manipulated - while removing technical complexity that only exists to support visual presentation. -### Usage +## How ITB Works -``` -itb_start +ITB operates as a bridge between an AI agent and a web browser. When you start ITB, it launches a browser process that runs independently, just like when you open a browser yourself. This browser loads and renders web pages normally, but instead of displaying them on screen, it communicates their content and state to the AI agent through a simplified XML format. + +Most of ITB's work happens through JavaScript that runs directly in the browser. Rather than making many small requests back and forth between Python and the browser, ITB bundles operations together into efficient JavaScript operations. This approach significantly improves performance while making the system more reliable. + +Consider a typical login form interaction. Instead of separate commands to find fields, click them, and type text, ITB might execute a single JavaScript operation that handles the entire interaction: + +```javascript +// Single efficient operation instead of multiple commands +function fillLoginForm(username, password) { + const usernameField = document.getElementById('username'); + const passwordField = document.getElementById('password'); + + // Perform all operations in one go + usernameField.focus(); + usernameField.value = username; + passwordField.focus(); + passwordField.value = password; + + return { + username_entered: true, + password_entered: true, + submit_button_enabled: document.querySelector('button[type="submit"]').disabled === false + }; +} ``` -### Output +## Page Representation -- Process ID (PID) of the browser session. -- WebDriver session ID. +ITB transforms complex HTML structures into a simplified XML format that represents what's actually visible and interactive on the page. Let's look at several examples to understand this transformation. -### Implementation Strategies +### Text Content -Use Selenium to start a browser session. -Ensure the browser runs in a detached process to allow other scripts to interact with it. -Output the PID and WebDriver session ID for subsequent scripts to use. +When a human reads a webpage, they naturally process related content together regardless of how it's structured in HTML: -## itb_screenshot +```html + +
+

Breaking News

+ +

Scientists announce breakthrough...

+
-Starts a virtual screenshot of the current browser view and outputs all visible information in text form. - -### Usage - -``` -itb_screenshot + + + Breaking News - 2024-12-20 - By John Smith - Scientists announce breakthrough... + ``` -### Output +The transformation merges related text elements into a single coherent flow, making it easier for AI agents to process content the way a human would read it. -- Scroll offset, visible range and limits. -- Text representation of all currently visible elements. -- IDs, size and offset for interactive elements (e.g., buttons, links, input fields). +### Interactive Elements -### Implementation Strategies +ITB describes elements by their interaction capabilities rather than their HTML structure: -Get the identifiers for all visible elements. -Identify elements with text content and interaction capabilities. -Extract text content without markup. -Create simplified XML structure. +```html + + -## itb_scroll - -Scrolls the browser window to a specified position. - -### Usage - -``` -itb_scroll + + + Username: + + Password: + + Login + ``` -## itb_click +### Media Content -Clicks on a specified viewport coordinates +Media elements like images, videos, and audio are represented simply with their source and alternative text: -### Usage +```html + +
+ System architecture diagram + + +
-``` -itb_click -x -y [--right] [--double] + + + System architecture diagram + ``` -### Implementation Strategies +### Embedded Content -Use a random short time to slowly move the cursor to the target position. -Click on a random position within the target element. +Modern web pages often embed content from other sources using iframes. While humans don't notice these technical boundaries, they affect how we can interact with the content. ITB handles this by representing iframes transparently: -## itb_input +```html + +
+

Welcome Back

+ +
-Inputs text or keystrokes into the focussed element. - -### Usage - -``` -itb_input -t -itb_input -k ... + + + Welcome Back +