# ITB: The Interactive Text Browser

## Understanding the Challenge

Modern websites are designed for human interaction. They rely on visual cues, spatial relationships, and complex interactions through mouse and keyboard. A human can instantly understand that a blue underlined piece of text is clickable, that a grid of product cards represents a catalog, or that a hamburger icon will reveal a navigation menu.

AI agents, however, process information differently. They need structured data that explicitly describes both content and interaction possibilities. Traditional web scraping tools can extract content but struggle with interaction. Screen readers come closer but are optimized for linear reading rather than complex interaction patterns.

ITB (Interactive Text Browser) bridges this gap. It transforms web pages into a format that AI agents can process effectively while preserving all the interaction capabilities a human would have. This transformation focuses on what matters for interaction - the content that's visible and the ways it can be manipulated - while removing technical complexity that only exists to support visual presentation.

## How ITB Works

ITB operates as a bridge between an AI agent and a web browser. When you start ITB, it launches a browser process that runs independently, just like when you open a browser yourself. This browser loads and renders web pages normally, but instead of displaying them on screen, it communicates their content and state to the AI agent through a simplified XML format.

## Page Representation

ITB transforms complex HTML structures into a simplified XML format that represents what's actually visible and interactive on the page. Let's look at several examples to understand this transformation.

### Text Content

When a human reads a webpage, they naturally process related content together regardless of how it's structured in HTML:

```html
<!-- Original HTML -->
<div class="article">
    <h2 class="title">Breaking News</h2>
    <div class="metadata">
        <span class="date">2024-12-20</span>
        <span class="author">By John Smith</span>
    </div>
    <p class="lead">Scientists announce breakthrough...</p>
</div>

<!-- ITB Output -->
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
    <text x="20" y="40" width="1024" height="106">Breaking News - 2024-12-20 - By John Smith - Scientists announce breakthrough...</text>
</viewport>
```

The transformation merges related text elements into a single coherent flow, making it easier for AI agents to process content the way a human would read it.

### Interactive Elements

ITB describes elements by their interaction capabilities rather than their HTML structure:

```html
<!-- Original HTML -->
<form class="login-form">
    <label for="username">Username:</label>
    <input type="text" id="username" class="form-control">
    <label for="password">Password:</label>
    <input type="password" id="password" class="form-control">
    <button type="submit" class="btn">Login</button>
</form>

<!-- ITB Output -->
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
    <text x="0" y="0" width="77" height="20">Username:</text>
    <interactive x="78" y="0" width="213" height="20" accepts_text="true" id="username" x="20" y="70" width="200" height="30"/>
    <text x="292" y="0" width="73" height="20">Password:</text>
    <interactive x="365" y="0" width="212" height="20" accepts_text="true" password="true" id="password" x="20" y="150" width="200" height="30"/>
    <interactive x="578" y="0" width="40" height="20" clickable="true" x="20" y="200" width="80" height="40">Login</interactive>
</viewport>
```

### Media Content

Media elements like images, videos, and audio are represented simply with their source and alternative text:

```html
<!-- Original HTML -->
<div class="media-content">
    <img src="diagram.png" alt="System architecture diagram"/>
    <video src="demo.mp4" controls>Video demonstration of the system</video>
    <audio src="recording.mp3" controls>Conference presentation audio</audio>
</div>

<!-- ITB Output -->
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
    <img src="diagram.png" alt="System architecture diagram" x="20" y="40" width="800" height="400"/>
    <video src="demo.mp4" alt="Video demonstration of the system" x="20" y="460" width="640" height="360"/>
    <audio src="recording.mp3" alt="Conference presentation audio" x="20" y="840" width="400" height="40"/>
</viewport>
```

### Embedded Content

Modern web pages often embed content from other sources using iframes. While humans don't notice these technical boundaries, they affect how we can interact with the content. ITB handles this by representing iframes transparently:

```html
<!-- Original HTML -->
<div class="content">
    <h1>Welcome Back</h1>
    <iframe src="https://auth.service.com/login" width="400" height="300"></iframe>
</div>

<!-- ITB Output -->
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
    <text x="20" y="40" width="203" height="20">Welcome Back</text>
    <iframe src="https://auth.service.com/login" x="20" y="80" width="400" height="300"/>
</viewport>
```

When an AI agent encounters an iframe, it can start a new browser instance to interact with that content directly. This maintains security boundaries while providing full access to embedded content.

### Interaction State

Just as humans need to know where their mouse pointer is and which text field they're typing in, AI agents need clear information about the current interaction state. ITB provides this through viewport attributes:

```xml
<viewport 
    width="1024" height="800" 
    scroll_x="0" scroll_y="0"
    pointer_x="150" pointer_y="45"
    focus_id="username"
    caret_position="5"
    selection_start="2" selection_end="5"
>
    <text x="20" y="30" width="174" height="20">Username:</text>
    <interactive accepts_text="true" id="username" x="100" y="30" width="200" height="30">
        alice
    </interactive>
</viewport>
```

This shows:
- The viewport's dimensions and scroll position
- The mouse pointer's location
- Which element has input focus
- The text cursor position
- Any selected text

### Helping AI Agents Click

Unlike humans who can easily click anywhere within an interactive element, AI agents need precise coordinates. For elements without IDs that could be targeted directly, ITB provides suggested click coordinates:

```xml
<interactive 
    accepts_text="true"
    suggested_click="135,45"
    x="100" y="30" width="200" height="30"
/>
```

## Command Reference

### Starting a Session

```bash
itb_start "https://example.com"
```

Starts a new browser session and returns:
```json
{
    "pid": 12345,
    "session_id": "550e8400-e29b-41d4-a716-446655440000",
    "status": "success"
}
```

### Reading Page State

```bash
itb_screenshot <session_id> [--no_wait_stable] [--timeout 1]
```

Captures the current page state in ITB's XML format. By default waits for the DOM to stop changing before capturing, useful for dynamic content. ITB detects stability by monitoring DOM mutations. Default timeout is 1 second.

### Controlling the Mouse Pointer

```bash
# Move pointer
itb_cursor <session_id> -x 150 -y 45

# Move and click
itb_cursor <session_id> -x 150 -y 45 --click

# Right click
itb_cursor <session_id> --right

# Double click by id
itb_cursor <session_id> --double --id "submit_button"
```

### Text Input

```bash
# Input by element ID
itb_input <session_id> -i "username" -t "alice@example.com"

# Input to focused element
itb_input <session_id> -t "hello world"

# Move text cursor
itb_input <session_id> --caret 5

# Select text
itb_input <session_id> --select 2 5
```

### File Upload

```bash
itb_upload <session_id> -i "profile_picture" -f "/path/to/avatar.jpg"
```

### Scrolling

ITB provides several ways to control scrolling:

```bash
# Absolute page position
itb_scroll <session_id> -x 0 -y 500

# Relative to current position
itb_scroll <session_id> --relative -x 0 -y 100
```

### Navigation

```bash
# Navigate to new URL
itb_navigate <session_id> <url>

# Refresh current page
itb_refresh <session_id>
```

## Debugging

Since most of ITB's logic runs as JavaScript in the browser, debugging can be enabled to provide detailed information about operations:

```bash
# Enable debug mode
itb_screenshot <session_id> --debug

# Output includes operation details:
# [Debug] Starting page analysis
# [Debug] Found 127 elements to process
# [Debug] Processing element 'username': visible at (150, 45)
# ...
```

## Implementation Guidelines

When implementing or extending ITB, keep these principles in mind:

### Efficient JavaScript

Bundle operations together to minimize round trips between Python and the browser. Instead of multiple small operations:

```python
# Inefficient - multiple Selenium calls
element = driver.find_element_by_id("username")
element.click()
element.clear()
element.send_keys("alice")
```

Use single JavaScript operations:

```python
# Efficient - one browser interaction
driver.execute_script("""
    const field = document.getElementById('username');
    field.focus();
    field.value = 'alice';
    return {success: true, value: field.value};
""")
```

### Text Processing

Text content should be merged when it forms a natural reading flow:
- Adjacent text elements should be combined
- Maintain accurate positioning information
- Preserve all text but no formatting

### Interactive Elements

Focus on describing interaction possibilities:
- What type of interaction is supported
- Any constraints or requirements
- Current state (disabled, selected, etc.)
- Position and size for interaction

### Testing

ITB's JavaScript operations can be tested using fixture files:

```python
class TestITBTransformations(unittest.TestCase):
    def test_text_merging(self):
        with open('fixtures/complex_text.html') as f:
            html = f.read()
        with open('fixtures/expected_output.xml') as f:
            expected = f.read()
            
        self.mock_driver.page_source = html
        result = itb_screenshot('dummy-session')
        self.assertEqual(result.to_xml(), expected)
```

## Technical Limitations

ITB intentionally does not attempt to:

1. Preserve HTML structure beyond what's necessary for interaction. The HTML structure exists to support visual presentation and doesn't matter to an AI agent.

2. Detect JavaScript event handlers or custom interaction patterns. These can't be reliably detected by examining the DOM alone.

3. Implement complex media playback controls. Media files should be downloaded and processed separately if needed.

4. Bypass browser security restrictions. Cross-origin iframes are handled by starting new browser sessions rather than trying to circumvent security measures.

These limitations keep the system focused and reliable, ensuring that AI agents can consistently interact with web pages through a simplified but functional interface.

## Example Scenario: Social Media Interaction

This example demonstrates how an AI agent uses ITB to interact with a social media interface. It showcases ITB's key features including text collapsing, state management, and complex interactions. The scenario walks through a complete social media interaction flow, from content creation to user engagement.

### social.html
```html
<!DOCTYPE html>
<html>
<head>
    <title>SocialConnect</title>
    <style>
        /* Core layout styles */
        body { margin: 0; font-family: Arial, sans-serif; background: #f0f2f5; }
        
        /* Fixed navigation bar */
        .navbar {
            position: fixed;
            top: 0;
            width: 100%;
            height: 60px;
            background: white;
            box-shadow: 0 2px 4px rgba(0,0,0,0.1);
            z-index: 100;
            display: flex;
            align-items: center;
            padding: 0 16px;
        }

        /* Main content area */
        .main-content {
            display: flex;
            margin-top: 60px;
            padding: 20px;
            max-width: 1200px;
            margin-left: auto;
            margin-right: auto;
        }

        /* Feed area */
        .feed {
            flex: 1;
            max-width: 680px;
            margin-right: 20px;
        }

        /* Post creation area */
        .create-post {
            background: white;
            border-radius: 8px;
            padding: 16px;
            margin-bottom: 16px;
            box-shadow: 0 1px 2px rgba(0,0,0,0.1);
        }

        /* Individual post styling */
        .post {
            background: white;
            border-radius: 8px;
            padding: 16px;
            margin-bottom: 16px;
            box-shadow: 0 1px 2px rgba(0,0,0,0.1);
        }

        /* Right sidebar */
        .sidebar {
            width: 300px;
            position: sticky;
            top: 80px;
            height: calc(100vh - 80px);
            overflow-y: auto;
        }

        /* Chat widget */
        .chat-widget {
            position: fixed;
            bottom: 0;
            right: 16px;
            width: 280px;
            background: white;
            border-radius: 8px 8px 0 0;
            box-shadow: 0 -2px 4px rgba(0,0,0,0.1);
        }
    </style>
</head>
<body>
    <!-- Fixed navigation bar -->
    <nav class="navbar">
        <h1>SocialConnect</h1>
        <input type="text" class="search-box" placeholder="Search posts..." aria-label="Search posts">
        <button id="messages" aria-label="Open messages">Messages</button>
        <button id="notifications" aria-label="View notifications">Notifications</button>
        <img src="https://picsum.photos/100" alt="Your profile picture" id="profile_menu" aria-label="Open profile menu">
    </nav>

    <div class="main-content">
        <!-- Main feed area -->
        <section class="feed">
            <!-- Post creation area -->
            <div class="create-post">
                <textarea placeholder="What's on your mind?" aria-label="Create a post"></textarea>
                <input type="file" id="post_images" multiple accept="image/*" aria-label="Add images to your post">
                <button type="button">Post</button>
            </div>

            <!-- Feed posts -->
            <div class="post">
                <div class="post-header">
                    <img src="https://picsum.photos/100?random=1" alt="Alice's profile picture" class="profile-pic">
                    <div class="post-meta">
                        <div class="username">Alice Johnson</div>
                        <div class="timestamp">2 hours ago</div>
                    </div>
                </div>
                <div class="post-content">
                    Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 
                    #MorningHike #Nature #Wellness
                </div>
                <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" class="post-image">
                <div class="post-actions">
                    <button class="action-button">Like</button>
                    <button class="action-button">Comment</button>
                    <button class="action-button">Share</button>
                </div>
                <div class="comments">
                    <div class="comment">
                        <img src="https://picsum.photos/100?random=3" alt="Bob's profile picture" class="profile-pic">
                        <div class="comment-content">Beautiful view! Which trail is this?</div>
                    </div>
                </div>
            </div>

            <!-- Long post with "Read more" -->
            <div class="post">
                <div class="post-header">
                    <img src="carol-avatar.jpg" alt="Carol's profile picture" class="profile-pic">
                    <div class="post-meta">
                        <div class="username">Carol White</div>
                        <div class="timestamp">3 hours ago</div>
                    </div>
                </div>
                <div class="post-content truncated">
                    Just finished reading an fascinating paper on artificial intelligence and its implications...
                    <button class="read-more">Read more</button>
                </div>
            </div>

            <!-- Multi-image post -->
            <div class="post">
                <div class="post-header">
                    <img src="https://picsum.photos/100?random=4" alt="David's profile picture" class="profile-pic">
                    <div class="post-meta">
                        <div class="username">David Brown</div>
                        <div class="timestamp">5 hours ago</div>
                    </div>
                </div>
                <div class="post-content">
                    Amazing day at the photography exhibition! Swipe to see more 📸
                </div>
                <div class="image-gallery">
                    <img src="https://picsum.photos/800/800?random=5" alt="Exhibition photo 1" class="gallery-image">
                    <img src="https://picsum.photos/800/800?random=6" alt="Exhibition photo 2" class="gallery-image">
                    <img src="https://picsum.photos/800/800?random=7" alt="Exhibition photo 3" class="gallery-image">
                    <button class="gallery-nav prev" aria-label="Previous image">←</button>
                    <button class="gallery-nav next" aria-label="Next image">→</button>
                </div>
            </div>

            <!-- Continue with more posts... -->
            <!-- Add 15-20 more posts with varying content -->
        </section>

        <!-- Right sidebar -->
        <aside class="sidebar">
            <!-- Online friends -->
            <div class="friends-list">
                <h2>Online Friends</h2>
                <div class="friend">
                    <img src="https://picsum.photos/100?random=8" alt="Emma's profile picture">
                    <span>Emma Davis</span>
                    <div class="online-indicator"></div>
                </div>
                <div class="friend">
                    <img src="https://picsum.photos/100?random=9" alt="Frank Wilson's profile picture">
                    <span>Frank Wilson</span>
                    <div class="online-indicator"></div>
                </div>
                <!-- Add more online friends -->
            </div>

            <!-- Trending topics -->
            <div class="trending">
                <h2>Trending Topics</h2>
                <div class="topic">#Photography</div>
                <div class="topic">#TechNews</div>
                <div class="topic">#Hiking</div>
            </div>
        </aside>
    </div>

    <!-- Chat widget -->
    <div class="chat-widget">
        <div class="chat-header">
            <h3>Chat</h3>
            <button class="minimize" aria-label="Minimize chat">−</button>
        </div>
        <div class="chat-messages">
            <!-- Chat messages would go here -->
        </div>
        <input type="text" class="chat-input" placeholder="Type a message..." aria-label="Type a message">
    </div>

    <!-- Image preview modal -->
    <div class="modal" id="image_preview" style="display: none;">
        <img src="" alt="Large preview of selected image">
        <button class="close-modal" aria-label="Close preview">×</button>
    </div>
</body>
</html>
```

### User Request to SIA

"I want you to help me create a social media post about my morning hike. I have a photo at '/photos/sunrise.jpg' to share. First, search for similar hiking posts to see what hashtags are popular. Then create a new post with my photo and some engaging text. After posting, find and interact with other hiking-related posts - like them and leave encouraging comments. Finally, send a chat message to anyone who's online and interested in hiking."

### Step 1: Initialize Session

```xml
<reasoning>
First, I'll start an ITB session and establish continuous monitoring of the page state.
</reasoning>

<single>
itb_start "social.html"
</single>
```

Expected Output:
```json
{
    "pid": 12345,
    "session_id": "550e8400-e29b-41d4-a716-446655440000",
    "status": "success"
}
```

```xml
<repeat>
itb_screenshot 550e8400-e29b-41d4-a716-446655440000
</repeat>
```

Initial viewport state:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
    <!-- Navigation bar with properly structured elements -->
    <text x="16" y="20" width="120" height="40">SocialConnect</text>
    <interactive accepts_text="true" x="152" y="15" width="300" height="40" 
                 placeholder="Search posts..." aria-label="Search posts"/>
    <interactive clickable="true" x="468" y="15" width="100" height="40" 
                 aria-label="Open messages">Messages</interactive>
    <interactive clickable="true" x="584" y="15" width="120" height="40" 
                 aria-label="View notifications">Notifications</interactive>
    <interactive clickable="true" id="profile_menu" x="720" y="15" width="40" height="40">
        <img src="https://picsum.photos/100" alt="Your profile picture" width="40" height="40"/>
    </interactive>
    
    <!-- Main content area -->
    <div class="main-content" x="0" y="60" width="1024" height="740">
        <!-- Post creation area -->
        <interactive accepts_text="true" x="20" y="80" width="680" height="100"
                     placeholder="What's on your mind?" aria-label="Create a post"/>
        <interactive accepts_file="true" id="post_images" x="20" y="190" width="200" height="40"
                     multiple="true" accept="image/*" aria-label="Add images to your post"/>
        <interactive clickable="true" x="20" y="240" width="100" height="40">Post</interactive>

        <!-- Feed posts -->
        <text x="20" y="300" width="680" height="60">Alice Johnson - 2 hours ago</text>
        <text x="20" y="360" width="680" height="80">Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
        <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" 
             x="20" y="450" width="680" height="510"/>
        <interactive clickable="true" x="20" y="970" width="100" height="40">Like</interactive>
        <interactive clickable="true" x="130" y="970" width="100" height="40">Comment</interactive>
        <interactive clickable="true" x="240" y="970" width="100" height="40">Share</interactive>
        
        <!-- Sidebar -->
        <text x="720" y="80" width="284" height="40">Online Friends</text>
        <interactive clickable="true" x="720" y="130" width="284" height="60">
            <img src="https://picsum.photos/100?random=8" alt="Emma's profile picture" 
                 width="40" height="40"/>
            <text x="770" y="140" width="200" height="40">Emma Davis</text>
            <div class="online-indicator" x="980" y="140" width="10" height="10"/>
        </interactive>
    </div>
</viewport>
```

### Step 2: Search for Hiking Posts

```xml
<reasoning>
I'll use the search box in the navigation bar to find hiking-related posts and analyze hashtag patterns.
</reasoning>

<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 302 -y 35 --click
</single>

<single>
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "#hiking"
</single>
```

Updated viewport showing search results:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
    <!-- Navigation bar remains unchanged -->
    
    <!-- Search results -->
    <text x="20" y="80" width="680" height="40">Search results for "#hiking" - 3 posts found</text>
    
    <!-- Search result posts -->
    <div class="post" x="20" y="140" width="680" height="600">
        <text x="20" y="140" width="680" height="60">Alice Johnson - 2 hours ago</text>
        <text x="20" y="200" width="680" height="80">Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
        <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" 
             x="20" y="290" width="680" height="510"/>
    </div>
    
    <div class="post" x="20" y="760" width="680" height="600">
        <text x="20" y="760" width="680" height="60">David Brown - 1 day ago</text>
        <text x="20" y="820" width="680" height="80">Weekend hiking adventure! Found this hidden waterfall trail. #Hiking #Adventure #NaturePhotography</text>
        <img src="https://picsum.photos/800/600?random=3" alt="Waterfall on hiking trail" 
             x="20" y="910" width="680" height="510"/>
    </div>
</viewport>
```

### Step 3: Create New Post

```xml
<reasoning>
Now I'll create our new post with the sunrise photo, incorporating popular hashtags from existing posts.
</reasoning>

<single>
itb_scroll 550e8400-e29b-41d4-a716-446655440000 -x 0 -y 0
</single>

<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 360 -y 130 --click
</single>

<single>
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews"
</single>

<single>
itb_upload 550e8400-e29b-41d4-a716-446655440000 -i "post_images" -f "/photos/sunrise.jpg"
</single>

<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 70 -y 260 --click
</single>

### Step 4: Interact with Related Posts

```xml
<reasoning>
I'll engage with Alice's hiking post through likes and comments.
</reasoning>

<single>
itb_scroll 550e8400-e29b-41d4-a716-446655440000 -x 0 -y 600
</single>

<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 70 -y 970 --click
</single>

<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 180 -y 970 --click
</single>

<single>
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "What a gorgeous view! I just did a morning hike too. The sunrise is always worth the early wake-up call! 🌅"
</single>

### Step 5: Start a Chat

```xml
<reasoning>
I'll initiate a conversation with Emma Davis about hiking.
</reasoning>

<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 862 -y 160 --click
</single>

Viewport showing chat widget:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
    <!-- Main content remains unchanged -->
    
    <!-- Chat widget -->
    <div class="chat-widget" x="744" y="400" width="280" height="400">
        <text x="754" y="420" width="260" height="40">Emma Davis (Online)</text>
        <div class="chat-messages" x="754" y="470" width="260" height="280"/>
        <interactive accepts_text="true" x="754" y="760" width="260" height="40"
                     placeholder="Type a message..." aria-label="Type a message"/>
    </div>
</viewport>
```

<single>
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!"
</single>

Final viewport state:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
    <!-- Chat widget with sent message -->
    <div class="chat-widget" x="744" y="400" width="280" height="400">
        <text x="754" y="420" width="260" height="40">Emma Davis (Online)</text>
        <div class="chat-messages" x="754" y="470" width="260" height="280">
            <text x="754" y="480" width="260" height="80">You - Just now - Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!</text>
        </div>
        <interactive accepts_text="true" x="754" y="760" width="260" height="40"
                     placeholder="Type a message..." aria-label="Type a message"/>
    </div>
</viewport>
```