Continue work on itb

tools/itb/README.md

@@ -1,151 +1,782 @@
 # ITB: The Interactive Text Browser
 
-ITB aims to create a set of Python scripts to enable a text-based AI agent to interact with web pages.
+## Understanding the Challenge
-The system is designed to start a detached browser process and interact with it through subsequent scripts.
-Each script performs a specific action, such as starting the browser, taking a virtual screenshot, scrolling, clicking, and inputting text.
-The output of these scripts is optimized to be evaluated by a Large Language Model (LLM).
-So it can understand and interact with the webpage in a text-only manner.
 
-# The Scripts
+Modern websites are designed for human interaction. They rely on visual cues, spatial relationships, and complex interactions through mouse and keyboard. A human can instantly understand that a blue underlined piece of text is clickable, that a grid of product cards represents a catalog, or that a hamburger icon will reveal a navigation menu.
 
-## itb_start
+AI agents, however, process information differently. They need structured data that explicitly describes both content and interaction possibilities. Traditional web scraping tools can extract content but struggle with interaction. Screen readers come closer but are optimized for linear reading rather than complex interaction patterns.
 
-Starts a detached browser session and outputs the necessary information to interact with the browser process.
+ITB (Interactive Text Browser) bridges this gap. It transforms web pages into a format that AI agents can process effectively while preserving all the interaction capabilities a human would have. This transformation focuses on what matters for interaction - the content that's visible and the ways it can be manipulated - while removing technical complexity that only exists to support visual presentation.
 
-### Usage
+## How ITB Works
 
-```
+ITB operates as a bridge between an AI agent and a web browser. When you start ITB, it launches a browser process that runs independently, just like when you open a browser yourself. This browser loads and renders web pages normally, but instead of displaying them on screen, it communicates their content and state to the AI agent through a simplified XML format.
-itb_start <url>
+
+Most of ITB's work happens through JavaScript that runs directly in the browser. Rather than making many small requests back and forth between Python and the browser, ITB bundles operations together into efficient JavaScript operations. This approach significantly improves performance while making the system more reliable.
+
+Consider a typical login form interaction. Instead of separate commands to find fields, click them, and type text, ITB might execute a single JavaScript operation that handles the entire interaction:
+
+```javascript
+// Single efficient operation instead of multiple commands
+function fillLoginForm(username, password) {
+    const usernameField = document.getElementById('username');
+    const passwordField = document.getElementById('password');
+    
+    // Perform all operations in one go
+    usernameField.focus();
+    usernameField.value = username;
+    passwordField.focus();
+    passwordField.value = password;
+    
+    return {
+        username_entered: true,
+        password_entered: true,
+        submit_button_enabled: document.querySelector('button[type="submit"]').disabled === false
+    };
+}
 ```
 
-### Output
+## Page Representation
 
-- Process ID (PID) of the browser session.
+ITB transforms complex HTML structures into a simplified XML format that represents what's actually visible and interactive on the page. Let's look at several examples to understand this transformation.
-- WebDriver session ID.
 
-### Implementation Strategies
+### Text Content
 
-Use Selenium to start a browser session.
+When a human reads a webpage, they naturally process related content together regardless of how it's structured in HTML:
-Ensure the browser runs in a detached process to allow other scripts to interact with it.
-Output the PID and WebDriver session ID for subsequent scripts to use.
 
-## itb_screenshot
+```html
+<!-- Original HTML -->
+<div class="article">
+    <h2 class="title">Breaking News</h2>
+    <div class="metadata">
+        <span class="date">2024-12-20</span>
+        <span class="author">By John Smith</span>
+    </div>
+    <p class="lead">Scientists announce breakthrough...</p>
+</div>
 
-Starts a virtual screenshot of the current browser view and outputs all visible information in text form.
+<!-- ITB Output -->
-
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
-### Usage
+    <text x="20" y="40">Breaking News - 2024-12-20 - By John Smith - Scientists announce breakthrough...</text>
-
+</viewport>
-```
-itb_screenshot <session_id>
 ```
 
-### Output
+The transformation merges related text elements into a single coherent flow, making it easier for AI agents to process content the way a human would read it.
 
-- Scroll offset, visible range and limits.
+### Interactive Elements
-- Text representation of all currently visible elements.
-- IDs, size and offset for interactive elements (e.g., buttons, links, input fields).
 
-### Implementation Strategies
+ITB describes elements by their interaction capabilities rather than their HTML structure:
 
-Get the identifiers for all visible elements.
+```html
-Identify elements with text content and interaction capabilities.
+<!-- Original HTML -->
-Extract text content without markup.
+<form class="login-form">
-Create simplified XML structure.
+    <label for="username">Username:</label>
+    <input type="text" id="username" class="form-control">
+    <label for="password">Password:</label>
+    <input type="password" id="password" class="form-control">
+    <button type="submit" class="btn">Login</button>
+</form>
 
-## itb_scroll
+<!-- ITB Output -->
-
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
-Scrolls the browser window to a specified position.
+    <text x="20" y="40">Username:</text>
-
+    <interactive accepts_text="true" id="username" x="20" y="70" width="200" height="30"/>
-### Usage
+    <text x="20" y="120">Password:</text>
-
+    <interactive accepts_text="true" password="true" id="password" x="20" y="150" width="200" height="30"/>
-```
+    <interactive clickable="true" x="20" y="200" width="80" height="40">Login</interactive>
-itb_scroll <session_id> <x> <y>
+</viewport>
 ```
 
-## itb_click
+### Media Content
 
-Clicks on a specified viewport coordinates
+Media elements like images, videos, and audio are represented simply with their source and alternative text:
 
-### Usage
+```html
+<!-- Original HTML -->
+<div class="media-content">
+    <img src="diagram.png" alt="System architecture diagram" class="full-width">
+    <video src="demo.mp4" controls>Video demonstration of the system</video>
+    <audio src="recording.mp3" controls>Conference presentation audio</audio>
+</div>
 
-```
+<!-- ITB Output -->
-itb_click <session_id> -x <x> -y <y> [--right] [--double]
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
+    <img src="diagram.png" alt="System architecture diagram" x="20" y="40" width="800" height="400"/>
+    <video src="demo.mp4" alt="Video demonstration of the system" x="20" y="460" width="640" height="360"/>
+    <audio src="recording.mp3" alt="Conference presentation audio" x="20" y="840" width="400" height="40"/>
+</viewport>
 ```
 
-### Implementation Strategies
+### Embedded Content
 
-Use a random short time to slowly move the cursor to the target position.
+Modern web pages often embed content from other sources using iframes. While humans don't notice these technical boundaries, they affect how we can interact with the content. ITB handles this by representing iframes transparently:
-Click on a random position within the target element.
 
-## itb_input
+```html
+<!-- Original HTML -->
+<div class="content">
+    <h1>Welcome Back</h1>
+    <iframe src="https://auth.service.com/login" width="400" height="300"></iframe>
+</div>
 
-Inputs text or keystrokes into the focussed element.
+<!-- ITB Output -->
-
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
-### Usage
+    <text x="20" y="40">Welcome Back</text>
-
+    <iframe src="https://auth.service.com/login" x="20" y="80" width="400" height="300"/>
-```
+</viewport>
-itb_input <session_id> -t <text>
-itb_input <session_id> -k <keystroke> ...
 ```
 
-### Implementation Strategies
+When an AI agent encounters an iframe, it can start a new browser instance to interact with that content directly. This maintains security boundaries while providing full access to embedded content.
 
-Use a random short delay between adding each character of text, to simulate human-like typing.
+### Interaction State
-Keystrokes are additive.
-e.g. for CTRL SHIFT a, this emulates the user pressing CTRL, keeping it pressed, then pressing SHIFT, keeping both CTRL and SHIFT pressed, then pressing 'a', keeping all three pressed, then releasing all three.
 
-## itb_links
+Just as humans need to know where their mouse pointer is and which text field they're typing in, AI agents need clear information about the current interaction state. ITB provides this through viewport attributes:
 
-Extracts all links from the full webpage.
+```xml
-
+<viewport 
-### Usage
+    width="1024" height="800" 
-
+    scroll_x="0" scroll_y="0"
-```
+    pointer_x="150" pointer_y="45"
-itb_links <session_id>
+    focus_id="username"
+    caret_position="5"
+    selection_start="2" selection_end="5"
+>
+    <text x="20" y="30">Username:</text>
+    <interactive accepts_text="true" id="username" x="100" y="30" width="200" height="30">
+        alice
+    </interactive>
+</viewport>
 ```
 
-### Output
+This shows:
+- The viewport's dimensions and scroll position
+- The mouse pointer's location
+- Which element has input focus
+- The text cursor position
+- Any selected text
 
-- List of all visible links with their text, URLs and position on the page.
+### Helping AI Agents Click
 
-## itb_forms
+Unlike humans who can easily click anywhere within an interactive element, AI agents need precise coordinates. For elements without IDs that could be targeted directly, ITB provides suggested click coordinates:
 
-Extracts all visible forms and their input fields from the current browser view.
+```xml
-
+<interactive 
-### Usage
+    accepts_text="true"
-
+    suggested_click="135,45"
-```
+    x="100" y="30" width="200" height="30"
-itb_forms <session_id>
+/>
 ```
 
-### Output
+## Command Reference
 
-List of all visible forms with their
+### Starting a Session
-- input fields
-- connected labels
-- identifier
-- position on the page.
-
-## itb_navigate
-
-Navigates to a specified URL within the existing browser session.
-
-### Usage
 
+```bash
+itb_start "https://example.com"
 ```
+
+Starts a new browser session and returns:
+```json
+{
+    "pid": 12345,
+    "session_id": "550e8400-e29b-41d4-a716-446655440000",
+    "status": "success"
+}
+```
+
+### Reading Page State
+
+```bash
+itb_screenshot <session_id> [--wait_stable] [--timeout 5]
+```
+
+Captures the current page state in ITB's XML format. The `--wait_stable` option waits for the DOM to stop changing before capturing, useful for dynamic content. ITB detects stability by monitoring DOM mutations.
+
+### Controlling the Mouse Pointer
+
+```bash
+# Move pointer
+itb_cursor <session_id> -x 150 -y 45
+
+# Move and click
+itb_cursor <session_id> -x 150 -y 45 --click
+```
+
+### Text Input
+
+```bash
+# Input by element ID
+itb_input <session_id> -i "username" -t "alice@example.com"
+
+# Input to focused element
+itb_input <session_id> -t "hello world"
+
+# Move text cursor
+itb_input <session_id> --caret 5
+
+# Select text
+itb_input <session_id> --select 2 5
+```
+
+### File Upload
+
+```bash
+itb_upload <session_id> -i "profile_picture" -f "/path/to/avatar.jpg"
+```
+
+### Scrolling
+
+ITB provides several ways to control scrolling:
+
+```bash
+# Absolute page position
+itb_scroll <session_id> -x 0 -y 500
+
+# Relative to current position
+itb_scroll <session_id> --relative -x 0 -y 100
+
+# To specific element
+itb_scroll <session_id> --to-element "section_3"
+```
+
+### Navigation
+
+```bash
+# Navigate to new URL
 itb_navigate <session_id> <url>
-```
 
-## itb_refresh
+# Refresh current page
-
-Refreshes the current browser page.
-
-### Usage
-
-```
 itb_refresh <session_id>
 ```
 
-### Implementation Strategies
+## Debugging
 
-Wait till the page is fully loaded and the DOM is stable.
+Since most of ITB's logic runs as JavaScript in the browser, debugging can be enabled to provide detailed information about operations:
+
+```bash
+# Enable debug mode
+itb_screenshot <session_id> --debug
+
+# Output includes operation details:
+# [Debug] Starting page analysis
+# [Debug] Found 127 elements to process
+# [Debug] Processing element 'username': visible at (150, 45)
+# ...
+```
+
+## Implementation Guidelines
+
+When implementing or extending ITB, keep these principles in mind:
+
+### Efficient JavaScript
+
+Bundle operations together to minimize round trips between Python and the browser. Instead of multiple small operations:
+
+```python
+# Inefficient - multiple Selenium calls
+element = driver.find_element_by_id("username")
+element.click()
+element.clear()
+element.send_keys("alice")
+```
+
+Use single JavaScript operations:
+
+```python
+# Efficient - one browser interaction
+driver.execute_script("""
+    const field = document.getElementById('username');
+    field.focus();
+    field.value = 'alice';
+    return {success: true, value: field.value};
+""")
+```
+
+### Text Processing
+
+Text content should be merged when it forms a natural reading flow:
+- Adjacent text elements should be combined
+- Maintain accurate positioning information
+- Preserve text exactly as it appears to users
+
+### Interactive Elements
+
+Focus on describing interaction possibilities:
+- What type of interaction is supported
+- Any constraints or requirements
+- Current state (disabled, selected, etc.)
+- Position and size for interaction
+
+### Testing
+
+ITB's JavaScript operations can be tested using fixture files:
+
+```python
+class TestITBTransformations(unittest.TestCase):
+    def test_text_merging(self):
+        with open('fixtures/complex_text.html') as f:
+            html = f.read()
+        with open('fixtures/expected_output.xml') as f:
+            expected = f.read()
+            
+        self.mock_driver.page_source = html
+        result = itb_screenshot('dummy-session')
+        self.assertEqual(result.to_xml(), expected)
+```
+
+## Technical Limitations
+
+ITB intentionally does not attempt to:
+
+1. Preserve HTML structure beyond what's necessary for interaction. The HTML structure exists to support visual presentation and doesn't matter to an AI agent.
+
+2. Detect JavaScript event handlers or custom interaction patterns. These can't be reliably detected by examining the DOM alone.
+
+3. Implement complex media playback controls. Media files should be downloaded and processed separately if needed.
+
+4. Bypass browser security restrictions. Cross-origin iframes are handled by starting new browser sessions rather than trying to circumvent security measures.
+
+These limitations keep the system focused and reliable, ensuring that AI agents can consistently interact with web pages through a simplified but functional interface.
+
+## Example Scenario: Social Media Interaction
+
+This example demonstrates how an AI agent uses ITB to interact with a social media interface. It showcases ITB's key features including text collapsing, state management, and complex interactions. The scenario walks through a complete social media interaction flow, from content creation to user engagement.
+
+### social.html
+```html
+<!DOCTYPE html>
+<html>
+<head>
+    <title>SocialConnect</title>
+    <style>
+        /* Core layout styles */
+        body { margin: 0; font-family: Arial, sans-serif; background: #f0f2f5; }
+        
+        /* Fixed navigation bar */
+        .navbar {
+            position: fixed;
+            top: 0;
+            width: 100%;
+            height: 60px;
+            background: white;
+            box-shadow: 0 2px 4px rgba(0,0,0,0.1);
+            z-index: 100;
+            display: flex;
+            align-items: center;
+            padding: 0 16px;
+        }
+
+        /* Main content area */
+        .main-content {
+            display: flex;
+            margin-top: 60px;
+            padding: 20px;
+            max-width: 1200px;
+            margin-left: auto;
+            margin-right: auto;
+        }
+
+        /* Feed area */
+        .feed {
+            flex: 1;
+            max-width: 680px;
+            margin-right: 20px;
+        }
+
+        /* Post creation area */
+        .create-post {
+            background: white;
+            border-radius: 8px;
+            padding: 16px;
+            margin-bottom: 16px;
+            box-shadow: 0 1px 2px rgba(0,0,0,0.1);
+        }
+
+        /* Individual post styling */
+        .post {
+            background: white;
+            border-radius: 8px;
+            padding: 16px;
+            margin-bottom: 16px;
+            box-shadow: 0 1px 2px rgba(0,0,0,0.1);
+        }
+
+        /* Right sidebar */
+        .sidebar {
+            width: 300px;
+            position: sticky;
+            top: 80px;
+            height: calc(100vh - 80px);
+            overflow-y: auto;
+        }
+
+        /* Chat widget */
+        .chat-widget {
+            position: fixed;
+            bottom: 0;
+            right: 16px;
+            width: 280px;
+            background: white;
+            border-radius: 8px 8px 0 0;
+            box-shadow: 0 -2px 4px rgba(0,0,0,0.1);
+        }
+    </style>
+</head>
+<body>
+    <!-- Fixed navigation bar -->
+    <nav class="navbar">
+        <h1>SocialConnect</h1>
+        <input type="text" class="search-box" placeholder="Search posts..." aria-label="Search posts">
+        <button id="messages" aria-label="Open messages">Messages</button>
+        <button id="notifications" aria-label="View notifications">Notifications</button>
+        <img src="https://picsum.photos/100" alt="Your profile picture" id="profile_menu" aria-label="Open profile menu">
+    </nav>
+
+    <div class="main-content">
+        <!-- Main feed area -->
+        <section class="feed">
+            <!-- Post creation area -->
+            <div class="create-post">
+                <textarea placeholder="What's on your mind?" aria-label="Create a post"></textarea>
+                <input type="file" id="post_images" multiple accept="image/*" aria-label="Add images to your post">
+                <button type="button">Post</button>
+            </div>
+
+            <!-- Feed posts -->
+            <div class="post">
+                <div class="post-header">
+                    <img src="https://picsum.photos/100?random=1" alt="Alice's profile picture" class="profile-pic">
+                    <div class="post-meta">
+                        <div class="username">Alice Johnson</div>
+                        <div class="timestamp">2 hours ago</div>
+                    </div>
+                </div>
+                <div class="post-content">
+                    Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 
+                    #MorningHike #Nature #Wellness
+                </div>
+                <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" class="post-image">
+                <div class="post-actions">
+                    <button class="action-button">Like</button>
+                    <button class="action-button">Comment</button>
+                    <button class="action-button">Share</button>
+                </div>
+                <div class="comments">
+                    <div class="comment">
+                        <img src="https://picsum.photos/100?random=3" alt="Bob's profile picture" class="profile-pic">
+                        <div class="comment-content">Beautiful view! Which trail is this?</div>
+                    </div>
+                </div>
+            </div>
+
+            <!-- Long post with "Read more" -->
+            <div class="post">
+                <div class="post-header">
+                    <img src="carol-avatar.jpg" alt="Carol's profile picture" class="profile-pic">
+                    <div class="post-meta">
+                        <div class="username">Carol White</div>
+                        <div class="timestamp">3 hours ago</div>
+                    </div>
+                </div>
+                <div class="post-content truncated">
+                    Just finished reading an fascinating paper on artificial intelligence and its implications...
+                    <button class="read-more">Read more</button>
+                </div>
+            </div>
+
+            <!-- Multi-image post -->
+            <div class="post">
+                <div class="post-header">
+                    <img src="https://picsum.photos/100?random=4" alt="David's profile picture" class="profile-pic">
+                    <div class="post-meta">
+                        <div class="username">David Brown</div>
+                        <div class="timestamp">5 hours ago</div>
+                    </div>
+                </div>
+                <div class="post-content">
+                    Amazing day at the photography exhibition! Swipe to see more 📸
+                </div>
+                <div class="image-gallery">
+                    <img src="https://picsum.photos/800/800?random=5" alt="Exhibition photo 1" class="gallery-image">
+                    <img src="https://picsum.photos/800/800?random=6" alt="Exhibition photo 2" class="gallery-image">
+                    <img src="https://picsum.photos/800/800?random=7" alt="Exhibition photo 3" class="gallery-image">
+                    <button class="gallery-nav prev" aria-label="Previous image">←</button>
+                    <button class="gallery-nav next" aria-label="Next image">→</button>
+                </div>
+            </div>
+
+            <!-- Continue with more posts... -->
+            <!-- Add 15-20 more posts with varying content -->
+        </section>
+
+        <!-- Right sidebar -->
+        <aside class="sidebar">
+            <!-- Online friends -->
+            <div class="friends-list">
+                <h2>Online Friends</h2>
+                <div class="friend">
+                    <img src="https://picsum.photos/100?random=8" alt="Emma's profile picture">
+                    <span>Emma Davis</span>
+                    <div class="online-indicator"></div>
+                </div>
+                <div class="friend">
+                    <img src="https://picsum.photos/100?random=9" alt="Frank Wilson's profile picture">
+                    <span>Frank Wilson</span>
+                    <div class="online-indicator"></div>
+                </div>
+                <!-- Add more online friends -->
+            </div>
+
+            <!-- Trending topics -->
+            <div class="trending">
+                <h2>Trending Topics</h2>
+                <div class="topic">#Photography</div>
+                <div class="topic">#TechNews</div>
+                <div class="topic">#Hiking</div>
+            </div>
+        </aside>
+    </div>
+
+    <!-- Chat widget -->
+    <div class="chat-widget">
+        <div class="chat-header">
+            <h3>Chat</h3>
+            <button class="minimize" aria-label="Minimize chat">−</button>
+        </div>
+        <div class="chat-messages">
+            <!-- Chat messages would go here -->
+        </div>
+        <input type="text" class="chat-input" placeholder="Type a message..." aria-label="Type a message">
+    </div>
+
+    <!-- Image preview modal -->
+    <div class="modal" id="image_preview" style="display: none;">
+        <img src="" alt="Large preview of selected image">
+        <button class="close-modal" aria-label="Close preview">×</button>
+    </div>
+</body>
+</html>
+```
+
+### User Request to SIA
+
+"I want you to help me create a social media post about my morning hike. I have a photo at '/photos/sunrise.jpg' to share. First, search for similar hiking posts to see what hashtags are popular. Then create a new post with my photo and some engaging text. After posting, find and interact with other hiking-related posts - like them and leave encouraging comments. Finally, send a chat message to anyone who's online and interested in hiking."
+
+### Step 1: Initialize Session
+
+```xml
+<reasoning>
+First, I'll start an ITB session and establish continuous monitoring of the page state.
+</reasoning>
+
+<single>
+itb_start "social.html"
+</single>
+```
+
+Expected Output:
+```json
+{
+    "pid": 12345,
+    "session_id": "550e8400-e29b-41d4-a716-446655440000",
+    "status": "success"
+}
+```
+
+```xml
+<repeat>
+itb_screenshot 550e8400-e29b-41d4-a716-446655440000 --wait_stable
+</repeat>
+```
+
+Initial viewport state:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
+    <!-- Navigation bar with collapsed text -->
+    <text x="20" y="30">SocialConnect | Search posts... | Messages | Notifications</text>
+    
+    <!-- Interactive navigation elements -->
+    <interactive accepts_text="true" x="200" y="30" width="300" height="30" 
+                 placeholder="Search posts..." aria-label="Search posts"/>
+    <interactive clickable="true" id="profile_menu" x="720" y="30" width="30" height="30">
+        <img src="https://picsum.photos/100" alt="Your profile picture"/>
+    </interactive>
+    
+    <!-- Post creation area -->
+    <interactive accepts_text="true" x="340" y="100" width="400" height="100"
+                 placeholder="What's on your mind?" aria-label="Create a post"/>
+    <interactive accepts_file="true" id="post_images" x="340" y="130" width="200" height="30"
+                 multiple="true" accept="image/*"/>
+    <interactive clickable="true" x="340" y="160" width="80" height="30">Post</interactive>
+
+    <!-- Feed posts with collapsed text content -->
+    <text x="100" y="220">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
+    <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="320" width="600" height="450"/>
+    
+    <!-- Interactive post actions -->
+    <interactive clickable="true" x="340" y="520" width="80" height="30">Like</interactive>
+    <interactive clickable="true" x="340" y="560" width="80" height="30">Comment</interactive>
+    <interactive clickable="true" x="340" y="600" width="80" height="30">Share</interactive>
+
+    <!-- Sidebar with collapsed text -->
+    <text x="960" y="30">Online Friends</text>
+    <interactive clickable="true" x="960" y="200" width="200" height="40">Emma Davis (Online)</interactive>
+</viewport>
+```
+
+### Step 2: Search for Hiking Posts
+
+```xml
+<reasoning>
+I see the search box in the navigation bar. I'll click it and search for hiking-related posts to analyze common hashtags.
+</reasoning>
+
+<single>
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 250 -y 30 --click
+</single>
+
+<single>
+itb_input 550e8400-e29b-41d4-a716-446655440000 -t "#hiking"
+</single>
+```
+
+Updated viewport showing search results:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
+    <!-- Navigation and search UI remain unchanged -->
+    
+    <!-- Search results header -->
+    <text x="100" y="80">Search results for "#hiking" - 3 posts found</text>
+    
+    <!-- Collapsed search result posts -->
+    <text x="100" y="120">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
+    <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="220" width="600" height="450"/>
+    
+    <text x="100" y="720">David Brown - 1 day ago - Weekend hiking adventure! Found this hidden waterfall trail. #Hiking #Adventure #NaturePhotography</text>
+    <img src="https://picsum.photos/800/600?random=3" alt="Waterfall on hiking trail" x="100" y="820" width="600" height="450"/>
+</viewport>
+```
+
+### Step 3: Create New Post
+
+```xml
+<reasoning>
+Having analyzed the existing posts and popular hashtags, I'll now create our new post with the sunrise photo.
+</reasoning>
+
+<single>
+itb_scroll 550e8400-e29b-41d4-a716-446655440000 -x 0 -y 0
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 100 --click
+</single>
+
+<single>
+itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews"
+</single>
+
+<single>
+itb_upload 550e8400-e29b-41d4-a716-446655440000 -i "post_images" -f "/photos/sunrise.jpg"
+</single>
+```
+
+Viewport showing post creation state:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
+    <!-- Navigation and header remain unchanged -->
+    
+    <!-- Post creation area with entered content -->
+    <text x="340" y="100">Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews</text>
+    <img src="file:///photos/sunrise.jpg" alt="Uploaded sunrise photo" x="340" y="220" width="400" height="300"/>
+    <interactive clickable="true" x="720" y="220" width="20" height="20" aria-label="Remove uploaded photo">×</interactive>
+</viewport>
+```
+
+```xml
+<single>
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 160 --click
+</single>
+```
+
+Viewport after post creation:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
+    <!-- Navigation remains unchanged -->
+    
+    <!-- Newly created post appears at top of feed -->
+    <text x="100" y="120">You - Just now - Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews</text>
+    <img src="file:///photos/sunrise.jpg" alt="Uploaded sunrise photo" x="100" y="220" width="600" height="450"/>
+</viewport>
+```
+
+### Step 4: Interact with Related Posts
+
+```xml
+<reasoning>
+Now I'll scroll to find Alice's hiking post and engage with it through likes and comments.
+</reasoning>
+
+<single>
+itb_scroll 550e8400-e29b-41d4-a716-446655440000 --to-element "post-actions"
+</single>
+
+<single>
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 520 --click
+</single>
+```
+
+Viewport after liking:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
+    <text x="100" y="120">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
+    <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="220" width="600" height="450"/>
+    <interactive clickable="true" x="340" y="520" width="80" height="30" state="active">Like</interactive>
+</viewport>
+```
+
+```xml
+<single>
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 580 --click
+itb_input 550e8400-e29b-41d4-a716-446655440000 -t "What a gorgeous view! I just did a morning hike too. The sunrise is always worth the early wake-up call! 🌅"
+</single>
+```
+
+### Step 5: Start a Chat
+
+```xml
+<reasoning>
+I notice Emma Davis is online and has hiking photos. I'll initiate a conversation about hiking.
+</reasoning>
+
+<single>
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 960 -y 200 --click
+</single>
+```
+
+Viewport showing chat window:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
+    <!-- Main content remains with added chat widget -->
+    <interactive class="chat-widget" x="744" y="400" width="280" height="400">
+        <text x="754" y="420">Emma Davis (Online)</text>
+        <interactive accepts_text="true" x="754" y="760" width="260" height="30"
+                     placeholder="Type a message..." aria-label="Type a message"/>
+    </interactive>
+</viewport>
+```
+
+```xml
+<single>
+itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!"
+</single>
+```
+
+Final viewport state:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
+    <!-- Chat widget with sent message -->
+    <interactive class="chat-widget" x="744" y="400" width="280" height="400">
+        <text x="754" y="420">Emma Davis (Online)</text>
+        <text x="754" y="480">You - Just now - Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!</text>
+        <interactive accepts_text="true" x="754" y="760" width="260" height="30"
+                     placeholder="Type a message..." aria-label="Type a message"/>
+    </interactive>
+</viewport>
+```

tools/itb/bin/itb_click

@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 import sys
+import os
 import time
 import random
 import argparse
@@ -21,6 +22,13 @@ class ClickExecutor:
         options.add_experimental_option("debuggerAddress", f"127.0.0.1:{debug_port}")
         self.driver = webdriver.Chrome(options=options)
         
+        # Load JavaScript files
+        js_dir = os.path.join(os.path.dirname(__file__), "..", "js")
+        self.js_code = ""
+        for filename in ["viewport.js", "mouse.js"]:
+            with open(os.path.join(js_dir, filename)) as f:
+                self.js_code += f.read() + "\n"
+        
         # Get initial window info
         self.window_size = self.driver.get_window_size()
         debug(f"Window size: {json.dumps(self.window_size)}")
@@ -33,36 +41,15 @@ class ClickExecutor:
 
     def get_viewport_size(self):
         """Get viewport dimensions"""
-        return self.driver.execute_script("""
+        return self.driver.execute_script(self.js_code + "; return getViewportInfo();")
-            return {
-                width: window.innerWidth,
-                height: window.innerHeight,
-                pageYOffset: window.pageYOffset,
-                pageXOffset: window.pageXOffset
-            };
-        """)
         
     def get_scroll_position(self):
         """Get current scroll position"""
-        return self.driver.execute_script("""
+        return self.driver.execute_script(self.js_code + "; return getViewportInfo();")
-            return {
-                scrollX: window.scrollX,
-                scrollY: window.scrollY,
-                pageXOffset: window.pageXOffset,
-                pageYOffset: window.pageYOffset
-            };
-        """)
         
     def get_mouse_position(self):
         """Get current mouse coordinates"""
-        return self.driver.execute_script("""
+        return self.driver.execute_script(self.js_code + "; return getMousePosition();")
-            return {
-                screenX: window.screenX,
-                screenY: window.screenY,
-                clientX: window.event ? window.event.clientX : 0,
-                clientY: window.event ? window.event.clientY : 0
-            };
-        """)
 
     def scroll_to_target(self, x: int, y: int):
         """Scroll to make target coordinates visible"""
@@ -78,9 +65,7 @@ class ClickExecutor:
 
     def get_element_at_position(self, x: int, y: int):
         """Get the element at the specified coordinates"""
-        return self.driver.execute_script("""
+        return self.driver.execute_script(self.js_code + "; return getElementAtPosition(arguments[0], arguments[1]);", x, y)
-            return document.elementFromPoint(arguments[0], arguments[1]);
-        """, x, y)
 
     def execute_click(self, x: int, y: int, double: bool = False, right: bool = False):
         """Execute click operation with improved coordinate handling"""

tools/itb/bin/itb_screenshot

@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 import sys
+import os
 from selenium import webdriver
 from selenium.webdriver.common.by import By
 import xml.etree.ElementTree as ET
@@ -10,84 +11,15 @@ class ScreenshotGenerator:
         options.add_experimental_option("debuggerAddress", f"127.0.0.1:{debug_port}")
         self.driver = webdriver.Chrome(options=options)
         
+        # Load JavaScript files
+        js_dir = os.path.join(os.path.dirname(__file__), "..", "js")
+        self.js_code = ""
+        for filename in ["viewport.js", "mouse.js", "dom_analyzer.js"]:
+            with open(os.path.join(js_dir, filename)) as f:
+                self.js_code += f.read() + "\n"
+
     def get_page_content(self) -> dict:
-        script = """
+        return self.driver.execute_script(self.js_code + "; return analyzePageContent();")
-        function processElement(element) {
-            // Check if element is actually visible in viewport
-            const rect = element.getBoundingClientRect();
-            if (!element.offsetParent && element !== document.body) return null;
-            if (rect.width === 0 || rect.height === 0) return null;
-            
-            // Check if element is within viewport bounds
-            const viewportWidth = window.innerWidth;
-            const viewportHeight = window.innerHeight;
-            if (rect.bottom < 0 || rect.top > viewportHeight || 
-                rect.right < 0 || rect.left > viewportWidth) {
-                return null;
-            }
-            
-            const styles = window.getComputedStyle(element);
-            
-            const text = Array.from(element.childNodes)
-                .filter(node => node.nodeType === Node.TEXT_NODE)
-                .map(node => node.textContent.trim())
-                .join(' ');
-                
-            const interactions = [];
-            if (element.tagName === 'BUTTON' || element.tagName === 'A' || 
-                element.getAttribute('role') === 'button' || 
-                element.getAttribute('onclick') || 
-                styles.cursor === 'pointer') {
-                interactions.push('click');
-            }
-            
-            if (element.tagName === 'INPUT' || element.tagName === 'TEXTAREA' ||
-                element.getAttribute('contenteditable') === 'true') {
-                interactions.push('text_input');
-                const placeholder = element.getAttribute('placeholder');
-                if (placeholder) interactions.push('placeholder:' + placeholder);
-            }
-            
-            if (element.tagName === 'SELECT' || 
-                element.getAttribute('role') === 'listbox' ||
-                element.getAttribute('type') === 'checkbox' ||
-                element.getAttribute('type') === 'radio') {
-                interactions.push('select');
-            }
-            
-            if (element.scrollHeight > element.clientHeight || 
-                element.scrollWidth > element.clientWidth) {
-                interactions.push('scroll');
-            }
-            
-            const children = Array.from(element.children)
-                .map(child => processElement(child))
-                .filter(child => child !== null);
-                
-            return {
-                tag: element.tagName.toLowerCase(),
-                text,
-                location: {x: Math.round(rect.left), y: Math.round(rect.top)},
-                size: {width: Math.round(rect.width), height: Math.round(rect.height)},
-                interactions,
-                children
-            };
-        }
-        
-        const body = document.body;
-        return {
-            viewport: {
-                scroll_x: window.pageXOffset,
-                scroll_y: window.pageYOffset,
-                width: document.documentElement.clientWidth,
-                height: document.documentElement.clientHeight,
-                max_scroll_y: Math.max(document.documentElement.scrollHeight - window.innerHeight, 0),
-                min_scroll_x: Math.max(document.documentElement.scrollLeft, 0)
-            },
-            content: body ? processElement(body) : null
-        };
-        """
-        return self.driver.execute_script(script)
 
     def format_xml(self, xml_str: str) -> str:
         root = ET.fromstring(xml_str)

tools/itb/bin/itb_scroll

@@ -1,5 +1,6 @@
 #!/usr/bin/env python3
 import sys
+import os
 import time
 import random
 from selenium import webdriver
@@ -10,16 +11,14 @@ class ScrollController:
         options.add_experimental_option("debuggerAddress", f"127.0.0.1:{debug_port}")
         self.driver = webdriver.Chrome(options=options)
         
+        # Load JavaScript files
+        js_dir = os.path.join(os.path.dirname(__file__), "..", "js")
+        self.js_code = ""
+        with open(os.path.join(js_dir, "viewport.js")) as f:
+            self.js_code = f.read()
+
     def get_viewport_info(self) -> dict:
-        script = """
+        return self.driver.execute_script(self.js_code + "; return getViewportInfo();")
-        return {
-            scroll_x: window.pageXOffset,
-            scroll_y: window.pageYOffset,
-            max_scroll_x: Math.max(document.documentElement.scrollWidth - window.innerWidth, 0),
-            max_scroll_y: Math.max(document.documentElement.scrollHeight - window.innerHeight, 0)
-        };
-        """
-        return self.driver.execute_script(script)
 
     def smooth_scroll(self, target_x: int, target_y: int):
         viewport = self.get_viewport_info()

tools/itb/bin/itb_start

@@ -7,7 +7,7 @@ def start(url):
     debug_port = random.randint(9000, 9999)
     chrome_cmd = [
         '/usr/bin/google-chrome',
-        '--headless=new',
+        #'--headless=new',
         '--no-sandbox',
         '--remote-debugging-port=' + str(debug_port),
         url

tools/itb/example.md

@@ -0,0 +1,267 @@
+# Comprehensive ITB Social Media Interaction Example
+
+## Overview
+
+This example demonstrates how an AI agent uses ITB to interact with a social media interface. It showcases ITB's key features including text collapsing, state management, and complex interactions. The scenario walks through a complete social media interaction flow, from content creation to user engagement.
+
+## User Request to SIA
+
+"I want you to help me create a social media post about my morning hike. I have a photo at '/photos/sunrise.jpg' to share. First, search for similar hiking posts to see what hashtags are popular. Then create a new post with my photo and some engaging text. After posting, find and interact with other hiking-related posts - like them and leave encouraging comments. Finally, send a chat message to anyone who's online and interested in hiking."
+
+## SIA's Response and Actions
+
+### Step 1: Initialize Session
+
+```xml
+<reasoning>
+First, I'll start an ITB session and establish continuous monitoring of the page state.
+</reasoning>
+
+<single>
+itb_start "social.html"
+</single>
+```
+
+Expected Output:
+```json
+{
+    "pid": 12345,
+    "session_id": "550e8400-e29b-41d4-a716-446655440000",
+    "status": "success"
+}
+```
+
+```xml
+<repeat>
+itb_screenshot 550e8400-e29b-41d4-a716-446655440000 --wait_stable
+</repeat>
+```
+
+Initial viewport state:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
+    <!-- Navigation bar with collapsed text -->
+    <text x="20" y="30">SocialConnect | Search posts... | Messages | Notifications</text>
+    
+    <!-- Interactive navigation elements -->
+    <interactive accepts_text="true" x="200" y="30" width="300" height="30" 
+                 placeholder="Search posts..." aria-label="Search posts"/>
+    <interactive clickable="true" id="profile_menu" x="720" y="30" width="30" height="30">
+        <img src="https://picsum.photos/100" alt="Your profile picture"/>
+    </interactive>
+    
+    <!-- Post creation area -->
+    <interactive accepts_text="true" x="340" y="100" width="400" height="100"
+                 placeholder="What's on your mind?" aria-label="Create a post"/>
+    <interactive accepts_file="true" id="post_images" x="340" y="130" width="200" height="30"
+                 multiple="true" accept="image/*"/>
+    <interactive clickable="true" x="340" y="160" width="80" height="30">Post</interactive>
+
+    <!-- Feed posts with collapsed text content -->
+    <text x="100" y="220">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
+    <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="320" width="600" height="450"/>
+    
+    <!-- Interactive post actions -->
+    <interactive clickable="true" x="340" y="520" width="80" height="30">Like</interactive>
+    <interactive clickable="true" x="340" y="560" width="80" height="30">Comment</interactive>
+    <interactive clickable="true" x="340" y="600" width="80" height="30">Share</interactive>
+
+    <!-- Sidebar with collapsed text -->
+    <text x="960" y="30">Online Friends</text>
+    <interactive clickable="true" x="960" y="200" width="200" height="40">Emma Davis (Online)</interactive>
+</viewport>
+```
+
+### Step 2: Search for Hiking Posts
+
+```xml
+<reasoning>
+I see the search box in the navigation bar. I'll click it and search for hiking-related posts to analyze common hashtags.
+</reasoning>
+
+<single>
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 250 -y 30 --click
+</single>
+
+<single>
+itb_input 550e8400-e29b-41d4-a716-446655440000 -t "#hiking"
+</single>
+```
+
+Updated viewport showing search results:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
+    <!-- Navigation and search UI remain unchanged -->
+    
+    <!-- Search results header -->
+    <text x="100" y="80">Search results for "#hiking" - 3 posts found</text>
+    
+    <!-- Collapsed search result posts -->
+    <text x="100" y="120">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
+    <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="220" width="600" height="450"/>
+    
+    <text x="100" y="720">David Brown - 1 day ago - Weekend hiking adventure! Found this hidden waterfall trail. #Hiking #Adventure #NaturePhotography</text>
+    <img src="https://picsum.photos/800/600?random=3" alt="Waterfall on hiking trail" x="100" y="820" width="600" height="450"/>
+</viewport>
+```
+
+### Step 3: Create New Post
+
+```xml
+<reasoning>
+Having analyzed the existing posts and popular hashtags, I'll now create our new post with the sunrise photo.
+</reasoning>
+
+<single>
+itb_scroll 550e8400-e29b-41d4-a716-446655440000 -x 0 -y 0
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 100 --click
+</single>
+
+<single>
+itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews"
+</single>
+
+<single>
+itb_upload 550e8400-e29b-41d4-a716-446655440000 -i "post_images" -f "/photos/sunrise.jpg"
+</single>
+```
+
+Viewport showing post creation state:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
+    <!-- Navigation and header remain unchanged -->
+    
+    <!-- Post creation area with entered content -->
+    <text x="340" y="100">Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews</text>
+    <img src="file:///photos/sunrise.jpg" alt="Uploaded sunrise photo" x="340" y="220" width="400" height="300"/>
+    <interactive clickable="true" x="720" y="220" width="20" height="20" aria-label="Remove uploaded photo">×</interactive>
+</viewport>
+```
+
+```xml
+<single>
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 160 --click
+</single>
+```
+
+Viewport after post creation:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
+    <!-- Navigation remains unchanged -->
+    
+    <!-- Newly created post appears at top of feed -->
+    <text x="100" y="120">You - Just now - Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews</text>
+    <img src="file:///photos/sunrise.jpg" alt="Uploaded sunrise photo" x="100" y="220" width="600" height="450"/>
+</viewport>
+```
+
+### Step 4: Interact with Related Posts
+
+```xml
+<reasoning>
+Now I'll scroll to find Alice's hiking post and engage with it through likes and comments.
+</reasoning>
+
+<single>
+itb_scroll 550e8400-e29b-41d4-a716-446655440000 --to-element "post-actions"
+</single>
+
+<single>
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 520 --click
+</single>
+```
+
+Viewport after liking:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
+    <text x="100" y="120">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
+    <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="220" width="600" height="450"/>
+    <interactive clickable="true" x="340" y="520" width="80" height="30" state="active">Like</interactive>
+</viewport>
+```
+
+```xml
+<single>
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 580 --click
+itb_input 550e8400-e29b-41d4-a716-446655440000 -t "What a gorgeous view! I just did a morning hike too. The sunrise is always worth the early wake-up call! 🌅"
+</single>
+```
+
+### Step 5: Start a Chat
+
+```xml
+<reasoning>
+I notice Emma Davis is online and has hiking photos. I'll initiate a conversation about hiking.
+</reasoning>
+
+<single>
+itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 960 -y 200 --click
+</single>
+```
+
+Viewport showing chat window:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
+    <!-- Main content remains with added chat widget -->
+    <interactive class="chat-widget" x="744" y="400" width="280" height="400">
+        <text x="754" y="420">Emma Davis (Online)</text>
+        <interactive accepts_text="true" x="754" y="760" width="260" height="30"
+                     placeholder="Type a message..." aria-label="Type a message"/>
+    </interactive>
+</viewport>
+```
+
+```xml
+<single>
+itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!"
+</single>
+```
+
+Final viewport state:
+```xml
+<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
+    <!-- Chat widget with sent message -->
+    <interactive class="chat-widget" x="744" y="400" width="280" height="400">
+        <text x="754" y="420">Emma Davis (Online)</text>
+        <text x="754" y="480">You - Just now - Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!</text>
+        <interactive accepts_text="true" x="754" y="760" width="260" height="30"
+                     placeholder="Type a message..." aria-label="Type a message"/>
+    </interactive>
+</viewport>
+```
+
+```xml
+<reasoning>
+I've completed all requested interactions:
+1. Analyzed existing hiking posts and hashtags
+2. Created a new post with photo and appropriate hashtags
+3. Engaged with Alice's hiking post through a like and comment
+4. Initiated a conversation with Emma about hiking
+The continuous screenshot monitoring remains active to catch any responses.
+</reasoning>
+```
+
+## Key Features Demonstrated
+
+This example illustrates several important ITB features:
+
+1. Text Collapsing: Related text elements are automatically combined into natural reading flows, such as "Author - Time - Content" for posts.
+
+2. State Management: The viewport maintains awareness of scroll position, focus state, and interactive element states throughout the session.
+
+3. Interactive Elements: The example shows how ITB handles various types of interaction including:
+   - Text input fields
+   - File uploads
+   - Clickable buttons
+   - Scrollable areas
+   - Modal windows (chat widget)
+
+4. Media Handling: Images are represented with their position, size, and alt text while maintaining proper layout relationships with surrounding content.
+
+5. Dynamic Content: The example demonstrates how ITB handles dynamic content updates like:
+   - Search results loading
+   - Post creation
+   - Chat window opening
+   - Comment threading
+
+This comprehensive example shows how ITB transforms complex web interfaces into a format that AI agents can effectively process and interact with while maintaining all the functionality a human user would have.

tools/itb/js/dom_analyzer.js

@@ -0,0 +1,67 @@
+function processElement(element) {
+    const rect = element.getBoundingClientRect();
+    if (!element.offsetParent && element !== document.body) return null;
+    if (rect.width === 0 || rect.height === 0) return null;
+    
+    const viewportWidth = window.innerWidth;
+    const viewportHeight = window.innerHeight;
+    if (rect.bottom < 0 || rect.top > viewportHeight || 
+        rect.right < 0 || rect.left > viewportWidth) {
+        return null;
+    }
+    
+    const styles = window.getComputedStyle(element);
+    
+    const text = Array.from(element.childNodes)
+        .filter(node => node.nodeType === Node.TEXT_NODE)
+        .map(node => node.textContent.trim())
+        .join(' ');
+        
+    const interactions = [];
+    if (element.tagName === 'BUTTON' || element.tagName === 'A' || 
+        element.getAttribute('role') === 'button' || 
+        element.getAttribute('onclick') || 
+        styles.cursor === 'pointer') {
+        interactions.push('click');
+    }
+    
+    if (element.tagName === 'INPUT' || element.tagName === 'TEXTAREA' ||
+        element.getAttribute('contenteditable') === 'true') {
+        interactions.push('text_input');
+        const placeholder = element.getAttribute('placeholder');
+        if (placeholder) interactions.push('placeholder:' + placeholder);
+    }
+    
+    if (element.tagName === 'SELECT' || 
+        element.getAttribute('role') === 'listbox' ||
+        element.getAttribute('type') === 'checkbox' ||
+        element.getAttribute('type') === 'radio') {
+        interactions.push('select');
+    }
+    
+    if (element.scrollHeight > element.clientHeight || 
+        element.scrollWidth > element.clientWidth) {
+        interactions.push('scroll');
+    }
+    
+    const children = Array.from(element.children)
+        .map(child => processElement(child))
+        .filter(child => child !== null);
+        
+    return {
+        tag: element.tagName.toLowerCase(),
+        text,
+        location: {x: Math.round(rect.left), y: Math.round(rect.top)},
+        size: {width: Math.round(rect.width), height: Math.round(rect.height)},
+        interactions,
+        children
+    };
+}
+
+function analyzePageContent() {
+    const body = document.body;
+    return {
+        viewport: getViewportInfo(),
+        content: body ? processElement(body) : null
+    };
+}

tools/itb/js/mouse.js

@@ -0,0 +1,12 @@
+function getMousePosition() {
+    return {
+        screenX: window.screenX,
+        screenY: window.screenY,
+        clientX: window.event ? window.event.clientX : 0,
+        clientY: window.event ? window.event.clientY : 0
+    };
+}
+
+function getElementAtPosition(x, y) {
+    return document.elementFromPoint(x, y);
+}

tools/itb/js/viewport.js

@@ -0,0 +1,12 @@
+function getViewportInfo() {
+    return {
+        scroll_x: window.pageXOffset,
+        scroll_y: window.pageYOffset,
+        width: window.innerWidth,
+        height: window.innerHeight,
+        max_scroll_x: Math.max(document.documentElement.scrollWidth - window.innerWidth, 0),
+        max_scroll_y: Math.max(document.documentElement.scrollHeight - window.innerHeight, 0),
+        pageYOffset: window.pageYOffset,
+        pageXOffset: window.pageXOffset
+    };
+}

tools/itb/social.html

@@ -3,11 +3,10 @@
 <head>
     <title>SocialConnect</title>
     <style>
-        body {
+        /* Core layout styles */
-            margin: 0;
+        body { margin: 0; font-family: Arial, sans-serif; background: #f0f2f5; }
-            font-family: Arial, sans-serif;
+        
-            background: #f0f2f5;
+        /* Fixed navigation bar */
-        }
         .navbar {
             position: fixed;
             top: 0;
@@ -20,20 +19,34 @@
             align-items: center;
             padding: 0 16px;
         }
-        .search-box {
+
-            margin-left: 16px;
+        /* Main content area */
-            padding: 8px;
-            border-radius: 20px;
-            border: 1px solid #ddd;
-            width: 240px;
-        }
         .main-content {
-            margin-top: 80px;
+            display: flex;
+            margin-top: 60px;
+            padding: 20px;
+            max-width: 1200px;
             margin-left: auto;
             margin-right: auto;
-            max-width: 680px;
-            padding: 20px;
         }
+
+        /* Feed area */
+        .feed {
+            flex: 1;
+            max-width: 680px;
+            margin-right: 20px;
+        }
+
+        /* Post creation area */
+        .create-post {
+            background: white;
+            border-radius: 8px;
+            padding: 16px;
+            margin-bottom: 16px;
+            box-shadow: 0 1px 2px rgba(0,0,0,0.1);
+        }
+
+        /* Individual post styling */
         .post {
             background: white;
             border-radius: 8px;
@@ -41,88 +54,52 @@
             margin-bottom: 16px;
             box-shadow: 0 1px 2px rgba(0,0,0,0.1);
         }
-        .post-header {
+
-            display: flex;
+        /* Right sidebar */
-            align-items: center;
+        .sidebar {
-            margin-bottom: 12px;
+            width: 300px;
+            position: sticky;
+            top: 80px;
+            height: calc(100vh - 80px);
+            overflow-y: auto;
         }
-        .profile-pic {
+
-            width: 40px;
+        /* Chat widget */
-            height: 40px;
+        .chat-widget {
-            border-radius: 50%;
+            position: fixed;
-            margin-right: 12px;
+            bottom: 0;
-            background: #ddd;
+            right: 16px;
-        }
+            width: 280px;
-        .post-meta {
+            background: white;
-            flex-grow: 1;
+            border-radius: 8px 8px 0 0;
-        }
+            box-shadow: 0 -2px 4px rgba(0,0,0,0.1);
-        .username {
-            font-weight: bold;
-            color: #1a1a1a;
-        }
-        .timestamp {
-            color: #65676b;
-            font-size: 13px;
-        }
-        .post-content {
-            margin-bottom: 12px;
-            line-height: 1.5;
-        }
-        .post-image {
-            width: 100%;
-            max-height: 400px;
-            object-fit: cover;
-            border-radius: 8px;
-            margin-bottom: 12px;
-        }
-        .post-actions {
-            display: flex;
-            border-top: 1px solid #ddd;
-            padding-top: 12px;
-        }
-        .action-button {
-            flex: 1;
-            text-align: center;
-            padding: 8px;
-            border: none;
-            background: none;
-            cursor: pointer;
-            color: #65676b;
-            font-weight: 600;
-        }
-        .action-button:hover {
-            background: #f2f2f2;
-            border-radius: 4px;
-        }
-        .comment-section {
-            margin-top: 12px;
-            padding-top: 12px;
-            border-top: 1px solid #ddd;
-        }
-        .comment {
-            display: flex;
-            margin-bottom: 8px;
-        }
-        .comment-content {
-            background: #f0f2f5;
-            padding: 8px 12px;
-            border-radius: 18px;
-            margin-left: 8px;
-            flex-grow: 1;
         }
     </style>
 </head>
 <body>
+    <!-- Fixed navigation bar -->
     <nav class="navbar">
         <h1>SocialConnect</h1>
         <input type="text" class="search-box" placeholder="Search posts..." aria-label="Search posts">
+        <button id="messages" aria-label="Open messages">Messages</button>
+        <button id="notifications" aria-label="View notifications">Notifications</button>
+        <img src="https://picsum.photos/100" alt="Your profile picture" id="profile_menu" aria-label="Open profile menu">
     </nav>
 
-    <main class="main-content">
+    <div class="main-content">
-        <!-- Generate 20 posts with varying content -->
+        <!-- Main feed area -->
+        <section class="feed">
+            <!-- Post creation area -->
+            <div class="create-post">
+                <textarea placeholder="What's on your mind?" aria-label="Create a post"></textarea>
+                <input type="file" id="post_images" multiple accept="image/*" aria-label="Add images to your post">
+                <button type="button">Post</button>
+            </div>
+
+            <!-- Feed posts -->
             <div class="post">
                 <div class="post-header">
-                <div class="profile-pic"></div>
+                    <img src="https://picsum.photos/100?random=1" alt="Alice's profile picture" class="profile-pic">
                     <div class="post-meta">
                         <div class="username">Alice Johnson</div>
                         <div class="timestamp">2 hours ago</div>
@@ -132,47 +109,104 @@
                     Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 
                     #MorningHike #Nature #Wellness
                 </div>
-            <img src="data:image/svg+xml,<svg xmlns='http://www.w3.org/2000/svg' width='648' height='400'><rect width='100%' height='100%' fill='%23ddd'/></svg>" alt="Sunrise view from mountain top" class="post-image">
+                <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" class="post-image">
                 <div class="post-actions">
                     <button class="action-button">Like</button>
                     <button class="action-button">Comment</button>
                     <button class="action-button">Share</button>
                 </div>
-            <div class="comment-section">
+                <div class="comments">
                     <div class="comment">
-                    <div class="profile-pic"></div>
+                        <img src="https://picsum.photos/100?random=3" alt="Bob's profile picture" class="profile-pic">
-                    <div class="comment-content">
+                        <div class="comment-content">Beautiful view! Which trail is this?</div>
-                        <div class="username">Bob Smith</div>
-                        Beautiful view! Which trail is this?
-                    </div>
                     </div>
                 </div>
             </div>
 
-        <!-- Repeat similar structure with different content 20 times -->
+            <!-- Long post with "Read more" -->
-        <!-- This is post 2 -->
             <div class="post">
                 <div class="post-header">
-                <div class="profile-pic"></div>
+                    <img src="carol-avatar.jpg" alt="Carol's profile picture" class="profile-pic">
                     <div class="post-meta">
-                    <div class="username">Sarah Chen</div>
+                        <div class="username">Carol White</div>
                         <div class="timestamp">3 hours ago</div>
                     </div>
                 </div>
-            <div class="post-content">
+                <div class="post-content truncated">
-                Just launched my new portfolio website! 🚀 Excited to share my latest projects with everyone.
+                    Just finished reading an fascinating paper on artificial intelligence and its implications...
-                Check it out and let me know what you think! #WebDevelopment #Portfolio #Coding
+                    <button class="read-more">Read more</button>
                 </div>
-            <div class="post-actions">
+            </div>
-                <button class="action-button">Like</button>
+
-                <button class="action-button">Comment</button>
+            <!-- Multi-image post -->
-                <button class="action-button">Share</button>
+            <div class="post">
+                <div class="post-header">
+                    <img src="https://picsum.photos/100?random=4" alt="David's profile picture" class="profile-pic">
+                    <div class="post-meta">
+                        <div class="username">David Brown</div>
+                        <div class="timestamp">5 hours ago</div>
+                    </div>
+                </div>
+                <div class="post-content">
+                    Amazing day at the photography exhibition! Swipe to see more 📸
+                </div>
+                <div class="image-gallery">
+                    <img src="https://picsum.photos/800/800?random=5" alt="Exhibition photo 1" class="gallery-image">
+                    <img src="https://picsum.photos/800/800?random=6" alt="Exhibition photo 2" class="gallery-image">
+                    <img src="https://picsum.photos/800/800?random=7" alt="Exhibition photo 3" class="gallery-image">
+                    <button class="gallery-nav prev" aria-label="Previous image">←</button>
+                    <button class="gallery-nav next" aria-label="Next image">→</button>
                 </div>
             </div>
 
             <!-- Continue with more posts... -->
-        <!-- Posts 3-20 would follow similar pattern with varied content -->
+            <!-- Add 15-20 more posts with varying content -->
+        </section>
 
-    </main>
+        <!-- Right sidebar -->
+        <aside class="sidebar">
+            <!-- Online friends -->
+            <div class="friends-list">
+                <h2>Online Friends</h2>
+                <div class="friend">
+                    <img src="https://picsum.photos/100?random=8" alt="Emma's profile picture">
+                    <span>Emma Davis</span>
+                    <div class="online-indicator"></div>
+                </div>
+                <div class="friend">
+                    <img src="https://picsum.photos/100?random=9" alt="Frank Wilson's profile picture">
+                    <span>Frank Wilson</span>
+                    <div class="online-indicator"></div>
+                </div>
+                <!-- Add more online friends -->
+            </div>
+
+            <!-- Trending topics -->
+            <div class="trending">
+                <h2>Trending Topics</h2>
+                <div class="topic">#Photography</div>
+                <div class="topic">#TechNews</div>
+                <div class="topic">#Hiking</div>
+            </div>
+        </aside>
+    </div>
+
+    <!-- Chat widget -->
+    <div class="chat-widget">
+        <div class="chat-header">
+            <h3>Chat</h3>
+            <button class="minimize" aria-label="Minimize chat">−</button>
+        </div>
+        <div class="chat-messages">
+            <!-- Chat messages would go here -->
+        </div>
+        <input type="text" class="chat-input" placeholder="Type a message..." aria-label="Type a message">
+    </div>
+
+    <!-- Image preview modal -->
+    <div class="modal" id="image_preview" style="display: none;">
+        <img src="" alt="Large preview of selected image">
+        <button class="close-modal" aria-label="Close preview">×</button>
+    </div>
 </body>
 </html>