llm/SIA
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Cleanup of itb readme

Browse Source
This commit is contained in:
geens
2024-12-26 18:19:27 +01:00
parent 63d1264f5a
commit fe72969b4f
1 changed files with 103 additions and 136 deletions
Download Patch File Download Diff File Expand all files Collapse all files

237
tools/itb/README.md
View File

@@ -12,30 +12,6 @@ ITB (Interactive Text Browser) bridges this gap. It transforms web pages into a
ITB operates as a bridge between an AI agent and a web browser. When you start ITB, it launches a browser process that runs independently, just like when you open a browser yourself. This browser loads and renders web pages normally, but instead of displaying them on screen, it communicates their content and state to the AI agent through a simplified XML format.
Most of ITB's work happens through JavaScript that runs directly in the browser. Rather than making many small requests back and forth between Python and the browser, ITB bundles operations together into efficient JavaScript operations. This approach significantly improves performance while making the system more reliable.
Consider a typical login form interaction. Instead of separate commands to find fields, click them, and type text, ITB might execute a single JavaScript operation that handles the entire interaction:
```javascript
// Single efficient operation instead of multiple commands
function fillLoginForm(username, password) {
const usernameField = document.getElementById('username');
const passwordField = document.getElementById('password');
// Perform all operations in one go
usernameField.focus();
usernameField.value = username;
passwordField.focus();
passwordField.value = password;
return {
username_entered: true,
password_entered: true,
submit_button_enabled: document.querySelector('button[type="submit"]').disabled === false
};
}
```
## Page Representation
ITB transforms complex HTML structures into a simplified XML format that represents what's actually visible and interactive on the page. Let's look at several examples to understand this transformation.
@@ -57,7 +33,7 @@ When a human reads a webpage, they naturally process related content together re
<!-- ITB Output -->
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<text x="20" y="40">Breaking News - 2024-12-20 - By John Smith - Scientists announce breakthrough...</text>
<text x="20" y="40" width="1024" height="106">Breaking News - 2024-12-20 - By John Smith - Scientists announce breakthrough...</text>
</viewport>
```
@@ -79,11 +55,11 @@ ITB describes elements by their interaction capabilities rather than their HTML
<!-- ITB Output -->
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<text x="20" y="40">Username:</text>
<interactive accepts_text="true" id="username" x="20" y="70" width="200" height="30"/>
<text x="20" y="120">Password:</text>
<interactive accepts_text="true" password="true" id="password" x="20" y="150" width="200" height="30"/>
<interactive clickable="true" x="20" y="200" width="80" height="40">Login</interactive>
<text x="0" y="0" width="77" height="20">Username:</text>
<interactive x="78" y="0" width="213" height="20" accepts_text="true" id="username" x="20" y="70" width="200" height="30"/>
<text x="292" y="0" width="73" height="20">Password:</text>
<interactive x="365" y="0" width="212" height="20" accepts_text="true" password="true" id="password" x="20" y="150" width="200" height="30"/>
<interactive x="578" y="0" width="40" height="20" clickable="true" x="20" y="200" width="80" height="40">Login</interactive>
</viewport>
```
@@ -94,7 +70,7 @@ Media elements like images, videos, and audio are represented simply with their
```html
<!-- Original HTML -->
<div class="media-content">
<img src="diagram.png" alt="System architecture diagram" class="full-width">
<img src="diagram.png" alt="System architecture diagram"/>
<video src="demo.mp4" controls>Video demonstration of the system</video>
<audio src="recording.mp3" controls>Conference presentation audio</audio>
</div>
@@ -120,7 +96,7 @@ Modern web pages often embed content from other sources using iframes. While hum
<!-- ITB Output -->
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<text x="20" y="40">Welcome Back</text>
<text x="20" y="40" width="203" height="20">Welcome Back</text>
<iframe src="https://auth.service.com/login" x="20" y="80" width="400" height="300"/>
</viewport>
```
@@ -140,7 +116,7 @@ Just as humans need to know where their mouse pointer is and which text field th
caret_position="5"
selection_start="2" selection_end="5"
>
<text x="20" y="30">Username:</text>
<text x="20" y="30" width="174" height="20">Username:</text>
<interactive accepts_text="true" id="username" x="100" y="30" width="200" height="30">
alice
</interactive>
@@ -186,10 +162,10 @@ Starts a new browser session and returns:
### Reading Page State
```bash
itb_screenshot <session_id> [--wait_stable] [--timeout 5]
itb_screenshot <session_id> [--no_wait_stable] [--timeout 1]
```
Captures the current page state in ITB's XML format. The `--wait_stable` option waits for the DOM to stop changing before capturing, useful for dynamic content. ITB detects stability by monitoring DOM mutations.
Captures the current page state in ITB's XML format. By default waits for the DOM to stop changing before capturing, useful for dynamic content. ITB detects stability by monitoring DOM mutations. Default timeout is 1 second.
### Controlling the Mouse Pointer
@@ -199,6 +175,12 @@ itb_cursor <session_id> -x 150 -y 45
# Move and click
itb_cursor <session_id> -x 150 -y 45 --click
# Right click
itb_cursor <session_id> --right
# Double click by id
itb_cursor <session_id> --double --id "submit_button"
```
### Text Input
@@ -233,9 +215,6 @@ itb_scroll <session_id> -x 0 -y 500
# Relative to current position
itb_scroll <session_id> --relative -x 0 -y 100
# To specific element
itb_scroll <session_id> --to-element "section_3"
```
### Navigation
@@ -296,7 +275,7 @@ driver.execute_script("""
Text content should be merged when it forms a natural reading flow:
- Adjacent text elements should be combined
- Maintain accurate positioning information
- Preserve text exactly as it appears to users
- Preserve all text but no formatting
### Interactive Elements
@@ -584,42 +563,52 @@ Expected Output:
```xml
<repeat>
itb_screenshot 550e8400-e29b-41d4-a716-446655440000 --wait_stable
itb_screenshot 550e8400-e29b-41d4-a716-446655440000
</repeat>
```
Initial viewport state:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<!-- Navigation bar with collapsed text -->
<text x="20" y="30">SocialConnect | Search posts... | Messages | Notifications</text>
<!-- Interactive navigation elements -->
<interactive accepts_text="true" x="200" y="30" width="300" height="30"
<!-- Navigation bar with properly structured elements -->
<text x="16" y="20" width="120" height="40">SocialConnect</text>
<interactive accepts_text="true" x="152" y="15" width="300" height="40"
placeholder="Search posts..." aria-label="Search posts"/>
<interactive clickable="true" id="profile_menu" x="720" y="30" width="30" height="30">
<img src="https://picsum.photos/100" alt="Your profile picture"/>
<interactive clickable="true" x="468" y="15" width="100" height="40"
aria-label="Open messages">Messages</interactive>
<interactive clickable="true" x="584" y="15" width="120" height="40"
aria-label="View notifications">Notifications</interactive>
<interactive clickable="true" id="profile_menu" x="720" y="15" width="40" height="40">
<img src="https://picsum.photos/100" alt="Your profile picture" width="40" height="40"/>
</interactive>
<!-- Post creation area -->
<interactive accepts_text="true" x="340" y="100" width="400" height="100"
placeholder="What's on your mind?" aria-label="Create a post"/>
<interactive accepts_file="true" id="post_images" x="340" y="130" width="200" height="30"
multiple="true" accept="image/*"/>
<interactive clickable="true" x="340" y="160" width="80" height="30">Post</interactive>
<!-- Main content area -->
<div class="main-content" x="0" y="60" width="1024" height="740">
<!-- Post creation area -->
<interactive accepts_text="true" x="20" y="80" width="680" height="100"
placeholder="What's on your mind?" aria-label="Create a post"/>
<interactive accepts_file="true" id="post_images" x="20" y="190" width="200" height="40"
multiple="true" accept="image/*" aria-label="Add images to your post"/>
<interactive clickable="true" x="20" y="240" width="100" height="40">Post</interactive>
<!-- Feed posts with collapsed text content -->
<text x="100" y="220">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
<img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="320" width="600" height="450"/>
<!-- Feed posts -->
<text x="20" y="300" width="680" height="60">Alice Johnson - 2 hours ago</text>
<text x="20" y="360" width="680" height="80">Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
<img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top"
x="20" y="450" width="680" height="510"/>
<interactive clickable="true" x="20" y="970" width="100" height="40">Like</interactive>
<interactive clickable="true" x="130" y="970" width="100" height="40">Comment</interactive>
<interactive clickable="true" x="240" y="970" width="100" height="40">Share</interactive>
<!-- Interactive post actions -->
<interactive clickable="true" x="340" y="520" width="80" height="30">Like</interactive>
<interactive clickable="true" x="340" y="560" width="80" height="30">Comment</interactive>
<interactive clickable="true" x="340" y="600" width="80" height="30">Share</interactive>
<!-- Sidebar with collapsed text -->
<text x="960" y="30">Online Friends</text>
<interactive clickable="true" x="960" y="200" width="200" height="40">Emma Davis (Online)</interactive>
<!-- Sidebar -->
<text x="720" y="80" width="284" height="40">Online Friends</text>
<interactive clickable="true" x="720" y="130" width="284" height="60">
<img src="https://picsum.photos/100?random=8" alt="Emma's profile picture"
width="40" height="40"/>
<text x="770" y="140" width="200" height="40">Emma Davis</text>
<div class="online-indicator" x="980" y="140" width="10" height="10"/>
</interactive>
</div>
</viewport>
```
@@ -627,11 +616,11 @@ Initial viewport state:
```xml
<reasoning>
I see the search box in the navigation bar. I'll click it and search for hiking-related posts to analyze common hashtags.
I'll use the search box in the navigation bar to find hiking-related posts and analyze hashtag patterns.
</reasoning>
<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 250 -y 30 --click
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 302 -y 35 --click
</single>
<single>
@@ -642,17 +631,25 @@ itb_input 550e8400-e29b-41d4-a716-446655440000 -t "#hiking"
Updated viewport showing search results:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<!-- Navigation and search UI remain unchanged -->
<!-- Navigation bar remains unchanged -->
<!-- Search results header -->
<text x="100" y="80">Search results for "#hiking" - 3 posts found</text>
<!-- Search results -->
<text x="20" y="80" width="680" height="40">Search results for "#hiking" - 3 posts found</text>
<!-- Collapsed search result posts -->
<text x="100" y="120">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
<img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="220" width="600" height="450"/>
<!-- Search result posts -->
<div class="post" x="20" y="140" width="680" height="600">
<text x="20" y="140" width="680" height="60">Alice Johnson - 2 hours ago</text>
<text x="20" y="200" width="680" height="80">Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
<img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top"
x="20" y="290" width="680" height="510"/>
</div>
<text x="100" y="720">David Brown - 1 day ago - Weekend hiking adventure! Found this hidden waterfall trail. #Hiking #Adventure #NaturePhotography</text>
<img src="https://picsum.photos/800/600?random=3" alt="Waterfall on hiking trail" x="100" y="820" width="600" height="450"/>
<div class="post" x="20" y="760" width="680" height="600">
<text x="20" y="760" width="680" height="60">David Brown - 1 day ago</text>
<text x="20" y="820" width="680" height="80">Weekend hiking adventure! Found this hidden waterfall trail. #Hiking #Adventure #NaturePhotography</text>
<img src="https://picsum.photos/800/600?random=3" alt="Waterfall on hiking trail"
x="20" y="910" width="680" height="510"/>
</div>
</viewport>
```
@@ -660,12 +657,15 @@ Updated viewport showing search results:
```xml
<reasoning>
Having analyzed the existing posts and popular hashtags, I'll now create our new post with the sunrise photo.
Now I'll create our new post with the sunrise photo, incorporating popular hashtags from existing posts.
</reasoning>
<single>
itb_scroll 550e8400-e29b-41d4-a716-446655440000 -x 0 -y 0
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 100 --click
</single>
<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 360 -y 130 --click
</single>
<single>
@@ -675,108 +675,75 @@ itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Starting my day with an invig
<single>
itb_upload 550e8400-e29b-41d4-a716-446655440000 -i "post_images" -f "/photos/sunrise.jpg"
</single>
```
Viewport showing post creation state:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<!-- Navigation and header remain unchanged -->
<!-- Post creation area with entered content -->
<text x="340" y="100">Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews</text>
<img src="file:///photos/sunrise.jpg" alt="Uploaded sunrise photo" x="340" y="220" width="400" height="300"/>
<interactive clickable="true" x="720" y="220" width="20" height="20" aria-label="Remove uploaded photo">×</interactive>
</viewport>
```
```xml
<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 160 --click
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 70 -y 260 --click
</single>
```
Viewport after post creation:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<!-- Navigation remains unchanged -->
<!-- Newly created post appears at top of feed -->
<text x="100" y="120">You - Just now - Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews</text>
<img src="file:///photos/sunrise.jpg" alt="Uploaded sunrise photo" x="100" y="220" width="600" height="450"/>
</viewport>
```
### Step 4: Interact with Related Posts
```xml
<reasoning>
Now I'll scroll to find Alice's hiking post and engage with it through likes and comments.
I'll engage with Alice's hiking post through likes and comments.
</reasoning>
<single>
itb_scroll 550e8400-e29b-41d4-a716-446655440000 --to-element "post-actions"
itb_scroll 550e8400-e29b-41d4-a716-446655440000 -x 0 -y 600
</single>
<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 520 --click
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 70 -y 970 --click
</single>
<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 180 -y 970 --click
</single>
```
Viewport after liking:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
<text x="100" y="120">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
<img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="220" width="600" height="450"/>
<interactive clickable="true" x="340" y="520" width="80" height="30" state="active">Like</interactive>
</viewport>
```
```xml
<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 580 --click
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "What a gorgeous view! I just did a morning hike too. The sunrise is always worth the early wake-up call! 🌅"
</single>
```
### Step 5: Start a Chat
```xml
<reasoning>
I notice Emma Davis is online and has hiking photos. I'll initiate a conversation about hiking.
I'll initiate a conversation with Emma Davis about hiking.
</reasoning>
<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 960 -y 200 --click
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 862 -y 160 --click
</single>
```
Viewport showing chat window:
Viewport showing chat widget:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
<!-- Main content remains with added chat widget -->
<interactive class="chat-widget" x="744" y="400" width="280" height="400">
<text x="754" y="420">Emma Davis (Online)</text>
<interactive accepts_text="true" x="754" y="760" width="260" height="30"
<!-- Main content remains unchanged -->
<!-- Chat widget -->
<div class="chat-widget" x="744" y="400" width="280" height="400">
<text x="754" y="420" width="260" height="40">Emma Davis (Online)</text>
<div class="chat-messages" x="754" y="470" width="260" height="280"/>
<interactive accepts_text="true" x="754" y="760" width="260" height="40"
placeholder="Type a message..." aria-label="Type a message"/>
</interactive>
</div>
</viewport>
```
```xml
<single>
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!"
</single>
```
Final viewport state:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
<!-- Chat widget with sent message -->
<interactive class="chat-widget" x="744" y="400" width="280" height="400">
<text x="754" y="420">Emma Davis (Online)</text>
<text x="754" y="480">You - Just now - Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!</text>
<interactive accepts_text="true" x="754" y="760" width="260" height="30"
<div class="chat-widget" x="744" y="400" width="280" height="400">
<text x="754" y="420" width="260" height="40">Emma Davis (Online)</text>
<div class="chat-messages" x="754" y="470" width="260" height="280">
<text x="754" y="480" width="260" height="80">You - Just now - Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!</text>
</div>
<interactive accepts_text="true" x="754" y="760" width="260" height="40"
placeholder="Type a message..." aria-label="Type a message"/>
</interactive>
</div>
</viewport>
```