llm/SIA
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Cleanup of itb readme

Browse Source
This commit is contained in:
geens
2024-12-26 18:19:27 +01:00
parent 63d1264f5a
commit fe72969b4f
1 changed files with 103 additions and 136 deletions
Download Patch File Download Diff File Expand all files Collapse all files

239
tools/itb/README.md
View File

@@ -12,30 +12,6 @@ ITB (Interactive Text Browser) bridges this gap. It transforms web pages into a
ITB operates as a bridge between an AI agent and a web browser. When you start ITB, it launches a browser process that runs independently, just like when you open a browser yourself. This browser loads and renders web pages normally, but instead of displaying them on screen, it communicates their content and state to the AI agent through a simplified XML format. ITB operates as a bridge between an AI agent and a web browser. When you start ITB, it launches a browser process that runs independently, just like when you open a browser yourself. This browser loads and renders web pages normally, but instead of displaying them on screen, it communicates their content and state to the AI agent through a simplified XML format.
Most of ITB's work happens through JavaScript that runs directly in the browser. Rather than making many small requests back and forth between Python and the browser, ITB bundles operations together into efficient JavaScript operations. This approach significantly improves performance while making the system more reliable.
Consider a typical login form interaction. Instead of separate commands to find fields, click them, and type text, ITB might execute a single JavaScript operation that handles the entire interaction:
```javascript
// Single efficient operation instead of multiple commands
function fillLoginForm(username, password) {
const usernameField = document.getElementById('username');
const passwordField = document.getElementById('password');
// Perform all operations in one go
usernameField.focus();
usernameField.value = username;
passwordField.focus();
passwordField.value = password;
return {
username_entered: true,
password_entered: true,
submit_button_enabled: document.querySelector('button[type="submit"]').disabled === false
};
}
```
## Page Representation ## Page Representation
ITB transforms complex HTML structures into a simplified XML format that represents what's actually visible and interactive on the page. Let's look at several examples to understand this transformation. ITB transforms complex HTML structures into a simplified XML format that represents what's actually visible and interactive on the page. Let's look at several examples to understand this transformation.
@@ -57,7 +33,7 @@ When a human reads a webpage, they naturally process related content together re
<!-- ITB Output --> <!-- ITB Output -->
<viewport width="1024" height="800" scroll_x="0" scroll_y="0"> <viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<text x="20" y="40">Breaking News - 2024-12-20 - By John Smith - Scientists announce breakthrough...</text> <text x="20" y="40" width="1024" height="106">Breaking News - 2024-12-20 - By John Smith - Scientists announce breakthrough...</text>
</viewport> </viewport>
``` ```
@@ -79,11 +55,11 @@ ITB describes elements by their interaction capabilities rather than their HTML
<!-- ITB Output --> <!-- ITB Output -->
<viewport width="1024" height="800" scroll_x="0" scroll_y="0"> <viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<text x="20" y="40">Username:</text> <text x="0" y="0" width="77" height="20">Username:</text>
<interactive accepts_text="true" id="username" x="20" y="70" width="200" height="30"/> <interactive x="78" y="0" width="213" height="20" accepts_text="true" id="username" x="20" y="70" width="200" height="30"/>
<text x="20" y="120">Password:</text> <text x="292" y="0" width="73" height="20">Password:</text>
<interactive accepts_text="true" password="true" id="password" x="20" y="150" width="200" height="30"/> <interactive x="365" y="0" width="212" height="20" accepts_text="true" password="true" id="password" x="20" y="150" width="200" height="30"/>
<interactive clickable="true" x="20" y="200" width="80" height="40">Login</interactive> <interactive x="578" y="0" width="40" height="20" clickable="true" x="20" y="200" width="80" height="40">Login</interactive>
</viewport> </viewport>
``` ```
@@ -94,7 +70,7 @@ Media elements like images, videos, and audio are represented simply with their
```html ```html
<!-- Original HTML --> <!-- Original HTML -->
<div class="media-content"> <div class="media-content">
<img src="diagram.png" alt="System architecture diagram" class="full-width"> <img src="diagram.png" alt="System architecture diagram"/>
<video src="demo.mp4" controls>Video demonstration of the system</video> <video src="demo.mp4" controls>Video demonstration of the system</video>
<audio src="recording.mp3" controls>Conference presentation audio</audio> <audio src="recording.mp3" controls>Conference presentation audio</audio>
</div> </div>
@@ -120,7 +96,7 @@ Modern web pages often embed content from other sources using iframes. While hum
<!-- ITB Output --> <!-- ITB Output -->
<viewport width="1024" height="800" scroll_x="0" scroll_y="0"> <viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<text x="20" y="40">Welcome Back</text> <text x="20" y="40" width="203" height="20">Welcome Back</text>
<iframe src="https://auth.service.com/login" x="20" y="80" width="400" height="300"/> <iframe src="https://auth.service.com/login" x="20" y="80" width="400" height="300"/>
</viewport> </viewport>
``` ```
@@ -140,7 +116,7 @@ Just as humans need to know where their mouse pointer is and which text field th
caret_position="5" caret_position="5"
selection_start="2" selection_end="5" selection_start="2" selection_end="5"
> >
<text x="20" y="30">Username:</text> <text x="20" y="30" width="174" height="20">Username:</text>
<interactive accepts_text="true" id="username" x="100" y="30" width="200" height="30"> <interactive accepts_text="true" id="username" x="100" y="30" width="200" height="30">
alice alice
</interactive> </interactive>
@@ -186,10 +162,10 @@ Starts a new browser session and returns:
### Reading Page State ### Reading Page State
```bash ```bash
itb_screenshot <session_id> [--wait_stable] [--timeout 5] itb_screenshot <session_id> [--no_wait_stable] [--timeout 1]
``` ```
Captures the current page state in ITB's XML format. The `--wait_stable` option waits for the DOM to stop changing before capturing, useful for dynamic content. ITB detects stability by monitoring DOM mutations. Captures the current page state in ITB's XML format. By default waits for the DOM to stop changing before capturing, useful for dynamic content. ITB detects stability by monitoring DOM mutations. Default timeout is 1 second.
### Controlling the Mouse Pointer ### Controlling the Mouse Pointer
@@ -199,6 +175,12 @@ itb_cursor <session_id> -x 150 -y 45
# Move and click # Move and click
itb_cursor <session_id> -x 150 -y 45 --click itb_cursor <session_id> -x 150 -y 45 --click
# Right click
itb_cursor <session_id> --right
# Double click by id
itb_cursor <session_id> --double --id "submit_button"
``` ```
### Text Input ### Text Input
@@ -233,9 +215,6 @@ itb_scroll <session_id> -x 0 -y 500
# Relative to current position # Relative to current position
itb_scroll <session_id> --relative -x 0 -y 100 itb_scroll <session_id> --relative -x 0 -y 100
# To specific element
itb_scroll <session_id> --to-element "section_3"
``` ```
### Navigation ### Navigation
@@ -296,7 +275,7 @@ driver.execute_script("""
Text content should be merged when it forms a natural reading flow: Text content should be merged when it forms a natural reading flow:
- Adjacent text elements should be combined - Adjacent text elements should be combined
- Maintain accurate positioning information - Maintain accurate positioning information
- Preserve text exactly as it appears to users - Preserve all text but no formatting
### Interactive Elements ### Interactive Elements
@@ -584,42 +563,52 @@ Expected Output:
```xml ```xml
<repeat> <repeat>
itb_screenshot 550e8400-e29b-41d4-a716-446655440000 --wait_stable itb_screenshot 550e8400-e29b-41d4-a716-446655440000
</repeat> </repeat>
``` ```
Initial viewport state: Initial viewport state:
```xml ```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="0"> <viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<!-- Navigation bar with collapsed text --> <!-- Navigation bar with properly structured elements -->
<text x="20" y="30">SocialConnect | Search posts... | Messages | Notifications</text> <text x="16" y="20" width="120" height="40">SocialConnect</text>
<interactive accepts_text="true" x="152" y="15" width="300" height="40"
<!-- Interactive navigation elements -->
<interactive accepts_text="true" x="200" y="30" width="300" height="30"
placeholder="Search posts..." aria-label="Search posts"/> placeholder="Search posts..." aria-label="Search posts"/>
<interactive clickable="true" id="profile_menu" x="720" y="30" width="30" height="30"> <interactive clickable="true" x="468" y="15" width="100" height="40"
<img src="https://picsum.photos/100" alt="Your profile picture"/> aria-label="Open messages">Messages</interactive>
<interactive clickable="true" x="584" y="15" width="120" height="40"
aria-label="View notifications">Notifications</interactive>
<interactive clickable="true" id="profile_menu" x="720" y="15" width="40" height="40">
<img src="https://picsum.photos/100" alt="Your profile picture" width="40" height="40"/>
</interactive> </interactive>
<!-- Post creation area --> <!-- Main content area -->
<interactive accepts_text="true" x="340" y="100" width="400" height="100" <div class="main-content" x="0" y="60" width="1024" height="740">
placeholder="What's on your mind?" aria-label="Create a post"/> <!-- Post creation area -->
<interactive accepts_file="true" id="post_images" x="340" y="130" width="200" height="30" <interactive accepts_text="true" x="20" y="80" width="680" height="100"
multiple="true" accept="image/*"/> placeholder="What's on your mind?" aria-label="Create a post"/>
<interactive clickable="true" x="340" y="160" width="80" height="30">Post</interactive> <interactive accepts_file="true" id="post_images" x="20" y="190" width="200" height="40"
multiple="true" accept="image/*" aria-label="Add images to your post"/>
<interactive clickable="true" x="20" y="240" width="100" height="40">Post</interactive>
<!-- Feed posts with collapsed text content --> <!-- Feed posts -->
<text x="100" y="220">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text> <text x="20" y="300" width="680" height="60">Alice Johnson - 2 hours ago</text>
<img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="320" width="600" height="450"/> <text x="20" y="360" width="680" height="80">Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
<img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top"
<!-- Interactive post actions --> x="20" y="450" width="680" height="510"/>
<interactive clickable="true" x="340" y="520" width="80" height="30">Like</interactive> <interactive clickable="true" x="20" y="970" width="100" height="40">Like</interactive>
<interactive clickable="true" x="340" y="560" width="80" height="30">Comment</interactive> <interactive clickable="true" x="130" y="970" width="100" height="40">Comment</interactive>
<interactive clickable="true" x="340" y="600" width="80" height="30">Share</interactive> <interactive clickable="true" x="240" y="970" width="100" height="40">Share</interactive>
<!-- Sidebar with collapsed text --> <!-- Sidebar -->
<text x="960" y="30">Online Friends</text> <text x="720" y="80" width="284" height="40">Online Friends</text>
<interactive clickable="true" x="960" y="200" width="200" height="40">Emma Davis (Online)</interactive> <interactive clickable="true" x="720" y="130" width="284" height="60">
<img src="https://picsum.photos/100?random=8" alt="Emma's profile picture"
width="40" height="40"/>
<text x="770" y="140" width="200" height="40">Emma Davis</text>
<div class="online-indicator" x="980" y="140" width="10" height="10"/>
</interactive>
</div>
</viewport> </viewport>
``` ```
@@ -627,11 +616,11 @@ Initial viewport state:
```xml ```xml
<reasoning> <reasoning>
I see the search box in the navigation bar. I'll click it and search for hiking-related posts to analyze common hashtags. I'll use the search box in the navigation bar to find hiking-related posts and analyze hashtag patterns.
</reasoning> </reasoning>
<single> <single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 250 -y 30 --click itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 302 -y 35 --click
</single> </single>
<single> <single>
@@ -642,17 +631,25 @@ itb_input 550e8400-e29b-41d4-a716-446655440000 -t "#hiking"
Updated viewport showing search results: Updated viewport showing search results:
```xml ```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="0"> <viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<!-- Navigation and search UI remain unchanged --> <!-- Navigation bar remains unchanged -->
<!-- Search results header --> <!-- Search results -->
<text x="100" y="80">Search results for "#hiking" - 3 posts found</text> <text x="20" y="80" width="680" height="40">Search results for "#hiking" - 3 posts found</text>
<!-- Collapsed search result posts --> <!-- Search result posts -->
<text x="100" y="120">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text> <div class="post" x="20" y="140" width="680" height="600">
<img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="220" width="600" height="450"/> <text x="20" y="140" width="680" height="60">Alice Johnson - 2 hours ago</text>
<text x="20" y="200" width="680" height="80">Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
<img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top"
x="20" y="290" width="680" height="510"/>
</div>
<text x="100" y="720">David Brown - 1 day ago - Weekend hiking adventure! Found this hidden waterfall trail. #Hiking #Adventure #NaturePhotography</text> <div class="post" x="20" y="760" width="680" height="600">
<img src="https://picsum.photos/800/600?random=3" alt="Waterfall on hiking trail" x="100" y="820" width="600" height="450"/> <text x="20" y="760" width="680" height="60">David Brown - 1 day ago</text>
<text x="20" y="820" width="680" height="80">Weekend hiking adventure! Found this hidden waterfall trail. #Hiking #Adventure #NaturePhotography</text>
<img src="https://picsum.photos/800/600?random=3" alt="Waterfall on hiking trail"
x="20" y="910" width="680" height="510"/>
</div>
</viewport> </viewport>
``` ```
@@ -660,12 +657,15 @@ Updated viewport showing search results:
```xml ```xml
<reasoning> <reasoning>
Having analyzed the existing posts and popular hashtags, I'll now create our new post with the sunrise photo. Now I'll create our new post with the sunrise photo, incorporating popular hashtags from existing posts.
</reasoning> </reasoning>
<single> <single>
itb_scroll 550e8400-e29b-41d4-a716-446655440000 -x 0 -y 0 itb_scroll 550e8400-e29b-41d4-a716-446655440000 -x 0 -y 0
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 100 --click </single>
<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 360 -y 130 --click
</single> </single>
<single> <single>
@@ -675,108 +675,75 @@ itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Starting my day with an invig
<single> <single>
itb_upload 550e8400-e29b-41d4-a716-446655440000 -i "post_images" -f "/photos/sunrise.jpg" itb_upload 550e8400-e29b-41d4-a716-446655440000 -i "post_images" -f "/photos/sunrise.jpg"
</single> </single>
```
Viewport showing post creation state:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<!-- Navigation and header remain unchanged -->
<!-- Post creation area with entered content -->
<text x="340" y="100">Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews</text>
<img src="file:///photos/sunrise.jpg" alt="Uploaded sunrise photo" x="340" y="220" width="400" height="300"/>
<interactive clickable="true" x="720" y="220" width="20" height="20" aria-label="Remove uploaded photo">×</interactive>
</viewport>
```
```xml
<single> <single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 160 --click itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 70 -y 260 --click
</single> </single>
```
Viewport after post creation:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
<!-- Navigation remains unchanged -->
<!-- Newly created post appears at top of feed -->
<text x="100" y="120">You - Just now - Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews</text>
<img src="file:///photos/sunrise.jpg" alt="Uploaded sunrise photo" x="100" y="220" width="600" height="450"/>
</viewport>
```
### Step 4: Interact with Related Posts ### Step 4: Interact with Related Posts
```xml ```xml
<reasoning> <reasoning>
Now I'll scroll to find Alice's hiking post and engage with it through likes and comments. I'll engage with Alice's hiking post through likes and comments.
</reasoning> </reasoning>
<single> <single>
itb_scroll 550e8400-e29b-41d4-a716-446655440000 --to-element "post-actions" itb_scroll 550e8400-e29b-41d4-a716-446655440000 -x 0 -y 600
</single> </single>
<single> <single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 520 --click itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 70 -y 970 --click
</single>
<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 180 -y 970 --click
</single> </single>
```
Viewport after liking:
```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
<text x="100" y="120">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
<img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="220" width="600" height="450"/>
<interactive clickable="true" x="340" y="520" width="80" height="30" state="active">Like</interactive>
</viewport>
```
```xml
<single> <single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 580 --click
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "What a gorgeous view! I just did a morning hike too. The sunrise is always worth the early wake-up call! 🌅" itb_input 550e8400-e29b-41d4-a716-446655440000 -t "What a gorgeous view! I just did a morning hike too. The sunrise is always worth the early wake-up call! 🌅"
</single> </single>
```
### Step 5: Start a Chat ### Step 5: Start a Chat
```xml ```xml
<reasoning> <reasoning>
I notice Emma Davis is online and has hiking photos. I'll initiate a conversation about hiking. I'll initiate a conversation with Emma Davis about hiking.
</reasoning> </reasoning>
<single> <single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 960 -y 200 --click itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 862 -y 160 --click
</single> </single>
```
Viewport showing chat window: Viewport showing chat widget:
```xml ```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="600"> <viewport width="1024" height="800" scroll_x="0" scroll_y="600">
<!-- Main content remains with added chat widget --> <!-- Main content remains unchanged -->
<interactive class="chat-widget" x="744" y="400" width="280" height="400">
<text x="754" y="420">Emma Davis (Online)</text> <!-- Chat widget -->
<interactive accepts_text="true" x="754" y="760" width="260" height="30" <div class="chat-widget" x="744" y="400" width="280" height="400">
<text x="754" y="420" width="260" height="40">Emma Davis (Online)</text>
<div class="chat-messages" x="754" y="470" width="260" height="280"/>
<interactive accepts_text="true" x="754" y="760" width="260" height="40"
placeholder="Type a message..." aria-label="Type a message"/> placeholder="Type a message..." aria-label="Type a message"/>
</interactive> </div>
</viewport> </viewport>
``` ```
```xml
<single> <single>
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!" itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!"
</single> </single>
```
Final viewport state: Final viewport state:
```xml ```xml
<viewport width="1024" height="800" scroll_x="0" scroll_y="600"> <viewport width="1024" height="800" scroll_x="0" scroll_y="600">
<!-- Chat widget with sent message --> <!-- Chat widget with sent message -->
<interactive class="chat-widget" x="744" y="400" width="280" height="400"> <div class="chat-widget" x="744" y="400" width="280" height="400">
<text x="754" y="420">Emma Davis (Online)</text> <text x="754" y="420" width="260" height="40">Emma Davis (Online)</text>
<text x="754" y="480">You - Just now - Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!</text> <div class="chat-messages" x="754" y="470" width="260" height="280">
<interactive accepts_text="true" x="754" y="760" width="260" height="30" <text x="754" y="480" width="260" height="80">You - Just now - Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!</text>
</div>
<interactive accepts_text="true" x="754" y="760" width="260" height="40"
placeholder="Type a message..." aria-label="Type a message"/> placeholder="Type a message..." aria-label="Type a message"/>
</interactive> </div>
</viewport> </viewport>
``` ```