llm/SIA
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
63d1264f5ae71394bffd2a05d39ef7b1c627f6ef
SIA/tools/itb/example.md
geens 63d1264f5a Continue work on itb
2024-12-26 17:36:59 +01:00

11 KiB
Raw Blame History

Comprehensive ITB Social Media Interaction Example

Overview

This example demonstrates how an AI agent uses ITB to interact with a social media interface. It showcases ITB's key features including text collapsing, state management, and complex interactions. The scenario walks through a complete social media interaction flow, from content creation to user engagement.

User Request to SIA

"I want you to help me create a social media post about my morning hike. I have a photo at '/photos/sunrise.jpg' to share. First, search for similar hiking posts to see what hashtags are popular. Then create a new post with my photo and some engaging text. After posting, find and interact with other hiking-related posts - like them and leave encouraging comments. Finally, send a chat message to anyone who's online and interested in hiking."

SIA's Response and Actions

Step 1: Initialize Session

<reasoning>
First, I'll start an ITB session and establish continuous monitoring of the page state.
</reasoning>

<single>
itb_start "social.html"
</single>

Expected Output:

{
    "pid": 12345,
    "session_id": "550e8400-e29b-41d4-a716-446655440000",
    "status": "success"
}

<repeat>
itb_screenshot 550e8400-e29b-41d4-a716-446655440000 --wait_stable
</repeat>

Initial viewport state:

<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
    <!-- Navigation bar with collapsed text -->
    <text x="20" y="30">SocialConnect | Search posts... | Messages | Notifications</text>
    
    <!-- Interactive navigation elements -->
    <interactive accepts_text="true" x="200" y="30" width="300" height="30" 
                 placeholder="Search posts..." aria-label="Search posts"/>
    <interactive clickable="true" id="profile_menu" x="720" y="30" width="30" height="30">
        <img src="https://picsum.photos/100" alt="Your profile picture"/>
    </interactive>
    
    <!-- Post creation area -->
    <interactive accepts_text="true" x="340" y="100" width="400" height="100"
                 placeholder="What's on your mind?" aria-label="Create a post"/>
    <interactive accepts_file="true" id="post_images" x="340" y="130" width="200" height="30"
                 multiple="true" accept="image/*"/>
    <interactive clickable="true" x="340" y="160" width="80" height="30">Post</interactive>

    <!-- Feed posts with collapsed text content -->
    <text x="100" y="220">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
    <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="320" width="600" height="450"/>
    
    <!-- Interactive post actions -->
    <interactive clickable="true" x="340" y="520" width="80" height="30">Like</interactive>
    <interactive clickable="true" x="340" y="560" width="80" height="30">Comment</interactive>
    <interactive clickable="true" x="340" y="600" width="80" height="30">Share</interactive>

    <!-- Sidebar with collapsed text -->
    <text x="960" y="30">Online Friends</text>
    <interactive clickable="true" x="960" y="200" width="200" height="40">Emma Davis (Online)</interactive>
</viewport>

Step 2: Search for Hiking Posts

<reasoning>
I see the search box in the navigation bar. I'll click it and search for hiking-related posts to analyze common hashtags.
</reasoning>

<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 250 -y 30 --click
</single>

<single>
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "#hiking"
</single>

Updated viewport showing search results:

<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
    <!-- Navigation and search UI remain unchanged -->
    
    <!-- Search results header -->
    <text x="100" y="80">Search results for "#hiking" - 3 posts found</text>
    
    <!-- Collapsed search result posts -->
    <text x="100" y="120">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
    <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="220" width="600" height="450"/>
    
    <text x="100" y="720">David Brown - 1 day ago - Weekend hiking adventure! Found this hidden waterfall trail. #Hiking #Adventure #NaturePhotography</text>
    <img src="https://picsum.photos/800/600?random=3" alt="Waterfall on hiking trail" x="100" y="820" width="600" height="450"/>
</viewport>

Step 3: Create New Post

<reasoning>
Having analyzed the existing posts and popular hashtags, I'll now create our new post with the sunrise photo.
</reasoning>

<single>
itb_scroll 550e8400-e29b-41d4-a716-446655440000 -x 0 -y 0
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 100 --click
</single>

<single>
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews"
</single>

<single>
itb_upload 550e8400-e29b-41d4-a716-446655440000 -i "post_images" -f "/photos/sunrise.jpg"
</single>

Viewport showing post creation state:

<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
    <!-- Navigation and header remain unchanged -->
    
    <!-- Post creation area with entered content -->
    <text x="340" y="100">Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews</text>
    <img src="file:///photos/sunrise.jpg" alt="Uploaded sunrise photo" x="340" y="220" width="400" height="300"/>
    <interactive clickable="true" x="720" y="220" width="20" height="20" aria-label="Remove uploaded photo">×</interactive>
</viewport>

<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 160 --click
</single>

Viewport after post creation:

<viewport width="1024" height="800" scroll_x="0" scroll_y="0">
    <!-- Navigation remains unchanged -->
    
    <!-- Newly created post appears at top of feed -->
    <text x="100" y="120">You - Just now - Starting my day with an invigorating mountain hike! The colors of dawn painted the sky in stunning shades of orange and purple. Nothing beats the serenity of nature in the early morning. 🌄 #MorningHike #Nature #Wellness #SunriseViews</text>
    <img src="file:///photos/sunrise.jpg" alt="Uploaded sunrise photo" x="100" y="220" width="600" height="450"/>
</viewport>

Step 4: Interact with Related Posts

<reasoning>
Now I'll scroll to find Alice's hiking post and engage with it through likes and comments.
</reasoning>

<single>
itb_scroll 550e8400-e29b-41d4-a716-446655440000 --to-element "post-actions"
</single>

<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 520 --click
</single>

Viewport after liking:

<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
    <text x="100" y="120">Alice Johnson - 2 hours ago - Just finished my morning hike! The sunrise was absolutely breathtaking today. 🌄 #MorningHike #Nature #Wellness</text>
    <img src="https://picsum.photos/800/600?random=2" alt="Sunrise view from mountain top" x="100" y="220" width="600" height="450"/>
    <interactive clickable="true" x="340" y="520" width="80" height="30" state="active">Like</interactive>
</viewport>

<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 340 -y 580 --click
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "What a gorgeous view! I just did a morning hike too. The sunrise is always worth the early wake-up call! 🌅"
</single>

Step 5: Start a Chat

<reasoning>
I notice Emma Davis is online and has hiking photos. I'll initiate a conversation about hiking.
</reasoning>

<single>
itb_cursor 550e8400-e29b-41d4-a716-446655440000 -x 960 -y 200 --click
</single>

Viewport showing chat window:

<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
    <!-- Main content remains with added chat widget -->
    <interactive class="chat-widget" x="744" y="400" width="280" height="400">
        <text x="754" y="420">Emma Davis (Online)</text>
        <interactive accepts_text="true" x="754" y="760" width="260" height="30"
                     placeholder="Type a message..." aria-label="Type a message"/>
    </interactive>
</viewport>

<single>
itb_input 550e8400-e29b-41d4-a716-446655440000 -t "Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!"
</single>

Final viewport state:

<viewport width="1024" height="800" scroll_x="0" scroll_y="600">
    <!-- Chat widget with sent message -->
    <interactive class="chat-widget" x="744" y="400" width="280" height="400">
        <text x="754" y="420">Emma Davis (Online)</text>
        <text x="754" y="480">You - Just now - Hi Emma! I noticed you're into hiking too. Would you be interested in joining for a sunrise hike sometime? I just discovered some beautiful trails!</text>
        <interactive accepts_text="true" x="754" y="760" width="260" height="30"
                     placeholder="Type a message..." aria-label="Type a message"/>
    </interactive>
</viewport>

<reasoning>
I've completed all requested interactions:
1. Analyzed existing hiking posts and hashtags
2. Created a new post with photo and appropriate hashtags
3. Engaged with Alice's hiking post through a like and comment
4. Initiated a conversation with Emma about hiking
The continuous screenshot monitoring remains active to catch any responses.
</reasoning>

Key Features Demonstrated

This example illustrates several important ITB features:

  1. Text Collapsing: Related text elements are automatically combined into natural reading flows, such as "Author - Time - Content" for posts.

  2. State Management: The viewport maintains awareness of scroll position, focus state, and interactive element states throughout the session.

  3. Interactive Elements: The example shows how ITB handles various types of interaction including:

    • Text input fields
    • File uploads
    • Clickable buttons
    • Scrollable areas
    • Modal windows (chat widget)

  4. Media Handling: Images are represented with their position, size, and alt text while maintaining proper layout relationships with surrounding content.

  5. Dynamic Content: The example demonstrates how ITB handles dynamic content updates like:

    • Search results loading
    • Post creation
    • Chat window opening
    • Comment threading

This comprehensive example shows how ITB transforms complex web interfaces into a format that AI agents can effectively process and interact with while maintaining all the functionality a human user would have.

Reference in New Issue View Git Blame Copy Permalink