152 lines
3.3 KiB
Markdown
152 lines
3.3 KiB
Markdown
# ITB: The Interactive Text Browser
|
|
|
|
ITB aims to create a set of Python scripts to enable a text-based AI agent to interact with web pages.
|
|
The system is designed to start a detached browser process and interact with it through subsequent scripts.
|
|
Each script performs a specific action, such as starting the browser, taking a virtual screenshot, scrolling, clicking, and inputting text.
|
|
The output of these scripts is optimized to be evaluated by a Large Language Model (LLM).
|
|
So it can understand and interact with the webpage in a text-only manner.
|
|
|
|
# The Scripts
|
|
|
|
## itb_start
|
|
|
|
Starts a detached browser session and outputs the necessary information to interact with the browser process.
|
|
|
|
### Usage
|
|
|
|
```
|
|
itb_start <url>
|
|
```
|
|
|
|
### Output
|
|
|
|
- Process ID (PID) of the browser session.
|
|
- WebDriver session ID.
|
|
|
|
### Implementation Strategies
|
|
|
|
Use Selenium to start a browser session.
|
|
Ensure the browser runs in a detached process to allow other scripts to interact with it.
|
|
Output the PID and WebDriver session ID for subsequent scripts to use.
|
|
|
|
## itb_screenshot
|
|
|
|
Starts a virtual screenshot of the current browser view and outputs all visible information in text form.
|
|
|
|
### Usage
|
|
|
|
```
|
|
itb_screenshot <session_id>
|
|
```
|
|
|
|
### Output
|
|
|
|
- Scroll offset, visible range and limits.
|
|
- Text representation of all currently visible elements.
|
|
- IDs, size and offset for interactive elements (e.g., buttons, links, input fields).
|
|
|
|
### Implementation Strategies
|
|
|
|
Get the identifiers for all visible elements.
|
|
Identify elements with text content and interaction capabilities.
|
|
Extract text content without markup.
|
|
Create simplified XML structure.
|
|
|
|
## itb_scroll
|
|
|
|
Scrolls the browser window to a specified position.
|
|
|
|
### Usage
|
|
|
|
```
|
|
itb_scroll <session_id> <x> <y>
|
|
```
|
|
|
|
## itb_click
|
|
|
|
Clicks on a specified element in the browser.
|
|
|
|
### Usage
|
|
|
|
```
|
|
itb_click <session_id> -e <element_id>
|
|
itb_click <session_id> -x <x> -y <y>
|
|
```
|
|
|
|
### Implementation Strategies
|
|
|
|
Use a random short time to slowly move the cursor to the target position.
|
|
Click on a random position within the target element.
|
|
|
|
## itb_input
|
|
|
|
Inputs text or keystrokes into the focussed element.
|
|
|
|
### Usage
|
|
|
|
```
|
|
itb_input <session_id> -t <text>
|
|
itb_input <session_id> -k <keystroke> ...
|
|
```
|
|
|
|
### Implementation Strategies
|
|
|
|
Use a random short delay between adding each character of text, to simulate human-like typing.
|
|
Keystrokes are additive.
|
|
e.g. for CTRL SHIFT a, this emulates the user pressing CTRL, keeping it pressed, then pressing SHIFT, keeping both CTRL and SHIFT pressed, then pressing 'a', keeping all three pressed, then releasing all three.
|
|
|
|
## itb_links
|
|
|
|
Extracts all links from the full webpage.
|
|
|
|
### Usage
|
|
|
|
```
|
|
itb_links <session_id>
|
|
```
|
|
|
|
### Output
|
|
|
|
- List of all visible links with their text, URLs and position on the page.
|
|
|
|
## itb_forms
|
|
|
|
Extracts all visible forms and their input fields from the current browser view.
|
|
|
|
### Usage
|
|
|
|
```
|
|
itb_forms <session_id>
|
|
```
|
|
|
|
### Output
|
|
|
|
List of all visible forms with their
|
|
- input fields
|
|
- connected labels
|
|
- identifier
|
|
- position on the page.
|
|
|
|
## itb_navigate
|
|
|
|
Navigates to a specified URL within the existing browser session.
|
|
|
|
### Usage
|
|
|
|
```
|
|
itb_navigate <session_id> <url>
|
|
```
|
|
|
|
## itb_refresh
|
|
|
|
Refreshes the current browser page.
|
|
|
|
### Usage
|
|
|
|
```
|
|
itb_refresh <session_id>
|
|
```
|
|
|
|
### Implementation Strategies
|
|
|
|
Wait till the page is fully loaded and the DOM is stable. |