# SIA - The Self Improving Agent

SIA is an agentic artificial intelligence system that autonomously completes complex tasks by writing and executing scripts.
It uses a Large Language Model (LLM) which operates in a loop.
Each iteration a context is updated with system info and a list of previous reasoning and actions.
The agent responds with a new reasoning or an action.
Context, reasoning and actions are stored in a file for each iteration.
SIA can read past iterations to improve its reasoning and actions.
It can improve in two ways:
- By finetuning the LLM with a better reasoning or action for a given context
- By modifying its own source code

## Example

This example shows a typical context with some monitored items and previous actions.
Between each of the responses, the context would be updated.

### Context

```xml
<context
  time="2024-10-18T12:00:00Z"
  cpu="12"
  gpu="26"
  memory_used="9556302234"
  memory_total="17179869184"
  disk_used="244434939904"
  disk_total="273145991168"
  context="3"
  stdin="0"
/>
  <repeat id="a3d89ee5-28ec-4c5a-b9e9-a30af53d43a0" exit_code="0">
    <![CDATA[ls -lah /]]>
    <stdout><![CDATA[total 16K
drwxr-xr-x 1 sia 1049089 0 Oct 28 13:40 ./
drwxr-xr-x 1 sia 1049089 0 Oct 28 13:40 ../
drwxr-xr-x 1 sia 1049089 0 Oct 28 13:40 tasks/
drwxr-xr-x 1 sia 1049089 0 Oct 28 13:40 user/
]]></stdout>
    <stderr/>
  </repeat>
  <repeat id="be8070f8-dbd2-47ee-a208-defe6fd49ae0" exit_code="0">
    <![CDATA[ls -lah /tasks]]>
    <stdout><![CDATA[total 0
drwxr-xr-x 1 ngeens 1049089 0 Oct 28 13:40 ./
drwxr-xr-x 1 ngeens 1049089 0 Oct 28 13:40 ../
]]></stdout>
    <stderr/>
  </repeat>
  <repeat id="375e1657-8140-456b-bda4-a8690bc4b3fb" exit_code="0">
    <![CDATA[cat /user/general_info.txt]]>
    <stdout><![CDATA[Name: John (I don't know his last name)
Location: Somewhere in Belgium
]]></stdout>
    <stderr/>
  </repeat>
  <reasoning id="c92d1594-4487-4a42-a153-f0a99da1762f"><![CDATA[There is data available on the standard input channel. I have no other running tasks to tend to and there is room in the context. I should read the standard input.]]></reasoning>
  <read_stdin id="5361ad0d-3ed1-4567-9f5a-70f3b462fd8d"><![CDATA[Remind me to feed the cat tomorrow morning]]></read_stdin>
</context>
```

### Responses

Start by reasoning about the task.
```xml
<reasoning>
  John did not specify an exact time.
  I'll suggest 9am.
  He also did not specify how to be reminded.
  I'll ask but if he doesn't respond I'll assume a text message on standard output is fine.
  I'll write down this task in a file so I remember it even on a power failure.
</reasoning>
```

Store important information on disk.
```xml
  <single_shot><![CDATA[echo 'Remind John to feed the cat on 2024-10-18T09:00:00+02:00. Use standard output.' > /tasks/reminder_to_feed_cat.txt]]></single_shot>
```

Respond to the user.
```xml
<write_stdout>I'll remind you to feed the cat tomorrow morning at 9am. Is a message on the standard output ok?</write_stdout>
```

Clear initial reasoning.
```xml
  <delete id="c92d1594-4487-4a42-a153-f0a99da1762f"/>
```

The conversation is kept in context to understand the user's expected response.
If the context was near full, it would be summarized and cleaned up.

The single shot is also kept in context.
If the file was updated often, it could be replaced by a repeated `cat`, like the general info.

## Working principles

The main context is regenerated for each iteration.
It contains info about the system and previous actions that have not been deleted.
Together with the system prompt and available core actions it forms the prompt for the LLM.
The LLM responds with one core action.

### Core Actions

There are only a few core actions:
- Starting a script
- Deleting data from context
- Stopping SIA
- Reading standard input
- Writing to standard output
- Reasoning

### Scripts

Scripts can run in one of 3 modes: single-shot, background or repeat.
Their mode, status and output (stdout and stderr) stay in the context until they are explicitly removed.
In this way the agent manages what information is available in the context.

#### Single-shot

The script is executed once.
This is useful for most operations e.g. writing to or moving a file or downloading content from the internet.
The next iteration starts after all single shot scripts have finished.

#### Repeat

The script is restarted on each iteration.
This is useful for monitoring files or the file system.
commands like `head` and `tail` can be used to limit the data in context.
Similar to single-shot scripts, the next iteration starts after all repeat scripts have finished.

#### Background

The script is started and keeps running.
This is useful for waiting for events, a communication channels or a process that requires attention.

Long-running processes e.g. a web server can be run as a service or detached process to keep the context small.
The output can be redirected to a file and monitored with a repeat script.

### Use of XML

The context and actions are formatted as XML.
For the context this adds clear rules for escaping.
This is usefull in case a previous context is embedded.

The LLM is free to escape data any way it wants,
as long as it results in valid XML.
The response is validated against a schema.

The Context is escaped using CDATA blocks.
Except when the data contains CDATA closing sequences.
Then the whole block is escaped using standard XML escaping.


### The SIA process

SIA is typically runs with the `restart.sh` script.
This is a simple shell script that runs SIA in a loop.
When stopped, SIA restarts and reloads the Python files.
This is how SIA can self-update.

SIA can also run SIA processes as script.
This can be used for testing updates to the LLM or core functionality.

### Server for debuggin and human input

SIA can be started with an optional `--server` flag.
This starts a web server that can be used to interact with SIA.
It is made, specifically for reinforcement learning by human feedback.
The web interface takes over standard input and output.
It will display the context for editing before handing it to the LLM.
After each run of the LLM, before parsing, it will display the reasoning and actions.
It interactively displays if the actions can be parsed.
At any time, the user can write to the standard input of SIA.

## Architecture

SIA follows a modular architecture centered around an agent that processes context through an LLM to generate actions.
The system can run in two modes: a standard command-line mode and an interactive web mode for debugging and human feedback.

### Core Components

#### Agent Architecture
The core of SIA is the agent, which exists in two variants:
- ProceduralAgent: Runs in a simple loop, processing context and executing actions directly
- WebAgent: Uses a state machine to allow human intervention and feedback through a web interface

Both agent types share common components:
- WorkingMemory: Maintains the current state through a collection of entries and system metrics
- ResponseParser: Converts LLM output into commands or entries
- XMLValidator: Validates responses against a schema
- IOBuffer: Handles input/output operations in an agent-appropriate way

#### Working Memory
The working memory stores the current state of the system through different types of entries:
- Script Entries:
  - SingleShotEntry: Results of one-time script executions
  - RepeatEntry: Continuously refreshed script outputs
  - BackgroundEntry: Status of long-running processes
- ReasoningEntry: LLM's thought process documentation
- ParseErrorEntry: XML validation or parsing errors
- IOEntry: Input/output operations

Each entry can be serialized to XML for inclusion in the LLM context.
Working memory is cleaned through explicit delete commands or when context size limits are reached.

#### Command Processing
SIA distinguishes between two types of LLM outputs:
1. Commands: Immediate actions that modify the system
   - DeleteCommand: Removes entries from working memory
   - StopCommand: Terminates the agent
2. Entries: Records that become part of working memory
   - Created from script executions, IO operations, reasoning, or errors
   - Persist until explicitly deleted or context limits are reached

#### IO Handling
IO operations are abstracted through an IOBuffer interface:
- StandardIOBuffer: Direct access to system stdin/stdout
- WebIOBuffer: Buffer for web interface communication
This abstraction allows the ResponseParser to generate consistent IOEntries regardless of agent type.

### Processing Flow

#### Standard Agent Flow
1. Update system metrics and context size
2. Compile context from working memory entries
3. Process context through LLM
4. Validate XML response
5. Parse response into command or entry
6. Execute command or update working memory
7. Return to step 1 unless stopped

#### Web Agent Flow
1. Update system metrics and context size
2. Compile context and await human approval
3. Process approved context through LLM
4. Validate XML response
5. Present validation results and await human review
6. Parse approved response into command or entry
7. Execute command or update working memory
8. Return to step 1 unless stopped

### Web Interface

The web interface provides interactive debugging and human feedback through a WebSocket-based protocol:

#### Server-Client Communication
- Server Messages:
  - Context updates for review
  - Validation results from LLM responses
  - Output updates from script execution
  - State updates for UI synchronization
- Client Messages:
  - Context approval decisions
  - Response modifications
  - User input for stdin operations

#### WebServer Architecture
The web server manages:
- WebSocket connections to clients
- Message routing to/from the WebAgent
- Broadcasting state updates to all connected clients

This architecture allows for:
- Real-time monitoring of agent state
- Interactive debugging of LLM responses
- Human-in-the-loop operation for testing and development
- Collection of human feedback for reinforcement learning

### Standard Agent Flow

```mermaid
stateDiagram-v2
    direction LR
    state "Standard Agent Flow" as standard_agent_flow {
        [*] --> UpdateSystem: Start
        UpdateSystem --> CompileContext: Updated Metrics & Size
        CompileContext --> ProcessLLM
        ProcessLLM --> ValidateXML: LLM Response
        ValidateXML --> ParseResponse: Valid XML
        ValidateXML --> UpdateEntries: Invalid XML\nCreate ParseErrorEntry
        ParseResponse --> ExecuteCommands: Command
        ParseResponse --> UpdateEntries: Entry
        ExecuteCommands --> [*]: Stop Command
        ExecuteCommands --> UpdateEntries: Delete Command
        UpdateEntries --> UpdateSystem: Continue Loop
    }
```

### Web Agent Flow

```mermaid
stateDiagram-v2
    direction LR
    state "Web Agent Flow" as web_agent_flow {
        [*] --> UpdateSystem: Start
        UpdateSystem --> CompileContext: Updated Metrics & Size
        CompileContext --> WaitForContextApproval: Send Context
        WaitForContextApproval --> ProcessLLM: Context Approved
        ProcessLLM --> ValidateXML: LLM Response
        ValidateXML --> WaitForResponseApproval: Send Validation Result
        ValidateXML --> UpdateEntries: Invalid XML\nCreate ParseErrorEntry

        WaitForResponseApproval --> ValidateXML: Modified Response
        WaitForResponseApproval --> ParseResponse: Approved Response

        ParseResponse --> ExecuteCommands: Command
        ParseResponse --> UpdateEntries: Entry
        
        ExecuteCommands --> [*]: Stop Command
        ExecuteCommands --> UpdateEntries: Delete Command
        
        UpdateEntries --> UpdateSystem: Continue Loop
    }
```

### Class Diagram

```mermaid
classDiagram
    direction LR
    class BaseAgent {
        <<abstract>>
        #working_memory WorkingMemory
        #metrics SystemMetrics
        #llm LLMEngine
        #parser ResponseParser
        #validator XMLValidator
        #io_buffer IOBuffer
        #compile_context() str
    }

    class LLMEngine {
        +LLMEngine(model_path str)
        +set_model_path(model_path str)
        +inference(context str) Iterator~str~
    }

    class StandardAgent {
        +run()
    }

    class WebAgent {
        -current_state WebAgentState
        +get_current_state() WebAgentState
        +proceed()
        +add_state_change_handler(handler func())
    }

    class WebAgentState {
        <<enumeration>>
        UPDATE
        CONTEXT_APPROVAL
        INFERENCE
        RESPONSE_APPROVAL
    }

    class WorkingMemory {
        +WorkingMemory()
        -entries List~Entry~
        +add_entry(entry Entry)
        +remove_entry(id str)
        +update()
        +generate_context() List~ElementTree~
    }

    class SystemMetrics {
        +generate_context(entries List~ElementTree~) ElementTree
    }

    class XMLValidator {
        -schema ElementTree
        +XMLValidator(schema str)
        +validate(xml str) Optional~str~
    }

    class ResponseParser {
        -io_buffer IOBuffer
        +ResponseParser(io_buffer IOBuffer)
        +parse(xml str) Command | Entry
    }

    class Entry {
        <<abstract>>
        +id str
        +timestamp datetime
        +Entry(id str, timestamp datetime)
        +update()*
        +generate_context() ElementTree*
    }

    class SingleShotEntry {
        +script str
        +stdout str
        +stderr str
        +exit_code int
        +SingleShotEntry(script str, id str, timestamp datetime)
        +update()
        +generate_context() ElementTree
    }

    class RepeatEntry {
        +script str
        +stdout str
        +stderr str
        +exit_code int
        +RepeatEntry(script str, id str, timestamp datetime)
        +update()
        +generate_context() ElementTree
    }

    class BackgroundEntry {
        +script str
        +stdout str
        +stderr str
        +process Process
        +BackgroundEntry(script str, id str, timestamp datetime)
        +update()
        +generate_context() ElementTree
    }

    class ReasoningEntry {
        +content str
        +ReasoningEntry(content str, id str, timestamp datetime)
        +update()
        +generate_context() ElementTree
    }

    class ParseErrorEntry {
        +content str
        +error str
        +ParseErrorEntry(content str, error str, id str, timestamp datetime)
        +update()
        +generate_context() ElementTree
    }

    class ReadEntry {
        +content str
        +ReadEntry(io_buffer IOBuffer, id str, timestamp datetime)
        +update()
        +generate_context() ElementTree
    }

    class WriteEntry {
        +content str
        +WriteEntry(content str, io_buffer IOBuffer, id str, timestamp datetime)
        +update()
        +generate_context() ElementTree
    }

    class IOBuffer {
        <<interface>>
        +read() str*
        +write(content str)*
        +buffer_length() int*
    }

    class StandardIOBuffer {
        +StandardIOBuffer()
        +read() str
        +write(content str)
        +buffer_length() int
    }

    class WebIOBuffer {
        -stdin_buffer str
        -stdout_buffer str
        +WebIOBuffer()
        +read() str
        +write(content str)
        +buffer_length() int
        +append_stdin(content str)
        +get_stdout() str
        +clear_stdout()
    }

    class CommandResult {
        +should_stop bool
        +success bool
        +message str
        +CommandResult(should_stop bool, success bool, message str)
        +static success() CommandResult
        +static failure(message str) CommandResult
        +static stop() CommandResult
    }

    class Command {
        <<abstract>>
        +execute(memory &WorkingMemory) CommandResult*
    }

    class DeleteCommand {
        +id str
        +DeleteCommand(id str)
        +execute(memory &WorkingMemory) CommandResult
    }

    class StopCommand {
        +execute(memory &WorkingMemory) CommandResult
    }

    class WebServer {
        -agent WebAgent
        -clients List~WebSocket~
        +WebServer(agent WebAgent)
        -broadcast_state_change()
    }

    BaseAgent <|-- WebAgent
    BaseAgent <|-- StandardAgent
    BaseAgent "1" *-- "1" IOBuffer
    BaseAgent "1" *-- "1" WorkingMemory
    BaseAgent "1" *-- "1" SystemMetrics
    BaseAgent "1" *-- "1" XMLValidator
    BaseAgent "1" *-- "1" LLMEngine
    BaseAgent "1" *-- "1" ResponseParser

    WebServer "1" *-- "1" WebIOBuffer
    WebServer "1" *-- "1" WebAgent

    WebAgent "1" *-- "1" WebAgentState

    WorkingMemory "1" *-- "*" Entry

    Entry <|-- ReasoningEntry
    Entry <|-- ParseErrorEntry
    Entry <|-- ReadEntry
    Entry <|-- WriteEntry
    Entry <|-- SingleShotEntry
    Entry <|-- RepeatEntry
    Entry <|-- BackgroundEntry

    Command <|-- DeleteCommand
    Command <|-- StopCommand
    Command -- CommandResult

    IOBuffer <|.. WebIOBuffer
    IOBuffer <|.. StandardIOBuffer
```