SIA/readme.md

# SIA - The Self Improving Agent

SIA is an agentic artificial intelligence system that autonomously completes complex tasks by writing and executing scripts.
It uses a Large Language Model (LLM) which operates in a loop, generating reasoning and actions based on an updating context.
SIA manages Docker containers for task execution.
These can be short-lived for e.g. bash one-liners or long-running for e.g. background tasks, training or communication.

The system implements reinforcement learning by analyzing past iterations to improve its LLM.
SIA can also modify its own source code, allowing it to adapt to new challenges.

## Working principles

High-level overview of the main components of SIA and how they work together.

### LLM-Powered Reasoning

SIA utilizes a Large Language Model (LLM) as its core reasoning engine.
This LLM can be updated and modified over time.

The LLM is inferred in a loop.
Each iteration of the loop the system prompt and main context are provided.
The LLM generates a response with reasoning and a list of core actions to take.

The main context is always regenerated and contains:
- System status and limits
  - time and date (ISO 8601 UTC)
  - CPU usage (%)
  - GPU usage (%)
  - memory usage and total (bytes)
  - disk usage and total (bytes)
  - context usage (%)
  - standard input buffer contents (bytes)
- List of containers
  - description
  - status (initializing, running, finished)
  - standard IO buffer usage
  - ports, volumes and environment variables
- The reasoning, actions and results of the previous iteration
- Files monitored from the filesystem

### Core Actions

Core actions are selected by the LLM by outputting an XML list of actions and parameters.
This system allows the agent to manage its memory, control containers, select wich version to run, and communicate.
The Agent Core parses the XML output and executes the corresponding functions.

Core actions for user interaction:
- Read standard input
- Write to standard output

### Docker Container Management

SIA utilizes Docker containers for anything not covered by core actions.
Containers can be short-lived, eg. for simple calculations.
They can also be long-lived, eg. to keep a communication channel open.
They can even run a complete SIA instance eg. for verifying updates to the LLM or core functionality.
The short-lived containers define a timeout.
The next iteration of the main loop starts when the container finishes or the timeout is reached.
The long-lived containers can also be waited on at a later point in time.

Core actions for container operations:
- Start container
- Stop container
- Write to container standard input
- Read from container standard output
- Read from container standard error
- Wait for container to finish

### Information Storage

The SIA main loop is ephemeral.
Therefore the agent needs to store information for future reference.
The agent has access to a Linux filesystem.
Files and directories in this filesystem can be mapped as volumes to containers.
The agent can load a file in its context, so it always has a view of the latest version.
The same can be done with directory listings.

Core actions for file operations:
- Monitor file (or folder)
- Unmonitor file

There are no core commands for creating, updating or deleting files.
This can be done using containers.

### Reinforcement Learning

For each iteration of the main loop, the context and the generated reasoning and actions are stored in the file system.
When the agent solves a problem it starts a search for the root cause by looking at previous iterations.
The iteration file is used for updating the LLM weights using functions in the SIA core.
The agent can access this by running a SIA container.
It can then test the updated LLM in another SIA instance in a container.
If it is acceptable it can change to this new version.

Core actions for reinforcement learning:
- Select LLM by file name

There are no specific commands for running a SIA instance in a container.
This can be done using the regular container commands.

### Self-Improvement

SIA has access to a git repository containing its source code.
It can also access a container repository with SIA builds.
With these it is possible for SIA to update and test new versions of itself.

If a new version is approved, the agent can switch to it and continue working.

Core actions for self-improvement:
- Update to docker tag

## Architecture

An overview of the key components and their interactions.

![SIA Component Model](./diagrams/SIA_Component_Model.svg)

### Modules

Modules execute core commands and provide data for the context template.

- Process Module
  - standard I/O operations
  - waiting
  - file monitoring
  - updating the SIA process to another version
- Docker Module
  - container operations
  - container status and buffer monitoring
- Reinforcement Learning Module
  - create dataset for fine-tuning the LLM
  - labeling trained models

### Agent Core

The Agent Core runs the SIA main loop.

- template the context
- run the LLM
- parse the LLM output
- execute the appropriate actions using the relevant modules

### LLM Engine

The LLM Engine is responsible for:
- Running inference based on the provided context
- Updating the model's weights during the learning process

## Implementation

This section explains technical details of the implementation.

### Use of XML

The context and actions are formatted as XML.
For the context this adds clear rules for escaping.
This is usefull in case a previous context is embedded.

The response starts with freeform reasoning followed by XML formatted actions.
In case the LLM makes a mistake it can start over.
Only the last XML block is evaluated.

XML is verbose by nature.
To avoid overflowing the context window, it should only be used where it adds value.
Directory listings, for instance, are formatted in the well known ls command format.

Parameters for actions can be passed as attributes or as child elements.
This allows the LLM to pass multiple volumes or environment variables in a clear way.
It also simplifies escaping of command line arguments.

Action results are added in the context as text nodes after the last parameter.

### Context Template

A Handlerbars template is used to create the context for the LLM.
A ContextTemplate object is created for each iteration of the main loop.
The template is filled with data by the Agent Core.

### Training datasets

A training dataset is a folder with these files:
- system_prompt.txt
- main_context.txt
- pre-reasoning.txt
- training_reasoning.txt
- post-reasoning.txt
- pre-actions.txt
- training_actions.txt
- post-actions.txt

The context window of the LLM is filled with all parts of the dataset in order.
The learning rate is only applied to the training reasoning and actions.
The pre and post files are optional.

To do an actual training round, a sia:latest container is started.
This is an example action that trains on two datasets with learning rate 0.1:

```xml
<start_container image="sia:latest" volumes="/tasks:/tasks">
    <volumes>
      <volume>/models/:/models/</volume>
      <volume>/datasets/description_of_a_problem/:/datasets/description_of_a_problem/</volume>
      <volume>/datasets/description_of_nother_problem/:/datasets/description_of_nother_problem/</volume>
    </volumes>
    <command>sia</command>
    <argument>train</argument>
    <argument>--learning-rate</argument>
    <argument>0.1</argument>
    <argument>--model</argument>
    <argument>/models/2024_10_19_08_21_41</argument>
    <argument>--out</argument>
    <argument>/models/2024_10_19_15_03_52</argument>
    <argument>/datasets/description_of_a_problem/</argument>
    <argument>/datasets/description_of_nother_problem/</argument>
</start_container>
```

### Reinforcement learning by human feedback

The SIA container can be used in 3 ways:
- To run a SIA instance
- To update LLM weights based on a dataset
- To host the interaction web interface

The web interface is an alternative way of interacting with SIA, specifically for reinforcement learning by human feedback.
The web interface takes over standard input and output.
It each time the LLM generates a response, the web interface will display it.
The user can modify the response before the actions are executed.

### Project structure

The SIA application is developed in the src directory.
The tests directory contains unit tests, mock objects and integration tests.
The model directory contains the trained model.
It is excluded from the git repository and the docker context because it is too large.

The docker file has a separate stage for testing.
The `test.sh` script builds this stage, runs the tests and removes the test image.

To use SIA several directories have to be mounted:
- `/root/model': The model directory
- `/root/sia_repo': The git repository
- the docker socket: to run sub-SIA instances

## Actions

A list of all available Core Actions.
Indicating how they are implemented and how SIA can use them.

### Read standard input

Module: Process

#### Function declaration
```python
def read_stdin(n: int = -1) -> str:
  ''' Read n bytes from standard input.

  Args:
    n: int, The number of bytes to read; -1 for all available bytes (default: -1)
  '''
  pass
```

#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="read_stdin">
    <xs:complexType>
      <xs:attribute name="n" type="xs:integer" use="optional"/>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

#### Example
```xml
<read_stdin n="42"/>
```

#### Results

An attribute `actual` is added with the amount of bytes read.
A text node is added with the data as `CDATA`.

### Write to standard output

Module: Process

Function declaration
```python
def write_stdout(text: str) -> None:
  ''' Write text to standard output. '''
  pass
```

Schema
```xml
  <?xml version="1.0" encoding="UTF-8"?>
  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <xs:element name="write_stdout">
      <xs:complexType>
        <xs:simpleContent>
          <xs:extension base="xs:string"/>
        </xs:simpleContent>
      </xs:complexType>
    </xs:element>
  </xs:schema>
```

  Example
```xml
  <write_stdout>Hello world!</write_stdout>
```

#### Results

No information is added.

### Monitor file

Module: Process

#### Function declaration
```python
def monitor_file(path: str, offset: int = 0, length: int = -1, unit: str = 'bytes') -> None:
  ''' Monitor a file for changes.

    Parameters:
    - path: str, the path to the file to monitor
    - offset: int, the starting point for reading (default: 0)
    - length: int, the amount to read; -1 means read to end (default: -1)
    - unit: str, 'bytes' or 'lines' (default: 'bytes')
  '''
  pass
```

#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="monitor_file">
    <xs:complexType mixed="true">
      <xs:simpleContent>
        <xs:extension base="xs:string">
          <xs:attribute name="offset" type="xs:integer" use="optional"/>
          <xs:attribute name="length" type="xs:integer" use="optional"/>
          <xs:attribute name="unit" use="optional">
            <xs:simpleType>
              <xs:restriction base="xs:string">
                <xs:enumeration value="bytes"/>
                <xs:enumeration value="lines"/>
              </xs:restriction>
            </xs:simpleType>
          </xs:attribute>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

#### Example
```xml
<monitor_file offset="12" unit="lines">
  /context_history/2024_10_20/13_43_51.56
</monitor_file>
```

#### Results

No information is added.
The monitored files are added to the context separately with an id.

### Unmonitor file

Module: Process

#### Function declaration
```python
def unmonitor_file(path: str = None, id: int = None) -> None:
  ''' Unmonitor a file.

  Parameters:
  - path: str, the path to the file
  - id: int, the id of the file as indicated in the context

  Either path or id must be provided.
  '''
```

#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="unmonitor_file">
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base="xs:string">
          <xs:attribute name="id" type="xs:integer" use="optional"/>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

#### Example
```xml
<unmonitor_file id="3"/>
<unmonitor_file>
  /context_history/2024_10_20/13_43_51.56
</unmonitor_file>
```

#### Results

No information is added.

### Start container

Module: Docker

#### Function declaration
```python
def start_container(
  image: str,
  name: str = None,
  timeout: int = -1,
  command: str = None,
  arguments: List[str] = None,
  volumes: Dict[str, str] = None,  # host_path: container_path
  ports: Dict[str, str] = None,    # host_port: container_port
  environment: Dict[str, str] = None,  # key: value
) -> None:
  """Start a new Docker container with the specified configuration.

  Args:
    image: Docker image to use
    name: Unique container name for long running containers
    timeout: Timeout in milliseconds for short running containers
    command: Main command to run in container
    arguments: List of command line arguments
    volumes: Dictionary mapping host paths to container paths
    ports: Dictionary mapping host ports to container ports
    environment: Dictionary of environment variables and values

  name or timeout must be provided.

  Example:
    start_container(
      image="busybox:latest",
      timeout=1000,
      command="sh",
      arguments=["-c", "echo 'Hello' > /data/output.txt"],
      volumes={
        "/host/data": "/data",
        "/host/config": "/config"
      },
      ports={
        "8080": "80",
        "2222": "22"
      },
      environment={
        "DEBUG": "1",
        "API_KEY": "secret123"
      }
    )
  """
  pass
```

#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="start_container">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="command" type="xs:string" minOccurs="0"/>
        <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
        <xs:element name="volumes" minOccurs="0">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="volume" type="xs:string" maxOccurs="unbounded"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="ports" minOccurs="0">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="port" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:string">
                      <xs:attribute name="host" type="xs:string" use="required"/>
                      <xs:attribute name="container" type="xs:string" use="required"/>
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="environment" minOccurs="0">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="variable" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:string">
                      <xs:attribute name="name" type="xs:string" use="required"/>
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
      <xs:attribute name="image" type="xs:string" use="required"/>
      <xs:attribute name="name" type="xs:string" use="optional"/>
      <xs:attribute name="timeout" type="xs:integer" use="optional"/>
      <xs:attribute name="volumes" type="xs:string" use="optional"/>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

#### Example
```xml
<start_container
    image="busybox:latest"
    timeout="1000">
    <command>sh</command>
    <argument>-c</argument>
    <argument><![CDATA[echo 'Hello' > /data/output.txt]]></argument>
    <volumes>
        <volume>/host/data:/data</volume>
        <volume>/host/config:/config</volume>
    </volumes>
    <ports>
        <port host="8080" container="80"/>
        <port host="2222" container="22"/>
    </ports>
    <environment>
        <variable name="DEBUG">1</variable>
        <variable name="API_KEY">secret123</variable>
    </environment>
</start_container>
```

#### Results

For short running containers, the exit status is added as attribute.
Standard output and standard error are merged and added as `CDATA` text node.

For long running containers, no information is added.
The container is started and represented in the containers section of the main context.

### Stop container

Module: Docker

#### Function declaration
```python
def stop_container(name: str) -> None:
  """Stop a Docker container by name.

  Args:
    name: Name of the container to stop
  """
  pass
```

#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="stop_container">
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base="xs:string"/>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

#### Example
```xml
<stop_container>my-long-running-container</stop_container>
```

#### Results

No information is added.
The container will be removed from the containers section of the main context.

### Write to container standard input

Module: Docker

#### Function declaration
```python
def write_container_stdin(name: str, data: str) -> None:
  """Write data to a container's standard input.

  Args:
    name: Name of the target container
    data: Data to write to stdin
  """
  pass
```

#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="write_container_stdin">
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base="xs:string">
          <xs:attribute name="container" type="xs:string" use="required"/>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

#### Example
```xml
<write_container_stdin container="interactive-shell">ls -la</write_container_stdin>
```

#### Results

No information is added.
The data will be written to the container's stdin buffer.

### Read from container standard output

Module: Docker

#### Function declaration
```python
def read_container_stdout(name: str, n: int = -1) -> str:
  """Read from a container's standard output buffer.

  Args:
    name: Name of the container
    n: Number of bytes to read; -1 means read all available (default: -1)

  Returns:
    Data read from stdout
  """
  pass
```

#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="read_container_stdout">
    <xs:complexType>
      <xs:attribute name="container" type="xs:string" use="required"/>
      <xs:attribute name="n" type="xs:integer" use="optional"/>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

#### Example
```xml
<read_container_stdout container="interactive-shell" n="1024"/>
```

#### Results

An attribute `actual` is added with the amount of bytes read.
A text node is added with the data as `CDATA`.

### Read from container standard error

Module: Docker

#### Function declaration
```python
def read_container_stderr(name: str, n: int = -1) -> str:
  """Read from a container's standard error buffer.

  Args:
    name: Name of the container
    n: Number of bytes to read; -1 means read all available (default: -1)

  Returns:
    Data read from stderr
  """
  pass
```

#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="read_container_stderr">
    <xs:complexType>
      <xs:attribute name="container" type="xs:string" use="required"/>
      <xs:attribute name="n" type="xs:integer" use="optional"/>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

#### Example
```xml
<read_container_stderr container="interactive-shell" n="1024"/>
```

#### Results
An attribute `actual` is added with the amount of bytes read.
A text node is added with the data as `CDATA`.

### Wait for container to finish

Module: Docker

#### Function declaration
```python
def wait_container(name: str, timeout: int):
  """Wait for a container to finish execution.

  Args:
    name: Name of the container to wait for
    timeout: Time to wait in milliseconds
  """
  pass
```

#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="wait_container">
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base="xs:string">
          <xs:attribute name="timeout" type="xs:integer" use="required"/>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

#### Example
```xml
<wait_container timeout="5000">background-task</wait_container>
```

#### Results
The exit status is added as an attribute.
All remaining data on stdout and stderr is merged and added as `CDATA` text node.

### Select LLM by file name

Module: Reinforcement Learning

#### Function declaration
```python
def select_llm(path: str) -> None:
  """Switch to a different LLM model file.

  Args:
    path: Path to the model file to load
  """
  pass
```

#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="select_llm">
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base="xs:string"/>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

#### Example
```xml
<select_llm>/models/2024_10_19_15_03_52</select_llm>
```

#### Results

No information is added.
The new model will be used starting from the next iteration.

### Update to commit id

Module: Process

#### Function declaration
```python
def update_to_commit(commit_id: str) -> None:
  """Update the running SIA process to a different version.
  This will terminate the current process immediately.
  Make sure the git commit is well tested and there is plenty of clear documentation for the new SIA instance to start.

  Args:
    commit_id: The git commit ID to update to
  """
  pass
```

#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="update_to_commit">
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base="xs:string"/>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>
```

#### Example
```xml
<update_to_commit>0015e27</update_to_commit>
```

#### Results

No information is added.
The process will be replaced by the new version.

## Example iterations

### Clarifying a task

This example shows how to work with standard IO, run simple scripts and monitor files.

#### Context

```xml
<context>
  <system
    time="2024-10-18T12:00:00Z"
    cpu="12"
    gpu="26"
    memory_used="9556302234"
    memory_total="17179869184"
    disk_used="244434939904"
    disk_total="273145991168"
    context="3"
    stdin="0"
  />
  <containers/>
  <previous>
    <reasoning>
      There is data available on the standard input channel. I should read it. I have no other running tasks to tend to.
    </reasoning>
    <actions>
      <read_stdin n="42">
        <![CDATA[Remind me to feed the cat tomorrow morning]]>
      </read_stdin>
    </actions>
  </previous>
  <files>
    <file name="/" type="dir" index="0">
      <![CDATA[
drwxr-xr-x 1 sia 197121 0 2024-10-16 23:02:16.486152500 +0200 tasks/
drwxr-xr-x 1 sia 197121 0 2024-10-16 22:35:31.806079500 +0200 user/
      ]]>
    </file>
    <file name="/tasks" type="dir" index="1">
    </file>
    <file name="/user" type="dir">
      <![CDATA[
-rw-r--r-- 1 sia 197121 71 2024-10-16 22:41:23.223580300 +0200 general_info.txt
      ]]>
    </file>
    <file name="/user/general_info.txt" type="file" index="2">
      <![CDATA[
Name: John (I don't know his last name)
Location: Somewhere in Belgium
      ]]>
    </file>
  </files>
</context>
```

#### LLM response

```xml
John did not specify an exact time. I'll suggest 9am. He also did not specify how to be reminded. I'll ask but if he doesn't respond I'll assume a text message on standard output is fine. I'll write down this task in a file so I can keep it in context. I can write simple files with busybox:latest and echo but I will need to use sh -c to do the redirect.
<actions>
  <write_stdout message="I'll remind you to feed the cat tomorrow morning at 9am. Is a message on the standard output ok?"/>
  <start_container image="busybox:latest" timeout="1000" volumes="/tasks:/tasks">
      <command>sh</command>
      <argtument>-c</argument>
      <argument><![CDATA[echo 'Remind John to feed the cat on 2024-10-18T09:00:00+02:00. Use standard output.' > /tasks/reminder.txt]]></argument>
  </start_container>
  <monitor_file path="/tasks/reminder.txt"/>
</actions>
```