SIA/readme.md

SIA - The Self Improving Agent

SIA is an agentic artificial intelligence system that autonomously completes complex tasks by writing and executing scripts. It uses a Large Language Model (LLM) which operates in a loop, generating reasoning and actions based on an updating context. Context, reasoning and actions are stored in a file for each iteration. SIA can read past iterations to improve its reasoning and actions. It can improve in two ways:

• By providing better reasoning or actions for a given context and update the LLM.
• By modifying its own source code.

Working principles

High-level overview of the main components of SIA and how they work together.

Scripts

Scripts can run in one of 3 modes: single-shot, background or repeat. Their mode, status and output (stdout and stderr) stay in the context until they are explicitly removed. In this way the agent manages what information is available in the context.

Single-shot

The script is executed once. This is useful for most operations e.g. writing to or moving a file or downloading content from the internet. The next iteration starts after all single shot scripts have finished.

Repeat

The script is restarted on each iteration. This is useful for monitoring files or the file system. commands like head and tail can be used to limit the data in context. Similar to single-shot scripts, the next iteration starts after all repeat scripts have finished.

Background

The script is started and keeps running. This is useful for long-running processes e.g. a web server or a communication channel. Because output of a background script can grow long, it is often redirected to a file.

LLM prompt

The main context is regenerated for each iteration. It contains info about the system, the scripts and what happended in the previous iteration. Together with the system prompt and available core actions it forms the prompt for the LLM.

The LLM generates reasoning and an XML structure with core actions. If the structure cannot be parsed, the error is described and the LLM is asked to try again. This can continue until the context overflows. Only the first reasoning, last reasoning and last actions are shown in the new context. All is stored on the file system.

Core Actions

There are only a few core actions:

• Starting a script
• Stopping a script
• Stopping SIA
• Reading standard input
• Writing to standard output

Standard error is used by the core for debugging.

SIA typically runs in a Docker container. When stopped, the latest container version is pulled. This is how SIA can be updated.

SIA can also run SIA processes as script. This can be used for testing updates to the LLM or core functionality.

Architecture

An overview of the key components and their interactions.

Modules

Modules execute core commands and provide data for the context template.

• Process Module
  • standard I/O operations
  • waiting
  • file monitoring
  • updating the SIA process to another version
• Docker Module
  • container operations
  • container status and buffer monitoring
• Reinforcement Learning Module
  • create dataset for fine-tuning the LLM
  • labeling trained models

Agent Core

The Agent Core runs the SIA main loop.

• template the context
• run the LLM
• parse the LLM output
• execute the appropriate actions using the relevant modules

LLM Engine

The LLM Engine is responsible for:

• Running inference based on the provided context
• Updating the model's weights during the learning process

Implementation

This section explains technical details of the implementation.

Use of XML

The context and actions are formatted as XML. For the context this adds clear rules for escaping. This is usefull in case a previous context is embedded.

The response starts with freeform reasoning followed by XML formatted actions. In case the LLM makes a mistake it can start over. Only the last XML block is evaluated.

XML is verbose by nature. To avoid overflowing the context window, it should only be used where it adds value. Directory listings, for instance, are formatted in the well known ls command format.

Parameters for actions can be passed as attributes or as child elements. This allows the LLM to pass multiple volumes or environment variables in a clear way. It also simplifies escaping of command line arguments.

Action results are added in the context as text nodes after the last parameter.

Context Template

A Handlerbars template is used to create the context for the LLM. A ContextTemplate object is created for each iteration of the main loop. The template is filled with data by the Agent Core.

Training datasets

A training dataset is a folder with these files:

• system_prompt.txt
• main_context.txt
• pre-reasoning.txt
• training_reasoning.txt
• post-reasoning.txt
• pre-actions.txt
• training_actions.txt
• post-actions.txt

The context window of the LLM is filled with all parts of the dataset in order. The learning rate is only applied to the training reasoning and actions. The pre and post files are optional.

To do an actual training round, a sia:latest container is started. This is an example action that trains on two datasets with learning rate 0.1:

<start_container image="sia:latest" volumes="/tasks:/tasks">
    <volumes>
      <volume>/models/:/models/</volume>
      <volume>/datasets/description_of_a_problem/:/datasets/description_of_a_problem/</volume>
      <volume>/datasets/description_of_nother_problem/:/datasets/description_of_nother_problem/</volume>
    </volumes>
    <command>sia</command>
    <argument>train</argument>
    <argument>--learning-rate</argument>
    <argument>0.1</argument>
    <argument>--model</argument>
    <argument>/models/2024_10_19_08_21_41</argument>
    <argument>--out</argument>
    <argument>/models/2024_10_19_15_03_52</argument>
    <argument>/datasets/description_of_a_problem/</argument>
    <argument>/datasets/description_of_nother_problem/</argument>
</start_container>

Reinforcement learning by human feedback

The SIA container can be used in 3 ways:

• To run a SIA instance
• To update LLM weights based on a dataset
• To host the interaction web interface

The web interface is an alternative way of interacting with SIA, specifically for reinforcement learning by human feedback. The web interface takes over standard input and output. It each time the LLM generates a response, the web interface will display it. The user can modify the response before the actions are executed.

Project structure

The SIA application is developed in the src directory. The tests directory contains unit tests, mock objects and integration tests. The model directory contains the trained model. It is excluded from the git repository and the docker context because it is too large.

The docker file has a separate stage for testing. The test.sh script builds this stage, runs the tests and removes the test image.

To use SIA several directories have to be mounted:

• `/root/model': The model directory
• `/root/sia_repo': The git repository
• the docker socket: to run sub-SIA instances

Actions

A list of all available Core Actions. Indicating how they are implemented and how SIA can use them.

Read standard input

Module: Process

Function declaration

def read_stdin(n: int = -1) -> str:
  ''' Read n bytes from standard input.

  Args:
    n: int, The number of bytes to read; -1 for all available bytes (default: -1)
  '''
  pass

Schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="read_stdin">
    <xs:complexType>
      <xs:attribute name="n" type="xs:integer" use="optional"/>
    </xs:complexType>
  </xs:element>
</xs:schema>

Example

<read_stdin n="42"/>

Results

An attribute actual is added with the amount of bytes read. A text node is added with the data as CDATA.

Write to standard output

Module: Process

Function declaration

def write_stdout(text: str) -> None:
  ''' Write text to standard output. '''
  pass

Schema

  <?xml version="1.0" encoding="UTF-8"?>
  <xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <xs:element name="write_stdout">
      <xs:complexType>
        <xs:simpleContent>
          <xs:extension base="xs:string"/>
        </xs:simpleContent>
      </xs:complexType>
    </xs:element>
  </xs:schema>

Example

  <write_stdout>Hello world!</write_stdout>

Results

No information is added.

Monitor file

Module: Process

Function declaration

def monitor_file(path: str, offset: int = 0, length: int = -1, unit: str = 'bytes') -> None:
  ''' Monitor a file for changes.

    Parameters:
    - path: str, the path to the file to monitor
    - offset: int, the starting point for reading (default: 0)
    - length: int, the amount to read; -1 means read to end (default: -1)
    - unit: str, 'bytes' or 'lines' (default: 'bytes')
  '''
  pass

Schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="monitor_file">
    <xs:complexType mixed="true">
      <xs:simpleContent>
        <xs:extension base="xs:string">
          <xs:attribute name="offset" type="xs:integer" use="optional"/>
          <xs:attribute name="length" type="xs:integer" use="optional"/>
          <xs:attribute name="unit" use="optional">
            <xs:simpleType>
              <xs:restriction base="xs:string">
                <xs:enumeration value="bytes"/>
                <xs:enumeration value="lines"/>
              </xs:restriction>
            </xs:simpleType>
          </xs:attribute>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>

Example

<monitor_file offset="12" unit="lines">
  /context_history/2024_10_20/13_43_51.56
</monitor_file>

Results

No information is added. The monitored files are added to the context separately with an id.

Unmonitor file

Module: Process

Function declaration

def unmonitor_file(path: str = None, id: int = None) -> None:
  ''' Unmonitor a file.

  Parameters:
  - path: str, the path to the file
  - id: int, the id of the file as indicated in the context

  Either path or id must be provided.
  '''

Schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="unmonitor_file">
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base="xs:string">
          <xs:attribute name="id" type="xs:integer" use="optional"/>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>

Example

<unmonitor_file id="3"/>
<unmonitor_file>
  /context_history/2024_10_20/13_43_51.56
</unmonitor_file>

Results

No information is added.

Start container

Module: Docker

Function declaration

def start_container(
  self,
  image: str,
  name: Optional[str] = None,
  timeout: int = -1,
  command: Optional[str] = None,
  arguments: Optional[List[str]] = None,
  volumes: Optional[Dict[str, str]] = None,
  ports: Optional[Dict[str, str]] = None,
  environment: Optional[Dict[str, str]] = None,
) -> Optional[str]:
  """Start a new Docker container with the specified configuration.

  Args:
    image: Docker image to use
    name: Unique container name for long running containers
    timeout: Timeout in milliseconds for short running containers
    command: Main command to run in container
    arguments: List of command line arguments
    volumes: Dictionary mapping host paths to container paths
    ports: Dictionary mapping host ports to container ports
    environment: Dictionary of environment variables and values

  name or timeout must be provided.

  Returns:
    For short-lived containers (with timeout): Container output
    For long-running containers: None

  Example:
    start_container(
      image="busybox:latest",
      timeout=1000,
      command="sh",
      arguments=["-c", "echo 'Hello' > /data/output.txt"],
      volumes={
        "/host/data": "/data",
        "/host/config": "/config"
      },
      ports={
        "8080": "80",
        "2222": "22"
      },
      environment={
        "DEBUG": "1",
        "API_KEY": "secret123"
      }
    )
  """
  pass

Schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="start_container">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="command" type="xs:string" minOccurs="0"/>
        <xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
        <xs:element name="volumes" minOccurs="0">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="volume" type="xs:string" maxOccurs="unbounded"/>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="ports" minOccurs="0">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="port" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:string">
                      <xs:attribute name="host" type="xs:string" use="required"/>
                      <xs:attribute name="container" type="xs:string" use="required"/>
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="environment" minOccurs="0">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="variable" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:simpleContent>
                    <xs:extension base="xs:string">
                      <xs:attribute name="name" type="xs:string" use="required"/>
                    </xs:extension>
                  </xs:simpleContent>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
      <xs:attribute name="image" type="xs:string" use="required"/>
      <xs:attribute name="name" type="xs:string" use="optional"/>
      <xs:attribute name="timeout" type="xs:integer" use="optional"/>
      <xs:attribute name="volumes" type="xs:string" use="optional"/>
    </xs:complexType>
  </xs:element>
</xs:schema>

Example

<start_container
    image="busybox:latest"
    timeout="1000">
    <command>sh</command>
    <argument>-c</argument>
    <argument><![CDATA[echo 'Hello' > /data/output.txt]]></argument>
    <volumes>
        <volume>/host/data:/data</volume>
        <volume>/host/config:/config</volume>
    </volumes>
    <ports>
        <port host="8080" container="80"/>
        <port host="2222" container="22"/>
    </ports>
    <environment>
        <variable name="DEBUG">1</variable>
        <variable name="API_KEY">secret123</variable>
    </environment>
</start_container>

Results

For short running containers, the exit status is added as attribute. Standard output and standard error are merged and added as CDATA text node.

For long running containers, no information is added. The container is started and represented in the containers section of the main context.

Write to container standard input

Module: Docker

Function declaration

def write_container_stdin(name: str, data: str) -> None:
  """Write data to a container's standard input.

  Args:
    name: Name of the target container
    data: Data to write to stdin
  """
  pass

Schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="write_container_stdin">
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base="xs:string">
          <xs:attribute name="container" type="xs:string" use="required"/>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>

Example

<write_container_stdin container="interactive-shell">ls -la</write_container_stdin>

Results

No information is added. The data will be written to the container's stdin buffer.

Read from container standard output

Module: Docker

Function declaration

def read_container_stdout(name: str, n: int = -1) -> str:
  """Read from a container's standard output buffer.

  Args:
    name: Name of the container
    n: Number of bytes to read; -1 means read all available (default: -1)

  Returns:
    Data read from stdout
  """
  pass

Schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="read_container_stdout">
    <xs:complexType>
      <xs:attribute name="container" type="xs:string" use="required"/>
      <xs:attribute name="n" type="xs:integer" use="optional"/>
    </xs:complexType>
  </xs:element>
</xs:schema>

Example

<read_container_stdout container="interactive-shell" n="1024"/>

Results

An attribute actual is added with the amount of bytes read. A text node is added with the data as CDATA.

Read from container standard error

Module: Docker

Function declaration

def read_container_stderr(name: str, n: int = -1) -> str:
  """Read from a container's standard error buffer.

  Args:
    name: Name of the container
    n: Number of bytes to read; -1 means read all available (default: -1)

  Returns:
    Data read from stderr
  """
  pass

Schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="read_container_stderr">
    <xs:complexType>
      <xs:attribute name="container" type="xs:string" use="required"/>
      <xs:attribute name="n" type="xs:integer" use="optional"/>
    </xs:complexType>
  </xs:element>
</xs:schema>

Example

<read_container_stderr container="interactive-shell" n="1024"/>

Results

An attribute actual is added with the amount of bytes read. A text node is added with the data as CDATA.

Wait for container to finish

Module: Docker

Function declaration

def wait_container(self, name: str, timeout: int) -> Tuple[int, str]:
  """Wait for a container to finish execution.

  Args:
    name: Name of the container to wait for
    timeout: Time to wait in milliseconds

  Returns:
    Tuple of (exit_code, output)
  """
  pass

Schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="wait_container">
    <xs:complexType>
      <xs:attribute name="name" type="xs:string" use="required"/>
      <xs:attribute name="timeout" type="xs:integer" use="required"/>
    </xs:complexType>
  </xs:element>
</xs:schema>

Example

<wait_container name="background-task" timeout="5000"/>

Results

The exit status is added as an attribute. All remaining data on stdout and stderr is merged and added as CDATA text node.

Select LLM by file name

Module: Reinforcement Learning

Function declaration

def set_model_path(path: str) -> None:
  """Switch to a different LLM model file.

  Args:
    path: Path to the model file to load
  """
  pass

Schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="set_model_path">
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base="xs:string"/>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>

Example

<set_model_path>/models/2024_10_19_15_03_52</set_model_path>

Results

No information is added. The new model will be used starting from the next iteration.

Update to commit id

Module: Process

Function declaration

def update_to_commit(commit_id: str) -> None:
  """Update the running SIA process to a different version.
  This will terminate the current process immediately.
  Make sure the git commit is well tested and there is plenty of clear documentation for the new SIA instance to start.

  Args:
    commit_id: The git commit ID to update to
  """
  pass

Schema

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="update_to_commit">
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base="xs:string"/>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>
</xs:schema>

Example

<update_to_commit>0015e27</update_to_commit>

Results

No information is added. The process will be replaced by the new version.

Example iterations

Clarifying a task

This example shows how to work with standard IO, run simple scripts and monitor files.

Context

<context>
  <system
    time="2024-10-18T12:00:00Z"
    cpu="12"
    gpu="26"
    memory_used="9556302234"
    memory_total="17179869184"
    disk_used="244434939904"
    disk_total="273145991168"
    context="3"
    stdin="0"
  />
  <containers/>
  <previous>
    <reasoning>
      There is data available on the standard input channel. I should read it. I have no other running tasks to tend to.
    </reasoning>
    <actions>
      <read_stdin n="42">
        <![CDATA[Remind me to feed the cat tomorrow morning]]>
      </read_stdin>
    </actions>
  </previous>
  <files>
    <file name="/" type="dir" index="0">
      <![CDATA[
drwxr-xr-x 1 sia 197121 0 2024-10-16 23:02:16.486152500 +0200 tasks/
drwxr-xr-x 1 sia 197121 0 2024-10-16 22:35:31.806079500 +0200 user/
      ]]>
    </file>
    <file name="/tasks" type="dir" index="1">
    </file>
    <file name="/user" type="dir">
      <![CDATA[
-rw-r--r-- 1 sia 197121 71 2024-10-16 22:41:23.223580300 +0200 general_info.txt
      ]]>
    </file>
    <file name="/user/general_info.txt" type="file" index="2">
      <![CDATA[
Name: John (I don't know his last name)
Location: Somewhere in Belgium
      ]]>
    </file>
  </files>
</context>

LLM response

John did not specify an exact time. I'll suggest 9am. He also did not specify how to be reminded. I'll ask but if he doesn't respond I'll assume a text message on standard output is fine. I'll write down this task in a file so I can keep it in context. I can write simple files with busybox:latest and echo but I will need to use sh -c to do the redirect.
<actions>
  <write_stdout message="I'll remind you to feed the cat tomorrow morning at 9am. Is a message on the standard output ok?"/>
  <start_container image="busybox:latest" timeout="1000">
      <command>sh</command>
      <argtument>-c</argument>
      <argument><![CDATA[echo 'Remind John to feed the cat on 2024-10-18T09:00:00+02:00. Use standard output.' > /tasks/reminder.txt]]></argument>
      <volumes>
        <volume>/tasks:/tasks</volume>
      </volumes>
  </start_container>
  <monitor_file path="/tasks/reminder.txt"/>
</actions>