SIA/readme.md

SIA - The Self Improving Agent

SIA is an agentic artificial intelligence system that autonomously completes complex tasks by writing and executing scripts. It uses a Large Language Model (LLM) which operates in a loop, generating reasoning and actions based on an updating context. SIA manages Docker containers for task execution. These can be short-lived for e.g. bash one-liners or long-running for e.g. background tasks, training or communication.

The system implements reinforcement learning by analyzing past iterations to improve its LLM. SIA can also modify its own source code, allowing it to adapt to new challenges.

Working principles

This section gives a high-level overview of the main components of SIA and how they work together.

LLM-Powered Reasoning

SIA utilizes a Large Language Model (LLM) as its core reasoning engine. This LLM can be updated and modified over time.

The LLM is inferred in a loop. Each iteration of the loop the system prompt and main context are provided. The LLM generates a response with reasoning and a list of core actions to take.

The main context is always regenerated and contains:

• System status and limits
  • time and date (ISO 8601 UTC)
  • CPU usage (%)
  • GPU usage (%)
  • memory usage and total (bytes)
  • disk usage and total (bytes)
  • context usage (%)
  • standard input buffer contents (bytes)
• List of containers
  • description
  • status (initializing, running, finished)
  • standard IO buffer usage
  • ports, volumes and environment variables
• The reasoning, actions and results of the previous iteration
• Files monitored from the filesystem

Core Action System

Core actions are selected by the LLM by outputting an XML list of actions and parameters. This system allows the agent to manage its memory, control containers, select wich version to run, and communicate. The Action System parses the XML output and executes the corresponding functions.

Core commands for user interaction:

• Read standard input
• Write to standard output

Docker Container Management

SIA utilizes Docker containers for anything not covered by core actions. Containers can be short-lived, eg. for simple calculations. They can also be long-lived, eg. to keep a communication channel open. They can even run a complete SIA instance eg. for verifying updates to the LLM or core functionality. The short-lived containers define a timeout. The next iteration of the main loop starts when the container finishes or the timeout is reached. The long-lived containers can also be waited on at a later point in time.

Core commands for container operations:

• Start container
• Stop container
• Write to container standard input
• Read from container standard output
• Read from container standard error
• Wait for container to finish

Information Storage

The SIA main loop is ephemeral. Therefore the agent needs to store information for future reference. The agent has access to a Linux filesystem. Files and directories in this filesystem can be mapped as volumes to containers. The agent can load a file in its context, so it always has a view of the latest version. The same can be done with directory listings.

Core commands for file operations:

• Monitor file (or folder)
• Unmonitor file

There are no core commands for creating, updating or deleting files. This can be done using containers.

Reinforcement Learning

For each iteration of the main loop, the context and the generated reasoning and actions are stored in the file system. When the agent solves a problem it starts a search for the root cause by looking at previous iterations. The iteration file is used for updating the LLM weights using functions in the SIA core. The agent can access this by running a SIA container. It can then test the updated LLM in another SIA instance in a container. If it is acceptable it can change to this new version.

Core commands for reinforcement learning:

• Select LLM by file name

There are no specific commands for running a SIA instance in a container. This can be done using the regular container commands.

Self-Improvement

SIA has access to a git repository containing its source code. It can also access a container repository with SIA builds. With these it is possible for SIA to update and test new versions of itself.

If a new version is approved, the agent can switch to it and continue working.

Core commands for self-improvement:

• Update to docker tag

Architecture

An overview of the key components and their interactions.

Modules

Modules execute core commands and provide data for the main context.

• Process Module
  • standard I/O operations
  • waiting
  • file monitoring
  • updating the SIA process to another version
• Docker Module
  • container operations
  • container status and buffer monitoring
• Reinforcement Learning Module
  • create dataset for fine-tuning the LLM
  • labeling trained models

Action System

The Action System runs the SIA main loop.

• request main context from the Context Manager
• run the LLM
• parse the LLM output
• execute the appropriate commands on the relevant modules

Context Template

The Context Template collects information from the modules and creates the input for the LLM.

LLM Engine

The LLM Engine is responsible for:

• Running inference based on the provided context
• Updating the model's weights during the learning process

Implementation

This section explains technical details of the implementation.

Use of XML

The context and actions are formatted as XML. For the context this adds clear rules for escaping. This is usefull in case a previous context is embedded.

The response starts with freeform reasoning followed by XML formatted actions. In case the LLM makes a mistake it can start over. Only the last XML block is evaluated.

XML is verbose by nature. To avoid overflowing the context window, it should only be used where it adds value. Directory listings, for instance, are formatted in the well known ls command format.

Parameters for actions can be passed as attributes or as child elements. This allows the LLM to pass multiple volumes or environment variables in a clear way. It also simplifies escaping of command line arguments.

Action results are added in the context as text nodes after the last parameter.

Training datasets

A training dataset is a folder with these files:

• system_prompt.txt
• main_context.txt
• pre-reasoning.txt
• training_reasoning.txt
• post-reasoning.txt
• pre-actions.txt
• training_actions.txt
• post-actions.txt

The context window of the LLM is filled with all parts of the dataset in order. The learning rate is only applied to the training reasoning and actions. The pre and post files are optional.

To do an actual training round, a sia:latest container is started. This is an example action that trains on two datasets with learning rate 0.1:

<start_container image="sia:latest" volumes="/tasks:/tasks">
    <volumes>
      <volume>/models/:/models/</volume>
      <volume>/datasets/description_of_a_problem/:/datasets/description_of_a_problem/</volume>
      <volume>/datasets/description_of_nother_problem/:/datasets/description_of_nother_problem/</volume>
    </volumes>
    <command>sia</command>
    <parameter>train</parameter>
    <parameter>--learning-rate</parameter>
    <parameter>0.1</parameter>
    <parameter>--model</parameter>
    <parameter>/models/2024_10_19_08_21_41</parameter>
    <parameter>--out</parameter>
    <parameter>/models/2024_10_19_15_03_52</parameter>
    <parameter>/datasets/description_of_a_problem/</parameter>
    <parameter>/datasets/description_of_nother_problem/</parameter>
</start_container>

Reinforcement learning by human feedback

The SIA container can be used in 3 ways:

• To run a SIA instance
• To update LLM weights based on a dataset
• To host the interaction web interface

The web interface is an alternative way of interacting with SIA, specifically for reinforcement learning by human feedback. The web interface takes over standard input and output. It each time the LLM generates a response, the web interface will display it. The user can modify the response before the actions are executed.

Example iterations

Clarifying a task

This example shows how to work with standard IO, run simple scripts and monitor files.

Context

<context>
  <system
    time="2024-10-18T12:00:00Z"
    cpu="12"
    gpu="26"
    memory_used="9556302234"
    memory_total="17179869184"
    disk_used="244434939904"
    disk_total="273145991168"
    context="3"
    stdin="0"
  />
  <containers/>
  <previous>
    <reasoning>
      There is data available on the standard input channel. I should read it. I have no other running tasks to tend to.
    </reasoning>
    <actions>
      <read_stdin n="42">
        Remind me to feed the cat tomorrow morning
      </read_stdin>
    </actions>
  </previous>
  <files>
    <file name="/" type="dir" index="0">
drwxr-xr-x 1 sia 197121 0 2024-10-16 23:02:16.486152500 +0200 tasks/
drwxr-xr-x 1 sia 197121 0 2024-10-16 22:35:31.806079500 +0200 user/
    </file>
    <file name="/tasks" type="dir" index="1">
    </file>
    <file name="/user" type="dir">
-rw-r--r-- 1 sia 197121 71 2024-10-16 22:41:23.223580300 +0200 general_info.txt
    </file>
    <file name="/user/general_info.txt" type="file" index="2">
Name: John (I don&apos;t know his last name)
Location: Somewhere in Belgium
    </file>
  </files>
</context>

LLM response

John did not specify an exact time. I'll suggest 9am. He also did not specify how to be reminded. I'll ask but if he doesn't respond I'll assume a text message on standard output is fine. I'll write down this task in a file so I can keep it in context. I can write simple files with busybox:latest and echo but I will need to use sh -c to do the redirect.
<actions>
  <write_stdout message="I'll remind you to feed the cat tomorrow morning at 9am. Is a message on the standard output ok?"/>
  <start_container image="busybox:latest" timeout="1000" volumes="/tasks:/tasks">
      <command>sh</command>
      <parameter>-c</parameter>
      <parameter><![CDATA[echo 'Remind John to feed the cat on 2024-10-18T09:00:00+02:00. Use standard output.' > /tasks/reminder.txt]]></parameter>
  </start_container>
  <monitor_file path="/tasks/reminder.txt"/>
</actions>