285 lines
10 KiB
Markdown
285 lines
10 KiB
Markdown
# SIA - The Self Improving Agent
|
|
|
|
SIA is an agentic artificial intelligence system that autonomously completes complex tasks by writing and executing scripts.
|
|
It uses a Large Language Model (LLM) which operates in a loop, generating reasoning and actions based on an updating context.
|
|
SIA manages Docker containers for task execution.
|
|
These can be short-lived for e.g. bash one-liners or long-running for e.g. background tasks, training or communication.
|
|
|
|
The system implements reinforcement learning by analyzing past iterations to improve its LLM.
|
|
SIA can also modify its own source code, allowing it to adapt to new challenges.
|
|
|
|
## Working principles
|
|
This section gives a high-level overview of the main components of SIA and how they work together.
|
|
|
|
### LLM-Powered Reasoning
|
|
|
|
SIA utilizes a Large Language Model (LLM) as its core reasoning engine.
|
|
This LLM can be updated and modified over time.
|
|
|
|
The LLM is inferred in a loop.
|
|
Each iteration of the loop the system prompt and main context are provided.
|
|
The LLM generates a response with reasoning and a list of core actions to take.
|
|
|
|
The main context is always regenerated and contains:
|
|
- System status and limits
|
|
- time and date (ISO 8601 UTC)
|
|
- CPU usage (%)
|
|
- GPU usage (%)
|
|
- memory usage and total (bytes)
|
|
- disk usage and total (bytes)
|
|
- context usage (%)
|
|
- standard input buffer contents (bytes)
|
|
- List of containers
|
|
- description
|
|
- status (initializing, running, finished)
|
|
- standard IO buffer usage
|
|
- ports, volumes and environment variables
|
|
- The reasoning, actions and results of the previous iteration
|
|
- Files monitored from the filesystem
|
|
|
|
### Core Action System
|
|
|
|
Core actions are selected by the LLM by outputting an XML list of actions and parameters.
|
|
This system allows the agent to manage its memory, control containers, select wich version to run, and communicate.
|
|
The Action System parses the XML output and executes the corresponding functions.
|
|
|
|
Core commands for user interaction:
|
|
- Read standard input
|
|
- Write to standard output
|
|
|
|
### Docker Container Management
|
|
|
|
SIA utilizes Docker containers for anything not covered by core actions.
|
|
Containers can be short-lived, eg. for simple calculations.
|
|
They can also be long-lived, eg. to keep a communication channel open.
|
|
They can even run a complete SIA instance eg. for verifying updates to the LLM or core functionality.
|
|
The short-lived containers define a timeout.
|
|
The next iteration of the main loop starts when the container finishes or the timeout is reached.
|
|
The long-lived containers can also be waited on at a later point in time.
|
|
|
|
Core commands for container operations:
|
|
- Start container
|
|
- Stop container
|
|
- Write to container standard input
|
|
- Read from container standard output
|
|
- Read from container standard error
|
|
- Wait for container to finish
|
|
|
|
### Information Storage
|
|
|
|
The SIA main loop is ephemeral.
|
|
Therefore the agent needs to store information for future reference.
|
|
The agent has access to a Linux filesystem.
|
|
Files and directories in this filesystem can be mapped as volumes to containers.
|
|
The agent can load a file in its context, so it always has a view of the latest version.
|
|
The same can be done with directory listings.
|
|
|
|
Core commands for file operations:
|
|
- Monitor file (or folder)
|
|
- Unmonitor file
|
|
|
|
There are no core commands for creating, updating or deleting files.
|
|
This can be done using containers.
|
|
|
|
### Reinforcement Learning
|
|
|
|
For each iteration of the main loop, the context and the generated reasoning and actions are stored in the file system.
|
|
When the agent solves a problem it starts a search for the root cause by looking at previous iterations.
|
|
The iteration file is used for updating the LLM weights using functions in the SIA core.
|
|
The agent can access this by running a SIA container.
|
|
It can then test the updated LLM in another SIA instance in a container.
|
|
If it is acceptable it can change to this new version.
|
|
|
|
Core commands for reinforcement learning:
|
|
- Select LLM by file name
|
|
|
|
There are no specific commands for running a SIA instance in a container.
|
|
This can be done using the regular container commands.
|
|
|
|
### Self-Improvement
|
|
|
|
SIA has access to a git repository containing its source code.
|
|
It can also access a container repository with SIA builds.
|
|
With these it is possible for SIA to update and test new versions of itself.
|
|
|
|
If a new version is approved, the agent can switch to it and continue working.
|
|
|
|
Core commands for self-improvement:
|
|
- Update to docker tag
|
|
|
|
## Architecture
|
|
|
|
An overview of the key components and their interactions.
|
|
|
|
|
|
|
|
### Modules
|
|
|
|
Modules execute core commands and provide data for the main context.
|
|
|
|
- Process Module
|
|
- standard I/O operations
|
|
- waiting
|
|
- file monitoring
|
|
- updating the SIA process to another version
|
|
- Docker Module
|
|
- container operations
|
|
- container status and buffer monitoring
|
|
- Reinforcement Learning Module
|
|
- create dataset for fine-tuning the LLM
|
|
- labeling trained models
|
|
|
|
### Action System
|
|
|
|
The Action System runs the SIA main loop.
|
|
|
|
- request main context from the Context Manager
|
|
- run the LLM
|
|
- parse the LLM output
|
|
- execute the appropriate commands on the relevant modules
|
|
|
|
### Context Template
|
|
|
|
The Context Template collects information from the modules and creates the input for the LLM.
|
|
|
|
### LLM Engine
|
|
|
|
The LLM Engine is responsible for:
|
|
- Running inference based on the provided context
|
|
- Updating the model's weights during the learning process
|
|
|
|
## Implementation
|
|
|
|
This section explains technical details of the implementation.
|
|
|
|
### Use of XML
|
|
|
|
The context and actions are formatted as XML.
|
|
For the context this adds clear rules for escaping.
|
|
This is usefull in case a previous context is embedded.
|
|
|
|
The response starts with freeform reasoning followed by XML formatted actions.
|
|
In case the LLM makes a mistake it can start over.
|
|
Only the last XML block is evaluated.
|
|
|
|
XML is verbose by nature.
|
|
To avoid overflowing the context window, it should only be used where it adds value.
|
|
Directory listings, for instance, are formatted in the well known ls command format.
|
|
|
|
Parameters for actions can be passed as attributes or as child elements.
|
|
This allows the LLM to pass multiple volumes or environment variables in a clear way.
|
|
It also simplifies escaping of command line arguments.
|
|
|
|
Action results are added in the context as text nodes after the last parameter.
|
|
|
|
### Training datasets
|
|
|
|
A training dataset is a folder with these files:
|
|
- system_prompt.txt
|
|
- main_context.txt
|
|
- pre-reasoning.txt
|
|
- training_reasoning.txt
|
|
- post-reasoning.txt
|
|
- pre-actions.txt
|
|
- training_actions.txt
|
|
- post-actions.txt
|
|
|
|
The context window of the LLM is filled with all parts of the dataset in order.
|
|
The learning rate is only applied to the training reasoning and actions.
|
|
The pre and post files are optional.
|
|
|
|
To do an actual training round, a sia:latest container is started.
|
|
This is an example action that trains on two datasets with learning rate 0.1:
|
|
```xml
|
|
<start_container image="sia:latest" volumes="/tasks:/tasks">
|
|
<volumes>
|
|
<volume>/models/:/models/</volume>
|
|
<volume>/datasets/description_of_a_problem/:/datasets/description_of_a_problem/</volume>
|
|
<volume>/datasets/description_of_nother_problem/:/datasets/description_of_nother_problem/</volume>
|
|
</volumes>
|
|
<command>sia</command>
|
|
<parameter>train</parameter>
|
|
<parameter>--learning-rate</parameter>
|
|
<parameter>0.1</parameter>
|
|
<parameter>--model</parameter>
|
|
<parameter>/models/2024_10_19_08_21_41</parameter>
|
|
<parameter>--out</parameter>
|
|
<parameter>/models/2024_10_19_15_03_52</parameter>
|
|
<parameter>/datasets/description_of_a_problem/</parameter>
|
|
<parameter>/datasets/description_of_nother_problem/</parameter>
|
|
</start_container>
|
|
```
|
|
|
|
### Reinforcement learning by human feedback
|
|
|
|
The SIA container can be used in 3 ways:
|
|
- To run a SIA instance
|
|
- To update LLM weights based on a dataset
|
|
- To host the interaction web interface
|
|
|
|
The web interface is an alternative way of interacting with SIA, specifically for reinforcement learning by human feedback.
|
|
The web interface takes over standard input and output.
|
|
It each time the LLM generates a response, the web interface will display it.
|
|
The user can modify the response before the actions are executed.
|
|
|
|
## Example iterations
|
|
|
|
### Clarifying a task
|
|
This example shows how to work with standard IO, run simple scripts and monitor files.
|
|
|
|
#### Context
|
|
```xml
|
|
<context>
|
|
<system
|
|
time="2024-10-18T12:00:00Z"
|
|
cpu="12"
|
|
gpu="26"
|
|
memory_used="9556302234"
|
|
memory_total="17179869184"
|
|
disk_used="244434939904"
|
|
disk_total="273145991168"
|
|
context="3"
|
|
stdin="0"
|
|
/>
|
|
<containers/>
|
|
<previous>
|
|
<reasoning>
|
|
There is data available on the standard input channel. I should read it. I have no other running tasks to tend to.
|
|
</reasoning>
|
|
<actions>
|
|
<read_stdin n="42">
|
|
Remind me to feed the cat tomorrow morning
|
|
</read_stdin>
|
|
</actions>
|
|
</previous>
|
|
<files>
|
|
<file name="/" type="dir" index="0">
|
|
drwxr-xr-x 1 sia 197121 0 2024-10-16 23:02:16.486152500 +0200 tasks/
|
|
drwxr-xr-x 1 sia 197121 0 2024-10-16 22:35:31.806079500 +0200 user/
|
|
</file>
|
|
<file name="/tasks" type="dir" index="1">
|
|
</file>
|
|
<file name="/user" type="dir">
|
|
-rw-r--r-- 1 sia 197121 71 2024-10-16 22:41:23.223580300 +0200 general_info.txt
|
|
</file>
|
|
<file name="/user/general_info.txt" type="file" index="2">
|
|
Name: John (I don't know his last name)
|
|
Location: Somewhere in Belgium
|
|
</file>
|
|
</files>
|
|
</context>
|
|
```
|
|
|
|
#### LLM response
|
|
```xml
|
|
John did not specify an exact time. I'll suggest 9am. He also did not specify how to be reminded. I'll ask but if he doesn't respond I'll assume a text message on standard output is fine. I'll write down this task in a file so I can keep it in context. I can write simple files with busybox:latest and echo but I will need to use sh -c to do the redirect.
|
|
<actions>
|
|
<write_stdout message="I'll remind you to feed the cat tomorrow morning at 9am. Is a message on the standard output ok?"/>
|
|
<start_container image="busybox:latest" timeout="1000" volumes="/tasks:/tasks">
|
|
<command>sh</command>
|
|
<parameter>-c</parameter>
|
|
<parameter><![CDATA[echo 'Remind John to feed the cat on 2024-10-18T09:00:00+02:00. Use standard output.' > /tasks/reminder.txt]]></parameter>
|
|
</start_container>
|
|
<monitor_file path="/tasks/reminder.txt"/>
|
|
</actions>
|
|
``` |