# SIA - The Self Improving Agent SIA is an agentic artificial intelligence system that autonomously completes complex tasks by writing and executing scripts. It uses a Large Language Model (LLM) which operates in a loop. Each iteration a context is updated with system info and a list of previous reasoning and actions. The agent responds with a new reasoning or an action. Context, reasoning and actions are stored in a file for each iteration. SIA can read past iterations to improve its reasoning and actions. It can improve in several ways: - By finetuning the LLM with a better reasoning or action for a given context - By modifying its own source code - By refining Procedures - By developing Tools ## Example This example shows a typical context with some monitored items and previous actions. Between each of the responses, the context would be updated. ### Context ```xml ``` ### Responses Start by reasoning about the task. ```xml John did not specify an exact time. I'll suggest 9am. He also did not specify how to be reminded. I'll ask but if he doesn't respond I'll assume a text message on standard output is fine. I'll write down this task in a file so I remember it even on a power failure. ``` Store important information on disk. ```xml /root/data/tasks/reminder_to_feed_cat.txt]]> ``` Respond to the user. ```xml I'll remind you to feed the cat tomorrow morning at 9am. Is a message on the standard output ok? ``` Clear initial reasoning. ```xml ``` The conversation is kept in context to understand the user's expected response. If the context was near full, it would be summarized and cleaned up. The `single` output is also kept in context. If the file was updated often, it could be replaced by a repeated `cat`, like the general info. ## Working principles The main context is regenerated for each iteration. It contains info about the system and previous actions that have not been deleted. Together with the system prompt and available core actions it forms the prompt for the LLM. The LLM responds with one core action. ### Core Actions There are only a few core actions: - Starting a script - Deleting data from context - Stopping SIA - Reading standard input - Writing to standard output - Reasoning #### Scripts Scripts can run in one of 2 modes: single-shot or repeat. Their mode and output (stdout and stderr) stay in the context until they are explicitly removed. In this way the agent manages what information is available in the context. ##### Single-shot script The script is executed once. This is useful for most operations e.g. writing to or moving a file or downloading content from the internet. The next iteration starts after the scripts has finished. ##### Repeat script The script is restarted on each iteration. This is useful for monitoring files or the file system. commands like `head` and `tail` can be used to limit the data in context. The next iteration starts after all repeat scripts in context have finished. ### Processes in SIA SIA operates through a coordinated system of processes, each with specialized responsibilities. This choice is driven by dependency isolation for the llm engine implementations, and the ability to use namespaces for process isolation of sia instances. #### Main SIA Process The core SIA application runs as a continuous process that cycles through its context generation, LLM interaction, and action execution loop. This process is typically managed by the `restart.sh` script, which ensures SIA restarts whenever it stops. This restart mechanism is a critical part of how SIA implements self-improvement: 1. When SIA makes changes to its own code, it terminates with a special exit code (42) 2. The restart script detects this exit code and restarts SIA 3. Upon restart, SIA loads the modified Python files, effectively "installing" its own updates #### Testing Instances SIA can create isolated test instances of itself to evaluate improvements and test capabilities. These instances run in separate process spaces with their own resources and filesystem views, managed by the tool Bubblewrap. This isolation ensures that test instances don't interfere with each other or the main SIA instance while allowing observation of their behavior. Sub instances are explained in `procedures/self_improvement/reasoning.md`. #### Web Server for Human Interaction SIA can be started with an optional `--server` flag. This starts a web server that can be used to interact with SIA. It is made for debugging and stearing the model until it is properly trained. The web interface takes over standard input and output. It will display the context for editing before handing it to the LLM. After each run of the LLM, before parsing, it will display the response. The web server uses WebSockets to maintain real-time communication with connected clients, broadcasting state updates as they occur and processing commands from the interface. #### LLM Engine Subprocesses SIA communicates with LLM engines through dedicated subprocesses rather than directly integrating them into the main application. Each LLM type (Gemma, QwQ, Mistral, etc.) runs in its own subprocess with a tailored environment. This architecture provides several advantages: 1. **Dependency Isolation**: Different LLM implementations often have conflicting dependency requirements. By running each in a separate subprocess with its own virtual environment, these conflicts are avoided. 2. **Resource Management**: LLM engines can be resource-intensive. The subprocess approach allows for clean termination and resource reclamation when switching between models or canceling generation. 3. **Implementation Simplicity**: New LLM types can be added by implementing a focused subprocess runner without modifying the core SIA agent code. ### Use of XML XML plays several crucial roles throughout SIA's architecture as a structured data format for different communication interfaces. The consistent use of XML throughout SIA provides a unified approach to data representation, validation, and communication between components. But because of how SIA operates it's necessary to treat some data as plain text. #### Context and Entry Representation The context and entries are formatted as XML before presenting them to the LLM. CDATA sections keep escaping to a minimum. This would not be the case when using e.g. json. Entry id's and the delete action allow the LLM to manage it's own context. **Example:** ```xml ``` #### XML formatting During context compilation, the XML formatter wraps text content in CDATA sections when possible, falling back to standard XML escaping when content contains CDATA closing sequences (`]]>`). Because of how newlines are added for formatting, all data should be trimmed from leading and trailing whitespace when reading. **Example of CDATA usage:** ```xml /tmp/test.txt]]> ``` **Example of XML escaping when CDATA cannot be used:** ```xml I noticed that the file contains a CDATA end marker like this: ]]> I need to be careful when processing this content. ``` Notice how the content of this rendered entry can differ from the generated text. The LLM needs to be trained to handle this properly. #### XML Schema Validation Responses from the LLM are validated against an XML schema (`action_schema.xsd`) that defines the structure and requirements for valid actions. This ensures only supported actions are executed with required attributes. **Example schema definition:** ```xml ``` **Example validation error:** ```xml Missing required attribute 'id' on element 'delete' ]]> ``` #### Logits Processing with XML Schema SIA uses a custom XML schema validator (`lib/xml_schema_validator`) that can operate on token probabilities during text generation. This guides the model toward valid XML structures. It is most helpful when creating training data. Logits processing is computationally expensive. It is not supported for all LLM implementations so the SIA core should not make assumptions on the generated text. #### Iteration Logging All iterations of context-response pairs are stored in XML files, providing a structured record of agent behavior. **Example iteration log file:** ```xml <reasoning> I should check what files are in the tasks directory to see if there are any pending tasks. </reasoning> ``` Notice how the response is stored as plaintext, even though it contains an xml reasoning action. When saving the iteration, the response is not parsed or validated yet. Storing the response as plaintext helps debugging and retains info that would otherwise be lost. E.g. delete actions do not create an entry and would be harder to find. Or xml comments used for inline reasoning are not saved after parsing. #### LLM Engine Communication LLM engine subprocesses receive input as XML documents containing paths to required files and the context. **Example input to LLM engine subprocess:** ```xml /root/sia/system_prompt.md /root/sia/action_schema.xsd ``` Though the LLM can output any text, the goal is to output valid xml. The logits processor can help enfoce this but ultimately the core application is responsible for parsing and interpreting the xml. Because the LLM can output any text, the core application can't wait until the returnd text is valid xml. To reliably indicate the end of text generation, we use the ASCII End of Transmission (EOT) character (ASCII code 4, `\u0004`). This character was chosen because: - It's specifically designed for this purpose in telecommunications protocols - It should not appear in normal generated text - It's a single byte, making it efficient to process - It's standard across all platforms **Example output from LLM engine subprocess:** ``` I should check the current state of the system and see if there are any pending tasks. \u0004 ``` The communication protocol between the SIA agent and LLM engine subprocesses has been designed with simplicity as the primary goal. Opting for a minimal approach that: - Keeps complexity on the agent side, not the llm engine - Uses familiar XML format for inputs to align with SIA's existing patterns - Utilizes standard ASCII chars to indicate EOT ### Core Components #### Agent The core of SIA is the agent, which exists in two variants: - ProceduralAgent: Runs in a simple state machine, processing context and executing actions directly - WebAgent: Gives more control on when to change state and allows human intervention and feedback through a web interface Both agent types share common components: - WorkingMemory - ResponseParser - IterationLogger - IOBuffer Interaction with these components and other shared behaviour is handled in BaseAgent. #### Working Memory The working memory stores the current state of the system through different types of entries: - SingleEntries: Output of single-shot script executions - RepeatEntry: Continuously refreshed script outputs - ReasoningEntry: LLM's thought process documentation - ParseErrorEntry: XML validation or parsing errors - IOEntry: Input/output operations Each entry can be serialized to XML for inclusion in the LLM context. Working memory is cleaned through explicit delete commands issued by the LLM. #### Command Processing SIA distinguishes between two types of LLM outputs: 1. Commands: Immediate actions that modify the system - DeleteCommand: Removes entries from working memory - StopCommand: Terminates the agent 2. Entries: Records that become part of working memory - Stay in the context until explicitly deleted - May execute once, each iteration or not at all depending on entry type #### IO Handling IO operations are abstracted through an IOBuffer interface: - StandardIOBuffer: Direct access to system stdin/stdout - WebIOBuffer: Buffer for web interface communication This abstraction allows the ResponseParser to generate consistent IOEntries regardless of agent type. ### Processing Flow #### Standard Agent Flow 1. Update system metrics and context size 2. Compile context from working memory entries 3. Process context through LLM 4. Validate XML response 5. Parse response into command or entry 6. Execute command or update working memory 7. Return to step 1 unless stopped #### Web Agent Flow 1. Update system metrics and context size 2. Compile context and await human approval 3. Process approved context through LLM 4. Validate XML response 5. Present validation results and await human review 6. Parse approved response into command or entry 7. Execute command or update working memory 8. Return to step 1 unless stopped ### Web Interface The web interface provides interactive debugging and human feedback through a WebSocket-based protocol: #### Server-Client Communication - Server Messages: - Context updates for review - Validation results from LLM responses - Output updates from script execution - State updates for UI synchronization - Client Messages: - Context approval decisions - Response modifications - User input for stdin operations #### WebServer Architecture The web server manages: - WebSocket connections to clients - Message routing to/from the WebAgent - Broadcasting state updates to all connected clients This architecture allows for: - Real-time monitoring of agent state - Interactive debugging of LLM responses - Human-in-the-loop operation for testing and development - Collection of human feedback for reinforcement learning ### Diagrams #### Core classes ```mermaid classDiagram class SystemMetrics { +SystemMetrics(sample_interval float) +generate_context(context_usage float) ElementTree +stop() void -monitor_loop() void } class LLMEngine { +LLMEngine(executable_path str) +infer(system_prompt str, main_context str, prefix str) Iterator~str~ } class BaseAgent { <> -working_memory: WorkingMemory -metrics: SystemMetrics -llm: LLMEngine -parser: ResponseParser -validator: XMLValidator -action_schema: str #_compile_context() str } class WorkingMemory { -entries: List~Entry~ +WorkingMemory() +add_entry(entry Entry) void +remove_entry(id str) void +clear() void +get_entry(id str) Optional~Entry~ +get_entries() List~Entry~ +get_entries_count() int +get_entries_by_type(type Type) List~Entry~ +update() void +generate_context() List~ElementTree~ } class XMLValidator { +XMLValidator(schema str) +validate(xml str) Optional~str~ +get_valid_root_elements() Set~str~ } class ResponseParser { -io_buffer: IOBuffer +ResponseParser(io_buffer IOBuffer) +parse(xml str) Command | Entry } class Entry { <> +id: str readonly +timestamp: datetime readonly +Entry(id str, timestamp datetime) +update() void* +generate_context() ElementTree* +cleanup() void* } class IOBuffer { <> +read() str* +write(content str) void* +buffer_length() int* } class Command { <> +execute(memory WorkingMemory) CommandResult* } SystemMetrics "1" --* "1" BaseAgent LLMEngine "1" --* "1" BaseAgent XMLValidator "1" --* "1" BaseAgent BaseAgent "1" *-- "1" IOBuffer BaseAgent "1" *-- "1" WorkingMemory BaseAgent "1" *-- "1" ResponseParser WorkingMemory "1" *-- "*" Entry ResponseParser ..> Entry ResponseParser ..> Command ``` #### Standard Agent Flow ```mermaid stateDiagram-v2 direction LR state "Standard Agent Flow" as standard_agent_flow { [*] --> UpdateSystem: Start UpdateSystem --> CompileContext: Updated Metrics & Size CompileContext --> ProcessLLM ProcessLLM --> ValidateXML: LLM Response ValidateXML --> ParseResponse: Valid XML ValidateXML --> UpdateEntries: Invalid XML\nCreate ParseErrorEntry ParseResponse --> ExecuteCommands: Command ParseResponse --> UpdateEntries: Entry ExecuteCommands --> [*]: Stop Command ExecuteCommands --> UpdateEntries: Delete Command UpdateEntries --> UpdateSystem: Continue Loop } ``` #### Web Agent ```mermaid classDiagram class BaseAgent { <> -working_memory: WorkingMemory -metrics: SystemMetrics -llm: LLMEngine -parser: ResponseParser -validator: XMLValidator -action_schema: str #_compile_context() str } class StandardAgent { +StandardAgent(model_path str, system_prompt str, action_schema str) +run() void } class WebAgent { +context: str +response: str readonly +current_state WebAgentState readonly +command_result Optional[CommandResult] readonly +validation_error Optional[str] readonly +add_state_change_handler(handler Callable) void +add_response_change_handler(handler Callable) void +approve_context() void +set__response(response str) void +approve_response() void } class WebAgentState { <> UPDATE CONTEXT_APPROVAL INFERENCE RESPONSE_APPROVAL STOPPED } class WebSocketManager { -web_sockets: Set~WebSocket~ +WebServer(agent WebAgent, io_buffer WebIOBuffer, static_files path, host str, port int) } class ClientMessage { <> APPROVE_CONTEXT APPROVE_RESPONSE MODIFY_RESPONSE SEND_INPUT } class ServerMessage { <> STATE_CHANGE CONTEXT_UPDATE RESPONSE_UPDATE OUTPUT_UPDATE VALIDATION_ERROR } class WebIOBuffer { -stdin_buffer: str -stdout_buffer: str +read() str +write(content str) void +buffer_length() int +append_stdin(content str) void +get_stdout() str +clear_stdout() void } BaseAgent <|-- WebAgent BaseAgent <|-- StandardAgent WebServer --> ClientMessage WebServer --> ServerMessage WebServer "1" *-- "1" WebIOBuffer WebServer "1" *-- "1" WebAgent WebAgent "1" *-- "1" WebAgentState ``` #### Web Agent Flow ```mermaid stateDiagram-v2 direction LR state "Web Agent Flow" as web_agent_flow { [*] --> UpdateSystem: Start UpdateSystem --> CompileContext: Updated Metrics & Size CompileContext --> WaitForContextApproval: Send Context WaitForContextApproval --> ProcessLLM: Context Approved ProcessLLM --> ValidateXML: LLM Response ValidateXML --> WaitForResponseApproval: Send Validation Result ValidateXML --> UpdateEntries: Invalid XML\nCreate ParseErrorEntry WaitForResponseApproval --> ValidateXML: Modified Response WaitForResponseApproval --> ParseResponse: Approved Response ParseResponse --> ExecuteCommands: Command ParseResponse --> UpdateEntries: Entry ExecuteCommands --> [*]: Stop Command ExecuteCommands --> UpdateEntries: Delete Command UpdateEntries --> UpdateSystem: Continue Loop } ``` #### Entry classes ```mermaid classDiagram class Entry { <> +id: str readonly +timestamp: datetime readonly +Entry(id str, timestamp datetime) +update() void* +generate_context() ElementTree* +cleanup() void* } class SingleEntry { +script: str readonly +stdout: str readonly +stderr: str readonly +exit_code: Optional~int~ readonly +Single(script str, id str, timestamp datetime) +update() void +generate_context() ElementTree } class RepeatEntry { +script: str readonly +stdout: str readonly +stderr: str readonly +exit_code: Optional~int~ readonly +RepeatEntry(script str, id str, timestamp datetime) +update() void +generate_context() ElementTree } class ReasoningEntry { +content: str readonly +ReasoningEntry(content str, id str, timestamp datetime) +update() void +generate_context() ElementTree } class ParseErrorEntry { +content: str readonly +error: str readonly +ParseErrorEntry(content str, error str, id str, timestamp datetime) +update() void +generate_context() ElementTree } class ReadEntry { +content: str readonly +ReadEntry(io_buffer IOBuffer, id str, timestamp datetime) +update() void +generate_context() ElementTree } class WriteEntry { +content: str readonly +WriteEntry(content str, io_buffer IOBuffer, id str, timestamp datetime) +update() void +generate_context() ElementTree } ReasoningEntry --|> Entry ParseErrorEntry --|> Entry ReadEntry --|> Entry Entry <|-- WriteEntry Entry <|-- SingleEntry Entry <|-- RepeatEntry ``` #### IO Buffer classes ```mermaid classDiagram class IOBuffer { <> +read() str* +write(content str) void* +buffer_length() int* } class StandardIOBuffer { +StandardIOBuffer() +read() str +write(content str) void +buffer_length() int } class WebIOBuffer { -stdin_buffer: str -stdout_buffer: str +read() str +write(content str) void +buffer_length() int +append_stdin(content str) void +get_stdout() str +clear_stdout() void } IOBuffer <|.. WebIOBuffer IOBuffer <|.. StandardIOBuffer ``` #### Command classes ```mermaid classDiagram direction LR class Command { <> +execute(memory WorkingMemory) CommandResult* } class DeleteCommand { +DeleteCommand(id str) +execute(memory WorkingMemory) CommandResult } class StopCommand { +StopCommand() +execute(memory WorkingMemory) CommandResult } class CommandResult { +message: str +success: bool +should_stop: bool +CommandResult(message str, success bool, should_stop bool) +static success() CommandResult +static failure(message str) CommandResult +static stop() CommandResult } Command <|-- DeleteCommand Command <|-- StopCommand Command -- CommandResult ```