llm/SIA
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

WIP continue update readme

Browse Source
This commit is contained in:
geens
2024-10-27 16:58:01 +01:00
parent 0bfc70a953
commit 705f17c47b
1 changed files with 41 additions and 674 deletions
Download Patch File Download Diff File Expand all files Collapse all files

715
readme.md
View File

@@ -67,47 +67,6 @@ This is how SIA can be updated.
SIA can also run SIA processes as script. SIA can also run SIA processes as script.
This can be used for testing updates to the LLM or core functionality. This can be used for testing updates to the LLM or core functionality.
## Architecture
An overview of the key components and their interactions.
![SIA Component Model](./diagrams/SIA_Component_Model.svg)
### Modules
Modules execute core commands and provide data for the context template.
- Process Module
- standard I/O operations
- waiting
- file monitoring
- updating the SIA process to another version
- Docker Module
- container operations
- container status and buffer monitoring
- Reinforcement Learning Module
- create dataset for fine-tuning the LLM
- labeling trained models
### Agent Core
The Agent Core runs the SIA main loop.
- template the context
- run the LLM
- parse the LLM output
- execute the appropriate actions using the relevant modules
### LLM Engine
The LLM Engine is responsible for:
- Running inference based on the provided context
- Updating the model's weights during the learning process
## Implementation
This section explains technical details of the implementation.
### Use of XML ### Use of XML
The context and actions are formatted as XML. The context and actions are formatted as XML.
@@ -118,655 +77,63 @@ The response starts with freeform reasoning followed by XML formatted actions.
In case the LLM makes a mistake it can start over. In case the LLM makes a mistake it can start over.
Only the last XML block is evaluated. Only the last XML block is evaluated.
XML is verbose by nature. Action results are added in the context in the previous_iteration section.
To avoid overflowing the context window, it should only be used where it adds value.
Directory listings, for instance, are formatted in the well known ls command format.
Parameters for actions can be passed as attributes or as child elements. ### Server for debuggin and human input
This allows the LLM to pass multiple volumes or environment variables in a clear way.
It also simplifies escaping of command line arguments.
Action results are added in the context as text nodes after the last parameter. SIA can be started with an optional `--server` flag.
This starts a web server that can be used to interact with SIA.
### Context Template It is made, specifically for reinforcement learning by human feedback.
A Handlerbars template is used to create the context for the LLM.
A ContextTemplate object is created for each iteration of the main loop.
The template is filled with data by the Agent Core.
### Training datasets
A training dataset is a folder with these files:
- system_prompt.txt
- main_context.txt
- pre-reasoning.txt
- training_reasoning.txt
- post-reasoning.txt
- pre-actions.txt
- training_actions.txt
- post-actions.txt
The context window of the LLM is filled with all parts of the dataset in order.
The learning rate is only applied to the training reasoning and actions.
The pre and post files are optional.
To do an actual training round, a sia:latest container is started.
This is an example action that trains on two datasets with learning rate 0.1:
```xml
<start_container image="sia:latest" volumes="/tasks:/tasks">
<volumes>
<volume>/models/:/models/</volume>
<volume>/datasets/description_of_a_problem/:/datasets/description_of_a_problem/</volume>
<volume>/datasets/description_of_nother_problem/:/datasets/description_of_nother_problem/</volume>
</volumes>
<command>sia</command>
<argument>train</argument>
<argument>--learning-rate</argument>
<argument>0.1</argument>
<argument>--model</argument>
<argument>/models/2024_10_19_08_21_41</argument>
<argument>--out</argument>
<argument>/models/2024_10_19_15_03_52</argument>
<argument>/datasets/description_of_a_problem/</argument>
<argument>/datasets/description_of_nother_problem/</argument>
</start_container>
```
### Reinforcement learning by human feedback
The SIA container can be used in 3 ways:
- To run a SIA instance
- To update LLM weights based on a dataset
- To host the interaction web interface
The web interface is an alternative way of interacting with SIA, specifically for reinforcement learning by human feedback.
The web interface takes over standard input and output. The web interface takes over standard input and output.
It each time the LLM generates a response, the web interface will display it. It will display the context for editing before handing it to the LLM.
The user can modify the response before the actions are executed. After each run of the LLM, before parsing, it will display the reasoning and actions.
It interactively displays if the actions can be parsed.
At any time, the user can write to the standard input of SIA.
### Project structure ## Architecture
The SIA application is developed in the src directory. An overview of the key components and their interactions.
The tests directory contains unit tests, mock objects and integration tests.
The model directory contains the trained model.
It is excluded from the git repository and the docker context because it is too large.
The docker file has a separate stage for testing. ![SIA Component Model](./diagrams/SIA_Component_Model.svg)
The `test.sh` script builds this stage, runs the tests and removes the test image.
To use SIA several directories have to be mounted: ### Modules
- `/root/model': The model directory
- `/root/sia_repo': The git repository
- the docker socket: to run sub-SIA instances
## Actions Modules execute core commands and provide data for the context template.
A list of all available Core Actions. - System Module
Indicating how they are implemented and how SIA can use them. - System information
- SIA stdio operations
- Stopping SIA (possibly triggering an update)
- Process Module
- Starting scripts
- Stopping scripts
- Managing process stdio and status
### Read standard input ### Agent Core
Module: Process The Agent Core runs the SIA main loop.
This loop consists of:
- Templating the context
- Running the LLM
- Parsing the LLM output
- Rerunning the LLM if the output cannot be parsed
- Executing the appropriate actions
#### Function declaration ### Server Core
```python
def read_stdin(n: int = -1) -> str:
''' Read n bytes from standard input.
Args: The Server Core is an alternative for the Agent Core.
n: int, The number of bytes to read; -1 for all available bytes (default: -1) It runs a modified main loop and ues the WebSystem Module.
''' This is an extension of the System Module redirecting stdio to the web interface.
pass
```
#### Schema ### LLM Engine
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="read_stdin">
<xs:complexType>
<xs:attribute name="n" type="xs:integer" use="optional"/>
</xs:complexType>
</xs:element>
</xs:schema>
```
#### Example The LLM Engine does the LLM inference.
```xml It takes a context and returns an iterator of tokens.
<read_stdin n="42"/>
```
#### Results ### Inference Result
An attribute `actual` is added with the amount of bytes read. An Inference Result object contains the resoning and parsed actions.
A text node is added with the data as `CDATA`. Parsing is part of the Inference Result constructor.
### Write to standard output
Module: Process
Function declaration
```python
def write_stdout(text: str) -> None:
''' Write text to standard output. '''
pass
```
Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="write_stdout">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string"/>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:schema>
```
Example
```xml
<write_stdout>Hello world!</write_stdout>
```
#### Results
No information is added.
### Monitor file
Module: Process
#### Function declaration
```python
def monitor_file(path: str, offset: int = 0, length: int = -1, unit: str = 'bytes') -> None:
''' Monitor a file for changes.
Parameters:
- path: str, the path to the file to monitor
- offset: int, the starting point for reading (default: 0)
- length: int, the amount to read; -1 means read to end (default: -1)
- unit: str, 'bytes' or 'lines' (default: 'bytes')
'''
pass
```
#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="monitor_file">
<xs:complexType mixed="true">
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="offset" type="xs:integer" use="optional"/>
<xs:attribute name="length" type="xs:integer" use="optional"/>
<xs:attribute name="unit" use="optional">
<xs:simpleType>
<xs:restriction base="xs:string">
<xs:enumeration value="bytes"/>
<xs:enumeration value="lines"/>
</xs:restriction>
</xs:simpleType>
</xs:attribute>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:schema>
```
#### Example
```xml
<monitor_file offset="12" unit="lines">
/context_history/2024_10_20/13_43_51.56
</monitor_file>
```
#### Results
No information is added.
The monitored files are added to the context separately with an id.
### Unmonitor file
Module: Process
#### Function declaration
```python
def unmonitor_file(path: str = None, id: int = None) -> None:
''' Unmonitor a file.
Parameters:
- path: str, the path to the file
- id: int, the id of the file as indicated in the context
Either path or id must be provided.
'''
```
#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="unmonitor_file">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="id" type="xs:integer" use="optional"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:schema>
```
#### Example
```xml
<unmonitor_file id="3"/>
<unmonitor_file>
/context_history/2024_10_20/13_43_51.56
</unmonitor_file>
```
#### Results
No information is added.
### Start container
Module: Docker
#### Function declaration
```python
def start_container(
self,
image: str,
name: Optional[str] = None,
timeout: int = -1,
command: Optional[str] = None,
arguments: Optional[List[str]] = None,
volumes: Optional[Dict[str, str]] = None,
ports: Optional[Dict[str, str]] = None,
environment: Optional[Dict[str, str]] = None,
) -> Optional[str]:
"""Start a new Docker container with the specified configuration.
Args:
image: Docker image to use
name: Unique container name for long running containers
timeout: Timeout in milliseconds for short running containers
command: Main command to run in container
arguments: List of command line arguments
volumes: Dictionary mapping host paths to container paths
ports: Dictionary mapping host ports to container ports
environment: Dictionary of environment variables and values
name or timeout must be provided.
Returns:
For short-lived containers (with timeout): Container output
For long-running containers: None
Example:
start_container(
image="busybox:latest",
timeout=1000,
command="sh",
arguments=["-c", "echo 'Hello' > /data/output.txt"],
volumes={
"/host/data": "/data",
"/host/config": "/config"
},
ports={
"8080": "80",
"2222": "22"
},
environment={
"DEBUG": "1",
"API_KEY": "secret123"
}
)
"""
pass
```
#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="start_container">
<xs:complexType>
<xs:sequence>
<xs:element name="command" type="xs:string" minOccurs="0"/>
<xs:element name="argument" type="xs:string" minOccurs="0" maxOccurs="unbounded"/>
<xs:element name="volumes" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element name="volume" type="xs:string" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="ports" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element name="port" maxOccurs="unbounded">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="host" type="xs:string" use="required"/>
<xs:attribute name="container" type="xs:string" use="required"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="environment" minOccurs="0">
<xs:complexType>
<xs:sequence>
<xs:element name="variable" maxOccurs="unbounded">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="name" type="xs:string" use="required"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
<xs:attribute name="image" type="xs:string" use="required"/>
<xs:attribute name="name" type="xs:string" use="optional"/>
<xs:attribute name="timeout" type="xs:integer" use="optional"/>
<xs:attribute name="volumes" type="xs:string" use="optional"/>
</xs:complexType>
</xs:element>
</xs:schema>
```
#### Example
```xml
<start_container
image="busybox:latest"
timeout="1000">
<command>sh</command>
<argument>-c</argument>
<argument><![CDATA[echo 'Hello' > /data/output.txt]]></argument>
<volumes>
<volume>/host/data:/data</volume>
<volume>/host/config:/config</volume>
</volumes>
<ports>
<port host="8080" container="80"/>
<port host="2222" container="22"/>
</ports>
<environment>
<variable name="DEBUG">1</variable>
<variable name="API_KEY">secret123</variable>
</environment>
</start_container>
```
#### Results
For short running containers, the exit status is added as attribute.
Standard output and standard error are merged and added as `CDATA` text node.
For long running containers, no information is added.
The container is started and represented in the containers section of the main context.
### Write to container standard input
Module: Docker
#### Function declaration
```python
def write_container_stdin(name: str, data: str) -> None:
"""Write data to a container's standard input.
Args:
name: Name of the target container
data: Data to write to stdin
"""
pass
```
#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="write_container_stdin">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string">
<xs:attribute name="container" type="xs:string" use="required"/>
</xs:extension>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:schema>
```
#### Example
```xml
<write_container_stdin container="interactive-shell">ls -la</write_container_stdin>
```
#### Results
No information is added.
The data will be written to the container's stdin buffer.
### Read from container standard output
Module: Docker
#### Function declaration
```python
def read_container_stdout(name: str, n: int = -1) -> str:
"""Read from a container's standard output buffer.
Args:
name: Name of the container
n: Number of bytes to read; -1 means read all available (default: -1)
Returns:
Data read from stdout
"""
pass
```
#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="read_container_stdout">
<xs:complexType>
<xs:attribute name="container" type="xs:string" use="required"/>
<xs:attribute name="n" type="xs:integer" use="optional"/>
</xs:complexType>
</xs:element>
</xs:schema>
```
#### Example
```xml
<read_container_stdout container="interactive-shell" n="1024"/>
```
#### Results
An attribute `actual` is added with the amount of bytes read.
A text node is added with the data as `CDATA`.
### Read from container standard error
Module: Docker
#### Function declaration
```python
def read_container_stderr(name: str, n: int = -1) -> str:
"""Read from a container's standard error buffer.
Args:
name: Name of the container
n: Number of bytes to read; -1 means read all available (default: -1)
Returns:
Data read from stderr
"""
pass
```
#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="read_container_stderr">
<xs:complexType>
<xs:attribute name="container" type="xs:string" use="required"/>
<xs:attribute name="n" type="xs:integer" use="optional"/>
</xs:complexType>
</xs:element>
</xs:schema>
```
#### Example
```xml
<read_container_stderr container="interactive-shell" n="1024"/>
```
#### Results
An attribute `actual` is added with the amount of bytes read.
A text node is added with the data as `CDATA`.
### Wait for container to finish
Module: Docker
#### Function declaration
```python
def wait_container(self, name: str, timeout: int) -> Tuple[int, str]:
"""Wait for a container to finish execution.
Args:
name: Name of the container to wait for
timeout: Time to wait in milliseconds
Returns:
Tuple of (exit_code, output)
"""
pass
```
#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="wait_container">
<xs:complexType>
<xs:attribute name="name" type="xs:string" use="required"/>
<xs:attribute name="timeout" type="xs:integer" use="required"/>
</xs:complexType>
</xs:element>
</xs:schema>
```
#### Example
```xml
<wait_container name="background-task" timeout="5000"/>
```
#### Results
The exit status is added as an attribute.
All remaining data on stdout and stderr is merged and added as `CDATA` text node.
### Select LLM by file name
Module: Reinforcement Learning
#### Function declaration
```python
def set_model_path(path: str) -> None:
"""Switch to a different LLM model file.
Args:
path: Path to the model file to load
"""
pass
```
#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="set_model_path">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string"/>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:schema>
```
#### Example
```xml
<set_model_path>/models/2024_10_19_15_03_52</set_model_path>
```
#### Results
No information is added.
The new model will be used starting from the next iteration.
### Update to commit id
Module: Process
#### Function declaration
```python
def update_to_commit(commit_id: str) -> None:
"""Update the running SIA process to a different version.
This will terminate the current process immediately.
Make sure the git commit is well tested and there is plenty of clear documentation for the new SIA instance to start.
Args:
commit_id: The git commit ID to update to
"""
pass
```
#### Schema
```xml
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:element name="update_to_commit">
<xs:complexType>
<xs:simpleContent>
<xs:extension base="xs:string"/>
</xs:simpleContent>
</xs:complexType>
</xs:element>
</xs:schema>
```
#### Example
```xml
<update_to_commit>0015e27</update_to_commit>
```
#### Results
No information is added.
The process will be replaced by the new version.
## Example iterations ## Example iterations
@@ -795,7 +162,7 @@ This example shows how to work with standard IO, run simple scripts and monitor
There is data available on the standard input channel. I should read it. I have no other running tasks to tend to. There is data available on the standard input channel. I should read it. I have no other running tasks to tend to.
</reasoning> </reasoning>
<actions> <actions>
<read_stdin n="42"> <read_stdin>
<![CDATA[Remind me to feed the cat tomorrow morning]]> <![CDATA[Remind me to feed the cat tomorrow morning]]>
</read_stdin> </read_stdin>
</actions> </actions>