From 0015e2731ad61913e186f5f2cafbde3e44107906 Mon Sep 17 00:00:00 2001 From: Geens Date: Sat, 19 Oct 2024 21:54:38 +0200 Subject: [PATCH] More info on training and datasets --- readme.md | 88 +++++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 66 insertions(+), 22 deletions(-) diff --git a/readme.md b/readme.md index 0a1083b..d32cac9 100644 --- a/readme.md +++ b/readme.md @@ -39,16 +39,13 @@ The main context is always regenerated and contains: ### Core Action System -Core actions can be executed by the LLM directly using a function call interface. -They allow the agent to manage its memory and containers. -They also allow the agent to select which SIA version and LLM to use. -It is also the standard way of communicating with the agent. +Core actions are selected by the LLM by outputting an XML list of actions and parameters. +This system allows the agent to manage its memory, control containers, select wich version to run, and communicate. +The Action System parses the XML output and executes the corresponding functions. -General core commands: +Core commands for user interaction: - Read standard input - Write to standard output -- Wait until time -- Wait for container to finish (with timeout) ### Docker Container Management @@ -56,6 +53,9 @@ SIA utilizes Docker containers for anything not covered by core actions. Containers can be short-lived, eg. for simple calculations. They can also be long-lived, eg. to keep a communication channel open. They can even run a complete SIA instance eg. for verifying updates to the LLM or core functionality. +The short-lived containers define a timeout. +The next iteration of the main loop starts when the container finishes or the timeout is reached. +The long-lived containers can also be waited on at a later point in time. Core commands for container operations: - Start container @@ -63,6 +63,7 @@ Core commands for container operations: - Write to container standard input - Read from container standard output - Read from container standard error +- Wait for container to finish ### Information Storage @@ -84,14 +85,13 @@ This can be done using containers. For each iteration of the main loop, the context and the generated reasoning and actions are stored in the file system. When the agent solves a problem it starts a search for the root cause by looking at previous iterations. -The iteration file is passed to the reenforcement learning system together with a description of the problem and how it was solved. -The reeinforcement learning system then updates the LLM's weights and returns a commit id for the new version of the LLM. -The agent can test this updated LLM in a SIA instance in a container. +The iteration file is used for updating the LLM weights using functions in the SIA core. +The agent can access this by running a SIA container. +It can then test the updated LLM in another SIA instance in a container. If it is acceptable it can change to this new version. Core commands for reinforcement learning: -- Learn from file -- Select LLM version by commit id +- Select LLM by file name There are no specific commands for running a SIA instance in a container. This can be done using the regular container commands. @@ -172,6 +172,56 @@ It also simplifies escaping of command line arguments. Action results are added in the context as text nodes after the last parameter. +### Training datasets + +A training dataset is a folder with these files: +- system_prompt.txt +- main_context.txt +- pre-reasoning.txt +- training_reasoning.txt +- post-reasoning.txt +- pre-actions.txt +- training_actions.txt +- post-actions.txt + +The context window of the LLM is filled with all parts of the dataset in order. +The learning rate is only applied to the training reasoning and actions. +The pre and post files are optional. + +To do an actual training round, a sia:latest container is started. +This is an example action that trains on two datasets with learning rate 0.1: +```xml + + + /models/:/models/ + /datasets/description_of_a_problem/:/datasets/description_of_a_problem/ + /datasets/description_of_nother_problem/:/datasets/description_of_nother_problem/ + + sia + train + --learning-rate + 0.1 + --model + /models/2024_10_19_08_21_41 + --out + /models/2024_10_19_15_03_52 + /datasets/description_of_a_problem/ + /datasets/description_of_nother_problem/ + +``` + +### Reinforcement learning by human feedback + +The SIA container can be used in 3 ways: +- To run a SIA instance +- To update LLM weights based on a dataset +- To host the interaction web interface + +The web interface is an alternative way of interacting with SIA, specifically for reinforcement learning by human feedback. +The web interface takes over standard input and output. +It each time the LLM generates a response, the web interface will display it. +The user can modify the response before the actions are executed. + ## Example iterations ### Clarifying a task @@ -203,16 +253,16 @@ This example shows how to work with standard IO, run simple scripts and monitor - + drwxr-xr-x 1 sia 197121 0 2024-10-16 23:02:16.486152500 +0200 tasks/ drwxr-xr-x 1 sia 197121 0 2024-10-16 22:35:31.806079500 +0200 user/ - + -rw-r--r-- 1 sia 197121 71 2024-10-16 22:41:23.223580300 +0200 general_info.txt - + Name: John (I don't know his last name) Location: Somewhere in Belgium @@ -232,10 +282,4 @@ John did not specify an exact time. I'll suggest 9am. He also did not specify ho -``` - -### Researching a topic - -### Modifying the SIA code - -### Learning from a mistake \ No newline at end of file +``` \ No newline at end of file