More info on training and datasets

readme.md

@@ -39,16 +39,13 @@ The main context is always regenerated and contains:
 
 ### Core Action System
 
-Core actions can be executed by the LLM directly using a function call interface.
-They allow the agent to manage its memory and containers.
-They also allow the agent to select which SIA version and LLM to use.
-It is also the standard way of communicating with the agent.
+Core actions are selected by the LLM by outputting an XML list of actions and parameters.
+This system allows the agent to manage its memory, control containers, select wich version to run, and communicate.
+The Action System parses the XML output and executes the corresponding functions.
 
-General core commands:
+Core commands for user interaction:
 - Read standard input
 - Write to standard output
-- Wait until time
-- Wait for container to finish (with timeout)
 
 ### Docker Container Management
 
@@ -56,6 +53,9 @@ SIA utilizes Docker containers for anything not covered by core actions.
 Containers can be short-lived, eg. for simple calculations.
 They can also be long-lived, eg. to keep a communication channel open.
 They can even run a complete SIA instance eg. for verifying updates to the LLM or core functionality.
+The short-lived containers define a timeout.
+The next iteration of the main loop starts when the container finishes or the timeout is reached.
+The long-lived containers can also be waited on at a later point in time.
 
 Core commands for container operations:
 - Start container
@@ -63,6 +63,7 @@ Core commands for container operations:
 - Write to container standard input
 - Read from container standard output
 - Read from container standard error
+- Wait for container to finish
 
 ### Information Storage
 
@@ -84,14 +85,13 @@ This can be done using containers.
 
 For each iteration of the main loop, the context and the generated reasoning and actions are stored in the file system.
 When the agent solves a problem it starts a search for the root cause by looking at previous iterations.
-The iteration file is passed to the reenforcement learning system together with a description of the problem and how it was solved.
-The reeinforcement learning system then updates the LLM's weights and returns a commit id for the new version of the LLM.
-The agent can test this updated LLM in a SIA instance in a container.
+The iteration file is used for updating the LLM weights using functions in the SIA core.
+The agent can access this by running a SIA container.
+It can then test the updated LLM in another SIA instance in a container.
 If it is acceptable it can change to this new version.
 
 Core commands for reinforcement learning:
-- Learn from file
-- Select LLM version by commit id
+- Select LLM by file name
 
 There are no specific commands for running a SIA instance in a container.
 This can be done using the regular container commands.
@@ -172,6 +172,56 @@ It also simplifies escaping of command line arguments.
 
 Action results are added in the context as text nodes after the last parameter.
 
+### Training datasets
+
+A training dataset is a folder with these files:
+- system_prompt.txt
+- main_context.txt
+- pre-reasoning.txt
+- training_reasoning.txt
+- post-reasoning.txt
+- pre-actions.txt
+- training_actions.txt
+- post-actions.txt
+
+The context window of the LLM is filled with all parts of the dataset in order.
+The learning rate is only applied to the training reasoning and actions.
+The pre and post files are optional.
+
+To do an actual training round, a sia:latest container is started.
+This is an example action that trains on two datasets with learning rate 0.1:
+```xml
+<start_container image="sia:latest" volumes="/tasks:/tasks">
+    <volumes>
+      <volume>/models/:/models/</volume>
+      <volume>/datasets/description_of_a_problem/:/datasets/description_of_a_problem/</volume>
+      <volume>/datasets/description_of_nother_problem/:/datasets/description_of_nother_problem/</volume>
+    </volumes>
+    <command>sia</command>
+    <parameter>train</parameter>
+    <parameter>--learning-rate</parameter>
+    <parameter>0.1</parameter>
+    <parameter>--model</parameter>
+    <parameter>/models/2024_10_19_08_21_41</parameter>
+    <parameter>--out</parameter>
+    <parameter>/models/2024_10_19_15_03_52</parameter>
+    <parameter>/datasets/description_of_a_problem/</parameter>
+    <parameter>/datasets/description_of_nother_problem/</parameter>
+</start_container>
+```
+
+### Reinforcement learning by human feedback
+
+The SIA container can be used in 3 ways:
+- To run a SIA instance
+- To update LLM weights based on a dataset
+- To host the interaction web interface
+
+The web interface is an alternative way of interacting with SIA, specifically for reinforcement learning by human feedback.
+The web interface takes over standard input and output.
+It each time the LLM generates a response, the web interface will display it.
+The user can modify the response before the actions are executed.
+
 ## Example iterations
 
 ### Clarifying a task
@@ -203,16 +253,16 @@ This example shows how to work with standard IO, run simple scripts and monitor
     </actions>
   </previous>
   <files>
-    <file name="/" type="dir">
+    <file name="/" type="dir" index="0">
 drwxr-xr-x 1 sia 197121 0 2024-10-16 23:02:16.486152500 +0200 tasks/
 drwxr-xr-x 1 sia 197121 0 2024-10-16 22:35:31.806079500 +0200 user/
     </file>
-    <file name="/tasks" type="dir">
+    <file name="/tasks" type="dir" index="1">
     </file>
     <file name="/user" type="dir">
 -rw-r--r-- 1 sia 197121 71 2024-10-16 22:41:23.223580300 +0200 general_info.txt
     </file>
-    <file name="/user/general_info.txt" type="file">
+    <file name="/user/general_info.txt" type="file" index="2">
 Name: John (I don&apos;t know his last name)
 Location: Somewhere in Belgium
     </file>
@@ -232,10 +282,4 @@ John did not specify an exact time. I'll suggest 9am. He also did not specify ho
   </start_container>
   <monitor_file path="/tasks/reminder.txt"/>
 </actions>
-```
-
-### Researching a topic
-
-### Modifying the SIA code
-
-### Learning from a mistake
+```