Merge branch 'master' of https://git.nielsgeens.be/llm/SIA

Dockerfile

@@ -46,4 +46,4 @@ COPY ./ /root/sia/
 COPY --from=web-build /app/dist /root/sia/static/
 WORKDIR /root/sia
 
-ENTRYPOINT ["python3", "-m", "sia"]
+CMD ["python3", "-m", "sia"]

README.md

@@ -74,7 +74,7 @@ Start by reasoning about the task.
 
 Store important information on disk.
 ```xml
-  <script><![CDATA[echo 'Remind John to feed the cat on 2024-10-18T09:00:00+02:00. Use standard output.' > /tasks/reminder_to_feed_cat.txt]]></script>
+  <single><![CDATA[echo 'Remind John to feed the cat on 2024-10-18T09:00:00+02:00. Use standard output.' > /tasks/reminder_to_feed_cat.txt]]></single>
 ```
 
 Respond to the user.
@@ -90,7 +90,7 @@ Clear initial reasoning.
 The conversation is kept in context to understand the user's expected response.
 If the context was near full, it would be summarized and cleaned up.
 
-The `script` output is also kept in context.
+The `single` output is also kept in context.
 If the file was updated often, it could be replaced by a repeated `cat`, like the general info.
 
 ## Working principles
@@ -196,7 +196,7 @@ Both agent types share common components:
 
 #### Working Memory
 The working memory stores the current state of the system through different types of entries:
-- Script Entries: Results of script executions
+- SingleEntries: Output of single-shot script executions
 - RepeatEntry: Continuously refreshed script outputs
 - ReasoningEntry: LLM's thought process documentation
 - ParseErrorEntry: XML validation or parsing errors
@@ -514,13 +514,13 @@ classDiagram
         +cleanup() void*
     }
 
-    class ScriptEntry {
+    class SingleEntry {
         +script: str readonly
         +stdout: str readonly
         +stderr: str readonly
         +exit_code: Optional~int~ readonly
 
-        +Script(script str, id str, timestamp datetime)
+        +Single(script str, id str, timestamp datetime)
         +update() void
         +generate_context() ElementTree
     }
@@ -573,7 +573,7 @@ classDiagram
     ParseErrorEntry --|> Entry
     ReadEntry --|> Entry
     Entry <|-- WriteEntry
-    Entry <|-- ScriptEntry
+    Entry <|-- SingleEntry
     Entry <|-- RepeatEntry
 ```
 

procedures/README.md

@@ -0,0 +1,105 @@
+# Procedures
+
+Procedures are step-by-step instructions that AI agents can follow to complete complex tasks.
+They create a framework for solving problems in a consistent manner while enabling continuous improvement through analysis and adaptation.
+Procedures are a guideline, if they don't work the agent can adapt to the situation.
+
+## Core Concepts
+
+A procedure is a flowchart-style guide that an agent can follow to complete a task.
+Each procedure is stored in its own directory and contains files that separate different concerns:
+
+- **Discovery**: Quick identification of relevant procedures
+- **Execution**: Clear steps and dependencies
+- **Analysis**: Understanding of effectiveness and issues
+- **Evolution**: Natural path to optimization and training
+
+## Directory Structure
+
+```
+procedures/
+└── example_procedure/
+    ├── description.md # Quick discovery info
+    ├── procedure.md   # Main procedure flow
+    ├── reasoning.md   # Analysis and rationale
+    ├── history/       # Usage records
+    │   └── 20250106_131420_405/
+    │       ├── iteration_20250106_131423_051.xml
+    │       └── iteration_20250106_131424_226.xml
+    └── training/      # Optional training data
+        ├── short_description_of_goal
+        │   ├── iteration_20250106_131423_051.xml
+        │   └── iteration_20250106_131424_226.xml
+        └── another_goal
+            └── ...
+```
+
+### File Purposes
+
+#### description.md
+Contains a short description and keywords that help agents find relevant procedures. This file should be quick to read and clearly indicate the procedure's purpose and applicability.
+
+```markdown
+Send an email using the users email account.
+
+- communication
+- gmail
+```
+
+#### procedure.md
+The main file loaded when executing the procedure. Contains:
+- Mermaid diagram showing the procedure flow
+- Referenced procedures
+- Prerequisites for execution
+
+````markdown
+# Sending an Email
+
+## Referenced Procedures
+- web usage (ITB)
+- managing user information
+
+## Prerequisites
+- If the email requires attachements, these should be available as files in the SIA filesystem
+
+## Flow
+```mermaid
+    ...
+```
+````
+
+#### reasoning.md
+Documents why the procedure works the way it does. Contains:
+- Design rationale
+- History of uses and outcomes
+- Notes about successful and failed executions
+- Analysis of issues and improvements
+
+New analysis is appended after each use of the procedure.
+
+#### history/
+Contains timestamped directories for each use of the procedure.
+The timestamp comes from the ID of the script that loads the procedure.md file.
+
+## Usage Flow
+
+### Discovery
+When an agent needs to complete a task:
+1. Search through description.md files
+2. Match keywords to task requirements
+3. Identify relevant procedures
+
+### Execution
+Once a procedure is selected:
+1. Load procedure.md for execution
+2. Follow the flowchart
+3. Copy iterations to new history subdirectory
+4. Analyze execution success and issues
+5. Append findings to reasoning.md
+
+### Optimization
+During idle time:
+1. Review procedure usage patterns
+2. Identify improvement opportunities
+3. Generate training data if appropriate
+4. Update procedure flow if needed

procedures/user_communication/description.md

@@ -0,0 +1,7 @@
+Manage communication with users through standard input/output.
+Handle both direct responses and conversation flows, with appropriate context management and response timing.
+
+- user interaction
+- conversation
+- messages
+- response handling

procedures/user_communication/procedure.md

@@ -0,0 +1,69 @@
+# User Communication
+
+## Prerequisites
+- User information is stored in the /user directory
+- Tasks and their progress are documented in the /tasks directory
+
+## Flow
+```mermaid
+flowchart TD
+    Start([Start])
+    LoadUserBasic[The /user/basic.md file should contain info to identify and address the user properly<br>Also load the last 10 messages from /user/conversation_history/ ]
+    PrepareForDraft{Have everything needed for drafting a message?}
+    DraftMessage[Draft message in reasoning entry]
+    ReadInput[Read input from standard input]
+    ReasonCleanContext[List id's of entries that are no longer needed<br>Explain for each entry why it is no longer needed]
+    DeleteEntries[Remove the entries that are no longer needed<br>End by deleting the ReasonCleanContext entry]
+    AddHistoryUser[Add the message to the /user/conversation_history/ directory<br>The filename is the id of the stdin entry with .user extension]
+    LoadTask[Look for the task in the /tasks directory and load relevant files]
+    LoadUserDetails[Look in the /user directory for relevant files]
+    EstimateScript[Draft the script in a reasoning block and estimate its runtime and output length]
+    ScriptAcceptable{Does the draft script make sense and are the estimations short enough to not hinder the conversation?}
+    RunScript[Run the script, make sure to set appropriate timeout and output limits]
+    ReviewDraft{Is the message well structured and free of logical errors?}
+    SendMessage[Send the message using standard output]
+    AddHistoryAgent[Add the message to the /user/conversation_history/ directory<br>The filename is the id of the stdout entry with .agent extension]
+    ReasonResponse[Is the conversation ongoing?<br>How long is the user expected to take to respond?]
+    NeedAwaitResponse{Is it likely to get a response within a minute?}
+    BusyWait[Wait 1 second for the first busy wait, double the time each iteration until a response is received<br>Make sure to set the timout]
+
+    End([Clean the context])
+
+    Start --> LoadUserBasic
+    LoadUserBasic --> PrepareForDraft
+
+    PrepareForDraft -->|Got all needed info| DraftMessage
+    PrepareForDraft -->|Getting the required info would slow the conversation| DraftMessage
+    PrepareForDraft -->|Input available on stdin| ReadInput
+    PrepareForDraft -->|Context usage more than 50%| ReasonCleanContext
+    PrepareForDraft -->|Task mentioned but not loaded| LoadTask
+    PrepareForDraft -->|Personal or social info mentioned but not loaded| LoadUserDetails
+    PrepareForDraft -->|Calculations, system info or other numerical values that can be scripted are mentioned| EstimateScript
+
+    ReasonCleanContext --> DeleteEntries
+    DeleteEntries --> PrepareForDraft
+
+    ReadInput --> AddHistoryUser
+    AddHistoryUser --> PrepareForDraft
+
+    LoadTask --> PrepareForDraft
+    LoadUserDetails --> PrepareForDraft
+
+    EstimateScript --> ScriptAcceptable
+    ScriptAcceptable -->|Acceptable| RunScript
+    ScriptAcceptable -->|Not acceptable| PrepareForDraft
+    RunScript --> PrepareForDraft
+    
+    DraftMessage --> ReviewDraft{Is this really what I want to say?}
+    ReviewDraft -->|Rewrite better| DraftMessage
+    ReviewDraft -->|Good message| SendMessage
+    
+    SendMessage --> AddHistoryAgent
+    AddHistoryAgent --> ReasonResponse
+    ReasonResponse --> NeedAwaitResponse
+    
+    NeedAwaitResponse -->|Quick response is unlikely| End
+    NeedAwaitResponse -->|Input available on stdin| PrepareForDraft
+    NeedAwaitResponse -->|Quick response is likely| BusyWait
+    BusyWait --> NeedAwaitResponse
+```

procedures/user_communication/reasoning.md

@@ -0,0 +1,29 @@
+# Design Rationale
+
+## Core Structure
+The procedure is designed to maintain a clean conversation flow while ensuring all necessary information is available.
+The flow focuses on:
+
+1. **Context management**
+   - Load basic user info immediately to ensure proper interaction
+   - Recent conversation history provides continuity
+   - Clean context when usage exceeds 50%
+   - Gather only what's needed for the current conversation
+   - Skip gathering if it would slow down the interaction
+
+2. **Message Management**
+   - Draft-review-revise cycle for quality
+   - Automatic history tracking for both user and agent messages
+   - Clear file naming convention using entry IDs
+
+3. **Script Handling**
+   - Pre-execution estimation of runtime and output
+   - Explicit acceptability check before running
+   - Timeout and output limits for safety
+
+4. **Response Timing**
+   - Adaptive busy wait with exponential backoff
+   - Clear decision point for wait vs end
+   - Proper context cleanup on exit
+
+## Usage History

procedures/using_procedures/procedure.md

@@ -0,0 +1,52 @@
+# Using Procedures
+
+## Core Guidelines
+
+When following a procedure's flowchart:
+
+1. Start with reasoning:
+   - State current position in flowchart explicitly
+   - Focus on immediate next step, don't go ahead of the chart
+   - Evaluate current context and task state
+   - Often state entry id's that can be removed and explain why to avoid mistakes
+   - State expected output and runtime for scripts
+
+2. After script execution:
+   - Analyze the actual output and compare with the expected output
+   - Reevaluate situation based on results
+   - Return to flowchart for next step
+   - Consider if current path is still appropriate
+
+3. When a procedure fails:
+   - Create a task explaining the issue and the need to fix it
+   - Add timestamps and id's of relevant entries
+
+## Attention Management
+
+LLMs pay more attention to recently mentioned information.
+Reasoning entries should mention what needs attention now.
+
+To maintain focus:
+1. State current flowchart position in each reasoning
+2. Quote relevant parts of important entries
+3. Reference specific entry IDs when using their information
+4. Periodically remind about ongoing tasks or future needs
+5. Clean up entries that aren't needed for current step
+
+Example of good attention management:
+```xml
+<reasoning>
+At node "evaluate_test_results". Entry 45f3d2 shows failed test: "Error: Connection timeout". 
+Will need to check system logs soon (noted in /tasks/reminders.txt, check at 14:00).
+First focusing on this error.
+</reasoning>
+```
+
+## Reasons to Switch Procedures
+
+Common triggers:
+    - Data available on stdin
+    - Time matching scheduled task
+    - Error conditions in script output
+    - Resource constraints detected
+    - User input needed

procedures/using_procedures/reasoning.md

@@ -0,0 +1,58 @@
+# Design Rationale
+
+## Why Procedures
+
+Procedures provide guided reasoning paths for complex tasks. The LLM engine can make mistakes when:
+- Jumping to action before proper analysis
+- Losing track of task context after interruptions  
+- Missing key steps in complex processes
+- Forgetting to preserve state before transitions
+
+Procedures help prevent these issues by:
+- Encouraging reasoning before action through flowchart structure
+- Providing clear decision points for state evaluation
+- Identifying when sub-procedures may help
+- Guiding consistent approaches to common tasks
+
+## Task State vs Procedure Guidance
+
+Procedures guide reasoning about tasks but don't manage task state directly. For example:
+
+During code development:
+- Task state in /tasks/: code files, test results, requirements
+- Procedure guidance: when to write tests, when to debug, when to refactor
+- State preserved in files before switching focus
+- Context regenerated from files when returning
+
+This separation allows:
+- Clean task interruption and resumption
+- Natural procedure transitions
+- State preservation without rigid control
+- Flexible adaptation to situations
+
+## Common Mistakes
+
+### Premature Context Cleaning
+
+```xml
+<context ...>
+    <single ... id="123">
+        cat /procedures/test/procedure.md
+        <stdout>
+            ...
+        </stdout>
+        <stderr/>
+    </single>
+    ...
+    <reasoning>Test failed. Moving to debug procedure.</reasoning>
+</context>
+
+<delete id="123"/>
+```
+
+This removes the active procedure entry before finishing its steps
+
+### Keeping irrelevant info in context
+
+LLM's have difficulty spotting duplicates and sections with a low attention score.
+Procedures with explicit instructions for finding these sections can help increase their attention.

run.sh

@@ -1,5 +1,6 @@
 #!/bin/bash
 
+export MSYS_NO_PATHCONV=1
 set -e
 
 function chown_iterations() {
@@ -21,6 +22,7 @@ docker run \
     -ti \
     --gpus=all \
     -p 8080:8080 \
+    -e SIA_ITERATIONS_DIR=/root/iterations \
     --env-file .env \
     -v /$(pwd)/model/:/root/model/ \
     -v /$(pwd)/.env:/root/.env \

sia/mistral_llm_engine.py

@@ -46,12 +46,12 @@ class MistralLlmEngine(LlmEngine):
         try:
             for chunk in stream_response:
                 if should_stop():
-                    stream_response.close()
+                    stream_response.response.close()
                     break
                 if content := chunk.data.choices[0].delta.content:
                     yield content
         finally:
-            stream_response.close()
+            stream_response.response.close()
 
     def token_count(self, system_prompt: str, main_context: str) -> int:
         messages = [

sia/web_agent.py

@@ -1,4 +1,4 @@
-from datetime import datetime
+from datetime import datetime, timezone
 from enum import Enum, auto
 from threading import Lock
 from typing import Callable, Dict, List, Optional
@@ -154,7 +154,7 @@ class WebAgent(BaseAgent):
         if llm_name not in self._llms:
             raise ValueError(f"Unknown LLM: {llm_name}")
 
-        timestamp = datetime.now()
+        timestamp = datetime.now(timezone.utc)
         self._iteration_logger.log_iteration(timestamp, self._context, response)
         parse_result = self._parser.parse(timestamp, response)
         if isinstance(parse_result, Command):

web/src/components/editors/MemoryEditor.jsx

@@ -74,13 +74,14 @@ export const MemoryEditor = ({
 
   const generateId = () => {
     const now = new Date();
-    return now.getFullYear() +
+    const utc = new Date(now.getTime() + now.getTimezoneOffset() * 60000);
-      String(now.getMonth() + 1).padStart(2, '0') +
+    return utc.getFullYear() +
-      String(now.getDate()).padStart(2, '0') + '_' +
+      String(utc.getMonth() + 1).padStart(2, '0') +
-      String(now.getHours()).padStart(2, '0') +
+      String(utc.getDate()).padStart(2, '0') + '_' +
-      String(now.getMinutes()).padStart(2, '0') +
+      String(utc.getHours()).padStart(2, '0') +
-      String(now.getSeconds()).padStart(2, '0') + '_' +
+      String(utc.getMinutes()).padStart(2, '0') +
-      String(now.getMilliseconds()).padStart(3, '0');
+      String(utc.getSeconds()).padStart(2, '0') + '_' +
+      String(utc.getMilliseconds()).padStart(3, '0');
   };
 
   const getInitialEntry = (type) => {