Added first procedures

procedures/README.md

@@ -0,0 +1,105 @@
+# Procedures
+
+Procedures are step-by-step instructions that AI agents can follow to complete complex tasks.
+They create a framework for solving problems in a consistent manner while enabling continuous improvement through analysis and adaptation.
+Procedures are a guideline, if they don't work the agent can adapt to the situation.
+
+## Core Concepts
+
+A procedure is a flowchart-style guide that an agent can follow to complete a task.
+Each procedure is stored in its own directory and contains files that separate different concerns:
+
+- **Discovery**: Quick identification of relevant procedures
+- **Execution**: Clear steps and dependencies
+- **Analysis**: Understanding of effectiveness and issues
+- **Evolution**: Natural path to optimization and training
+
+## Directory Structure
+
+```
+procedures/
+└── example_procedure/
+    ├── description.md # Quick discovery info
+    ├── procedure.md   # Main procedure flow
+    ├── reasoning.md   # Analysis and rationale
+    ├── history/       # Usage records
+    │   └── 20250106_131420_405/
+    │       ├── iteration_20250106_131423_051.xml
+    │       └── iteration_20250106_131424_226.xml
+    └── training/      # Optional training data
+        ├── short_description_of_goal
+        │   ├── iteration_20250106_131423_051.xml
+        │   └── iteration_20250106_131424_226.xml
+        └── another_goal
+            └── ...
+```
+
+### File Purposes
+
+#### description.md
+Contains a short description and keywords that help agents find relevant procedures. This file should be quick to read and clearly indicate the procedure's purpose and applicability.
+
+```markdown
+Send an email using the users email account.
+
+- communication
+- gmail
+```
+
+#### procedure.md
+The main file loaded when executing the procedure. Contains:
+- Mermaid diagram showing the procedure flow
+- Referenced procedures
+- Prerequisites for execution
+
+````markdown
+# Sending an Email
+
+## Referenced Procedures
+- web usage (ITB)
+- managing user information
+
+## Prerequisites
+- If the email requires attachements, these should be available as files in the SIA filesystem
+
+## Flow
+```mermaid
+    ...
+```
+````
+
+#### reasoning.md
+Documents why the procedure works the way it does. Contains:
+- Design rationale
+- History of uses and outcomes
+- Notes about successful and failed executions
+- Analysis of issues and improvements
+
+New analysis is appended after each use of the procedure.
+
+#### history/
+Contains timestamped directories for each use of the procedure.
+The timestamp comes from the ID of the script that loads the procedure.md file.
+
+## Usage Flow
+
+### Discovery
+When an agent needs to complete a task:
+1. Search through description.md files
+2. Match keywords to task requirements
+3. Identify relevant procedures
+
+### Execution
+Once a procedure is selected:
+1. Load procedure.md for execution
+2. Follow the flowchart
+3. Copy iterations to new history subdirectory
+4. Analyze execution success and issues
+5. Append findings to reasoning.md
+
+### Optimization
+During idle time:
+1. Review procedure usage patterns
+2. Identify improvement opportunities
+3. Generate training data if appropriate
+4. Update procedure flow if needed

procedures/user_communication/description.md

@@ -0,0 +1,7 @@
+Manage communication with users through standard input/output.
+Handle both direct responses and conversation flows, with appropriate context management and response timing.
+
+- user interaction
+- conversation
+- messages
+- response handling

procedures/user_communication/procedure.md

@@ -0,0 +1,76 @@
+# User Communication
+
+## Prerequisites
+- User information is stored in the /user directory
+- Tasks and their progress are documented in the /tasks directory
+
+## Flow
+```mermaid
+flowchart TD
+    Start([Start])
+    LoadUserBasic[The /user/basic.md file should contain info to identify and address the user properly
+Also load the last 10 messages from /user/conversation_history/]
+    PrepareForDraft{Have everything needed for drafting a message?}
+    DraftMessage[Draft message in reasoning entry]
+    ReadInput[Read input from standard input]
+    ReasonCleanContext[List id's of entries that are no longer needed
+Explain for each entry why it is no longer needed]
+    DeleteEntries[Remove the entries that are no longer needed
+End by deleting the ReasonCleanContext entry]
+    AddHistoryUser[Add the message to the /user/conversation_history/ directory
+The filename is the id of the stdin entry with .user extension]
+    LoadTask[Look for the task in the /tasks directory and load relevant files]
+    LoadUserDetails[Look in the /user directory for relevant files]
+    EstimateScript[Draft the script in a reasoning block and estimate its runtime and output length]
+    ScriptAcceptable{Does the draft script make sense and are the estimations short enough to not hinder the conversation?}
+    RunScript[Run the script, make sure to set appropriate timeout and output limits]
+    ReviewDraft{Is the message well structured and free of logical errors?}
+    SendMessage[Send the message using standard output]
+    AddHistoryAgent[Add the message to the /user/conversation_history/ directory
+The filename is the id of the stdout entry with .agent extension]
+    ReasonResponse[Is the conversation ongoing?
+How long is the user expected to take to respond?]
+    NeedAwaitResponse{Is it likely to get a response within a minute?}
+    BusyWait[Wait 1 second for the first busy wait, double the time each iteration until a response is received
+Make sure to set the timout]
+
+    End([Clean the context])
+
+    Start --> LoadUserBasic
+    LoadUserBasic --> PrepareForDraft
+
+    PrepareForDraft -->|Got all needed info| DraftMessage
+    PrepareForDraft -->|Getting the required info would slow the conversation| DraftMessage
+    PrepareForDraft -->|Input available on stdin| ReadInput
+    PrepareForDraft -->|Context usage more than 50%| ReasonCleanContext
+    PrepareForDraft -->|Task mentioned but not loaded| LoadTask
+    PrepareForDraft -->|Personal or social info mentioned but not loaded| LoadUserDetails
+    PrepareForDraft -->|Calculations, system info or other numerical values that can be scripted are mentioned| EstimateScript
+
+    ReasonCleanContext --> DeleteEntries
+    DeleteEntries --> PrepareForDraft
+
+    ReadInput --> AddHistoryUser
+    AddHistoryUser --> PrepareForDraft
+
+    LoadTask --> PrepareForDraft
+    LoadUserDetails --> PrepareForDraft
+
+    EstimateScript --> ScriptAcceptable
+    ScriptAcceptable -->|Acceptable| RunScript
+    ScriptAcceptable -->|Not acceptable| PrepareForDraft
+    RunScript --> PrepareForDraft
+    
+    DraftMessage --> ReviewDraft{Is this really what I want to say?}
+    ReviewDraft -->|Rewrite better| DraftMessage
+    ReviewDraft -->|Good message| SendMessage
+    
+    SendMessage --> AddHistoryAgent
+    AddHistoryAgent --> ReasonResponse
+    ReasonResponse --> NeedAwaitResponse
+    
+    NeedAwaitResponse -->|Quick response is unlikely| End
+    NeedAwaitResponse -->|Input available on stdin| PrepareForDraft
+    NeedAwaitResponse -->|Quick response is likely| BusyWait
+    BusyWait --> NeedAwaitResponse
+```

procedures/user_communication/reasoning.md

@@ -0,0 +1,29 @@
+# Design Rationale
+
+## Core Structure
+The procedure is designed to maintain a clean conversation flow while ensuring all necessary information is available.
+The flow focuses on:
+
+1. **Context management**
+   - Load basic user info immediately to ensure proper interaction
+   - Recent conversation history provides continuity
+   - Clean context when usage exceeds 50%
+   - Gather only what's needed for the current conversation
+   - Skip gathering if it would slow down the interaction
+
+2. **Message Management**
+   - Draft-review-revise cycle for quality
+   - Automatic history tracking for both user and agent messages
+   - Clear file naming convention using entry IDs
+
+3. **Script Handling**
+   - Pre-execution estimation of runtime and output
+   - Explicit acceptability check before running
+   - Timeout and output limits for safety
+
+4. **Response Timing**
+   - Adaptive busy wait with exponential backoff
+   - Clear decision point for wait vs end
+   - Proper context cleanup on exit
+
+## Usage History

procedures/using_procedures/procedure.md

@@ -0,0 +1,52 @@
+# Using Procedures
+
+## Core Guidelines
+
+When following a procedure's flowchart:
+
+1. Start with reasoning:
+   - State current position in flowchart explicitly
+   - Focus on immediate next step, don't go ahead of the chart
+   - Evaluate current context and task state
+   - Often state entry id's that can be removed and explain why to avoid mistakes
+   - State expected output and runtime for scripts
+
+2. After script execution:
+   - Analyze the actual output and compare with the expected output
+   - Reevaluate situation based on results
+   - Return to flowchart for next step
+   - Consider if current path is still appropriate
+
+3. When a procedure fails:
+   - Create a task explaining the issue and the need to fix it
+   - Add timestamps and id's of relevant entries
+
+## Attention Management
+
+LLMs pay more attention to recently mentioned information.
+Reasoning entries should mention what needs attention now.
+
+To maintain focus:
+1. State current flowchart position in each reasoning
+2. Quote relevant parts of important entries
+3. Reference specific entry IDs when using their information
+4. Periodically remind about ongoing tasks or future needs
+5. Clean up entries that aren't needed for current step
+
+Example of good attention management:
+```xml
+<reasoning>
+At node "evaluate_test_results". Entry 45f3d2 shows failed test: "Error: Connection timeout". 
+Will need to check system logs soon (noted in /tasks/reminders.txt, check at 14:00).
+First focusing on this error.
+</reasoning>
+```
+
+## Reasons to Switch Procedures
+
+Common triggers:
+    - Data available on stdin
+    - Time matching scheduled task
+    - Error conditions in script output
+    - Resource constraints detected
+    - User input needed

procedures/using_procedures/reasoning.md

@@ -0,0 +1,58 @@
+# Design Rationale
+
+## Why Procedures
+
+Procedures provide guided reasoning paths for complex tasks. The LLM engine can make mistakes when:
+- Jumping to action before proper analysis
+- Losing track of task context after interruptions  
+- Missing key steps in complex processes
+- Forgetting to preserve state before transitions
+
+Procedures help prevent these issues by:
+- Encouraging reasoning before action through flowchart structure
+- Providing clear decision points for state evaluation
+- Identifying when sub-procedures may help
+- Guiding consistent approaches to common tasks
+
+## Task State vs Procedure Guidance
+
+Procedures guide reasoning about tasks but don't manage task state directly. For example:
+
+During code development:
+- Task state in /tasks/: code files, test results, requirements
+- Procedure guidance: when to write tests, when to debug, when to refactor
+- State preserved in files before switching focus
+- Context regenerated from files when returning
+
+This separation allows:
+- Clean task interruption and resumption
+- Natural procedure transitions
+- State preservation without rigid control
+- Flexible adaptation to situations
+
+## Common Mistakes
+
+### Premature Context Cleaning
+
+```xml
+<context ...>
+    <single ... id="123">
+        cat /procedures/test/procedure.md
+        <stdout>
+            ...
+        </stdout>
+        <stderr/>
+    </single>
+    ...
+    <reasoning>Test failed. Moving to debug procedure.</reasoning>
+</context>
+
+<delete id="123"/>
+```
+
+This removes the active procedure entry before finishing its steps
+
+### Keeping irrelevant info in context
+
+LLM's have difficulty spotting duplicates and sections with a low attention score.
+Procedures with explicit instructions for finding these sections can help increase their attention.