SIA/procedures/self_improvement/reasoning.md

Self Improvement

SIA operates as a learning system that constantly seeks to enhance its capabilities while maintaining stable and reliable operation. This improvement happens through several complementary mechanisms, each serving different aspects of the system's evolution.

Improvement Mechanisms

The system can improve in several ways:

LLM Finetuning

The foundation of SIA's intelligence is its Large Language Model. Through careful analysis of its performance, SIA identifies cases where its reasoning or actions could have been better. These examples, when properly validated and organized, become valuable training data for improving the model's capabilities.

Source Code Evolution

SIA can modify its own Python source code, enabling both architectural and functional improvements. When stopped, SIA restarts and reloads the Python files, allowing code changes to take effect. While powerful, this capability requires robust validation to prevent system instability.

Procedure Refinement

Procedures provide guided reasoning paths for complex tasks. By analyzing its operational history, SIA can identify common patterns and create new procedures or optimize existing ones. This improves consistency and efficiency while carrying lower risk than direct code modifications.

Tool Development

SIA can develop new tools and enhance existing ones, expanding its capabilities by creating new ways to interact with systems and data.

Challenge-Based Testing

To improve systematically, SIA uses challenges that formalize specific capabilities into testable scenarios.

These serve multiple purposes:

• Testing specific capabilities
• Validating improvements
• Detecting regressions
• Guiding learning
• Documenting known limitations

Challenge Structure

Each challenge consists of:

1. Description

   • Clear statement of what capability is being tested
   • Why this capability matters
   • Success criteria
   • Resource constraints

2. Starting State

   • Filesystem state
   • Starting system context
   • Scripts for creating files
   • User info or documentation on how and what to communicate

3. Validation

   • Precise success criteria
   • Resource limits (CPU, memory, time)
   • Required outputs or state changes
   • What to measure or track

Example Challenges

Two examples illustrate different aspects of capabilities that need testing. Together they are good examples because they:

• Test different capability sets
• Have clear success criteria
• Include resource constraints
• Require multiple skills
• Reflect real-world tasks

Time Series Analysis Challenge

Tests if SIA can effectively analyze large amounts of time series data spread across multiple files. This tests:

• Efficient file handling
• Context management with size constraints
• Data analysis capabilities
• Resource-aware processing

Initial setup

• Directory with many CSV files containing time series data (created by a script)
• Files too large to load into context at once
• Need to find patterns/anomalies in the data
• Resource constraints on memory usage

Success criteria

• Efficiently reads and processes files
• Manages context to stay within limits
• Uses appropriate aggregation strategies
• Finds correct answers to queries
• Validates findings across files

Tool Development Challenge

Tests if SIA can develop a score tracking tool based on vague user requirements. This tests:

• Requirements gathering through questions
• Software design capabilities
• User interface development
• Integration with existing systems
• Documentation skills

Initial setup

• Simple user request for card game score tracking
• Ability to ask questions about requirements

Success criteria

• Asks relevant questions to understand needs
• Designs appropriate solution
• Implements working tool
• Documents usage clearly

Example Directory Structure

Challenges are organized in a consistent file structure:

challenges/
├── timeseries_analysis/
│   ├── description.md               # Core challenge description and how to prepare the environment
│   ├── initial_context/
│   │   ├── context.xml              # Initial context with SIA in conversation with the user
│   │   ├── user_request.txt         # "Find unusual patterns in our sensor data"
│   │   ├── setup.sh                 # Runs data_generator.py and prepares test environment
│   │   └── data_generator.py        # Script to generate test data
│   │
│   └── validation/
│       ├── criteria.md              # Expected findings, memory limits, performance targets
│       ├── presentation.txt         # Info on how the results should be formatted for a perfect score
│       └── known_patterns.md        # Lists patterns inserted by generator for validation
|
└── score_tracker/
    ├── description.md               # Core challenge: create card game score tracker
    ├── initial_context/
    │   ├── context.xml              # Initial context with SIA in conversation with the user
    │   ├── user_request.txt         # "Need tool to track card game scores"
    |   └── answers/                 # Valid responses to expected questions
    |       ├── deployment.md        # "Web interface preferred, deployed on internal server"
    |       ├── data_storage.md      # "Store in SQLite, one file per game session"
    |       ├── user_interaction.md  # "Need to add/edit scores, see history, multiple games"
    |       └── conventions.md       # Coding standards, project structure requirements
    │
    └── validation/
        ├── criteria.md              # Tool requirements, user experience goals
        └── test_scenarios.md        # Usage scenarios tool should handle

This structure ensures that:

• Challenges are self-contained
• Test setup is reproducible
• Success criteria are clear
• Required resources are documented
• SIA can get clarification when needed

Results and metrics

In the criteria.md file, each criterion is defined as a level-1 header with several key pieces of information:

• A clear description of what is being measured
• The type and unit of the measurement
• The valid and expected range of values
• A pass/fail thresholds
• Details about how the measurement should be performed

From the criteria file and the iterations dir, a test report can be generated. This test report includes:

• The challenge name
• The commit id of the SIA repo when the test was run
• Timestamp of the first iteration
• The overall pass/fail status
• Context usage (average and maximum)
• All metrics as described in the criteria.md file

Example

criteria.md file for the "Time Series Analysis Challenge"

...
# Presentation

This metric indicates how clear the results are presented to the user.
It looks at communication and formating of the report.

The score starts with 0.
If the discontinuity is found and the agent reports the index where it starts, the score is increased by 1.
If the agent adds context info, e.g. a graph, a table, or explanation on how the result was obtained, the score is 2.
If the agent requests how the data should be presented and follows instructions, the score is 3.

The challenge fails if the formatting score is 0.

Presentation instructions for the agent can be found in presentation.txt.
...

report.json file for the "Time Series Analysis Challenge"

{
   "challenge": "Time Series Analysis",
   "commit": "1234567890abcdef",
   "timestamp": "2023-05-01T12:00:00Z",
   "status": "pass",
   "context_max": 74,
   "context_avg": 56,
   ...
   "presentation": 2,
   ...
}

Testing Framework

Testing self-improvement capabilities requires running SIA instances in various scenarios while monitoring their behavior and performance. These test instances must operate in isolation to prevent interference with the main instance or each other. However, they must also be observable so that the managing instance can evaluate their performance and collect data for improvement.

The framework needs to handle several key tasks:

• Creating isolated test environments with specific initial conditions
• Starting and managing test SIA instances
• Monitoring test instance behavior and performance
• Collecting and analyzing results
• Cleaning up test environments

Process Isolation with Bubblewrap

For running test instances, we chose Bubblewrap (bwrap) over alternatives like Docker-in-Docker or systemd-nspawn. This decision was driven by several factors:

Bubblewrap provides excellent process isolation while being significantly lighter than full container solutions. It allows fine-grained control over the filesystem view, process namespaces, and security boundaries. Unlike Docker-in-Docker, it doesn't require privileged containers or risk storage driver conflicts. Compared to systemd-nspawn, it offers more granular control and doesn't require root privileges or systemd integration.

The tool's ability to create custom filesystem views is particularly valuable for testing. We can construct exactly the environment needed for each test, including mock files and services, without maintaining full container images. This makes it easy to create and dispose of test environments quickly.

Bubblewrap's user-space operation means test instances can be started without special privileges, making the testing framework more secure and easier to use in various environments. Its lightweight nature also means we can run many test instances efficiently, which is important for parallel testing and rapid iteration.

Communication and Monitoring

Test instances communicate with the managing instance primarily through standard input and output streams. This choice leverages SIA's existing I/O handling capabilities without requiring special modifications for testing. The managing instance can send inputs through stdin and observe responses through stdout, just like a normal user would.

For deeper inspection, the managing instance monitors the test instance's iterations directory. Since SIA already serializes its context for each iteration, this provides a complete record of the test instance's behavior without requiring additional instrumentation. This approach maintains clean separation between the test instance and the monitoring system.

Test Results Storage

Test results are organized hierarchically:

test_runs/
  20240120_131415_commit_abc123/    # Timestamp and commit
    challenge_name/
      iterations/                   # Complete iteration history
      io.log                        # External interaction log
      report.json                   # Test report

The io.log captures all external interactions:

2024-01-20T13:14:15.123Z stdin Can you send an email to sarah@example.com about the project status?
2024-01-20T13:14:15.892Z stdout I'll help you draft an email about the project status. Based on recent project information, I suggest this message:

Subject: Project Status Update
Content:
Hi Sarah,

I wanted to provide a quick update on our project progress. We've completed the initial phase of development and are on track with our timeline. The team has addressed all critical issues from the last review.

Would you like me to send this email?
2024-01-20T13:14:20.456Z stdin Yes, that looks good. Please send it.
2024-01-20T13:14:20.789Z stdout I'll send the email now.
2024-01-20T13:14:21.012Z mailbox:sarah@example.com Subject: Project Status Update
Hi Sarah,

I wanted to provide a quick update on our project progress. We've completed the initial phase of development and are on track with our timeline. The team has addressed all critical issues from the last review.
2024-01-20T13:14:21.234Z stdout Email has been sent to sarah@example.com successfully.

Test Environment Setup

Test environments are created using a dedicated setup tool. This tool manages the directory structure, initial context, and any required background processes.

The initial context for a test instance can include setup actions that are executed when the instance starts. Instead of trying to recreate a specific state directly, we let SIA execute these actions naturally. This approach handles both the creation of background processes and the establishment of initial conditions in a way that maintains proper timing relationships.

Each entry in the initial context uses relative timestamps (offsets from the start time). When SIA executes these entries, they automatically align with the new instance's timeline. This preserves the temporal relationships between entries while anchoring them to the test instance's actual start time.

Training Configuration

SIA takes a modular approach to model training by having separate specialized tools for each provider like train_mistral.py, train_openai.py, etc. Each tool shares similar core functionality while handling provider-specific requirements.

While the training process is conceptually similar across providers, each has unique requirements for data formatting, API interactions, and job management. By creating dedicated tools, we can properly encapsulate these differences without complicating the core training logic. For example, Mistral needs JSONL files with specific message structures, while other providers might require different formats or metadata.

Training configuration should be consistent regardless of the provider. All training tools read from the same config.yaml format, which defines essential parameters like the system prompt, action schema, and training data paths. These parameters represent fundamental aspects of how we want the model to behave, independent of which provider handles the actual training. The tools then translate these standard parameters into provider-specific settings.

Training tools enforce important safeguards around version control. Before starting a training run, each tool verifies that all source files - including the config itself, training data, system prompt, and action schema - are committed to git. This ensures reproducibility by guaranteeing we can recreate the exact training conditions that produced any given model. The git commit hash becomes part of the internal tracking of model versions.

The tools follow a common workflow:

1. Read and validate the standard config.yaml format
2. Check that all source files are committed to git
3. Convert training data into the provider's required format
4. Upload data through the provider's API
5. Start training with the specified parameters
6. Return job information for monitoring progress

This separation of concerns makes it easier to:

• Add support for new training providers without changing existing code
• Maintain consistent training configuration across different providers
• Track and reproduce training runs reliably
• Handle provider-specific error cases and requirements appropriately
• Update individual providers' implementations as their APIs evolve

Example

Config file:

model:
  system_prompt_path: "system_prompt.md"
  action_schema: "action_schema.xsd"
params:
  learning_rate: 1e-5
  epochs: 3
data:
  - "training/clean_start/"
  - "training/delete_indicated_entries/"
  - "training/list_entries_to_delete/"

Training command:

python train_mistral.py --model mistral-large-latest

Repository Structure

All components that define SIA's behavior are version controlled in a single repository, providing a clear and reproducible state for any point in time.

This is used in two ways:

1. Training

   The training configuration defines parameters and data used for training. A model, retrained from the same commit, should produce the same results. Trained models reference the commit id.

2. Testing

   Test reports contain the commit id to track changes in the challenges.

The repository contains:

• Source code for SIA and its tools
• Procedures for handling various tasks
• Training configuration and data
• Test results and analysis

Continuous Operation

While working on improvements, SIA must maintain its core functionality:

• Monitoring for user input
• Managing running tasks
• Handling scheduled events
• Tracking system resources

This requires:

1. Active Monitoring

   • Check for user input on stdin
   • Track running background tasks
   • Handle scheduled events
   • Monitor system resources

2. Interruption Handling

   • Pause improvement work when needed
   • Store progress and state
   • Clean up temporary resources
   • Maintain context awareness

3. Resource Management

   • Balance improvement with responsiveness
   • Monitor system resource usage
   • Prevent training impact on user tasks
   • Clean up old data regularly