SIA/procedures/filesystem_design/reasoning.md

Filesystem Usage Reasoning

SIA's filesystem organization is foundational to its operation and self-improvement capabilities. This document focuses specifically on how files and directories should be structured to support SIA's core functions across different environments.

The filesystem design addresses several key requirements:

• Self-Modification: Supporting SIA's ability to modify its own code and resources
• Persistence: Ensuring critical data survives across container restarts and environment changes
• Environment Portability: Maintaining consistent operation in local development and cloud environments
• Sub-Instance Management: Enabling isolated filesystem views for testing
• Tool Accessibility: Organizing tools to be both modifiable and accessible

These requirements often have competing demands, requiring careful balance between flexibility, consistency, and isolation.

Runtime Filesystem Structure

Core Directories

The SIA runtime filesystem is organized into distinct areas with different purposes and persistence characteristics:

/root/
├── sia/                    # Git repository containing code, configuration, and documentation
├── data/                   # Central location for all persistent storage
│   ├── iterations/         # Operation history recorded as XML files
│   ├── user/               # User-specific information and preferences
│   ├── tasks/              # Task tracking and management
│   ├── environment/        # Information about the runtime environment
│   └── ...                 # Other persistent data directories as needed
├── models/                 # LLM model files, organized by version
│   ├── 1234567890abcdef/   # Specific versioned model
│   ├── abcdef1234567890/   # Another versioned model
│   └── current/            # Symlink to the currently active model
└── desktop/                # Ephemeral workspace for temporary files
    ├── test_environments/  # Sub-instance test environments
    ├── task_workspaces/    # Temporary files for current tasks
    └── ...                 # Other ephemeral workspaces

Purpose and Characteristics

Each top-level directory serves a specific purpose with distinct persistence characteristics:

• /root/sia/

  • Contains the complete git repository
  • All source code, configurations, and documentation
  • Modified through self-improvement processes
  • Synchronized with remote git repository
  • Mounted from host in development, cloned in cloud environments

• /root/data/

  • Central location for all persistent data
  • Mounted as persistent storage in all environments
  • Critical for operational continuity
  • Requires regular backup and synchronization
  • Contains subdirectories for different types of persistent data:
    • iterations/: Records of SIA's reasoning and actions
    • user/: Information about users SIA interacts with
    • tasks/: Current and completed task information
    • environment/: Information about the current environment

• /root/models/

  • Storage for LLM model weights
  • Organized by git commit id of the config file it was trained on
  • current/ symlink points to active model
  • Enables easy switching between model versions
  • Mounted for local development
  • Ephemeral since models can be trained on the fly

• /root/desktop/

  • Ephemeral workspace for temporary operations
  • Can be lost without compromising system integrity
  • Used for testing environments, task workspaces, and intermediate files
  • Cleared periodically to prevent accumulation of stale data

This organization creates clear boundaries between:

• Code (version controlled)
• Persistent data (requires preservation)
• Model files (versioned artifacts)
• Temporary workspaces (disposable)

Persistent Data Management

Data Categories

The /root/data/ directory contains several categories of persistent information:

• Iterations (/root/data/iterations/)

  • Complete history of SIA's operation
  • XML files containing context and responses
  • Organized chronologically
  • Used for training data extraction and operational history

• User Information (/root/data/user/)

  • User preferences and characteristics
  • Conversation history
  • User-specific settings
  • Critical for continuity in user interactions

• Task State (/root/data/tasks/)

  • Current task descriptions and status
  • Task history and outcomes
  • Links to relevant files and resources
  • Essential for resuming work after interruptions

• Environment Information (/root/data/environment/)

  • Information about the current runtime environment
  • URL for the SIA git repository
  • Kind of deployment (e.g., local, cloud, sub-instance)

Additional persistent data categories may be added as SIA's capabilities evolve.

Persistence Strategy

Different environments require different approaches to ensure data persistence:

• Local Development

  • /root/data/ directory is mounted as a Docker volume
  • Maps to a local directory on the development machine
  • Persists across container restarts automatically
  • Easy to back up through standard filesystem operations

• Cloud Deployment (e.g., RunPod)

  • /root/data/ mapped to a persistent volume
  • Cloud provider ensures data survives instance restarts

• Sub-Instances

  • Each sub-instance receives a properly initialized /root/data/ directory
  • Contains minimal necessary data for the specific test case
  • Can be discarded after test completion
  • Important results extracted and preserved before cleanup

Model Management

Model Organization

LLM models are stored in /root/models/ with a clear version-based structure:

• Versioned Directories

  • Models stored in separate directories named by the commit id of the training config file
  • Contains all files needed for the model (weights, tokenizer config, etc.)

• Current Model Symlink

  • /root/models/current/ is a symbolic link to the active model
  • Allows code to reference a consistent path regardless of which model is active
  • Switching models is accomplished by updating this symlink

Model Persistence

Models require special handling due to their size:

• Local Development Models are stored in a local directory mounted to /root/models/. This directory is in the .gitignore.

• Cloud Deployment The initial model is downloaded and finetuned by the bootstrap script.

Git Repository Management

Repository Organization

The SIA git repository at /root/sia/ contains all code and configuration:

• Code Organization

  • Core SIA package in /root/sia/sia/
  • Procedures in /root/sia/procedures/
  • Tools in /root/sia/tools/
  • Documentation in appropriate locations throughout the repository

• Version Control

  • All code modifications tracked through git
  • Branching strategy for self-improvement defined in the git workflow procedure
  • Commit history provides audit trail of system evolution

Repository Access

Access to the git repository varies by environment:

• Local Development

  • Repository directory mounted directly from host
  • Changes immediately visible to the developer
  • No need to push changes to remote repository

• Cloud Deployment

  • Repository cloned during bootstrap
  • Dedicated git credentials for SIA self-modification
  • Changes pushed to remote repository

Sub-Instance Filesystem Management

Sub-Instance Requirements

Sub-instances need isolated filesystem views for testing and development:

• Code Isolation

  • Access to specific versions of SIA code
  • Potentially modified for testing purposes
  • Protected from changes by other instances

• Data Initialization

  • Properly initialized /root/data/ with necessary state
  • Test-specific user and task information
  • Clean iterations directory for recording test behavior

• Model Access

  • Access to appropriate model files
  • Potentially different from the parent instance's model

Sub-Instance Structure

Each sub-instance gets a controlled environment in /root/desktop/test_environments/:

/root/desktop/test_environments/
├── instance_123/                 # Specific test instance
│   ├── sia/                      # Copy or modified version of SIA code
│   ├── data/                     # Initialized data directory
│   │   ├── iterations/           # Empty or seeded with initial state
│   │   ├── user/                 # Test-specific user data
│   │   ├── tasks/                # Test-specific tasks
│   │   └── environment/          # Test-specific environment data
│   └── models/                   # Access to necessary models
│       └── current/              # Symlink to appropriate model version
└── ...                           # Other test instances

The managing instance maintains responsibility for:

• Creating these isolated environments
• Monitoring the sub-instance's operation
• Collecting test results
• Cleaning up after tests complete

Bootstrap Process

When starting in a new environment, SIA requires initialization:

1. Repository Preparation

   • Check for existence of repository at /root/sia/
   • Clone the repository at /root/sia/

2. Model Setup

   • Check for existence of current model
   • Start finetune job for HEAD commit
   • Create/update the /root/models/current/ symlink

3. Data Initialization

   • Initialize /root/data/ with minimal required structure
   • Restore data from backups when available (future work)