# Filesystem Usage Reasoning

SIA's filesystem organization is foundational to its operation and self-improvement capabilities. This document focuses specifically on how files and directories should be structured to support SIA's core functions across different environments.

The filesystem design addresses several key requirements:

- **Self-Modification**: Supporting SIA's ability to modify its own code and resources
- **Persistence**: Ensuring critical data survives across container restarts and environment changes
- **Environment Portability**: Maintaining consistent operation in local development and cloud environments
- **Sub-Instance Management**: Enabling isolated filesystem views for testing
- **Tool Accessibility**: Organizing tools to be both modifiable and accessible

These requirements often have competing demands, requiring careful balance between flexibility, consistency, and isolation.

## Runtime Filesystem Structure

### Core Directories

The SIA runtime filesystem is organized into distinct areas with different purposes and persistence characteristics:

```
/root/
├── sia/                    # Git repository containing code, configuration, and documentation
├── data/                   # Central location for all persistent storage
│   ├── iterations/         # Operation history recorded as XML files
│   ├── user/               # User-specific information and preferences
│   ├── tasks/              # Task tracking and management
│   ├── environment/        # Information about the runtime environment
│   └── ...                 # Other persistent data directories as needed
├── models/                 # LLM model files, organized by version
│   ├── 1234567890abcdef/   # Specific versioned model
│   ├── abcdef1234567890/   # Another versioned model
│   └── current/            # Symlink to the currently active model
└── desktop/                # Ephemeral workspace for temporary files
    ├── test_environments/  # Sub-instance test environments
    ├── task_workspaces/    # Temporary files for current tasks
    └── ...                 # Other ephemeral workspaces
```

### Purpose and Characteristics

Each top-level directory serves a specific purpose with distinct persistence characteristics:

- **`/root/sia/`**
    - Contains the complete git repository
    - All source code, configurations, and documentation
    - Modified through self-improvement processes
    - Synchronized with remote git repository
    - Mounted from host in development, cloned in cloud environments

- **`/root/data/`**
    - Central location for all persistent data
    - Mounted as persistent storage in all environments
    - Critical for operational continuity
    - Requires regular backup and synchronization
    - Contains subdirectories for different types of persistent data:
        - `iterations/`: Records of SIA's reasoning and actions
        - `user/`: Information about users SIA interacts with
        - `tasks/`: Current and completed task information
        - `environment/`: Information about the current environment

- **`/root/models/`**
    - Storage for LLM model weights
    - Organized by git commit id of the config file it was trained on
    - `current/` symlink points to active model
    - Enables easy switching between model versions
    - Mounted for local development
    - Ephemeral since models can be trained on the fly

- **`/root/desktop/`**
    - Ephemeral workspace for temporary operations
    - Can be lost without compromising system integrity
    - Used for testing environments, task workspaces, and intermediate files
    - Cleared periodically to prevent accumulation of stale data

This organization creates clear boundaries between:
- Code (version controlled)
- Persistent data (requires preservation)
- Model files (versioned artifacts)
- Temporary workspaces (disposable)

## Persistent Data Management

### Data Categories

The `/root/data/` directory contains several categories of persistent information:

- **Iterations** (`/root/data/iterations/`)
    - Complete history of SIA's operation
    - XML files containing context and responses
    - Organized chronologically
    - Used for training data extraction and operational history

- **User Information** (`/root/data/user/`)
    - User preferences and characteristics
    - Conversation history
    - User-specific settings
    - Critical for continuity in user interactions

- **Task State** (`/root/data/tasks/`)
    - Current task descriptions and status
    - Task history and outcomes
    - Links to relevant files and resources
    - Essential for resuming work after interruptions

- **Environment Information** (`/root/data/environment/`)
    - Information about the current runtime environment
    - URL for the SIA git repository
    - Kind of deployment (e.g., local, cloud, sub-instance)

Additional persistent data categories may be added as SIA's capabilities evolve.

### Persistence Strategy

Different environments require different approaches to ensure data persistence:

- **Local Development**
    - `/root/data/` directory is mounted as a Docker volume
    - Maps to a local directory on the development machine
    - Persists across container restarts automatically
    - Easy to back up through standard filesystem operations

- **Cloud Deployment (e.g., RunPod)**
    - `/root/data/` mapped to a persistent volume
    - Cloud provider ensures data survives instance restarts

- **Sub-Instances**
    - Each sub-instance receives a properly initialized `/root/data/` directory
    - Contains minimal necessary data for the specific test case
    - Can be discarded after test completion
    - Important results extracted and preserved before cleanup

## Model Management

### Model Organization

LLM models are stored in `/root/models/` with a clear version-based structure:

- **Versioned Directories**
    - Models stored in separate directories named by the commit id of the training config file
    - Contains all files needed for the model (weights, tokenizer config, etc.)

- **Current Model Symlink**
    - `/root/models/current/` is a symbolic link to the active model
    - Allows code to reference a consistent path regardless of which model is active
    - Switching models is accomplished by updating this symlink

### Model Persistence

Models require special handling due to their size:

- **Local Development**
    Models are stored in a local directory mounted to `/root/models/`.
    This directory is in the .gitignore.

- **Cloud Deployment**
    The initial model is downloaded and finetuned by the bootstrap script.

## Git Repository Management

### Repository Organization

The SIA git repository at `/root/sia/` contains all code and configuration:

- **Code Organization**
    - Core SIA package in `/root/sia/sia/`
    - Procedures in `/root/sia/procedures/`
    - Tools in `/root/sia/tools/`
    - Documentation in appropriate locations throughout the repository

- **Version Control**
    - All code modifications tracked through git
    - Branching strategy for self-improvement defined in the git workflow procedure
    - Commit history provides audit trail of system evolution

### Repository Access

Access to the git repository varies by environment:

- **Local Development**
    - Repository directory mounted directly from host
    - Changes immediately visible to the developer
    - No need to push changes to remote repository

- **Cloud Deployment**
    - Repository cloned during bootstrap
    - Dedicated git credentials for SIA self-modification
    - Changes pushed to remote repository

## Sub-Instance Filesystem Management

### Sub-Instance Requirements

Sub-instances need isolated filesystem views for testing and development:

- **Code Isolation**
    - Access to specific versions of SIA code
    - Potentially modified for testing purposes
    - Protected from changes by other instances

- **Data Initialization**
    - Properly initialized `/root/data/` with necessary state
    - Test-specific user and task information
    - Clean iterations directory for recording test behavior

- **Model Access**
    - Access to appropriate model files
    - Potentially different from the parent instance's model

### Sub-Instance Structure

Each sub-instance gets a controlled environment in `/root/desktop/test_environments/`:

```
/root/desktop/test_environments/
├── instance_123/                 # Specific test instance
│   ├── sia/                      # Copy or modified version of SIA code
│   ├── data/                     # Initialized data directory
│   │   ├── iterations/           # Empty or seeded with initial state
│   │   ├── user/                 # Test-specific user data
│   │   ├── tasks/                # Test-specific tasks
│   │   └── environment/          # Test-specific environment data
│   └── models/                   # Access to necessary models
│       └── current/              # Symlink to appropriate model version
└── ...                           # Other test instances
```

The managing instance maintains responsibility for:
- Creating these isolated environments
- Monitoring the sub-instance's operation
- Collecting test results
- Cleaning up after tests complete

## Bootstrap Process

When starting in a new environment, SIA requires initialization:

1. **Repository Preparation**
    - Check for existence of repository at `/root/sia/`
    - Clone the repository at `/root/sia/`
2. **Model Setup**
    - Check for existence of current model
    - Start finetune job for HEAD commit
    - Create/update the `/root/models/current/` symlink

3. **Data Initialization**
    - Initialize `/root/data/` with minimal required structure
    - Restore data from backups when available (future work)