248 lines
9.7 KiB
Markdown
248 lines
9.7 KiB
Markdown
# Filesystem Usage Reasoning
|
|
|
|
SIA's filesystem organization is foundational to its operation and self-improvement capabilities. This document focuses specifically on how files and directories should be structured to support SIA's core functions across different environments.
|
|
|
|
The filesystem design addresses several key requirements:
|
|
|
|
- **Self-Modification**: Supporting SIA's ability to modify its own code and resources
|
|
- **Persistence**: Ensuring critical data survives across container restarts and environment changes
|
|
- **Environment Portability**: Maintaining consistent operation in local development and cloud environments
|
|
- **Sub-Instance Management**: Enabling isolated filesystem views for testing
|
|
- **Tool Accessibility**: Organizing tools to be both modifiable and accessible
|
|
|
|
These requirements often have competing demands, requiring careful balance between flexibility, consistency, and isolation.
|
|
|
|
## Runtime Filesystem Structure
|
|
|
|
### Core Directories
|
|
|
|
The SIA runtime filesystem is organized into distinct areas with different purposes and persistence characteristics:
|
|
|
|
```
|
|
/root/
|
|
├── sia/ # Git repository containing code, configuration, and documentation
|
|
├── data/ # Central location for all persistent storage
|
|
│ ├── iterations/ # Operation history recorded as XML files
|
|
│ ├── user/ # User-specific information and preferences
|
|
│ ├── tasks/ # Task tracking and management
|
|
│ ├── environment/ # Information about the runtime environment
|
|
│ └── ... # Other persistent data directories as needed
|
|
├── models/ # LLM model files, organized by version
|
|
│ ├── 1234567890abcdef/ # Specific versioned model
|
|
│ ├── abcdef1234567890/ # Another versioned model
|
|
│ └── current/ # Symlink to the currently active model
|
|
└── desktop/ # Ephemeral workspace for temporary files
|
|
├── test_environments/ # Sub-instance test environments
|
|
├── task_workspaces/ # Temporary files for current tasks
|
|
└── ... # Other ephemeral workspaces
|
|
```
|
|
|
|
### Purpose and Characteristics
|
|
|
|
Each top-level directory serves a specific purpose with distinct persistence characteristics:
|
|
|
|
- **`/root/sia/`**
|
|
- Contains the complete git repository
|
|
- All source code, configurations, and documentation
|
|
- Modified through self-improvement processes
|
|
- Synchronized with remote git repository
|
|
- Mounted from host in development, cloned in cloud environments
|
|
|
|
- **`/root/data/`**
|
|
- Central location for all persistent data
|
|
- Mounted as persistent storage in all environments
|
|
- Critical for operational continuity
|
|
- Requires regular backup and synchronization
|
|
- Contains subdirectories for different types of persistent data:
|
|
- `iterations/`: Records of SIA's reasoning and actions
|
|
- `user/`: Information about users SIA interacts with
|
|
- `tasks/`: Current and completed task information
|
|
- `environment/`: Information about the current environment
|
|
|
|
- **`/root/models/`**
|
|
- Storage for LLM model weights
|
|
- Organized by git commit id of the config file it was trained on
|
|
- `current/` symlink points to active model
|
|
- Enables easy switching between model versions
|
|
- Mounted for local development
|
|
- Ephemeral since models can be trained on the fly
|
|
|
|
- **`/root/desktop/`**
|
|
- Ephemeral workspace for temporary operations
|
|
- Can be lost without compromising system integrity
|
|
- Used for testing environments, task workspaces, and intermediate files
|
|
- Cleared periodically to prevent accumulation of stale data
|
|
|
|
This organization creates clear boundaries between:
|
|
- Code (version controlled)
|
|
- Persistent data (requires preservation)
|
|
- Model files (versioned artifacts)
|
|
- Temporary workspaces (disposable)
|
|
|
|
## Persistent Data Management
|
|
|
|
### Data Categories
|
|
|
|
The `/root/data/` directory contains several categories of persistent information:
|
|
|
|
- **Iterations** (`/root/data/iterations/`)
|
|
- Complete history of SIA's operation
|
|
- XML files containing context and responses
|
|
- Organized chronologically
|
|
- Used for training data extraction and operational history
|
|
|
|
- **User Information** (`/root/data/user/`)
|
|
- User preferences and characteristics
|
|
- Conversation history
|
|
- User-specific settings
|
|
- Critical for continuity in user interactions
|
|
|
|
- **Task State** (`/root/data/tasks/`)
|
|
- Current task descriptions and status
|
|
- Task history and outcomes
|
|
- Links to relevant files and resources
|
|
- Essential for resuming work after interruptions
|
|
|
|
- **Environment Information** (`/root/data/environment/`)
|
|
- Information about the current runtime environment
|
|
- URL for the SIA git repository
|
|
- Kind of deployment (e.g., local, cloud, sub-instance)
|
|
|
|
Additional persistent data categories may be added as SIA's capabilities evolve.
|
|
|
|
### Persistence Strategy
|
|
|
|
Different environments require different approaches to ensure data persistence:
|
|
|
|
- **Local Development**
|
|
- `/root/data/` directory is mounted as a Docker volume
|
|
- Maps to a local directory on the development machine
|
|
- Persists across container restarts automatically
|
|
- Easy to back up through standard filesystem operations
|
|
|
|
- **Cloud Deployment (e.g., RunPod)**
|
|
- `/root/data/` mapped to a persistent volume
|
|
- Cloud provider ensures data survives instance restarts
|
|
|
|
- **Sub-Instances**
|
|
- Each sub-instance receives a properly initialized `/root/data/` directory
|
|
- Contains minimal necessary data for the specific test case
|
|
- Can be discarded after test completion
|
|
- Important results extracted and preserved before cleanup
|
|
|
|
## Model Management
|
|
|
|
### Model Organization
|
|
|
|
LLM models are stored in `/root/models/` with a clear version-based structure:
|
|
|
|
- **Versioned Directories**
|
|
- Models stored in separate directories named by the commit id of the training config file
|
|
- Contains all files needed for the model (weights, tokenizer config, etc.)
|
|
|
|
- **Current Model Symlink**
|
|
- `/root/models/current/` is a symbolic link to the active model
|
|
- Allows code to reference a consistent path regardless of which model is active
|
|
- Switching models is accomplished by updating this symlink
|
|
|
|
### Model Persistence
|
|
|
|
Models require special handling due to their size:
|
|
|
|
- **Local Development**
|
|
Models are stored in a local directory mounted to `/root/models/`.
|
|
This directory is in the .gitignore.
|
|
|
|
- **Cloud Deployment**
|
|
The initial model is downloaded and finetuned by the bootstrap script.
|
|
|
|
## Git Repository Management
|
|
|
|
### Repository Organization
|
|
|
|
The SIA git repository at `/root/sia/` contains all code and configuration:
|
|
|
|
- **Code Organization**
|
|
- Core SIA package in `/root/sia/sia/`
|
|
- Procedures in `/root/sia/procedures/`
|
|
- Tools in `/root/sia/tools/`
|
|
- Documentation in appropriate locations throughout the repository
|
|
|
|
- **Version Control**
|
|
- All code modifications tracked through git
|
|
- Branching strategy for self-improvement defined in the git workflow procedure
|
|
- Commit history provides audit trail of system evolution
|
|
|
|
### Repository Access
|
|
|
|
Access to the git repository varies by environment:
|
|
|
|
- **Local Development**
|
|
- Repository directory mounted directly from host
|
|
- Changes immediately visible to the developer
|
|
- No need to push changes to remote repository
|
|
|
|
- **Cloud Deployment**
|
|
- Repository cloned during bootstrap
|
|
- Dedicated git credentials for SIA self-modification
|
|
- Changes pushed to remote repository
|
|
|
|
## Sub-Instance Filesystem Management
|
|
|
|
### Sub-Instance Requirements
|
|
|
|
Sub-instances need isolated filesystem views for testing and development:
|
|
|
|
- **Code Isolation**
|
|
- Access to specific versions of SIA code
|
|
- Potentially modified for testing purposes
|
|
- Protected from changes by other instances
|
|
|
|
- **Data Initialization**
|
|
- Properly initialized `/root/data/` with necessary state
|
|
- Test-specific user and task information
|
|
- Clean iterations directory for recording test behavior
|
|
|
|
- **Model Access**
|
|
- Access to appropriate model files
|
|
- Potentially different from the parent instance's model
|
|
|
|
### Sub-Instance Structure
|
|
|
|
Each sub-instance gets a controlled environment in `/root/desktop/test_environments/`:
|
|
|
|
```
|
|
/root/desktop/test_environments/
|
|
├── instance_123/ # Specific test instance
|
|
│ ├── sia/ # Copy or modified version of SIA code
|
|
│ ├── data/ # Initialized data directory
|
|
│ │ ├── iterations/ # Empty or seeded with initial state
|
|
│ │ ├── user/ # Test-specific user data
|
|
│ │ ├── tasks/ # Test-specific tasks
|
|
│ │ └── environment/ # Test-specific environment data
|
|
│ └── models/ # Access to necessary models
|
|
│ └── current/ # Symlink to appropriate model version
|
|
└── ... # Other test instances
|
|
```
|
|
|
|
The managing instance maintains responsibility for:
|
|
- Creating these isolated environments
|
|
- Monitoring the sub-instance's operation
|
|
- Collecting test results
|
|
- Cleaning up after tests complete
|
|
|
|
## Bootstrap Process
|
|
|
|
When starting in a new environment, SIA requires initialization:
|
|
|
|
1. **Repository Preparation**
|
|
- Check for existence of repository at `/root/sia/`
|
|
- Clone the repository at `/root/sia/`
|
|
2. **Model Setup**
|
|
- Check for existence of current model
|
|
- Start finetune job for HEAD commit
|
|
- Create/update the `/root/models/current/` symlink
|
|
|
|
3. **Data Initialization**
|
|
- Initialize `/root/data/` with minimal required structure
|
|
- Restore data from backups when available (future work) |