Added info about filesystem, tools and git as procedures
This commit is contained in:
Niels Geens
248
procedures/filesystem_design/reasoning.md
Normal file
248
procedures/filesystem_design/reasoning.md
Normal file
@@ -0,0 +1,248 @@
|
||||
# Filesystem Usage Reasoning
|
||||
|
||||
SIA's filesystem organization is foundational to its operation and self-improvement capabilities. This document focuses specifically on how files and directories should be structured to support SIA's core functions across different environments.
|
||||
|
||||
The filesystem design addresses several key requirements:
|
||||
|
||||
- **Self-Modification**: Supporting SIA's ability to modify its own code and resources
|
||||
- **Persistence**: Ensuring critical data survives across container restarts and environment changes
|
||||
- **Environment Portability**: Maintaining consistent operation in local development and cloud environments
|
||||
- **Sub-Instance Management**: Enabling isolated filesystem views for testing
|
||||
- **Tool Accessibility**: Organizing tools to be both modifiable and accessible
|
||||
|
||||
These requirements often have competing demands, requiring careful balance between flexibility, consistency, and isolation.
|
||||
|
||||
## Runtime Filesystem Structure
|
||||
|
||||
### Core Directories
|
||||
|
||||
The SIA runtime filesystem is organized into distinct areas with different purposes and persistence characteristics:
|
||||
|
||||
```
|
||||
/root/
|
||||
├── sia/ # Git repository containing code, configuration, and documentation
|
||||
├── data/ # Central location for all persistent storage
|
||||
│ ├── iterations/ # Operation history recorded as XML files
|
||||
│ ├── user/ # User-specific information and preferences
|
||||
│ ├── tasks/ # Task tracking and management
|
||||
│ ├── environment/ # Information about the runtime environment
|
||||
│ └── ... # Other persistent data directories as needed
|
||||
├── models/ # LLM model files, organized by version
|
||||
│ ├── 1234567890abcdef/ # Specific versioned model
|
||||
│ ├── abcdef1234567890/ # Another versioned model
|
||||
│ └── current/ # Symlink to the currently active model
|
||||
└── desktop/ # Ephemeral workspace for temporary files
|
||||
├── test_environments/ # Sub-instance test environments
|
||||
├── task_workspaces/ # Temporary files for current tasks
|
||||
└── ... # Other ephemeral workspaces
|
||||
```
|
||||
|
||||
### Purpose and Characteristics
|
||||
|
||||
Each top-level directory serves a specific purpose with distinct persistence characteristics:
|
||||
|
||||
- **`/root/sia/`**
|
||||
- Contains the complete git repository
|
||||
- All source code, configurations, and documentation
|
||||
- Modified through self-improvement processes
|
||||
- Synchronized with remote git repository
|
||||
- Mounted from host in development, cloned in cloud environments
|
||||
|
||||
- **`/root/data/`**
|
||||
- Central location for all persistent data
|
||||
- Mounted as persistent storage in all environments
|
||||
- Critical for operational continuity
|
||||
- Requires regular backup and synchronization
|
||||
- Contains subdirectories for different types of persistent data:
|
||||
- `iterations/`: Records of SIA's reasoning and actions
|
||||
- `user/`: Information about users SIA interacts with
|
||||
- `tasks/`: Current and completed task information
|
||||
- `environment/`: Information about the current environment
|
||||
|
||||
- **`/root/models/`**
|
||||
- Storage for LLM model weights
|
||||
- Organized by git commit id of the config file it was trained on
|
||||
- `current/` symlink points to active model
|
||||
- Enables easy switching between model versions
|
||||
- Mounted for local development
|
||||
- Ephemeral since models can be trained on the fly
|
||||
|
||||
- **`/root/desktop/`**
|
||||
- Ephemeral workspace for temporary operations
|
||||
- Can be lost without compromising system integrity
|
||||
- Used for testing environments, task workspaces, and intermediate files
|
||||
- Cleared periodically to prevent accumulation of stale data
|
||||
|
||||
This organization creates clear boundaries between:
|
||||
- Code (version controlled)
|
||||
- Persistent data (requires preservation)
|
||||
- Model files (versioned artifacts)
|
||||
- Temporary workspaces (disposable)
|
||||
|
||||
## Persistent Data Management
|
||||
|
||||
### Data Categories
|
||||
|
||||
The `/root/data/` directory contains several categories of persistent information:
|
||||
|
||||
- **Iterations** (`/root/data/iterations/`)
|
||||
- Complete history of SIA's operation
|
||||
- XML files containing context and responses
|
||||
- Organized chronologically
|
||||
- Used for training data extraction and operational history
|
||||
|
||||
- **User Information** (`/root/data/user/`)
|
||||
- User preferences and characteristics
|
||||
- Conversation history
|
||||
- User-specific settings
|
||||
- Critical for continuity in user interactions
|
||||
|
||||
- **Task State** (`/root/data/tasks/`)
|
||||
- Current task descriptions and status
|
||||
- Task history and outcomes
|
||||
- Links to relevant files and resources
|
||||
- Essential for resuming work after interruptions
|
||||
|
||||
- **Environment Information** (`/root/data/environment/`)
|
||||
- Information about the current runtime environment
|
||||
- URL for the SIA git repository
|
||||
- Kind of deployment (e.g., local, cloud, sub-instance)
|
||||
|
||||
Additional persistent data categories may be added as SIA's capabilities evolve.
|
||||
|
||||
### Persistence Strategy
|
||||
|
||||
Different environments require different approaches to ensure data persistence:
|
||||
|
||||
- **Local Development**
|
||||
- `/root/data/` directory is mounted as a Docker volume
|
||||
- Maps to a local directory on the development machine
|
||||
- Persists across container restarts automatically
|
||||
- Easy to back up through standard filesystem operations
|
||||
|
||||
- **Cloud Deployment (e.g., RunPod)**
|
||||
- `/root/data/` mapped to a persistent volume
|
||||
- Cloud provider ensures data survives instance restarts
|
||||
|
||||
- **Sub-Instances**
|
||||
- Each sub-instance receives a properly initialized `/root/data/` directory
|
||||
- Contains minimal necessary data for the specific test case
|
||||
- Can be discarded after test completion
|
||||
- Important results extracted and preserved before cleanup
|
||||
|
||||
## Model Management
|
||||
|
||||
### Model Organization
|
||||
|
||||
LLM models are stored in `/root/models/` with a clear version-based structure:
|
||||
|
||||
- **Versioned Directories**
|
||||
- Models stored in separate directories named by the commit id of the training config file
|
||||
- Contains all files needed for the model (weights, tokenizer config, etc.)
|
||||
|
||||
- **Current Model Symlink**
|
||||
- `/root/models/current/` is a symbolic link to the active model
|
||||
- Allows code to reference a consistent path regardless of which model is active
|
||||
- Switching models is accomplished by updating this symlink
|
||||
|
||||
### Model Persistence
|
||||
|
||||
Models require special handling due to their size:
|
||||
|
||||
- **Local Development**
|
||||
Models are stored in a local directory mounted to `/root/models/`.
|
||||
This directory is in the .gitignore.
|
||||
|
||||
- **Cloud Deployment**
|
||||
The initial model is downloaded and finetuned by the bootstrap script.
|
||||
|
||||
## Git Repository Management
|
||||
|
||||
### Repository Organization
|
||||
|
||||
The SIA git repository at `/root/sia/` contains all code and configuration:
|
||||
|
||||
- **Code Organization**
|
||||
- Core SIA package in `/root/sia/sia/`
|
||||
- Procedures in `/root/sia/procedures/`
|
||||
- Tools in `/root/sia/tools/`
|
||||
- Documentation in appropriate locations throughout the repository
|
||||
|
||||
- **Version Control**
|
||||
- All code modifications tracked through git
|
||||
- Branching strategy for self-improvement defined in the git workflow procedure
|
||||
- Commit history provides audit trail of system evolution
|
||||
|
||||
### Repository Access
|
||||
|
||||
Access to the git repository varies by environment:
|
||||
|
||||
- **Local Development**
|
||||
- Repository directory mounted directly from host
|
||||
- Changes immediately visible to the developer
|
||||
- No need to push changes to remote repository
|
||||
|
||||
- **Cloud Deployment**
|
||||
- Repository cloned during bootstrap
|
||||
- Dedicated git credentials for SIA self-modification
|
||||
- Changes pushed to remote repository
|
||||
|
||||
## Sub-Instance Filesystem Management
|
||||
|
||||
### Sub-Instance Requirements
|
||||
|
||||
Sub-instances need isolated filesystem views for testing and development:
|
||||
|
||||
- **Code Isolation**
|
||||
- Access to specific versions of SIA code
|
||||
- Potentially modified for testing purposes
|
||||
- Protected from changes by other instances
|
||||
|
||||
- **Data Initialization**
|
||||
- Properly initialized `/root/data/` with necessary state
|
||||
- Test-specific user and task information
|
||||
- Clean iterations directory for recording test behavior
|
||||
|
||||
- **Model Access**
|
||||
- Access to appropriate model files
|
||||
- Potentially different from the parent instance's model
|
||||
|
||||
### Sub-Instance Structure
|
||||
|
||||
Each sub-instance gets a controlled environment in `/root/desktop/test_environments/`:
|
||||
|
||||
```
|
||||
/root/desktop/test_environments/
|
||||
├── instance_123/ # Specific test instance
|
||||
│ ├── sia/ # Copy or modified version of SIA code
|
||||
│ ├── data/ # Initialized data directory
|
||||
│ │ ├── iterations/ # Empty or seeded with initial state
|
||||
│ │ ├── user/ # Test-specific user data
|
||||
│ │ ├── tasks/ # Test-specific tasks
|
||||
│ │ └── environment/ # Test-specific environment data
|
||||
│ └── models/ # Access to necessary models
|
||||
│ └── current/ # Symlink to appropriate model version
|
||||
└── ... # Other test instances
|
||||
```
|
||||
|
||||
The managing instance maintains responsibility for:
|
||||
- Creating these isolated environments
|
||||
- Monitoring the sub-instance's operation
|
||||
- Collecting test results
|
||||
- Cleaning up after tests complete
|
||||
|
||||
## Bootstrap Process
|
||||
|
||||
When starting in a new environment, SIA requires initialization:
|
||||
|
||||
1. **Repository Preparation**
|
||||
- Check for existence of repository at `/root/sia/`
|
||||
- Clone the repository at `/root/sia/`
|
||||
2. **Model Setup**
|
||||
- Check for existence of current model
|
||||
- Start finetune job for HEAD commit
|
||||
- Create/update the `/root/models/current/` symlink
|
||||
|
||||
3. **Data Initialization**
|
||||
- Initialize `/root/data/` with minimal required structure
|
||||
- Restore data from backups when available (future work)
|
||||
Reference in New Issue
Block a user