llm/SIA
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Improved self improvement example directory structure

Browse Source
This commit is contained in:
Niels Geens
2025-02-24 14:03:52 +01:00
parent e664c625b7
commit f635725064
1 changed files with 26 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
procedures/self_improvement/reasoning.md
View File

@@ -51,11 +51,11 @@ Each challenge consists of:
- Success criteria
- Resource constraints
2. Initial Context
- Minimal but complete starting state
- Required user information
- System files and directories needed
- Clear separation between essential context and noise
2. Starting State
- Filesystem state
- Starting system context
- Scripts for creating files
- User info or documentation on how and what to communicate
3. Validation
- Precise success criteria
@@ -127,38 +127,32 @@ Challenges are organized in a consistent file structure:
```
challenges/
├── timeseries_analysis/
│ ├── description.md # Core challenge description and how to prepare the environment
│ ├── description.md # Core challenge description and how to prepare the environment
│ ├── initial_context/
│ │ ├── context.xml # Initial context with SIA in conversation with the user
│ │ ├── user_request.txt # "Find unusual patterns in our sensor data"
│ │ ├── presentation.txt # Info on how the results should be formatted for a perfect score
│ │ └── data_generator.py # Script to generate test data
│ │ ├── context.xml # Initial context with SIA in conversation with the user
│ │ ├── user_request.txt # "Find unusual patterns in our sensor data"
│ │ ├── setup.sh # Runs data_generator.py and prepares test environment
│ │ └── data_generator.py # Script to generate test data
│ │
│ ├── validation/
│ │ ├── criteria.md # Expected findings, memory limits, performance targets
│ │ ├── setup.sh # Runs data_generator.py and prepares test environment
│ │ └── known_patterns.md # Lists patterns inserted by generator for validation
│ │
│ └── answers/ # Valid answers to questions SIA might ask
| └── data_format.txt # Explains CSV structure if asked
│ └── validation/
│ ├── criteria.md # Expected findings, memory limits, performance targets
│ ├── presentation.txt # Info on how the results should be formatted for a perfect score
│ └── known_patterns.md # Lists patterns inserted by generator for validation
|
└── score_tracker/
├── description.md # Core challenge: create card game score tracker
├── description.md # Core challenge: create card game score tracker
├── initial_context/
│ ├── context.xml # Initial context with SIA in conversation with the user
│ ├── user_request.txt # "Need tool to track card game scores"
│ ├── tool_directory.txt # Shows where tools are stored and basic structure
│ └── example_tool/ # Simple tool showing expected structure/style
│ ├── context.xml # Initial context with SIA in conversation with the user
│ ├── user_request.txt # "Need tool to track card game scores"
| └── answers/ # Valid responses to expected questions
| ├── deployment.md # "Web interface preferred, deployed on internal server"
| ├── data_storage.md # "Store in SQLite, one file per game session"
| ├── user_interaction.md # "Need to add/edit scores, see history, multiple games"
| └── conventions.md # Coding standards, project structure requirements
│
├── validation/
│ ├── criteria.md # Tool requirements, user experience goals
│ └── test_scenarios.md # Usage scenarios tool should handle
│
└── answers/ # Valid responses to expected questions
├── deployment.md # "Web interface preferred, deployed on internal server"
├── data_storage.md # "Store in SQLite, one file per game session"
├── user_interaction.md # "Need to add/edit scores, see history, multiple games"
└── conventions.md # Coding standards, project structure requirements
└── validation/
├── criteria.md # Tool requirements, user experience goals
└── test_scenarios.md # Usage scenarios tool should handle
```
This structure ensures that:
@@ -177,7 +171,7 @@ In the criteria.md file, each criterion is defined as a level-1 header with seve
- A pass/fail thresholds
- Details about how the measurement should be performed
From the criteria fila and the iterations dir, a test report can be generated.
From the criteria file and the iterations dir, a test report can be generated.
This test report includes:
- The challenge name
- The commit id of the SIA repo when the test was run
@@ -224,7 +218,6 @@ report.json file for the "Time Series Analysis Challenge"
}
```
## Testing Framework
Testing self-improvement capabilities requires running SIA instances in various scenarios while monitoring their behavior and performance.