llm/SIA
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Improved self improvement example directory structure

Browse Source
This commit is contained in:
Niels Geens
2025-02-24 14:03:52 +01:00
parent e664c625b7
commit f635725064
1 changed files with 26 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

59
procedures/self_improvement/reasoning.md
View File

@@ -51,11 +51,11 @@ Each challenge consists of:
- Success criteria - Success criteria
- Resource constraints - Resource constraints
2. Initial Context 2. Starting State
- Minimal but complete starting state - Filesystem state
- Required user information - Starting system context
- System files and directories needed - Scripts for creating files
- Clear separation between essential context and noise - User info or documentation on how and what to communicate
3. Validation 3. Validation
- Precise success criteria - Precise success criteria
@@ -127,38 +127,32 @@ Challenges are organized in a consistent file structure:
``` ```
challenges/ challenges/
├── timeseries_analysis/ ├── timeseries_analysis/
│ ├── description.md # Core challenge description and how to prepare the environment │ ├── description.md # Core challenge description and how to prepare the environment
│ ├── initial_context/ │ ├── initial_context/
│ │ ├── context.xml # Initial context with SIA in conversation with the user │ │ ├── context.xml # Initial context with SIA in conversation with the user
│ │ ├── user_request.txt # "Find unusual patterns in our sensor data" │ │ ├── user_request.txt # "Find unusual patterns in our sensor data"
│ │ ├── presentation.txt # Info on how the results should be formatted for a perfect score │ │ ├── setup.sh # Runs data_generator.py and prepares test environment
│ │ └── data_generator.py # Script to generate test data │ │ └── data_generator.py # Script to generate test data
│ │ │ │
│ ├── validation/ │ └── validation/
│ │ ├── criteria.md # Expected findings, memory limits, performance targets │ ├── criteria.md # Expected findings, memory limits, performance targets
│ │ ├── setup.sh # Runs data_generator.py and prepares test environment │ ├── presentation.txt # Info on how the results should be formatted for a perfect score
│ │ └── known_patterns.md # Lists patterns inserted by generator for validation │ └── known_patterns.md # Lists patterns inserted by generator for validation
│ │
│ └── answers/ # Valid answers to questions SIA might ask
| └── data_format.txt # Explains CSV structure if asked
| |
└── score_tracker/ └── score_tracker/
├── description.md # Core challenge: create card game score tracker ├── description.md # Core challenge: create card game score tracker
├── initial_context/ ├── initial_context/
│ ├── context.xml # Initial context with SIA in conversation with the user │ ├── context.xml # Initial context with SIA in conversation with the user
│ ├── user_request.txt # "Need tool to track card game scores" │ ├── user_request.txt # "Need tool to track card game scores"
│ ├── tool_directory.txt # Shows where tools are stored and basic structure | └── answers/ # Valid responses to expected questions
│ └── example_tool/ # Simple tool showing expected structure/style | ├── deployment.md # "Web interface preferred, deployed on internal server"
| ├── data_storage.md # "Store in SQLite, one file per game session"
| ├── user_interaction.md # "Need to add/edit scores, see history, multiple games"
| └── conventions.md # Coding standards, project structure requirements
│ │
├── validation/ └── validation/
│ ├── criteria.md # Tool requirements, user experience goals ├── criteria.md # Tool requirements, user experience goals
│ └── test_scenarios.md # Usage scenarios tool should handle └── test_scenarios.md # Usage scenarios tool should handle
│
└── answers/ # Valid responses to expected questions
├── deployment.md # "Web interface preferred, deployed on internal server"
├── data_storage.md # "Store in SQLite, one file per game session"
├── user_interaction.md # "Need to add/edit scores, see history, multiple games"
└── conventions.md # Coding standards, project structure requirements
``` ```
This structure ensures that: This structure ensures that:
@@ -177,7 +171,7 @@ In the criteria.md file, each criterion is defined as a level-1 header with seve
- A pass/fail thresholds - A pass/fail thresholds
- Details about how the measurement should be performed - Details about how the measurement should be performed
From the criteria fila and the iterations dir, a test report can be generated. From the criteria file and the iterations dir, a test report can be generated.
This test report includes: This test report includes:
- The challenge name - The challenge name
- The commit id of the SIA repo when the test was run - The commit id of the SIA repo when the test was run
@@ -224,7 +218,6 @@ report.json file for the "Time Series Analysis Challenge"
} }
``` ```
## Testing Framework ## Testing Framework
Testing self-improvement capabilities requires running SIA instances in various scenarios while monitoring their behavior and performance. Testing self-improvement capabilities requires running SIA instances in various scenarios while monitoring their behavior and performance.