WIP self improvement procedure

procedures/self_improvement/reasoning_2.md

@@ -0,0 +1,54 @@
+I'm working on the self improvement procedure.
+I want to think about the testing Framework. How to run challenge tests and how to validate improvements.
+To do so we need to know how to track progress. What metrics to collect and how to store these results.
+To run a challenge a new SIA instance needs to be started and managed.
+This instance needs to have its own environment and be able to communicate with the controlling instance.
+Or we could have a managing instance and an instance under test, both managed by the main instance.
+How do you suggest we handle all aspects of testing?
+
+2. **Training Data Management**
+   - Organizing labeled iterations
+   - Storing validation notes
+   - Managing finetuning data
+   - Tracking training results
+
+3. **Progress Tracking**
+   - What metrics to collect
+   - How to store results
+   - When to update progress
+   - How to resume interrupted work
+
+3. **Testing Framework**
+   - How to run challenge tests
+   - What to measure and track
+   - How to validate improvements
+   - When to run regression tests
+
+## Common Mistakes to Avoid
+
+1. **Context Management**
+   - Keeping too much history in context
+   - Losing track of current state after interruptions
+   - Missing important user information
+   - Not cleaning up unused entries
+
+2. **Resource Usage**
+   - Running expensive tests during user interaction
+   - Accumulating too much training data
+   - Not cleaning up after experiments
+   - Overloading system resources
+
+3. **Improvement Strategy**
+   - Making changes without clear benefits
+   - Breaking existing capabilities
+   - Over-optimizing edge cases
+   - Not documenting why changes work
+   - Losing track of improvements after interruptions
+
+4. **Training Data**
+   - Mixing good and bad examples
+   - Not validating training data
+   - Missing important context
+   - Duplicating existing data
+
+