55 lines
1.8 KiB
Markdown
55 lines
1.8 KiB
Markdown
I'm working on the self improvement procedure.
|
|
I want to think about the testing Framework. How to run challenge tests and how to validate improvements.
|
|
To do so we need to know how to track progress. What metrics to collect and how to store these results.
|
|
To run a challenge a new SIA instance needs to be started and managed.
|
|
This instance needs to have its own environment and be able to communicate with the controlling instance.
|
|
Or we could have a managing instance and an instance under test, both managed by the main instance.
|
|
How do you suggest we handle all aspects of testing?
|
|
|
|
2. **Training Data Management**
|
|
- Organizing labeled iterations
|
|
- Storing validation notes
|
|
- Managing finetuning data
|
|
- Tracking training results
|
|
|
|
3. **Progress Tracking**
|
|
- What metrics to collect
|
|
- How to store results
|
|
- When to update progress
|
|
- How to resume interrupted work
|
|
|
|
3. **Testing Framework**
|
|
- How to run challenge tests
|
|
- What to measure and track
|
|
- How to validate improvements
|
|
- When to run regression tests
|
|
|
|
## Common Mistakes to Avoid
|
|
|
|
1. **Context Management**
|
|
- Keeping too much history in context
|
|
- Losing track of current state after interruptions
|
|
- Missing important user information
|
|
- Not cleaning up unused entries
|
|
|
|
2. **Resource Usage**
|
|
- Running expensive tests during user interaction
|
|
- Accumulating too much training data
|
|
- Not cleaning up after experiments
|
|
- Overloading system resources
|
|
|
|
3. **Improvement Strategy**
|
|
- Making changes without clear benefits
|
|
- Breaking existing capabilities
|
|
- Over-optimizing edge cases
|
|
- Not documenting why changes work
|
|
- Losing track of improvements after interruptions
|
|
|
|
4. **Training Data**
|
|
- Mixing good and bad examples
|
|
- Not validating training data
|
|
- Missing important context
|
|
- Duplicating existing data
|
|
|
|
|