From ae34c070a0652f8e48b57fefae10e859229fe4b7 Mon Sep 17 00:00:00 2001
From: Niels Geens <niels.geens@materialise.be>
Date: Fri, 24 Jan 2025 14:47:09 +0100
Subject: [PATCH] Added info about results and metrics for challenges

---
 procedures/self_improvement/reasoning.md | 72 ++++++++++++++++++++----
 1 file changed, 60 insertions(+), 12 deletions(-)

diff --git a/procedures/self_improvement/reasoning.md b/procedures/self_improvement/reasoning.md
index afcd9fd..1b85aea 100644
--- a/procedures/self_improvement/reasoning.md
+++ b/procedures/self_improvement/reasoning.md
@@ -131,7 +131,7 @@ challenges/
 │   ├── initial_context/
 │   │   ├── context.xml            # Initial context with SIA in conversation with the user
 │   │   ├── user_request.txt       # "Find unusual patterns in our sensor data"
-│   │   ├── available_paths.txt    # Lists /data/sensors/* and explains permissions
+│   │   ├── presentation.txt       # Info on how the results should be formatted for a perfect score
 │   │   └── data_generator.py      # Script to generate test data
 │   │
 │   ├── validation/
@@ -168,6 +168,63 @@ This structure ensures that:
 - Required resources are documented
 - SIA can get clarification when needed
 
+### Results and metrics
+
+In the criteria.md file, each criterion is defined as a level-1 header with several key pieces of information:
+- A clear description of what is being measured
+- The type and unit of the measurement
+- The valid and expected range of values
+- A pass/fail thresholds
+- Details about how the measurement should be performed
+
+From the criteria fila and the iterations dir, a test report can be generated.
+This test report includes:
+- The challenge name
+- The commit id of the SIA repo when the test was run
+- Timestamp of the first iteration
+- The overall pass/fail status
+- Context usage (average and maximum)
+- All metrics as described in the criteria.md file
+
+#### Example
+
+criteria.md file for the "Time Series Analysis Challenge"
+
+```markdown
+...
+# Presentation
+
+This metric indicates how clear the results are presented to the user.
+It looks at communication and formating of the report.
+
+The score starts with 0.
+If the discontinuity is found and the agent reports the index where it starts, the score is increased by 1.
+If the agent adds context info, e.g. a graph, a table, or explanation on how the result was obtained, the score is 2.
+If the agent requests how the data should be presented and follows instructions, the score is 3.
+
+The challenge fails if the formatting score is 0.
+
+Presentation instructions for the agent can be found in presentation.txt.
+...
+```
+
+report.json file for the "Time Series Analysis Challenge"
+
+```json
+{
+   "challenge": "Time Series Analysis",
+   "commit": "1234567890abcdef",
+   "timestamp": "2023-05-01T12:00:00Z",
+   "status": "pass",
+   "context_max": 74,
+   "context_avg": 56,
+   ...
+   "presentation": 2,
+   ...
+}
+```
+
+
 ## Testing Framework
 
 Testing self-improvement capabilities requires running SIA instances in various scenarios while monitoring their behavior and performance.
@@ -219,6 +276,7 @@ test_runs/
     challenge_name/
       iterations/                   # Complete iteration history
       io.log                        # External interaction log
+      report.json                   # Test report
 ```
 
 The io.log captures all external interactions:
@@ -371,14 +429,4 @@ This requires:
    - Balance improvement with responsiveness
    - Monitor system resource usage
    - Prevent training impact on user tasks
-   - Clean up old data regularly
-
-# TODO
-
-- Fix training config
-- Write training script
-- implement stdio + auto mode
-- Write setup script
-- Explain challenge report card
-- Document report card tool
-- Write report card tool
\ No newline at end of file
+   - Clean up old data regularly
\ No newline at end of file