projects/fcb_looper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Updated audio_engine with better solution for sync offset

Browse Source
This commit is contained in:
Geens 2025-06-09 11:54:55 +02:00
parent 6ab1884553
commit 9b4fd1685c
1 changed files with 171 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

184
audio_engine.md
View File

@ -35,6 +35,66 @@ Track ownership is shared between threads after buffers are written.
This enables file operations without copying audio data,
ensuring commands and data transfers never block the critical audio processing path.
## Multi-Track Matrix Organization
The system implements a 5×5 matrix of tracks organized into columns and rows:
- **Columns (5)**: Share synchronized timing behavior. The first recording in a column establishes the beat count for all subsequent recordings in that column.
- **Rows (5)**: Individual tracks within each column that can be recorded and played independently.
- **Total Tracks (25)**: Each cell in the matrix can contain one audio loop with independent state and volume.
### Column Synchronization and Sync Offsets
When multiple tracks are recorded in the same column at different times, they must maintain their relative timing relationships. This is achieved through sync offsets that track where each recording started relative to the column's beat cycle.
#### Detailed Example: Understanding Sync Offsets
**Step 1: First Recording (Row 1)**
```
Column Beat: | 1 | 2 | 3 | 4 |
Row 1: | A | Bb| B | C |
```
- Buffer contains: `[A, Bb, B, C]`
- Sync offset: 0 (defines beat 1 for the column)
- Column beat count: 4
**Step 2: Second Recording (Row 2, starting at column beat 3)**
```
Column Beat: | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 |
Row 1: | A | Bb| B | C | A | Bb| B | C |
Row 2: | | | Dm|Ebm| Em| Fm| | |
^recording starts
```
- Row 2 buffer contains: `[Dm, Ebm, Em, Fm]` (recording order)
- Row 2 sync offset: 2 (buffer[0] corresponds to column beat 3)
**Step 3: Understanding Playback Mapping**
Row 2's logical relationship to column beats:
```
Column Beat: | 1 | 2 | 3 | 4 |
Row 2 Content: | Em| Fm| Dm|Ebm|
Buffer Index: | 2 | 3 | 0 | 1 |
```
**Key Insight**: When Row 2 plays alone, it should start with Em (the chord recorded during column beat 1), not Dm (the first recorded chord).
#### Synchronized Playback Scenarios
**Scenario 1: Row 2 starts alone (from stopped state)**
- Always starts at column beat 1 = Em
- Plays: Em → Fm → Dm → Ebm → (loop)
**Scenario 2: Row 1 joins while Row 2 plays Fm**
- Row 2 playing Fm = column beat 2
- Next beat = column beat 3
- Row 1 starts at beat 3 = B
- We hear: B (Row 1) + Dm (Row 2)
**Scenario 3: Both stopped, both start together**
- Both start at column beat 1
- Row 1 plays A, Row 2 plays Em simultaneously
## Audio Buffer Management and Ownership
Audio buffers use Arc-wrapped chunks to enable safe sharing between threads without sacrificing RT performance.
@ -53,9 +113,26 @@ After saving, it consolidates linked lists into single buffers.
This means only unsaved or actively-recording tracks have linked-list overhead,
while frequently-played loops benefit from optimized layout.
Arc destruction happens in the IO thread via the Tracks channel as a Delete message,
keeping deallocation costs out of the RT thread.
The RT thread's buffer pool operations involve only taking and returning Arc references, with minimal atomic operations.
### AudioData Enum for Sync Offset Support
To efficiently handle sync offsets while maintaining performance, audio data is represented by an enum that supports both unconsolidated (with offset) and consolidated (pre-aligned) formats:
- **Unconsolidated**: Contains Arc-wrapped linked list chunks with a sync offset. Used during and immediately after recording for instant playback capability.
- **Consolidated**: Contains a single optimized buffer where the sync offset has been applied during reordering. Used for long-term playback performance.
- **Empty**: No audio data present.
### Consolidation Process
**Workflow**:
1. Recording completes → Track has Unconsolidated data with sync offset
2. RT thread sends Arc-wrapped chunks + sync offset to IO thread
3. RT thread continues using Unconsolidated data (works fine, just slower)
4. IO thread consolidates in background (reorders audio to remove offset)
5. IO thread sends back Consolidated data via channel
6. RT thread swaps at start of next `process()` call
The consolidation process reorders the audio data so that the first sample corresponds to column beat 1, eliminating the need for offset calculations during playback.
This provides a significant performance benefit for frequently-played loops while maintaining the flexibility to play immediately after recording.
```
System State Hierarchy:
@ -69,20 +146,23 @@ GlobalState
│
├── columns[0]
│ ├── beats: usize
│ ├── column_position: usize (current sample position in column cycle)
│ └── tracks: TrackState[]
│ │
│ ├── tracks[0]
│ │ ├── current_state: Idle | Recording | Playing | Solo
│ │ ├── next_state: Idle | Recording | Playing | Solo
│ │ ├── volume: f32
│ │ └── audio: Option<Arc<AudioChunk>>
│ │ └── audio_data: AudioData
│ │ │
│ │ └── AudioChunk
│ │ ├── samples: [f32]
│ │ ├── sample_count: usize
│ │ └── next: Option<Arc<AudioChunk>>
│ │ │
│ │ └── AudioChunk (next in linked list)
│ │ ├── AudioData::Unconsolidated
│ │ │ ├── chunks: Arc<AudioChunk>
│ │ │ ├── sync_offset: usize
│ │ │ └── length: usize
│ │ │
│ │ └── AudioData::Consolidated
│ │ ├── buffer: Box<[f32]>
│ │ └── length: usize
│ │
│ ├── TrackState[1] ...
│ └── TrackState[2] ...
@ -109,7 +189,7 @@ Commands update the next beat state, which becomes active at the next beat bound
Audio processing follows these major steps:
- **Process updated audio buffers**: Store loaded and optimized audio data.
- **Process updated audio buffers**: Store loaded and optimized audio data from consolidation.
- **Process MIDI commands**: Update track volumes (immediate) and next beat states.
Last command wins if multiple commands arrive for the same track during a buffer.
@ -137,7 +217,7 @@ Click generation operates alongside the beat detection system,
triggering the pre-computed waveform when beats occur.
Click volume and enable/disable control operate through the standard command system for real-time adjustment.
### Xrun detection and recovery
## Xrun detection and recovery
Audio buffer underruns and overruns (xruns) disrupt the continuous flow of audio data and can break musical timing if not handled properly.
JACK provides xrun detection only through a callback mechanism that runs in a separate thread.
@ -157,4 +237,82 @@ Playback positions advance by the missed sample count, wrapping at loop boundari
If beats were missed during the xrun,
the normal beat detection in subsequent stages will handle state transitions appropriately,
since the missed time has already been accounted for in the position calculations.
since the missed time has already been accounted for in the position calculations.
## Appendix: Full recording example
Let's say I first record the chords A, Bb, B, C, one chord per beat.
Then I switch to row 2 and start recording minor chords Dm, Ebm, Em, Fm.
The switching Rows and record start take some time, so I get this.
```
I press the record button. Let's call the first beat after the press beat 1.
During this beat I play the A chord
Beat 1 2 3 4
Row 1 | A | | |
Row 2 | | | |
Next beat I play Bb.
Beat 1 2 3 4
Row 1 | A | Bb | |
Row 2 | | | |
During beat 3 I play B.
Beat 1 2 3 4
Row 1 | A | Bb | B |
Row 2 | | | |
Finally I play C and press the record button again.
Beat 1 2 3 4
Row 1 | A | Bb | B | C
Row 2 | | | |
This leaves us with the first track recorded and looping.
I now switch to Row 2 and when playback is in beat 2 I start recording again.
Actual recording only starts at beat 3 because of the beat sync.
I play the Dm chord here.
Beat 1 2 3 4
Row 1 | A | Bb | B | C
Row 2 | | | Dm |
Next I play the Ebm chord.
Beat 1 2 3 4
Row 1 | A | Bb | B | C
Row 2 | | | Dm | Ebm
The column wraps around and starts at beat 1.
During this beat I play the Em chord.
Beat 1 2 3 4
Row 1 | A | Bb | B | C
Row 2 | Em | | Dm | Ebm
Finally I play Fm.
I don't need to press the record button to stop because the track length is already set.
Beat 1 2 3 4
Row 1 | A | Bb | B | C
Row 2 | Em | Fm | Dm | Ebm
```
After a while I stop playback for both tracks.
A while later again, I start Row 2 playback.
It should start with the Em chord because it was recorded during beat 1.
It continues playing Fm.
If I start playback of Row 1 while Fm is playing, it will start on beat 3 and I should hear B and Dm.
Say I stop both tracks again and start recording Row 3.
I play the G chord first.
This G chord is recorded in beat 1, independent of other tracks, assuming no tracks in this column where playing.
But is will stop after 4 beats because the track length is set to 4 beats by the very first recording.