watch-finished-turbo/docs/architecture.md

Watch Finished Turbo - System Architecture & Flows

Overview

Watch Finished Turbo is a video processing system that automatically detects new video files in configured directories and provides a web interface for manual processing control. The system uses HandBrake for video encoding and provides real-time progress updates through a modern web UI.

System Architecture

graph TB
    subgraph "File System"
        FS[Video Files<br/>Directories]
    end

    subgraph "Backend Services (NestJS)"
        WS[WatcherService<br/>Chokidar]
        TQS[TaskQueueService<br/>Auto Processing]
        HS[HandbrakeService<br/>HandBrakeCLI]
        DS[DatasetsService<br/>Configuration]
        DBS[DbService<br/>SQLite]
        CS[ConfigService<br/>Settings]
        EG[EventsGateway<br/>Socket.io]
        MS[MaintenanceService<br/>Cleanup]
    end

    subgraph "Web Interface (Next.js)"
        WUI[Web UI<br/>React/TypeScript]
        WSOCK[WebSocket Client<br/>Socket.io]
    end

    subgraph "Database"
        DB[(SQLite<br/>Database)]
    end

    FS --> WS
    WS --> TQS
    TQS --> HS
    HS --> EG
    EG --> WSOCK
    WSOCK --> WUI

    WUI --> TQS
    TQS --> EG

    DS --> WS
    DS --> TQS
    DBS --> DB
    CS --> DB
    MS --> DB

    style FS fill:#e1f5fe
    style WUI fill:#f3e5f5
    style DB fill:#e8f5e8

Core Components

Backend Services

• WatcherService: Monitors file system changes using Chokidar
• HandbrakeService: Manages video encoding processes with HandBrakeCLI
• DatasetsService: Manages dataset configurations and paths
• DbService: Handles SQLite database operations for file tracking
• ConfigService: Manages application settings and configurations
• EventsGateway: WebSocket server for real-time event broadcasting
• MaintenanceService: Handles cleanup and maintenance operations

Web Interface

• React/Next.js Application: Modern web UI with TypeScript
• WebSocket Client: Real-time communication with backend
• File Management: Browse, filter, and manage video files
• Processing Control: Manual queue management and progress monitoring

File Processing Workflow

stateDiagram-v2
    [*] --> FileDetected: New video file added to<br/>watched directory
    FileDetected --> TaskCreated: Task automatically created<br/>with processing details
    TaskCreated --> Queued: Task added to processing queue<br/>(priority-based ordering)

    Queued --> Processing: Task picked up by<br/>TaskQueueService
    Processing --> Encoding: HandBrakeCLI process<br/>starts encoding
    Encoding --> ProgressUpdates: Real-time progress<br/>via WebSocket events

    ProgressUpdates --> Success: Encoding completed<br/>successfully
    ProgressUpdates --> Error: Encoding failed

    Success --> [*]: File ready for use
    Error --> Queued: Can be retried or<br/>manually requeued

    note right of FileDetected : Automatic detection
    note right of TaskCreated : No manual intervention needed
    note right of Processing : Background processing

Event Flow Architecture

sequenceDiagram
    participant FS as File System
    participant WS as WatcherService
    participant TQS as TaskQueueService
    participant EG as EventsGateway
    participant WSOCK as WebSocket Client
    participant WUI as Web UI
    participant CTRL as AppController
    participant HS as HandbrakeService

    FS->>WS: Video file added
    WS->>WS: Validate file & determine dataset
    WS->>TQS: createTask()
    TQS->>EG: emitFileUpdate() & emitTaskUpdate()
    EG->>WSOCK: fileUpdate & taskUpdate events
    WSOCK->>WUI: CustomEvent dispatch
    WUI->>WUI: UI updates in real-time

    TQS->>TQS: Check for pending tasks (every 5s)
    TQS->>HS: processWithHandbrake(taskId)
    HS->>HS: Spawn HandBrakeCLI process
    loop Progress Updates
        HS->>EG: emitTaskUpdate(progress)
        EG->>WSOCK: taskUpdate event
        WSOCK->>WUI: Real-time progress
    end

    HS->>TQS: Process completed
    TQS->>TQS: Update task & file status
    TQS->>EG: emitTaskUpdate() & emitFileUpdate()
    EG->>WSOCK: completion events

    WUI->>CTRL: POST /files/:dataset/:file/requeue (manual)
    CTRL->>TQS: createTask() with high priority
    TQS->>EG: emit events

## Data Flow

```mermaid
flowchart TD
    A[Video File Added to Directory] --> B[Chokidar Detects Change]
    B --> C[WatcherService.handleFileAdded()]
    C --> D{Is Video File?}
    D -->|No| E[Skip File]
    D -->|Yes| F[Determine Dataset & Preset]
    F --> G[TaskQueueService.createTask()]
    G --> H[Database: Insert Task]
    H --> I[EventsGateway: emitFileUpdate & emitTaskUpdate]
    I --> J[WebSocket Broadcast]
    J --> K[Web UI Receives Event]
    K --> L[React Query Invalidation]
    L --> M[UI Updates with New Task]

    N[TaskQueueService Polls DB] --> O{Tasks Pending?}
    O -->|No| N
    O -->|Yes| P[Pick Highest Priority Task]
    P --> Q[Update Task Status: processing]
    Q --> R[HandbrakeService.processWithHandbrake(taskId)]
    R --> S[Spawn HandBrakeCLI Process]
    S --> T[Progress Event Loop]
    T --> U[Database: Update Progress]
    U --> V[EventsGateway: emitTaskUpdate]
    V --> W[WebSocket: Real-time Progress]
    W --> X[UI Shows Live Progress]

    T --> Y{Process Complete?}
    Y -->|No| T
    Y -->|Yes| Z[Update Task: completed/failed]
    Z --> BB[EventsGateway: emit Completion Events]
    BB --> CC[UI Shows Final Status]

    DD[User Manual Requeue] --> EE[HTTP POST to Controller]
    EE --> FF[AppController.requeueFile()]
    FF --> GG[TaskQueueService.createTask() High Priority]
    GG --> H

    style A fill:#e3f2fd
    style DD fill:#f3e5f5
    style S fill:#e8f5e8
    style Z fill:#fff3e0

Service Interactions

WatcherService Flow

flowchart TD
    A[Start Watcher] --> B[Get Enabled Dataset Paths]
    B --> C[Initialize Chokidar]
    C --> D[Setup Event Handlers]

    D --> E[on 'add']
    D --> F[on 'change']
    D --> G[on 'unlink']
    D --> H[on 'error']

    E --> I[handleFileAdded()]
    I --> J{Is Video File?}
    J -->|No| K[Skip File]
    J -->|Yes| L[Determine Dataset]
    L --> M[Get Preset from Config]
    M --> N[Create Output Path]
    N --> O[TaskQueueService.createTask()]
    O --> P[emitFileUpdate Event]
    P --> Q[Broadcast via Socket.io]

    F --> R[Log File Changed]
    R --> S[emitFileUpdate Event]

    G --> T[Log File Removed]
    T --> U[emitFileUpdate Event]

    H --> V[Log Error]
    V --> W[emitWatcherUpdate Event]

TaskQueueService Flow

flowchart TD
    A[Service Starts] --> B[onModuleInit()]
    B --> C[Start Processing Interval]
    C --> D[Check for Pending Tasks Every 5s]

    D --> E[Query DB: Pending Tasks]
    E --> F{Tasks Found?}
    F -->|No| D
    F -->|Yes| G[Sort by Priority & Time]
    G --> H[Pick Next Task]
    H --> I[Update Task Status: processing]
    I --> J[Call HandbrakeService.processWithHandbrake(taskId)]
    J --> K[Monitor Process]
    K --> L{Process Running?}
    L -->|Yes| K
    L -->|No| M[Process Completed]
    M --> N{Exit Code 0?}
    N -->|Yes| O[Update Task: completed]
    N -->|No| P[Update Task: failed]
    O --> S[emit Completion Events]
    P --> S
    S --> D

    T[Manual Requeue] --> U[createTask() High Priority]
    U --> V[Insert into DB]
    V --> W[emit Task Events]
    W --> D

Advanced Queue Configuration

The TaskQueueService supports configurable processing parameters that control how tasks are managed and processed:

• Batch Size (batchSize): Number of tasks pulled from the queue for consideration in each processing cycle (default: 10). This determines how many pending tasks are evaluated for processing at once.
• Concurrency (concurrency): Maximum number of tasks that can be actively processing simultaneously (default: 1). This limits parallel execution to prevent system overload.
• Retry Logic: Automatic retry for failed tasks (default: enabled)
• Max Retries (maxRetries): Maximum retry attempts for failed tasks (default: 3). Total attempts = maxRetries + 1 (original attempt + retries).
• Retry Delay (retryDelay): Delay between retry attempts in milliseconds (default: 30 seconds)
• Processing Interval (processingInterval): How often to check for new tasks in milliseconds (default: 5 seconds)

Queue Processing Behavior:

• In each processing cycle, up to batchSize pending tasks are retrieved from the database
• Tasks are started for processing only if the number of currently active tasks is below concurrency
• If batchSize = 3 and concurrency = 1, the system will pull 3 tasks but only process 1 at a time
• Remaining tasks from the batch will be processed in subsequent cycles as active tasks complete

Queue settings are persisted in the database and can be updated via API or CLI.

Retry Logic Flow

flowchart TD
    A[Task Fails] --> B[Update Task: failed]
    B --> C[Check Retry Settings]
    C --> D{Retries Enabled?}
    D -->|No| E[Task Remains Failed]
    D -->|Yes| F[Check Retry Count]
    F --> G{retry_count < max_retries?}
    G -->|No| E
    G -->|Yes| H[Wait retry_delay]
    H --> I[Increment retry_count]
    I --> J[Reset Task: pending]
    J --> K[Requeue for Processing]
    K --> L[emit Retry Event]

Processing Flow

flowchart TD
    A[Task Started] --> B[HandbrakeService.processWithHandbrake()]
    B --> C[Spawn HandBrakeCLI Process]
    C --> D[Monitor stdout & stderr]
    D --> E[Parse Progress: "Encoding: task X of X, Y.Y%"]
    E --> F[Update Task Progress in DB]
    F --> G[emitTaskUpdate with Progress]
    G --> H[Broadcast via WebSocket]
    H --> I[UI Shows Real-time Progress]
    I --> J{Process Complete?}
    J -->|No| D
    J -->|Yes| K[Check Exit Code]
    K --> L{Exit Code 0?}
    L -->|Yes| M[emitTaskUpdate: completed]
    L -->|No| N[emitTaskUpdate: failed]
    M --> O[TaskQueueService Updates Status]
    N --> O
    O --> P[Final UI Update]

Configuration Management

flowchart TD
    A[Dataset Configuration] --> B[JSON Config Files]
    B --> C[DatasetsService]
    C --> D[Parse Configuration]
    D --> E[Extract Paths & Presets]
    E --> F[Validate Enabled Datasets]
    F --> G[Provide to WatcherService]

    H[Application Settings] --> I[SQLite Database]
    I --> J[ConfigService]
    J --> K[CRUD Operations]
    K --> L[Settings API Endpoints]

    M[Web UI Settings] --> N[SettingsCrud Component]
    N --> O[Form Validation]
    O --> P[API Calls to ConfigService]
    P --> Q[Real-time Updates]

User Interface Flows

File Management Flow

flowchart TD
    A[User Opens File List] --> B[Load Dataset List]
    B --> C[Load Files by Dataset]
    C --> D[Apply Filters & Search]
    D --> E[Display File Grid/List]
    E --> F[Real-time Updates via WebSocket]

    G[User Selects File] --> H[Show File Details]
    H --> I[Display Processing Status]
    I --> J[Show Action Buttons]

    K[User Clicks Requeue] --> L[Confirmation Dialog]
    L --> M[API Call to Requeue]
    M --> N[Show Processing Progress]
    N --> O[Update UI on Completion]

Settings Management Flow

flowchart TD
    A[User Opens Settings] --> B[Load Current Settings]
    B --> C[Display Settings Editor]
    C --> D[Validate Input Changes]
    D --> E[Save to Backend]
    E --> F[Update Watcher Configuration]
    F --> G[Restart Watcher if Needed]
    G --> H[Show Success/Error Feedback]

Database Schema

erDiagram
    settings {
        string key PK
        string value
    }

    datasets {
        string name PK
        string data
    }

    files {
        integer id PK
        string dataset
        string input
        string output
        string date
    }

    tasks {
        integer id PK
        string type
        string status
        integer progress
        string dataset
        string input
        string output
        string preset
        integer priority
        integer retry_count
        integer max_retries
        string error_message
        string created_at
        string updated_at
    }

    settings ||--o{ files : "configures"
    datasets ||--o{ files : "groups"
    files ||--o{ tasks : "tracked_by"

API Endpoints

File Management

• GET /files - List enabled datasets
• GET /files/all-datasets - List all datasets
• GET /files/:dataset - Get all files for a dataset
• POST /files/:dataset/:file/requeue - Requeue file for processing
• DELETE /files/:dataset/:file - Delete file record

Watcher Management

• GET /watcher/status - Get watcher status
• POST /watcher/start - Start file watcher
• POST /watcher/stop - Stop file watcher

Task Management

• GET /tasks - Get all tasks
• GET /tasks/queue/status - Get queue processing status
• GET /tasks/queue/settings - Get queue configuration settings
• POST /tasks/queue/settings - Update queue configuration settings
• GET /tasks/:id - Get specific task by ID
• DELETE /tasks/:id - Delete a task

Configuration

• GET /config/settings - Get all settings
• GET /config/settings/:key - Get specific setting
• POST /config/settings - Update settings
• GET /config/file/:name - Get dataset configuration

Statistics

• GET /files/stats/successful - Total successful files
• GET /files/stats/processed - Total processed files

Command Line Interface (CLI)

The system provides a comprehensive CLI for automation, monitoring, and management:

CLI Architecture

graph TB
    CLI[CLI Application<br/>Commander.js]
    API[HTTP Client<br/>REST API]
    WS[WebSocket Client<br/>Real-time Events]
    CONF[Configuration<br/>Environment]

    CLI --> API
    CLI --> WS
    CLI --> CONF

    API --> AS[AppService<br/>Backend]
    WS --> EG[EventsGateway<br/>WebSocket]

    style CLI fill:#e1f5fe
    style API fill:#f3e5f5
    style WS fill:#fff3e0

CLI Command Categories

• Task Management: task:list, task:get, task:delete, task:queue:*
• File Operations: list, file:get, file:set, file:remove
• Configuration: config:* commands for settings management
• Watcher Control: watcher:start, watcher:stop, watcher:status
• Maintenance: maintenance:cleanup, maintenance:purge, maintenance:prune
• HandBrake: handbrake:presets, handbrake:process

CLI Integration

The CLI enables:

• Automation Scripts: Batch operations and scheduled tasks
• Monitoring: Queue status, task progress, system health
• Configuration Management: Settings updates and validation
• Troubleshooting: Detailed task and file inspection
• Integration: Easy integration with external tools and scripts

Deployment Architecture

graph TB
    subgraph "Docker Environment"
        DC[Docker Compose]
        WEB[Web Container<br/>Next.js:3000]
        API[API Container<br/>NestJS:3001]
        DB[(SQLite Volume)]
    end

    subgraph "Host System"
        FS2[Host File System]
        HB[HandBrakeCLI]
    end

    DC --> WEB
    DC --> API
    DC --> DB

    WEB --> API
    API --> FS2
    API --> HB
    API --> DB

    style DC fill:#e8f5e8
    style FS2 fill:#e1f5fe
    style HB fill:#fff3e0

Error Handling & Recovery

flowchart TD
    A[Error Occurs] --> B{Error Type}
    B -->|File System| C[WatcherService Error]
    B -->|Processing| D[HandbrakeService Error]
    B -->|Database| E[DbService Error]
    B -->|WebSocket| F[EventsGateway Error]

    C --> G[Log Error]
    G --> H[Emit watcherUpdate Event]
    H --> I[UI Shows Error State]

    D --> J[Kill HandBrake Process]
    J --> K[Update File Status: 'error']
    K --> L[Emit taskUpdate Event]
    L --> M[UI Shows Error Details]

    E --> N[Log Database Error]
    N --> O[Return Error Response]
    O --> P[UI Shows Error Message]

    F --> Q[Attempt Reconnection]
    Q --> R{Reconnect Success?}
    R -->|Yes| S[Resume Normal Operation]
    R -->|No| T[Show Connection Error]

Performance Considerations

• File System Monitoring: Chokidar efficiently watches multiple directories
• Database Operations: SQLite provides fast local storage
• WebSocket Communication: Real-time updates without polling
• Process Management: HandBrake processes are properly cleaned up
• Memory Management: Large file lists are paginated in the UI

Security Considerations

• File System Access: Limited to configured dataset directories
• WebSocket Security: CORS configuration for allowed origins
• Input Validation: All API inputs are validated
• Process Isolation: HandBrake runs as separate system processes
• Database Security: SQLite files stored in data directory

Monitoring & Maintenance

flowchart TD
    A[Maintenance Tasks] --> B[File Cleanup]
    A --> C[Database Pruning]
    A --> D[Log Rotation]

    B --> E[Remove Deleted Records]
    B --> F[Clean Orphaned Files]

    C --> G[Purge Old Records]
    C --> H[Optimize Database]

    D --> I[Compress Old Logs]
    D --> J[Remove Expired Logs]

    K[Scheduled Tasks] --> L[Cron Jobs]
    L --> B
    L --> C
    L --> D

    M[Health Checks] --> N[API Endpoints]
    N --> O[Database Connectivity]
    N --> P[File System Access]
    N --> Q[WebSocket Status]

Recent Improvements & Optimizations

Database Schema Simplification (January 2026)

The files table schema was simplified by removing the status column:

• Before: Files had status tracking (discovered, success, error) causing complexity in queries
• After: Files table is now a simple observation log with just dataset, input, output, and date
• Benefit: Eliminates confusion between file observations and task processing states
• Migration: Existing databases automatically migrated while preserving all file records

UI/UX Enhancements (January 2026)

localStorage Persistence:

• Filter selections (datasets, statuses) persist across page reloads
• Search terms maintained between sessions
• Sort preferences saved per-page
• Page size preferences remembered
• Namespaced keys prevent conflicts: fileList:*, taskList:*

Loading State Improvements:

• Fixed "flash of empty content" on fresh page loads
• Proper loading indicators during data fetching
• Handles multi-step query dependencies (datasets → files)
• Shows spinner until data is fully loaded, preventing wrong impression

Real-time Progress Tracking:

• HandBrake progress regex updated to handle space before % sign
• Format: "Encoding: task 1 of 1, 79.60 %" now correctly parsed
• React Query cache updates for smooth progress transitions
• Visual progress bar with color-coded states
• WebSocket events processed efficiently without full refetches

Code Quality:

• Fixed React key warnings in list rendering
• Proper React.Fragment usage with keys in mapped components
• Improved error boundary handling

Performance Optimizations

• Efficient Cache Updates: Progress events update React Query cache directly instead of triggering full refetches
• Reduced API Calls: localStorage caching reduces redundant settings requests
• Optimistic UI Updates: Immediate feedback for user actions with background sync
• Proper Loading States: Prevents unnecessary re-renders and empty state flashing

This documentation provides a comprehensive overview of the Watch Finished Turbo system architecture, data flows, and operational procedures. The Mermaid.js diagrams illustrate the complex interactions between components, while the detailed explanations help developers understand how to work with and extend the system.