# Watch Finished Turbo - System Architecture & Flows

## Overview

Watch Finished Turbo is a video processing system that automatically detects new video files in configured directories and provides a web interface for manual processing control. The system uses HandBrake for video encoding and provides real-time progress updates through a modern web UI.

## System Architecture

```mermaid
graph TB
    subgraph "File System"
        FS[Video Files<br/>Directories]
    end

    subgraph "Backend Services (NestJS)"
        WS[WatcherService<br/>Chokidar]
        TQS[TaskQueueService<br/>Auto Processing]
        HS[HandbrakeService<br/>HandBrakeCLI]
        DS[DatasetsService<br/>Configuration]
        DBS[DbService<br/>SQLite]
        CS[ConfigService<br/>Settings]
        EG[EventsGateway<br/>Socket.io]
        MS[MaintenanceService<br/>Cleanup]
    end

    subgraph "Web Interface (Next.js)"
        WUI[Web UI<br/>React/TypeScript]
        WSOCK[WebSocket Client<br/>Socket.io]
    end

    subgraph "Database"
        DB[(SQLite<br/>Database)]
    end

    FS --> WS
    WS --> TQS
    TQS --> HS
    HS --> EG
    EG --> WSOCK
    WSOCK --> WUI

    WUI --> TQS
    TQS --> EG

    DS --> WS
    DS --> TQS
    DBS --> DB
    CS --> DB
    MS --> DB

    style FS fill:#e1f5fe
    style WUI fill:#f3e5f5
    style DB fill:#e8f5e8
```

## Core Components

### Backend Services

- **WatcherService**: Monitors file system changes using Chokidar
- **HandbrakeService**: Manages video encoding processes with HandBrakeCLI
- **DatasetsService**: Manages dataset configurations and paths
- **DbService**: Handles SQLite database operations for file tracking
- **ConfigService**: Manages application settings and configurations
- **EventsGateway**: WebSocket server for real-time event broadcasting
- **MaintenanceService**: Handles cleanup and maintenance operations

### Web Interface

- **React/Next.js Application**: Modern web UI with TypeScript
- **WebSocket Client**: Real-time communication with backend
- **File Management**: Browse, filter, and manage video files
- **Processing Control**: Manual queue management and progress monitoring

## File Processing Workflow

```mermaid
stateDiagram-v2
    [*] --> FileDetected: New video file added to<br/>watched directory
    FileDetected --> TaskCreated: Task automatically created<br/>with processing details
    TaskCreated --> Queued: Task added to processing queue<br/>(priority-based ordering)

    Queued --> Processing: Task picked up by<br/>TaskQueueService
    Processing --> Encoding: HandBrakeCLI process<br/>starts encoding
    Encoding --> ProgressUpdates: Real-time progress<br/>via WebSocket events

    ProgressUpdates --> Success: Encoding completed<br/>successfully
    ProgressUpdates --> Error: Encoding failed

    Success --> [*]: File ready for use
    Error --> Queued: Can be retried or<br/>manually requeued

    note right of FileDetected : Automatic detection
    note right of TaskCreated : No manual intervention needed
    note right of Processing : Background processing
```

## Event Flow Architecture

```mermaid
sequenceDiagram
    participant FS as File System
    participant WS as WatcherService
    participant TQS as TaskQueueService
    participant EG as EventsGateway
    participant WSOCK as WebSocket Client
    participant WUI as Web UI
    participant CTRL as AppController
    participant HS as HandbrakeService

    FS->>WS: Video file added
    WS->>WS: Validate file & determine dataset
    WS->>TQS: createTask()
    TQS->>EG: emitFileUpdate() & emitTaskUpdate()
    EG->>WSOCK: fileUpdate & taskUpdate events
    WSOCK->>WUI: CustomEvent dispatch
    WUI->>WUI: UI updates in real-time

    TQS->>TQS: Check for pending tasks (every 5s)
    TQS->>HS: processWithHandbrake(taskId)
    HS->>HS: Spawn HandBrakeCLI process
    loop Progress Updates
        HS->>EG: emitTaskUpdate(progress)
        EG->>WSOCK: taskUpdate event
        WSOCK->>WUI: Real-time progress
    end

    HS->>TQS: Process completed
    TQS->>TQS: Update task & file status
    TQS->>EG: emitTaskUpdate() & emitFileUpdate()
    EG->>WSOCK: completion events

    WUI->>CTRL: POST /files/:dataset/:file/requeue (manual)
    CTRL->>TQS: createTask() with high priority
    TQS->>EG: emit events
```

````

## Data Flow

```mermaid
flowchart TD
    A[Video File Added to Directory] --> B[Chokidar Detects Change]
    B --> C[WatcherService.handleFileAdded()]
    C --> D{Is Video File?}
    D -->|No| E[Skip File]
    D -->|Yes| F[Determine Dataset & Preset]
    F --> G[TaskQueueService.createTask()]
    G --> H[Database: Insert Task]
    H --> I[EventsGateway: emitFileUpdate & emitTaskUpdate]
    I --> J[WebSocket Broadcast]
    J --> K[Web UI Receives Event]
    K --> L[React Query Invalidation]
    L --> M[UI Updates with New Task]

    N[TaskQueueService Polls DB] --> O{Tasks Pending?}
    O -->|No| N
    O -->|Yes| P[Pick Highest Priority Task]
    P --> Q[Update Task Status: processing]
    Q --> R[HandbrakeService.processWithHandbrake(taskId)]
    R --> S[Spawn HandBrakeCLI Process]
    S --> T[Progress Event Loop]
    T --> U[Database: Update Progress]
    U --> V[EventsGateway: emitTaskUpdate]
    V --> W[WebSocket: Real-time Progress]
    W --> X[UI Shows Live Progress]

    T --> Y{Process Complete?}
    Y -->|No| T
    Y -->|Yes| Z[Update Task Status: completed/failed]
    Z --> AA[Update File Status]
    AA --> BB[EventsGateway: emit Completion Events]
    BB --> CC[UI Shows Final Status]

    DD[User Manual Requeue] --> EE[HTTP POST to Controller]
    EE --> FF[AppController.requeueFile()]
    FF --> GG[TaskQueueService.createTask() High Priority]
    GG --> H

    style A fill:#e3f2fd
    style DD fill:#f3e5f5
    style S fill:#e8f5e8
    style Z fill:#fff3e0
````

## Service Interactions

### WatcherService Flow

```mermaid
flowchart TD
    A[Start Watcher] --> B[Get Enabled Dataset Paths]
    B --> C[Initialize Chokidar]
    C --> D[Setup Event Handlers]

    D --> E[on 'add']
    D --> F[on 'change']
    D --> G[on 'unlink']
    D --> H[on 'error']

    E --> I[handleFileAdded()]
    I --> J{Is Video File?}
    J -->|No| K[Skip File]
    J -->|Yes| L[Determine Dataset]
    L --> M[Get Preset from Config]
    M --> N[Create Output Path]
    N --> O[TaskQueueService.createTask()]
    O --> P[emitFileUpdate Event]
    P --> Q[Broadcast via Socket.io]

    F --> R[Log File Changed]
    R --> S[emitFileUpdate Event]

    G --> T[Log File Removed]
    T --> U[emitFileUpdate Event]

    H --> V[Log Error]
    V --> W[emitWatcherUpdate Event]
```

### TaskQueueService Flow

```mermaid
flowchart TD
    A[Service Starts] --> B[onModuleInit()]
    B --> C[Start Processing Interval]
    C --> D[Check for Pending Tasks Every 5s]

    D --> E[Query DB: Pending Tasks]
    E --> F{Tasks Found?}
    F -->|No| D
    F -->|Yes| G[Sort by Priority & Time]
    G --> H[Pick Next Task]
    H --> I[Update Task Status: processing]
    I --> J[Call HandbrakeService.processWithHandbrake(taskId)]
    J --> K[Monitor Process]
    K --> L{Process Running?}
    L -->|Yes| K
    L -->|No| M[Process Completed]
    M --> N{Exit Code 0?}
    N -->|Yes| O[Update Task: completed]
    N -->|No| P[Update Task: failed]
    O --> Q[Update File Status: success]
    P --> R[Update File Status: error]
    Q --> S[emit Completion Events]
    R --> S
    S --> D

    T[Manual Requeue] --> U[createTask() High Priority]
    U --> V[Insert into DB]
    V --> W[emit Task Events]
    W --> D
```

### Advanced Queue Configuration

The TaskQueueService supports configurable processing parameters that control how tasks are managed and processed:

- **Batch Size** (`batchSize`): Number of tasks pulled from the queue for consideration in each processing cycle (default: 10). This determines how many pending tasks are evaluated for processing at once.
- **Concurrency** (`concurrency`): Maximum number of tasks that can be actively processing simultaneously (default: 1). This limits parallel execution to prevent system overload.
- **Retry Logic**: Automatic retry for failed tasks (default: enabled)
- **Max Retries** (`maxRetries`): Maximum retry attempts for failed tasks (default: 3). Total attempts = maxRetries + 1 (original attempt + retries).
- **Retry Delay** (`retryDelay`): Delay between retry attempts in milliseconds (default: 30 seconds)
- **Processing Interval** (`processingInterval`): How often to check for new tasks in milliseconds (default: 5 seconds)

**Queue Processing Behavior:**
- In each processing cycle, up to `batchSize` pending tasks are retrieved from the database
- Tasks are started for processing only if the number of currently active tasks is below `concurrency`
- If `batchSize = 3` and `concurrency = 1`, the system will pull 3 tasks but only process 1 at a time
- Remaining tasks from the batch will be processed in subsequent cycles as active tasks complete

Queue settings are persisted in the database and can be updated via API or CLI.

### Retry Logic Flow

```mermaid
flowchart TD
    A[Task Fails] --> B[Update Task: failed]
    B --> C[Check Retry Settings]
    C --> D{Retries Enabled?}
    D -->|No| E[Task Remains Failed]
    D -->|Yes| F[Check Retry Count]
    F --> G{retry_count < max_retries?}
    G -->|No| E
    G -->|Yes| H[Wait retry_delay]
    H --> I[Increment retry_count]
    I --> J[Reset Task: pending]
    J --> K[Requeue for Processing]
    K --> L[emit Retry Event]
```

### Processing Flow

```mermaid
flowchart TD
    A[Task Started] --> B[HandbrakeService.processWithHandbrake()]
    B --> C[Spawn HandBrakeCLI Process]
    C --> D[Monitor stdout & stderr]
    D --> E[Parse Progress: "Encoding: task X of X, Y.Y%"]
    E --> F[Update Task Progress in DB]
    F --> G[emitTaskUpdate with Progress]
    G --> H[Broadcast via WebSocket]
    H --> I[UI Shows Real-time Progress]
    I --> J{Process Complete?}
    J -->|No| D
    J -->|Yes| K[Check Exit Code]
    K --> L{Exit Code 0?}
    L -->|Yes| M[emitTaskUpdate: completed]
    L -->|No| N[emitTaskUpdate: failed]
    M --> O[TaskQueueService Updates Status]
    N --> O
    O --> P[Final UI Update]
```

## Configuration Management

```mermaid
flowchart TD
    A[Dataset Configuration] --> B[JSON Config Files]
    B --> C[DatasetsService]
    C --> D[Parse Configuration]
    D --> E[Extract Paths & Presets]
    E --> F[Validate Enabled Datasets]
    F --> G[Provide to WatcherService]

    H[Application Settings] --> I[SQLite Database]
    I --> J[ConfigService]
    J --> K[CRUD Operations]
    K --> L[Settings API Endpoints]

    M[Web UI Settings] --> N[SettingsCrud Component]
    N --> O[Form Validation]
    O --> P[API Calls to ConfigService]
    P --> Q[Real-time Updates]
```

## User Interface Flows

### File Management Flow

```mermaid
flowchart TD
    A[User Opens File List] --> B[Load Dataset List]
    B --> C[Load Files by Dataset]
    C --> D[Apply Filters & Search]
    D --> E[Display File Grid/List]
    E --> F[Real-time Updates via WebSocket]

    G[User Selects File] --> H[Show File Details]
    H --> I[Display Processing Status]
    I --> J[Show Action Buttons]

    K[User Clicks Requeue] --> L[Confirmation Dialog]
    L --> M[API Call to Requeue]
    M --> N[Show Processing Progress]
    N --> O[Update UI on Completion]
```

### Settings Management Flow

```mermaid
flowchart TD
    A[User Opens Settings] --> B[Load Current Settings]
    B --> C[Display Settings Editor]
    C --> D[Validate Input Changes]
    D --> E[Save to Backend]
    E --> F[Update Watcher Configuration]
    F --> G[Restart Watcher if Needed]
    G --> H[Show Success/Error Feedback]
```

## Database Schema

```mermaid
erDiagram
    settings {
        string key PK
        string value
    }

    datasets {
        string name PK
        string data
    }

    files {
        integer id PK
        string dataset
        string input
        string output
        string status
        string date
        string preset
        integer progress
        string error_message
    }

    tasks {
        integer id PK
        string type
        string status
        integer progress
        string dataset
        string input
        string output
        string preset
        integer priority
        integer retry_count
        integer max_retries
        string error_message
        string created_at
        string updated_at
    }

    settings ||--o{ files : "configures"
    datasets ||--o{ files : "groups"
    files ||--o{ tasks : "tracked_by"
```

## API Endpoints

### File Management

- `GET /files` - List enabled datasets
- `GET /files/all-datasets` - List all datasets
- `GET /files/:dataset/status/:status` - Get files by status
- `POST /files/:dataset/:file/requeue` - Requeue file for processing
- `DELETE /files/:dataset/:file` - Delete file record

### Watcher Management

- `GET /watcher/status` - Get watcher status
- `POST /watcher/start` - Start file watcher
- `POST /watcher/stop` - Stop file watcher

### Task Management

- `GET /tasks` - Get all tasks
- `GET /tasks/queue/status` - Get queue processing status
- `GET /tasks/queue/settings` - Get queue configuration settings
- `POST /tasks/queue/settings` - Update queue configuration settings
- `GET /tasks/:id` - Get specific task by ID
- `DELETE /tasks/:id` - Delete a task

### Configuration

- `GET /config/settings` - Get all settings
- `GET /config/settings/:key` - Get specific setting
- `POST /config/settings` - Update settings
- `GET /config/file/:name` - Get dataset configuration

### Statistics

- `GET /files/stats/successful` - Total successful files
- `GET /files/stats/processed` - Total processed files

## Command Line Interface (CLI)

The system provides a comprehensive CLI for automation, monitoring, and management:

### CLI Architecture

```mermaid
graph TB
    CLI[CLI Application<br/>Commander.js]
    API[HTTP Client<br/>REST API]
    WS[WebSocket Client<br/>Real-time Events]
    CONF[Configuration<br/>Environment]

    CLI --> API
    CLI --> WS
    CLI --> CONF

    API --> AS[AppService<br/>Backend]
    WS --> EG[EventsGateway<br/>WebSocket]

    style CLI fill:#e1f5fe
    style API fill:#f3e5f5
    style WS fill:#fff3e0
```

### CLI Command Categories

- **Task Management**: `task:list`, `task:get`, `task:delete`, `task:queue:*`
- **File Operations**: `list`, `file:get`, `file:set`, `file:remove`
- **Configuration**: `config:*` commands for settings management
- **Watcher Control**: `watcher:start`, `watcher:stop`, `watcher:status`
- **Maintenance**: `maintenance:cleanup`, `maintenance:purge`, `maintenance:prune`
- **HandBrake**: `handbrake:presets`, `handbrake:process`

### CLI Integration

The CLI enables:

- **Automation Scripts**: Batch operations and scheduled tasks
- **Monitoring**: Queue status, task progress, system health
- **Configuration Management**: Settings updates and validation
- **Troubleshooting**: Detailed task and file inspection
- **Integration**: Easy integration with external tools and scripts

## Deployment Architecture

```mermaid
graph TB
    subgraph "Docker Environment"
        DC[Docker Compose]
        WEB[Web Container<br/>Next.js:3000]
        API[API Container<br/>NestJS:3001]
        DB[(SQLite Volume)]
    end

    subgraph "Host System"
        FS2[Host File System]
        HB[HandBrakeCLI]
    end

    DC --> WEB
    DC --> API
    DC --> DB

    WEB --> API
    API --> FS2
    API --> HB
    API --> DB

    style DC fill:#e8f5e8
    style FS2 fill:#e1f5fe
    style HB fill:#fff3e0
```

## Error Handling & Recovery

```mermaid
flowchart TD
    A[Error Occurs] --> B{Error Type}
    B -->|File System| C[WatcherService Error]
    B -->|Processing| D[HandbrakeService Error]
    B -->|Database| E[DbService Error]
    B -->|WebSocket| F[EventsGateway Error]

    C --> G[Log Error]
    G --> H[Emit watcherUpdate Event]
    H --> I[UI Shows Error State]

    D --> J[Kill HandBrake Process]
    J --> K[Update File Status: 'error']
    K --> L[Emit taskUpdate Event]
    L --> M[UI Shows Error Details]

    E --> N[Log Database Error]
    N --> O[Return Error Response]
    O --> P[UI Shows Error Message]

    F --> Q[Attempt Reconnection]
    Q --> R{Reconnect Success?}
    R -->|Yes| S[Resume Normal Operation]
    R -->|No| T[Show Connection Error]
```

## Performance Considerations

- **File System Monitoring**: Chokidar efficiently watches multiple directories
- **Database Operations**: SQLite provides fast local storage
- **WebSocket Communication**: Real-time updates without polling
- **Process Management**: HandBrake processes are properly cleaned up
- **Memory Management**: Large file lists are paginated in the UI

## Security Considerations

- **File System Access**: Limited to configured dataset directories
- **WebSocket Security**: CORS configuration for allowed origins
- **Input Validation**: All API inputs are validated
- **Process Isolation**: HandBrake runs as separate system processes
- **Database Security**: SQLite files stored in data directory

## Monitoring & Maintenance

```mermaid
flowchart TD
    A[Maintenance Tasks] --> B[File Cleanup]
    A --> C[Database Pruning]
    A --> D[Log Rotation]

    B --> E[Remove Deleted Records]
    B --> F[Clean Orphaned Files]

    C --> G[Purge Old Records]
    C --> H[Optimize Database]

    D --> I[Compress Old Logs]
    D --> J[Remove Expired Logs]

    K[Scheduled Tasks] --> L[Cron Jobs]
    L --> B
    L --> C
    L --> D

    M[Health Checks] --> N[API Endpoints]
    N --> O[Database Connectivity]
    N --> P[File System Access]
    N --> Q[WebSocket Status]
```

This documentation provides a comprehensive overview of the Watch Finished Turbo system architecture, data flows, and operational procedures. The Mermaid.js diagrams illustrate the complex interactions between components, while the detailed explanations help developers understand how to work with and extend the system.