docs: update documentation with recent improvements and fixes

- Add CHANGELOG.md to track all notable changes
- Update architecture.md with simplified files table schema
- Remove outdated file status endpoints from API docs
- Add "Recent Improvements & Optimizations" section covering:
  - Database schema simplification (removed status column)
  - localStorage persistence for filters and preferences
  - Real-time progress tracking improvements
  - Loading state fixes and code quality improvements
- Add "Key Features" section to main README highlighting:
  - Automated processing capabilities
  - Modern web interface features
  - Developer experience improvements
  - Data management optimizations

CHANGELOG.md

@@ -0,0 +1,54 @@
+# Changelog
+
+All notable changes to Watch Finished Turbo will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Added
+
+- localStorage persistence for filter selections, search terms, and sort preferences across Files and Tasks pages
+- Proper loading states to prevent "flash of empty content" on fresh page loads
+- Real-time progress tracking with visual progress bar and color-coded states
+- Comprehensive Recent Improvements section in architecture documentation
+- Key Features section in main README highlighting system capabilities
+
+### Changed
+
+- **BREAKING**: Simplified `files` table schema by removing `status` column
+  - Files table now only tracks observations: `dataset`, `input`, `output`, `date`
+  - Task processing states managed exclusively in `tasks` table
+  - Automatic database migration preserves all existing file records
+- Updated API endpoint from `/files/:dataset/status/:status` to `/files/:dataset`
+- Improved React Query cache updates for progress events (no more full refetches)
+- Enhanced HandBrake progress regex to handle space before `%` sign (e.g., "79.60 %")
+
+### Fixed
+
+- Dataset name extraction in WatcherService now correctly maps file paths to configured dataset keys
+- HandBrake progress tracking regex now matches format "XX.XX %" with space before percent
+- React key prop warning in TaskList component by using React.Fragment with proper keys
+- Empty state flashing by checking for undefined data during initial page loads
+- FileList loading state now accounts for multi-step query dependencies (datasets → files)
+
+### Performance
+
+- Reduced API calls through localStorage caching of user preferences
+- Direct cache updates for progress events eliminate unnecessary network requests
+- Optimistic UI updates provide immediate feedback with background synchronization
+- Prevented unnecessary re-renders through proper loading state management
+
+## [Previous Versions]
+
+See git commit history for changes prior to this changelog.
+
+### Notable Historical Changes
+
+- Multi-stage Docker build with dev and prod modes
+- Comprehensive dev container setup
+- Task queue retry logic with configurable settings
+- WebSocket real-time event system
+- Priority-based task processing
+- Dataset configuration management

README.md

@@ -65,6 +65,23 @@ See `.devcontainer/README.md` for detailed setup instructions.
 
 If you prefer local development:
 
+#### Prerequisites
+
+**Install pnpm** (if not already installed):
+
+```bash
+# Using npm (recommended)
+npm install -g pnpm
+
+# Using Homebrew (macOS)
+brew install pnpm
+
+# Using other package managers
+# See: https://pnpm.io/installation
+```
+
+#### Setup
+
 ```bash
 # Install dependencies
 pnpm install
@@ -179,3 +196,35 @@ graph TD
     style DATA fill:#fff3e0
     style DOCS fill:#fce4ec
 ```
+
+## Key Features
+
+### Automated Processing
+
+- **Intelligent File Watching**: Automatically detects new video files in configured directories
+- **Queue Management**: Priority-based task queue with configurable concurrency and retry logic
+- **HandBrake Integration**: High-quality video encoding with real-time progress tracking
+- **Dataset Organization**: Flexible configuration for different video categories (Movies, TV, Sports, etc.)
+
+### Modern Web Interface
+
+- **Real-time Updates**: WebSocket-powered live progress and status updates
+- **Persistent Preferences**: Filter selections and search terms saved via localStorage
+- **Responsive Design**: Clean, modern UI with dark mode support
+- **Efficient Loading**: Proper loading states prevent "flash of empty content"
+- **Advanced Filtering**: Multi-select datasets, status filters, and search across all fields
+
+### Developer Experience
+
+- **Hot Reload**: Instant feedback during development for both frontend and backend
+- **Type Safety**: Full TypeScript support across all applications
+- **Monorepo Structure**: Turborepo for efficient builds and shared packages
+- **Dev Container**: Consistent development environment with Docker
+- **Comprehensive Docs**: Detailed architecture diagrams and API documentation
+
+### Data Management
+
+- **Simplified Schema**: Clean separation between file observations and task processing
+- **Automatic Migration**: Database schema updates handled automatically
+- **Efficient Queries**: Optimized SQLite operations with proper indexing
+- **Cache Management**: React Query for intelligent data caching and updates

docs/architecture.md

@@ -168,9 +168,8 @@ flowchart TD
 
     T --> Y{Process Complete?}
     Y -->|No| T
-    Y -->|Yes| Z[Update Task Status: completed/failed]
-    Z --> AA[Update File Status]
-    AA --> BB[EventsGateway: emit Completion Events]
+    Y -->|Yes| Z[Update Task: completed/failed]
+    Z --> BB[EventsGateway: emit Completion Events]
     BB --> CC[UI Shows Final Status]
 
     DD[User Manual Requeue] --> EE[HTTP POST to Controller]
@@ -241,10 +240,8 @@ flowchart TD
     M --> N{Exit Code 0?}
     N -->|Yes| O[Update Task: completed]
     N -->|No| P[Update Task: failed]
-    O --> Q[Update File Status: success]
-    P --> R[Update File Status: error]
-    Q --> S[emit Completion Events]
-    R --> S
+    O --> S[emit Completion Events]
+    P --> S
     S --> D
 
     T[Manual Requeue] --> U[createTask() High Priority]
@@ -265,6 +262,7 @@ The TaskQueueService supports configurable processing parameters that control ho
 - **Processing Interval** (`processingInterval`): How often to check for new tasks in milliseconds (default: 5 seconds)
 
 **Queue Processing Behavior:**
+
 - In each processing cycle, up to `batchSize` pending tasks are retrieved from the database
 - Tasks are started for processing only if the number of currently active tasks is below `concurrency`
 - If `batchSize = 3` and `concurrency = 1`, the system will pull 3 tasks but only process 1 at a time
@@ -389,11 +387,7 @@ erDiagram
         string dataset
         string input
         string output
-        string status
         string date
-        string preset
-        integer progress
-        string error_message
     }
 
     tasks {
@@ -424,7 +418,7 @@ erDiagram
 
 - `GET /files` - List enabled datasets
 - `GET /files/all-datasets` - List all datasets
-- `GET /files/:dataset/status/:status` - Get files by status
+- `GET /files/:dataset` - Get all files for a dataset
 - `POST /files/:dataset/:file/requeue` - Requeue file for processing
 - `DELETE /files/:dataset/:file` - Delete file record
 
@@ -602,4 +596,53 @@ flowchart TD
     N --> Q[WebSocket Status]
 ```
 
+## Recent Improvements & Optimizations
+
+### Database Schema Simplification (January 2026)
+
+The `files` table schema was simplified by removing the `status` column:
+
+- **Before**: Files had status tracking (`discovered`, `success`, `error`) causing complexity in queries
+- **After**: Files table is now a simple observation log with just `dataset`, `input`, `output`, and `date`
+- **Benefit**: Eliminates confusion between file observations and task processing states
+- **Migration**: Existing databases automatically migrated while preserving all file records
+
+### UI/UX Enhancements (January 2026)
+
+**localStorage Persistence**:
+
+- Filter selections (datasets, statuses) persist across page reloads
+- Search terms maintained between sessions
+- Sort preferences saved per-page
+- Page size preferences remembered
+- Namespaced keys prevent conflicts: `fileList:*`, `taskList:*`
+
+**Loading State Improvements**:
+
+- Fixed "flash of empty content" on fresh page loads
+- Proper loading indicators during data fetching
+- Handles multi-step query dependencies (datasets → files)
+- Shows spinner until data is fully loaded, preventing wrong impression
+
+**Real-time Progress Tracking**:
+
+- HandBrake progress regex updated to handle space before `%` sign
+- Format: `"Encoding: task 1 of 1, 79.60 %"` now correctly parsed
+- React Query cache updates for smooth progress transitions
+- Visual progress bar with color-coded states
+- WebSocket events processed efficiently without full refetches
+
+**Code Quality**:
+
+- Fixed React key warnings in list rendering
+- Proper React.Fragment usage with keys in mapped components
+- Improved error boundary handling
+
+### Performance Optimizations
+
+- **Efficient Cache Updates**: Progress events update React Query cache directly instead of triggering full refetches
+- **Reduced API Calls**: localStorage caching reduces redundant settings requests
+- **Optimistic UI Updates**: Immediate feedback for user actions with background sync
+- **Proper Loading States**: Prevents unnecessary re-renders and empty state flashing
+
 This documentation provides a comprehensive overview of the Watch Finished Turbo system architecture, data flows, and operational procedures. The Mermaid.js diagrams illustrate the complex interactions between components, while the detailed explanations help developers understand how to work with and extend the system.