From fb5dcbeec2bf00fdf398e3b3d88bf506d41774a6 Mon Sep 17 00:00:00 2001 From: Benjamin Slingo Date: Mon, 8 Sep 2025 13:06:31 -0400 Subject: [PATCH] Add comprehensive documentation with event-driven architecture details --- README.md | 75 ++++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 57 insertions(+), 18 deletions(-) diff --git a/README.md b/README.md index 9092aa2..c134d77 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,11 @@ # Livestream Archiver -A Rust application that monitors a directory for livestream recordings, processes them with hardware-accelerated transcoding, and syncs them to a remote server. +A Rust application that monitors a directory for livestream recordings, processes them with hardware-accelerated transcoding, and syncs them to a remote server using an event-driven architecture. ## Features -- **Automatic Detection**: Monitors a configured directory for new MP4 files +- **Event-Driven Architecture**: Uses filesystem events for instant file detection +- **Automatic Detection**: Monitors a configured directory for new MP4 files - **Smart Processing**: Waits for files to be completely written before processing - **Remote Sync**: Uses rsync to sync files to a remote server immediately after detection - **Hardware Acceleration**: Converts files to AV1 using Intel QSV hardware acceleration @@ -12,6 +13,7 @@ A Rust application that monitors a directory for livestream recordings, processe - **Metadata Generation**: Creates NFO files for media servers - **Retry Logic**: Caches and retries failed sync operations - **Service Integration**: Runs as a systemd service with automatic restart +- **Environment Configuration**: Uses .env file for easy configuration management ## Prerequisites @@ -39,9 +41,10 @@ cargo build --release 1. Copy and customize the example files: ```bash +cp .env.example .env cp livestream-archiver.service.example livestream-archiver.service cp deploy.sh.example deploy.sh -# Edit both files with your specific configuration +# Edit all files with your specific configuration ``` 2. Run the deployment: @@ -51,12 +54,21 @@ cp deploy.sh.example deploy.sh ## Configuration -### Environment Variables +### Environment Variables (.env file) -- `INPUT_DIR`: Directory to monitor for new MP4 files (default: `/home/user/livestreams`) -- `OUTPUT_DIR`: Local output directory for processed files (default: `/media/archive/livestreams`) +Create a `.env` file in the project root with the following variables: + +- `INPUT_DIR`: Directory to monitor for new MP4 files +- `OUTPUT_DIR`: Local output directory for processed files - `PC_SYNC_TARGET`: Remote sync destination (e.g., `user@server:/path/to/destination/`) +Example `.env` file: +```bash +INPUT_DIR=/home/user/livestreams +OUTPUT_DIR=/media/archive/livestreams +PC_SYNC_TARGET=user@server:/remote/path/to/livestreams/ +``` + ### Service Configuration 1. Copy the example service file: @@ -67,7 +79,7 @@ cp livestream-archiver.service.example livestream-archiver.service 2. Edit the service file with your configuration: - Update `User` and `Group` - Set correct paths for `WorkingDirectory` and `ExecStart` -- Configure the `PC_SYNC_TARGET` environment variable +- Ensure the service can access your `.env` file 3. Install and start the service: ```bash @@ -82,7 +94,7 @@ sudo systemctl start livestream-archiver ### Running Manually ```bash -# Run directly +# Run directly (uses .env file automatically) cargo run # Or run the compiled binary @@ -104,17 +116,19 @@ sudo systemctl restart livestream-archiver ## How It Works -1. **Monitoring**: Continuously watches the input directory for new MP4 files -2. **Validation**: Waits for files to be completely written (stable size) -3. **Sync**: Immediately syncs new files to the remote server via rsync -4. **Processing**: Converts to AV1 format using hardware acceleration -5. **Organization**: Moves processed files to date-based directories -6. **Metadata**: Creates NFO files for media server compatibility +The application uses an event-driven architecture for efficient file monitoring: + +1. **Event Monitoring**: Uses filesystem events to detect new files instantly (no polling) +2. **File Validation**: Waits for files to be completely written before processing +3. **Immediate Sync**: Syncs new files to remote server via rsync as soon as they're detected +4. **Hardware Processing**: Converts to AV1 format using Intel QSV hardware acceleration +5. **Smart Organization**: Moves processed files to date-based directories +6. **Metadata Generation**: Creates NFO files for media server compatibility 7. **Cleanup**: Optionally removes original files after successful processing ## Directory Structure -Output files are organized by date: +Output files are organized by date for easy browsing: ``` / ├── 2024/ @@ -132,18 +146,26 @@ Output files are organized by date: ### Service not starting - Check logs: `journalctl -u livestream-archiver -f` -- Verify all paths exist and have proper permissions -- Ensure ffmpeg has QSV support: `ffmpeg -hwaccels | grep qsv` +- Verify `.env` file exists and has correct permissions +- Ensure all paths exist and have proper permissions +- Verify ffmpeg has QSV support: `ffmpeg -hwaccels | grep qsv` ### Sync failures - Verify SSH key authentication to remote server - Check network connectivity +- Test rsync manually: `rsync -av /local/path/ user@server:/remote/path/` - Review cached sync attempts in `~/.cache/livestream-archiver/` ### Processing issues - Ensure Intel QSV drivers are installed -- Check available disk space +- Check available disk space in both input and output directories - Verify file permissions in input/output directories +- Test hardware acceleration: `ffmpeg -hwaccel qsv -i input.mp4 -c:v av1_qsv output.mp4` + +### Files not detected +- Ensure the input directory exists and is accessible +- Check that the application has read permissions on the input directory +- Verify filesystem events are working (may not work on some network filesystems) ## Development @@ -157,6 +179,23 @@ cargo test cargo build ``` +### Development Environment +```bash +# Copy example env for development +cp .env.example .env +# Edit .env with your development paths + +# Run with debug output +RUST_LOG=debug cargo run +``` + +## Performance Notes + +- Uses filesystem events instead of polling for better performance +- Hardware acceleration significantly reduces CPU usage during transcoding +- Processes files as they arrive rather than batch processing +- Minimal memory footprint due to streaming processing approach + ## License This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.