From ce4b30b9a9347952bf1f98a87f699784829a0c6e Mon Sep 17 00:00:00 2001 From: Benjamin Slingo Date: Mon, 8 Sep 2025 12:58:36 -0400 Subject: [PATCH] Update README.md --- README.md | 303 +++++++++++++++++++++++++++++++++++++----------------- 1 file changed, 211 insertions(+), 92 deletions(-) diff --git a/README.md b/README.md index 9092aa2..cbf1aa6 100644 --- a/README.md +++ b/README.md @@ -1,24 +1,25 @@ # Livestream Archiver -A Rust application that monitors a directory for livestream recordings, processes them with hardware-accelerated transcoding, and syncs them to a remote server. +An event-driven Rust service that automatically detects livestream recordings, converts them for efficient storage, and syncs originals to PC for editing. Uses the shared `video_processing` crate for optimal performance. ## Features -- **Automatic Detection**: Monitors a configured directory for new MP4 files -- **Smart Processing**: Waits for files to be completely written before processing -- **Remote Sync**: Uses rsync to sync files to a remote server immediately after detection -- **Hardware Acceleration**: Converts files to AV1 using Intel QSV hardware acceleration -- **Organized Storage**: Creates date-based directory structure (Jellyfin-compatible) -- **Metadata Generation**: Creates NFO files for media servers -- **Retry Logic**: Caches and retries failed sync operations -- **Service Integration**: Runs as a systemd service with automatic restart +- **Event-driven file detection** - No polling, instant response to new files +- **PC sync for editing** - Syncs original files to Mac/PC for post-production +- **AV1 video conversion** with Intel QSV hardware acceleration +- **Smart program naming** - Detects Divine Worship vs Afternoon Program +- **File stability checking** - Ensures files are fully written before processing +- **Date-based organization** - Jellyfin-compatible directory structure +- **NFO file generation** - Complete media server metadata +- **Efficient resource usage** - Only processes when files appear ## Prerequisites -- Rust toolchain (1.70 or later) +- Rust toolchain (latest stable) - `ffmpeg` with Intel QSV support - `rsync` for file synchronization -- SSH key authentication for passwordless rsync (if using remote sync) +- SSH key authentication for passwordless rsync to PC +- Intel Media SDK (for QSV acceleration) ## Installation @@ -35,115 +36,227 @@ cargo build --release # Binary will be at target/release/livestream_archiver ``` -### Quick Deploy - -1. Copy and customize the example files: -```bash -cp livestream-archiver.service.example livestream-archiver.service -cp deploy.sh.example deploy.sh -# Edit both files with your specific configuration -``` - -2. Run the deployment: -```bash -./deploy.sh -``` - ## Configuration -### Environment Variables +Create a `.env` file based on `.env.example`: -- `INPUT_DIR`: Directory to monitor for new MP4 files (default: `/home/user/livestreams`) -- `OUTPUT_DIR`: Local output directory for processed files (default: `/media/archive/livestreams`) -- `PC_SYNC_TARGET`: Remote sync destination (e.g., `user@server:/path/to/destination/`) - -### Service Configuration - -1. Copy the example service file: ```bash -cp livestream-archiver.service.example livestream-archiver.service +cp .env.example .env +nano .env ``` -2. Edit the service file with your configuration: -- Update `User` and `Group` -- Set correct paths for `WorkingDirectory` and `ExecStart` -- Configure the `PC_SYNC_TARGET` environment variable +### Required Environment Variables -3. Install and start the service: ```bash -sudo cp livestream-archiver.service /etc/systemd/system/ -sudo systemctl daemon-reload +# Directory paths (REQUIRED) +INPUT_DIR=/home/rockvilleav/.rtsda/livestreams +OUTPUT_DIR=/media/archive/jellyfin/livestreams + +# PC sync target for editing (REQUIRED) +PC_SYNC_TARGET=user@macbook.local:~/rtsda/livestreams/ + +# Program naming +DIVINE_WORSHIP_NAME=Divine Worship Service - RTSDA +AFTERNOON_PROGRAM_NAME=Afternoon Program - RTSDA + +# FFmpeg Configuration +FFMPEG_BINARY=ffmpeg +VIDEO_CODEC=av1_qsv # AV1 with Intel QSV +HW_ACCEL=qsv +HW_DEVICE=qsv=hw +FFMPEG_PRESET=4 +VIDEO_BITRATE=6M +MAX_BITRATE=12M +BUFFER_SIZE=24M +AUDIO_CODEC=copy # Preserve original audio +AUDIO_BITRATE=192k + +# File Stability Settings +STABILITY_CHECK_INTERVAL=2 # seconds between stability checks +STABILITY_REQUIRED_CHECKS=15 # consecutive stable checks required +MAX_STABILITY_WAIT_HOURS=4 # maximum time to wait for stability + +# Directory Settings +CREATE_YEAR_MONTH_DIRS=true # organize by date +PRESERVE_ORIGINAL_FILES=true # keep originals after conversion + +# NFO Settings +SHOW_TITLE=LiveStreams +CREATE_NFO_FILES=true # or false to disable +``` + +## Expected File Format + +Livestream files should follow OBS naming convention: +``` +YYYY-MM-DD_HH-MM-SS.mp4 +``` + +Examples: +- `2024-12-25_10-30-00.mp4` → Divine Worship Service (Christmas 2024) +- `2024-12-25_14-00-00.mp4` → Afternoon Program (Christmas 2024) + +The service automatically detects service type based on file timing and existing files. + +## Running as a System Service + +Create a systemd service file at `/etc/systemd/system/livestream-archiver.service`: + +```ini +[Unit] +Description=Livestream Archive Service +After=network.target media-archive.mount + +[Service] +Type=simple +User=rockvilleav +Group=rockvilleav +ExecStart=/usr/local/bin/livestream_archiver +WorkingDirectory=/home/rockvilleav/livestream-archiver + +# Load environment variables from .env file +EnvironmentFile=/home/rockvilleav/livestream-archiver/.env + +# Environment variables for home directory access +Environment=HOME=/home/rockvilleav +Environment=USER=rockvilleav +Environment=XDG_RUNTIME_DIR=/run/user/1000 + +# Intel Media Stack environment +Environment=LIBVA_DRIVER_NAME=iHD +Environment=LIBVA_DRIVERS_PATH=/opt/intel/media/lib64 +Environment=LD_LIBRARY_PATH=/opt/intel/media/lib64 +Environment=LIBVA_DRIVER_PATH=/opt/intel/media/lib64 +Environment=DISPLAY=:0 + +# Restart settings +Restart=always +RestartSec=10 + +# Make sure service has access to needed directories +RequiresMountsFor=/media/archive + +[Install] +WantedBy=multi-user.target +``` + +Enable and start the service: +```bash sudo systemctl enable livestream-archiver sudo systemctl start livestream-archiver -``` - -## Usage - -### Running Manually - -```bash -# Run directly -cargo run - -# Or run the compiled binary -./target/release/livestream_archiver -``` - -### Running as a Service - -```bash -# Check service status sudo systemctl status livestream-archiver +``` -# View logs +View logs: +```bash sudo journalctl -u livestream-archiver -f - -# Restart service -sudo systemctl restart livestream-archiver ``` ## How It Works -1. **Monitoring**: Continuously watches the input directory for new MP4 files -2. **Validation**: Waits for files to be completely written (stable size) -3. **Sync**: Immediately syncs new files to the remote server via rsync -4. **Processing**: Converts to AV1 format using hardware acceleration -5. **Organization**: Moves processed files to date-based directories -6. **Metadata**: Creates NFO files for media server compatibility -7. **Cleanup**: Optionally removes original files after successful processing +1. **Event Detection**: Uses `inotify` to instantly detect new MP4 files (no polling!) +2. **File Stability**: Waits for OBS to finish writing the file completely +3. **PC Sync**: Immediately syncs the **original** file to Mac/PC for editing via rsync +4. **AV1 Conversion**: Converts to efficient AV1 format for long-term storage +5. **Smart Naming**: Determines Divine Worship vs Afternoon Program automatically +6. **Organization**: Places files in date-based directories for Jellyfin +7. **NFO Generation**: Creates metadata files for media server compatibility +8. **Preservation**: Keeps original files on server for backup ## Directory Structure Output files are organized by date: ``` -/ +/media/archive/jellyfin/livestreams/ ├── 2024/ -│ ├── 01-January/ -│ │ ├── Recording - January 01 2024.mp4 -│ │ ├── Recording - January 01 2024.nfo +│ ├── 12-December/ +│ │ ├── Divine Worship Service - RTSDA (December 25 2024).mp4 +│ │ ├── Divine Worship Service - RTSDA (December 25 2024).nfo +│ │ ├── Afternoon Program - RTSDA (December 25 2024).mp4 +│ │ ├── Afternoon Program - RTSDA (December 25 2024).nfo │ │ └── ... -│ └── 02-February/ -│ └── ... +│ └── 01-January/ └── 2025/ - └── ... ``` +PC sync creates: +``` +~/rtsda/livestreams/ +├── 2024-12-25_10-30-00.mp4 # Original for editing +├── 2024-12-25_14-00-00.mp4 # Original for editing +└── ... +``` + +## Architecture + +This service uses a clean, event-driven architecture: + +- **Event Detection**: Uses Linux `inotify` for instant file detection (no polling) +- **File Stability**: Proper completion detection before processing +- **Shared Logic**: Uses `video_processing` crate for common functionality +- **Dual Output**: Converted files for storage + original sync for editing +- **Resource Efficient**: Only uses resources when files actually appear + +## Video Conversion Details + +Default settings (configurable via .env): +- **Video Codec**: AV1 with Intel QSV hardware acceleration +- **Audio Codec**: Copy (preserves original quality) +- **Hardware Acceleration**: Intel Quick Sync Video +- **Video Bitrate**: 6Mbps (efficient for streaming) +- **Max Bitrate**: 12Mbps (for complex scenes) + ## Troubleshooting -### Service not starting -- Check logs: `journalctl -u livestream-archiver -f` -- Verify all paths exist and have proper permissions -- Ensure ffmpeg has QSV support: `ffmpeg -hwaccels | grep qsv` +### Service fails to start +```bash +# Check if .env file exists and has correct paths +ls -la /home/rockvilleav/livestream-archiver/.env -### Sync failures -- Verify SSH key authentication to remote server -- Check network connectivity -- Review cached sync attempts in `~/.cache/livestream-archiver/` +# Verify directories exist and are accessible +ls -la /home/rockvilleav/.rtsda/livestreams +ls -la /media/archive/jellyfin/livestreams -### Processing issues -- Ensure Intel QSV drivers are installed -- Check available disk space -- Verify file permissions in input/output directories +# Check systemd service file +sudo systemctl cat livestream-archiver +``` + +### PC sync failures +```bash +# Test SSH connection manually +ssh user@macbook.local + +# Test rsync manually +rsync -avz /test/file user@macbook.local:~/rtsda/livestreams/ + +# Check SSH key authentication +ssh-copy-id user@macbook.local +``` + +### FFmpeg/QSV issues +```bash +# Test FFmpeg with Intel QSV +ffmpeg -encoders | grep av1_qsv +ffmpeg -hwaccels | grep qsv + +# Check Intel Media SDK +vainfo +``` + +### Files not being detected +```bash +# Check logs for file events +sudo journalctl -u livestream-archiver -f + +# Test file detection manually +touch /home/rockvilleav/.rtsda/livestreams/test-2024-01-01_10-00-00.mp4 +``` + +### Disk space issues +```bash +# Service checks for 5GB minimum free space +df -h /media/archive/jellyfin/livestreams +``` ## Development @@ -152,9 +265,15 @@ Output files are organized by date: cargo test ``` -### Building for Debug +### Running Manually ```bash -cargo build +# Set environment variables +export INPUT_DIR="/path/to/input" +export OUTPUT_DIR="/path/to/output" +export PC_SYNC_TARGET="user@host:~/path/" + +# Run with debug logging +RUST_LOG=debug cargo run ``` ## License @@ -163,4 +282,4 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file ## Contributing -Contributions are welcome! Please feel free to submit a Pull Request. \ No newline at end of file +Contributions are welcome! This service is part of a larger church media processing system. \ No newline at end of file