diff --git a/README.md b/README.md index cbf1aa6..9092aa2 100644 --- a/README.md +++ b/README.md @@ -1,25 +1,24 @@ # Livestream Archiver -An event-driven Rust service that automatically detects livestream recordings, converts them for efficient storage, and syncs originals to PC for editing. Uses the shared `video_processing` crate for optimal performance. +A Rust application that monitors a directory for livestream recordings, processes them with hardware-accelerated transcoding, and syncs them to a remote server. ## Features -- **Event-driven file detection** - No polling, instant response to new files -- **PC sync for editing** - Syncs original files to Mac/PC for post-production -- **AV1 video conversion** with Intel QSV hardware acceleration -- **Smart program naming** - Detects Divine Worship vs Afternoon Program -- **File stability checking** - Ensures files are fully written before processing -- **Date-based organization** - Jellyfin-compatible directory structure -- **NFO file generation** - Complete media server metadata -- **Efficient resource usage** - Only processes when files appear +- **Automatic Detection**: Monitors a configured directory for new MP4 files +- **Smart Processing**: Waits for files to be completely written before processing +- **Remote Sync**: Uses rsync to sync files to a remote server immediately after detection +- **Hardware Acceleration**: Converts files to AV1 using Intel QSV hardware acceleration +- **Organized Storage**: Creates date-based directory structure (Jellyfin-compatible) +- **Metadata Generation**: Creates NFO files for media servers +- **Retry Logic**: Caches and retries failed sync operations +- **Service Integration**: Runs as a systemd service with automatic restart ## Prerequisites -- Rust toolchain (latest stable) +- Rust toolchain (1.70 or later) - `ffmpeg` with Intel QSV support - `rsync` for file synchronization -- SSH key authentication for passwordless rsync to PC -- Intel Media SDK (for QSV acceleration) +- SSH key authentication for passwordless rsync (if using remote sync) ## Installation @@ -36,227 +35,115 @@ cargo build --release # Binary will be at target/release/livestream_archiver ``` +### Quick Deploy + +1. Copy and customize the example files: +```bash +cp livestream-archiver.service.example livestream-archiver.service +cp deploy.sh.example deploy.sh +# Edit both files with your specific configuration +``` + +2. Run the deployment: +```bash +./deploy.sh +``` + ## Configuration -Create a `.env` file based on `.env.example`: +### Environment Variables +- `INPUT_DIR`: Directory to monitor for new MP4 files (default: `/home/user/livestreams`) +- `OUTPUT_DIR`: Local output directory for processed files (default: `/media/archive/livestreams`) +- `PC_SYNC_TARGET`: Remote sync destination (e.g., `user@server:/path/to/destination/`) + +### Service Configuration + +1. Copy the example service file: ```bash -cp .env.example .env -nano .env +cp livestream-archiver.service.example livestream-archiver.service ``` -### Required Environment Variables +2. Edit the service file with your configuration: +- Update `User` and `Group` +- Set correct paths for `WorkingDirectory` and `ExecStart` +- Configure the `PC_SYNC_TARGET` environment variable +3. Install and start the service: ```bash -# Directory paths (REQUIRED) -INPUT_DIR=/home/rockvilleav/.rtsda/livestreams -OUTPUT_DIR=/media/archive/jellyfin/livestreams - -# PC sync target for editing (REQUIRED) -PC_SYNC_TARGET=user@macbook.local:~/rtsda/livestreams/ - -# Program naming -DIVINE_WORSHIP_NAME=Divine Worship Service - RTSDA -AFTERNOON_PROGRAM_NAME=Afternoon Program - RTSDA - -# FFmpeg Configuration -FFMPEG_BINARY=ffmpeg -VIDEO_CODEC=av1_qsv # AV1 with Intel QSV -HW_ACCEL=qsv -HW_DEVICE=qsv=hw -FFMPEG_PRESET=4 -VIDEO_BITRATE=6M -MAX_BITRATE=12M -BUFFER_SIZE=24M -AUDIO_CODEC=copy # Preserve original audio -AUDIO_BITRATE=192k - -# File Stability Settings -STABILITY_CHECK_INTERVAL=2 # seconds between stability checks -STABILITY_REQUIRED_CHECKS=15 # consecutive stable checks required -MAX_STABILITY_WAIT_HOURS=4 # maximum time to wait for stability - -# Directory Settings -CREATE_YEAR_MONTH_DIRS=true # organize by date -PRESERVE_ORIGINAL_FILES=true # keep originals after conversion - -# NFO Settings -SHOW_TITLE=LiveStreams -CREATE_NFO_FILES=true # or false to disable -``` - -## Expected File Format - -Livestream files should follow OBS naming convention: -``` -YYYY-MM-DD_HH-MM-SS.mp4 -``` - -Examples: -- `2024-12-25_10-30-00.mp4` → Divine Worship Service (Christmas 2024) -- `2024-12-25_14-00-00.mp4` → Afternoon Program (Christmas 2024) - -The service automatically detects service type based on file timing and existing files. - -## Running as a System Service - -Create a systemd service file at `/etc/systemd/system/livestream-archiver.service`: - -```ini -[Unit] -Description=Livestream Archive Service -After=network.target media-archive.mount - -[Service] -Type=simple -User=rockvilleav -Group=rockvilleav -ExecStart=/usr/local/bin/livestream_archiver -WorkingDirectory=/home/rockvilleav/livestream-archiver - -# Load environment variables from .env file -EnvironmentFile=/home/rockvilleav/livestream-archiver/.env - -# Environment variables for home directory access -Environment=HOME=/home/rockvilleav -Environment=USER=rockvilleav -Environment=XDG_RUNTIME_DIR=/run/user/1000 - -# Intel Media Stack environment -Environment=LIBVA_DRIVER_NAME=iHD -Environment=LIBVA_DRIVERS_PATH=/opt/intel/media/lib64 -Environment=LD_LIBRARY_PATH=/opt/intel/media/lib64 -Environment=LIBVA_DRIVER_PATH=/opt/intel/media/lib64 -Environment=DISPLAY=:0 - -# Restart settings -Restart=always -RestartSec=10 - -# Make sure service has access to needed directories -RequiresMountsFor=/media/archive - -[Install] -WantedBy=multi-user.target -``` - -Enable and start the service: -```bash +sudo cp livestream-archiver.service /etc/systemd/system/ +sudo systemctl daemon-reload sudo systemctl enable livestream-archiver sudo systemctl start livestream-archiver -sudo systemctl status livestream-archiver ``` -View logs: +## Usage + +### Running Manually + ```bash +# Run directly +cargo run + +# Or run the compiled binary +./target/release/livestream_archiver +``` + +### Running as a Service + +```bash +# Check service status +sudo systemctl status livestream-archiver + +# View logs sudo journalctl -u livestream-archiver -f + +# Restart service +sudo systemctl restart livestream-archiver ``` ## How It Works -1. **Event Detection**: Uses `inotify` to instantly detect new MP4 files (no polling!) -2. **File Stability**: Waits for OBS to finish writing the file completely -3. **PC Sync**: Immediately syncs the **original** file to Mac/PC for editing via rsync -4. **AV1 Conversion**: Converts to efficient AV1 format for long-term storage -5. **Smart Naming**: Determines Divine Worship vs Afternoon Program automatically -6. **Organization**: Places files in date-based directories for Jellyfin -7. **NFO Generation**: Creates metadata files for media server compatibility -8. **Preservation**: Keeps original files on server for backup +1. **Monitoring**: Continuously watches the input directory for new MP4 files +2. **Validation**: Waits for files to be completely written (stable size) +3. **Sync**: Immediately syncs new files to the remote server via rsync +4. **Processing**: Converts to AV1 format using hardware acceleration +5. **Organization**: Moves processed files to date-based directories +6. **Metadata**: Creates NFO files for media server compatibility +7. **Cleanup**: Optionally removes original files after successful processing ## Directory Structure Output files are organized by date: ``` -/media/archive/jellyfin/livestreams/ +/ ├── 2024/ -│ ├── 12-December/ -│ │ ├── Divine Worship Service - RTSDA (December 25 2024).mp4 -│ │ ├── Divine Worship Service - RTSDA (December 25 2024).nfo -│ │ ├── Afternoon Program - RTSDA (December 25 2024).mp4 -│ │ ├── Afternoon Program - RTSDA (December 25 2024).nfo +│ ├── 01-January/ +│ │ ├── Recording - January 01 2024.mp4 +│ │ ├── Recording - January 01 2024.nfo │ │ └── ... -│ └── 01-January/ +│ └── 02-February/ +│ └── ... └── 2025/ + └── ... ``` -PC sync creates: -``` -~/rtsda/livestreams/ -├── 2024-12-25_10-30-00.mp4 # Original for editing -├── 2024-12-25_14-00-00.mp4 # Original for editing -└── ... -``` - -## Architecture - -This service uses a clean, event-driven architecture: - -- **Event Detection**: Uses Linux `inotify` for instant file detection (no polling) -- **File Stability**: Proper completion detection before processing -- **Shared Logic**: Uses `video_processing` crate for common functionality -- **Dual Output**: Converted files for storage + original sync for editing -- **Resource Efficient**: Only uses resources when files actually appear - -## Video Conversion Details - -Default settings (configurable via .env): -- **Video Codec**: AV1 with Intel QSV hardware acceleration -- **Audio Codec**: Copy (preserves original quality) -- **Hardware Acceleration**: Intel Quick Sync Video -- **Video Bitrate**: 6Mbps (efficient for streaming) -- **Max Bitrate**: 12Mbps (for complex scenes) - ## Troubleshooting -### Service fails to start -```bash -# Check if .env file exists and has correct paths -ls -la /home/rockvilleav/livestream-archiver/.env +### Service not starting +- Check logs: `journalctl -u livestream-archiver -f` +- Verify all paths exist and have proper permissions +- Ensure ffmpeg has QSV support: `ffmpeg -hwaccels | grep qsv` -# Verify directories exist and are accessible -ls -la /home/rockvilleav/.rtsda/livestreams -ls -la /media/archive/jellyfin/livestreams +### Sync failures +- Verify SSH key authentication to remote server +- Check network connectivity +- Review cached sync attempts in `~/.cache/livestream-archiver/` -# Check systemd service file -sudo systemctl cat livestream-archiver -``` - -### PC sync failures -```bash -# Test SSH connection manually -ssh user@macbook.local - -# Test rsync manually -rsync -avz /test/file user@macbook.local:~/rtsda/livestreams/ - -# Check SSH key authentication -ssh-copy-id user@macbook.local -``` - -### FFmpeg/QSV issues -```bash -# Test FFmpeg with Intel QSV -ffmpeg -encoders | grep av1_qsv -ffmpeg -hwaccels | grep qsv - -# Check Intel Media SDK -vainfo -``` - -### Files not being detected -```bash -# Check logs for file events -sudo journalctl -u livestream-archiver -f - -# Test file detection manually -touch /home/rockvilleav/.rtsda/livestreams/test-2024-01-01_10-00-00.mp4 -``` - -### Disk space issues -```bash -# Service checks for 5GB minimum free space -df -h /media/archive/jellyfin/livestreams -``` +### Processing issues +- Ensure Intel QSV drivers are installed +- Check available disk space +- Verify file permissions in input/output directories ## Development @@ -265,15 +152,9 @@ df -h /media/archive/jellyfin/livestreams cargo test ``` -### Running Manually +### Building for Debug ```bash -# Set environment variables -export INPUT_DIR="/path/to/input" -export OUTPUT_DIR="/path/to/output" -export PC_SYNC_TARGET="user@host:~/path/" - -# Run with debug logging -RUST_LOG=debug cargo run +cargo build ``` ## License @@ -282,4 +163,4 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file ## Contributing -Contributions are welcome! This service is part of a larger church media processing system. \ No newline at end of file +Contributions are welcome! Please feel free to submit a Pull Request. \ No newline at end of file