Benjamin Slingo
parent
ce4b30b9a9
commit
b9c916167e
299
README.md
299
README.md
|
|
@ -1,25 +1,24 @@
|
|||
# Livestream Archiver
|
||||
|
||||
An event-driven Rust service that automatically detects livestream recordings, converts them for efficient storage, and syncs originals to PC for editing. Uses the shared `video_processing` crate for optimal performance.
|
||||
A Rust application that monitors a directory for livestream recordings, processes them with hardware-accelerated transcoding, and syncs them to a remote server.
|
||||
|
||||
## Features
|
||||
|
||||
- **Event-driven file detection** - No polling, instant response to new files
|
||||
- **PC sync for editing** - Syncs original files to Mac/PC for post-production
|
||||
- **AV1 video conversion** with Intel QSV hardware acceleration
|
||||
- **Smart program naming** - Detects Divine Worship vs Afternoon Program
|
||||
- **File stability checking** - Ensures files are fully written before processing
|
||||
- **Date-based organization** - Jellyfin-compatible directory structure
|
||||
- **NFO file generation** - Complete media server metadata
|
||||
- **Efficient resource usage** - Only processes when files appear
|
||||
- **Automatic Detection**: Monitors a configured directory for new MP4 files
|
||||
- **Smart Processing**: Waits for files to be completely written before processing
|
||||
- **Remote Sync**: Uses rsync to sync files to a remote server immediately after detection
|
||||
- **Hardware Acceleration**: Converts files to AV1 using Intel QSV hardware acceleration
|
||||
- **Organized Storage**: Creates date-based directory structure (Jellyfin-compatible)
|
||||
- **Metadata Generation**: Creates NFO files for media servers
|
||||
- **Retry Logic**: Caches and retries failed sync operations
|
||||
- **Service Integration**: Runs as a systemd service with automatic restart
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Rust toolchain (latest stable)
|
||||
- Rust toolchain (1.70 or later)
|
||||
- `ffmpeg` with Intel QSV support
|
||||
- `rsync` for file synchronization
|
||||
- SSH key authentication for passwordless rsync to PC
|
||||
- Intel Media SDK (for QSV acceleration)
|
||||
- SSH key authentication for passwordless rsync (if using remote sync)
|
||||
|
||||
## Installation
|
||||
|
||||
|
|
@ -36,227 +35,115 @@ cargo build --release
|
|||
# Binary will be at target/release/livestream_archiver
|
||||
```
|
||||
|
||||
### Quick Deploy
|
||||
|
||||
1. Copy and customize the example files:
|
||||
```bash
|
||||
cp livestream-archiver.service.example livestream-archiver.service
|
||||
cp deploy.sh.example deploy.sh
|
||||
# Edit both files with your specific configuration
|
||||
```
|
||||
|
||||
2. Run the deployment:
|
||||
```bash
|
||||
./deploy.sh
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Create a `.env` file based on `.env.example`:
|
||||
### Environment Variables
|
||||
|
||||
- `INPUT_DIR`: Directory to monitor for new MP4 files (default: `/home/user/livestreams`)
|
||||
- `OUTPUT_DIR`: Local output directory for processed files (default: `/media/archive/livestreams`)
|
||||
- `PC_SYNC_TARGET`: Remote sync destination (e.g., `user@server:/path/to/destination/`)
|
||||
|
||||
### Service Configuration
|
||||
|
||||
1. Copy the example service file:
|
||||
```bash
|
||||
cp .env.example .env
|
||||
nano .env
|
||||
cp livestream-archiver.service.example livestream-archiver.service
|
||||
```
|
||||
|
||||
### Required Environment Variables
|
||||
2. Edit the service file with your configuration:
|
||||
- Update `User` and `Group`
|
||||
- Set correct paths for `WorkingDirectory` and `ExecStart`
|
||||
- Configure the `PC_SYNC_TARGET` environment variable
|
||||
|
||||
3. Install and start the service:
|
||||
```bash
|
||||
# Directory paths (REQUIRED)
|
||||
INPUT_DIR=/home/rockvilleav/.rtsda/livestreams
|
||||
OUTPUT_DIR=/media/archive/jellyfin/livestreams
|
||||
|
||||
# PC sync target for editing (REQUIRED)
|
||||
PC_SYNC_TARGET=user@macbook.local:~/rtsda/livestreams/
|
||||
|
||||
# Program naming
|
||||
DIVINE_WORSHIP_NAME=Divine Worship Service - RTSDA
|
||||
AFTERNOON_PROGRAM_NAME=Afternoon Program - RTSDA
|
||||
|
||||
# FFmpeg Configuration
|
||||
FFMPEG_BINARY=ffmpeg
|
||||
VIDEO_CODEC=av1_qsv # AV1 with Intel QSV
|
||||
HW_ACCEL=qsv
|
||||
HW_DEVICE=qsv=hw
|
||||
FFMPEG_PRESET=4
|
||||
VIDEO_BITRATE=6M
|
||||
MAX_BITRATE=12M
|
||||
BUFFER_SIZE=24M
|
||||
AUDIO_CODEC=copy # Preserve original audio
|
||||
AUDIO_BITRATE=192k
|
||||
|
||||
# File Stability Settings
|
||||
STABILITY_CHECK_INTERVAL=2 # seconds between stability checks
|
||||
STABILITY_REQUIRED_CHECKS=15 # consecutive stable checks required
|
||||
MAX_STABILITY_WAIT_HOURS=4 # maximum time to wait for stability
|
||||
|
||||
# Directory Settings
|
||||
CREATE_YEAR_MONTH_DIRS=true # organize by date
|
||||
PRESERVE_ORIGINAL_FILES=true # keep originals after conversion
|
||||
|
||||
# NFO Settings
|
||||
SHOW_TITLE=LiveStreams
|
||||
CREATE_NFO_FILES=true # or false to disable
|
||||
```
|
||||
|
||||
## Expected File Format
|
||||
|
||||
Livestream files should follow OBS naming convention:
|
||||
```
|
||||
YYYY-MM-DD_HH-MM-SS.mp4
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `2024-12-25_10-30-00.mp4` → Divine Worship Service (Christmas 2024)
|
||||
- `2024-12-25_14-00-00.mp4` → Afternoon Program (Christmas 2024)
|
||||
|
||||
The service automatically detects service type based on file timing and existing files.
|
||||
|
||||
## Running as a System Service
|
||||
|
||||
Create a systemd service file at `/etc/systemd/system/livestream-archiver.service`:
|
||||
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Livestream Archive Service
|
||||
After=network.target media-archive.mount
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=rockvilleav
|
||||
Group=rockvilleav
|
||||
ExecStart=/usr/local/bin/livestream_archiver
|
||||
WorkingDirectory=/home/rockvilleav/livestream-archiver
|
||||
|
||||
# Load environment variables from .env file
|
||||
EnvironmentFile=/home/rockvilleav/livestream-archiver/.env
|
||||
|
||||
# Environment variables for home directory access
|
||||
Environment=HOME=/home/rockvilleav
|
||||
Environment=USER=rockvilleav
|
||||
Environment=XDG_RUNTIME_DIR=/run/user/1000
|
||||
|
||||
# Intel Media Stack environment
|
||||
Environment=LIBVA_DRIVER_NAME=iHD
|
||||
Environment=LIBVA_DRIVERS_PATH=/opt/intel/media/lib64
|
||||
Environment=LD_LIBRARY_PATH=/opt/intel/media/lib64
|
||||
Environment=LIBVA_DRIVER_PATH=/opt/intel/media/lib64
|
||||
Environment=DISPLAY=:0
|
||||
|
||||
# Restart settings
|
||||
Restart=always
|
||||
RestartSec=10
|
||||
|
||||
# Make sure service has access to needed directories
|
||||
RequiresMountsFor=/media/archive
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Enable and start the service:
|
||||
```bash
|
||||
sudo cp livestream-archiver.service /etc/systemd/system/
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable livestream-archiver
|
||||
sudo systemctl start livestream-archiver
|
||||
sudo systemctl status livestream-archiver
|
||||
```
|
||||
|
||||
View logs:
|
||||
## Usage
|
||||
|
||||
### Running Manually
|
||||
|
||||
```bash
|
||||
# Run directly
|
||||
cargo run
|
||||
|
||||
# Or run the compiled binary
|
||||
./target/release/livestream_archiver
|
||||
```
|
||||
|
||||
### Running as a Service
|
||||
|
||||
```bash
|
||||
# Check service status
|
||||
sudo systemctl status livestream-archiver
|
||||
|
||||
# View logs
|
||||
sudo journalctl -u livestream-archiver -f
|
||||
|
||||
# Restart service
|
||||
sudo systemctl restart livestream-archiver
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **Event Detection**: Uses `inotify` to instantly detect new MP4 files (no polling!)
|
||||
2. **File Stability**: Waits for OBS to finish writing the file completely
|
||||
3. **PC Sync**: Immediately syncs the **original** file to Mac/PC for editing via rsync
|
||||
4. **AV1 Conversion**: Converts to efficient AV1 format for long-term storage
|
||||
5. **Smart Naming**: Determines Divine Worship vs Afternoon Program automatically
|
||||
6. **Organization**: Places files in date-based directories for Jellyfin
|
||||
7. **NFO Generation**: Creates metadata files for media server compatibility
|
||||
8. **Preservation**: Keeps original files on server for backup
|
||||
1. **Monitoring**: Continuously watches the input directory for new MP4 files
|
||||
2. **Validation**: Waits for files to be completely written (stable size)
|
||||
3. **Sync**: Immediately syncs new files to the remote server via rsync
|
||||
4. **Processing**: Converts to AV1 format using hardware acceleration
|
||||
5. **Organization**: Moves processed files to date-based directories
|
||||
6. **Metadata**: Creates NFO files for media server compatibility
|
||||
7. **Cleanup**: Optionally removes original files after successful processing
|
||||
|
||||
## Directory Structure
|
||||
|
||||
Output files are organized by date:
|
||||
```
|
||||
/media/archive/jellyfin/livestreams/
|
||||
<output-directory>/
|
||||
├── 2024/
|
||||
│ ├── 12-December/
|
||||
│ │ ├── Divine Worship Service - RTSDA (December 25 2024).mp4
|
||||
│ │ ├── Divine Worship Service - RTSDA (December 25 2024).nfo
|
||||
│ │ ├── Afternoon Program - RTSDA (December 25 2024).mp4
|
||||
│ │ ├── Afternoon Program - RTSDA (December 25 2024).nfo
|
||||
│ ├── 01-January/
|
||||
│ │ ├── Recording - January 01 2024.mp4
|
||||
│ │ ├── Recording - January 01 2024.nfo
|
||||
│ │ └── ...
|
||||
│ └── 01-January/
|
||||
│ └── 02-February/
|
||||
│ └── ...
|
||||
└── 2025/
|
||||
```
|
||||
|
||||
PC sync creates:
|
||||
```
|
||||
~/rtsda/livestreams/
|
||||
├── 2024-12-25_10-30-00.mp4 # Original for editing
|
||||
├── 2024-12-25_14-00-00.mp4 # Original for editing
|
||||
└── ...
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
This service uses a clean, event-driven architecture:
|
||||
|
||||
- **Event Detection**: Uses Linux `inotify` for instant file detection (no polling)
|
||||
- **File Stability**: Proper completion detection before processing
|
||||
- **Shared Logic**: Uses `video_processing` crate for common functionality
|
||||
- **Dual Output**: Converted files for storage + original sync for editing
|
||||
- **Resource Efficient**: Only uses resources when files actually appear
|
||||
|
||||
## Video Conversion Details
|
||||
|
||||
Default settings (configurable via .env):
|
||||
- **Video Codec**: AV1 with Intel QSV hardware acceleration
|
||||
- **Audio Codec**: Copy (preserves original quality)
|
||||
- **Hardware Acceleration**: Intel Quick Sync Video
|
||||
- **Video Bitrate**: 6Mbps (efficient for streaming)
|
||||
- **Max Bitrate**: 12Mbps (for complex scenes)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Service fails to start
|
||||
```bash
|
||||
# Check if .env file exists and has correct paths
|
||||
ls -la /home/rockvilleav/livestream-archiver/.env
|
||||
### Service not starting
|
||||
- Check logs: `journalctl -u livestream-archiver -f`
|
||||
- Verify all paths exist and have proper permissions
|
||||
- Ensure ffmpeg has QSV support: `ffmpeg -hwaccels | grep qsv`
|
||||
|
||||
# Verify directories exist and are accessible
|
||||
ls -la /home/rockvilleav/.rtsda/livestreams
|
||||
ls -la /media/archive/jellyfin/livestreams
|
||||
### Sync failures
|
||||
- Verify SSH key authentication to remote server
|
||||
- Check network connectivity
|
||||
- Review cached sync attempts in `~/.cache/livestream-archiver/`
|
||||
|
||||
# Check systemd service file
|
||||
sudo systemctl cat livestream-archiver
|
||||
```
|
||||
|
||||
### PC sync failures
|
||||
```bash
|
||||
# Test SSH connection manually
|
||||
ssh user@macbook.local
|
||||
|
||||
# Test rsync manually
|
||||
rsync -avz /test/file user@macbook.local:~/rtsda/livestreams/
|
||||
|
||||
# Check SSH key authentication
|
||||
ssh-copy-id user@macbook.local
|
||||
```
|
||||
|
||||
### FFmpeg/QSV issues
|
||||
```bash
|
||||
# Test FFmpeg with Intel QSV
|
||||
ffmpeg -encoders | grep av1_qsv
|
||||
ffmpeg -hwaccels | grep qsv
|
||||
|
||||
# Check Intel Media SDK
|
||||
vainfo
|
||||
```
|
||||
|
||||
### Files not being detected
|
||||
```bash
|
||||
# Check logs for file events
|
||||
sudo journalctl -u livestream-archiver -f
|
||||
|
||||
# Test file detection manually
|
||||
touch /home/rockvilleav/.rtsda/livestreams/test-2024-01-01_10-00-00.mp4
|
||||
```
|
||||
|
||||
### Disk space issues
|
||||
```bash
|
||||
# Service checks for 5GB minimum free space
|
||||
df -h /media/archive/jellyfin/livestreams
|
||||
```
|
||||
### Processing issues
|
||||
- Ensure Intel QSV drivers are installed
|
||||
- Check available disk space
|
||||
- Verify file permissions in input/output directories
|
||||
|
||||
## Development
|
||||
|
||||
|
|
@ -265,15 +152,9 @@ df -h /media/archive/jellyfin/livestreams
|
|||
cargo test
|
||||
```
|
||||
|
||||
### Running Manually
|
||||
### Building for Debug
|
||||
```bash
|
||||
# Set environment variables
|
||||
export INPUT_DIR="/path/to/input"
|
||||
export OUTPUT_DIR="/path/to/output"
|
||||
export PC_SYNC_TARGET="user@host:~/path/"
|
||||
|
||||
# Run with debug logging
|
||||
RUST_LOG=debug cargo run
|
||||
cargo build
|
||||
```
|
||||
|
||||
## License
|
||||
|
|
@ -282,4 +163,4 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
|
|||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! This service is part of a larger church media processing system.
|
||||
Contributions are welcome! Please feel free to submit a Pull Request.
|
||||
Loading…
Reference in a new issue