RTSDA/livestream-archiver
1
0
Fork 0
Code Issues Pull requests Actions Packages Projects Releases Wiki Activity

Update README.md

Browse source
This commit is contained in:
Benjamin Slingo 2025-09-08 12:58:36 -04:00
parent ab9c293a01
commit ce4b30b9a9
1 changed files with 211 additions and 92 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

303
README.md
View file

@ -1,24 +1,25 @@
# Livestream Archiver # Livestream Archiver
A Rust application that monitors a directory for livestream recordings, processes them with hardware-accelerated transcoding, and syncs them to a remote server. An event-driven Rust service that automatically detects livestream recordings, converts them for efficient storage, and syncs originals to PC for editing. Uses the shared `video_processing` crate for optimal performance.
## Features ## Features
- **Automatic Detection**: Monitors a configured directory for new MP4 files - **Event-driven file detection** - No polling, instant response to new files
- **Smart Processing**: Waits for files to be completely written before processing - **PC sync for editing** - Syncs original files to Mac/PC for post-production
- **Remote Sync**: Uses rsync to sync files to a remote server immediately after detection - **AV1 video conversion** with Intel QSV hardware acceleration
- **Hardware Acceleration**: Converts files to AV1 using Intel QSV hardware acceleration - **Smart program naming** - Detects Divine Worship vs Afternoon Program
- **Organized Storage**: Creates date-based directory structure (Jellyfin-compatible) - **File stability checking** - Ensures files are fully written before processing
- **Metadata Generation**: Creates NFO files for media servers - **Date-based organization** - Jellyfin-compatible directory structure
- **Retry Logic**: Caches and retries failed sync operations - **NFO file generation** - Complete media server metadata
- **Service Integration**: Runs as a systemd service with automatic restart - **Efficient resource usage** - Only processes when files appear
## Prerequisites ## Prerequisites
- Rust toolchain (1.70 or later) - Rust toolchain (latest stable)
- `ffmpeg` with Intel QSV support - `ffmpeg` with Intel QSV support
- `rsync` for file synchronization - `rsync` for file synchronization
- SSH key authentication for passwordless rsync (if using remote sync) - SSH key authentication for passwordless rsync to PC
- Intel Media SDK (for QSV acceleration)
## Installation ## Installation
@ -35,115 +36,227 @@ cargo build --release
# Binary will be at target/release/livestream_archiver # Binary will be at target/release/livestream_archiver
``` ```
### Quick Deploy
1. Copy and customize the example files:
```bash
cp livestream-archiver.service.example livestream-archiver.service
cp deploy.sh.example deploy.sh
# Edit both files with your specific configuration
```
2. Run the deployment:
```bash
./deploy.sh
```
## Configuration ## Configuration
### Environment Variables Create a `.env` file based on `.env.example`:
- `INPUT_DIR`: Directory to monitor for new MP4 files (default: `/home/user/livestreams`)
- `OUTPUT_DIR`: Local output directory for processed files (default: `/media/archive/livestreams`)
- `PC_SYNC_TARGET`: Remote sync destination (e.g., `user@server:/path/to/destination/`)
### Service Configuration
1. Copy the example service file:
```bash ```bash
cp livestream-archiver.service.example livestream-archiver.service cp .env.example .env
nano .env
``` ```
2. Edit the service file with your configuration: ### Required Environment Variables
- Update `User` and `Group`
- Set correct paths for `WorkingDirectory` and `ExecStart`
- Configure the `PC_SYNC_TARGET` environment variable
3. Install and start the service:
```bash ```bash
sudo cp livestream-archiver.service /etc/systemd/system/ # Directory paths (REQUIRED)
sudo systemctl daemon-reload INPUT_DIR=/home/rockvilleav/.rtsda/livestreams
OUTPUT_DIR=/media/archive/jellyfin/livestreams
# PC sync target for editing (REQUIRED)
PC_SYNC_TARGET=user@macbook.local:~/rtsda/livestreams/
# Program naming
DIVINE_WORSHIP_NAME=Divine Worship Service - RTSDA
AFTERNOON_PROGRAM_NAME=Afternoon Program - RTSDA
# FFmpeg Configuration
FFMPEG_BINARY=ffmpeg
VIDEO_CODEC=av1_qsv # AV1 with Intel QSV
HW_ACCEL=qsv
HW_DEVICE=qsv=hw
FFMPEG_PRESET=4
VIDEO_BITRATE=6M
MAX_BITRATE=12M
BUFFER_SIZE=24M
AUDIO_CODEC=copy # Preserve original audio
AUDIO_BITRATE=192k
# File Stability Settings
STABILITY_CHECK_INTERVAL=2 # seconds between stability checks
STABILITY_REQUIRED_CHECKS=15 # consecutive stable checks required
MAX_STABILITY_WAIT_HOURS=4 # maximum time to wait for stability
# Directory Settings
CREATE_YEAR_MONTH_DIRS=true # organize by date
PRESERVE_ORIGINAL_FILES=true # keep originals after conversion
# NFO Settings
SHOW_TITLE=LiveStreams
CREATE_NFO_FILES=true # or false to disable
```
## Expected File Format
Livestream files should follow OBS naming convention:
```
YYYY-MM-DD_HH-MM-SS.mp4
```
Examples:
- `2024-12-25_10-30-00.mp4` → Divine Worship Service (Christmas 2024)
- `2024-12-25_14-00-00.mp4` → Afternoon Program (Christmas 2024)
The service automatically detects service type based on file timing and existing files.
## Running as a System Service
Create a systemd service file at `/etc/systemd/system/livestream-archiver.service`:
```ini
[Unit]
Description=Livestream Archive Service
After=network.target media-archive.mount
[Service]
Type=simple
User=rockvilleav
Group=rockvilleav
ExecStart=/usr/local/bin/livestream_archiver
WorkingDirectory=/home/rockvilleav/livestream-archiver
# Load environment variables from .env file
EnvironmentFile=/home/rockvilleav/livestream-archiver/.env
# Environment variables for home directory access
Environment=HOME=/home/rockvilleav
Environment=USER=rockvilleav
Environment=XDG_RUNTIME_DIR=/run/user/1000
# Intel Media Stack environment
Environment=LIBVA_DRIVER_NAME=iHD
Environment=LIBVA_DRIVERS_PATH=/opt/intel/media/lib64
Environment=LD_LIBRARY_PATH=/opt/intel/media/lib64
Environment=LIBVA_DRIVER_PATH=/opt/intel/media/lib64
Environment=DISPLAY=:0
# Restart settings
Restart=always
RestartSec=10
# Make sure service has access to needed directories
RequiresMountsFor=/media/archive
[Install]
WantedBy=multi-user.target
```
Enable and start the service:
```bash
sudo systemctl enable livestream-archiver sudo systemctl enable livestream-archiver
sudo systemctl start livestream-archiver sudo systemctl start livestream-archiver
```
## Usage
### Running Manually
```bash
# Run directly
cargo run
# Or run the compiled binary
./target/release/livestream_archiver
```
### Running as a Service
```bash
# Check service status
sudo systemctl status livestream-archiver sudo systemctl status livestream-archiver
```
# View logs View logs:
```bash
sudo journalctl -u livestream-archiver -f sudo journalctl -u livestream-archiver -f
# Restart service
sudo systemctl restart livestream-archiver
``` ```
## How It Works ## How It Works
1. **Monitoring**: Continuously watches the input directory for new MP4 files 1. **Event Detection**: Uses `inotify` to instantly detect new MP4 files (no polling!)
2. **Validation**: Waits for files to be completely written (stable size) 2. **File Stability**: Waits for OBS to finish writing the file completely
3. **Sync**: Immediately syncs new files to the remote server via rsync 3. **PC Sync**: Immediately syncs the **original** file to Mac/PC for editing via rsync
4. **Processing**: Converts to AV1 format using hardware acceleration 4. **AV1 Conversion**: Converts to efficient AV1 format for long-term storage
5. **Organization**: Moves processed files to date-based directories 5. **Smart Naming**: Determines Divine Worship vs Afternoon Program automatically
6. **Metadata**: Creates NFO files for media server compatibility 6. **Organization**: Places files in date-based directories for Jellyfin
7. **Cleanup**: Optionally removes original files after successful processing 7. **NFO Generation**: Creates metadata files for media server compatibility
8. **Preservation**: Keeps original files on server for backup
## Directory Structure ## Directory Structure
Output files are organized by date: Output files are organized by date:
``` ```
<output-directory>/ /media/archive/jellyfin/livestreams/
├── 2024/ ├── 2024/
│ ├── 01-January/ │ ├── 12-December/
│ │ ├── Recording - January 01 2024.mp4 │ │ ├── Divine Worship Service - RTSDA (December 25 2024).mp4
│ │ ├── Recording - January 01 2024.nfo │ │ ├── Divine Worship Service - RTSDA (December 25 2024).nfo
│ │ ├── Afternoon Program - RTSDA (December 25 2024).mp4
│ │ ├── Afternoon Program - RTSDA (December 25 2024).nfo
│ │ └── ... │ │ └── ...
│ └── 02-February/ │ └── 01-January/
│ └── ...
└── 2025/ └── 2025/
└── ...
``` ```
PC sync creates:
```
~/rtsda/livestreams/
├── 2024-12-25_10-30-00.mp4 # Original for editing
├── 2024-12-25_14-00-00.mp4 # Original for editing
└── ...
```
## Architecture
This service uses a clean, event-driven architecture:
- **Event Detection**: Uses Linux `inotify` for instant file detection (no polling)
- **File Stability**: Proper completion detection before processing
- **Shared Logic**: Uses `video_processing` crate for common functionality
- **Dual Output**: Converted files for storage + original sync for editing
- **Resource Efficient**: Only uses resources when files actually appear
## Video Conversion Details
Default settings (configurable via .env):
- **Video Codec**: AV1 with Intel QSV hardware acceleration
- **Audio Codec**: Copy (preserves original quality)
- **Hardware Acceleration**: Intel Quick Sync Video
- **Video Bitrate**: 6Mbps (efficient for streaming)
- **Max Bitrate**: 12Mbps (for complex scenes)
## Troubleshooting ## Troubleshooting
### Service not starting ### Service fails to start
- Check logs: `journalctl -u livestream-archiver -f` ```bash
- Verify all paths exist and have proper permissions # Check if .env file exists and has correct paths
- Ensure ffmpeg has QSV support: `ffmpeg -hwaccels | grep qsv` ls -la /home/rockvilleav/livestream-archiver/.env
### Sync failures # Verify directories exist and are accessible
- Verify SSH key authentication to remote server ls -la /home/rockvilleav/.rtsda/livestreams
- Check network connectivity ls -la /media/archive/jellyfin/livestreams
- Review cached sync attempts in `~/.cache/livestream-archiver/`
### Processing issues # Check systemd service file
- Ensure Intel QSV drivers are installed sudo systemctl cat livestream-archiver
- Check available disk space ```
- Verify file permissions in input/output directories
### PC sync failures
```bash
# Test SSH connection manually
ssh user@macbook.local
# Test rsync manually
rsync -avz /test/file user@macbook.local:~/rtsda/livestreams/
# Check SSH key authentication
ssh-copy-id user@macbook.local
```
### FFmpeg/QSV issues
```bash
# Test FFmpeg with Intel QSV
ffmpeg -encoders | grep av1_qsv
ffmpeg -hwaccels | grep qsv
# Check Intel Media SDK
vainfo
```
### Files not being detected
```bash
# Check logs for file events
sudo journalctl -u livestream-archiver -f
# Test file detection manually
touch /home/rockvilleav/.rtsda/livestreams/test-2024-01-01_10-00-00.mp4
```
### Disk space issues
```bash
# Service checks for 5GB minimum free space
df -h /media/archive/jellyfin/livestreams
```
## Development ## Development
@ -152,9 +265,15 @@ Output files are organized by date:
cargo test cargo test
``` ```
### Building for Debug ### Running Manually
```bash ```bash
cargo build # Set environment variables
export INPUT_DIR="/path/to/input"
export OUTPUT_DIR="/path/to/output"
export PC_SYNC_TARGET="user@host:~/path/"
# Run with debug logging
RUST_LOG=debug cargo run
``` ```
## License ## License
@ -163,4 +282,4 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
## Contributing ## Contributing
Contributions are welcome! Please feel free to submit a Pull Request. Contributions are welcome! This service is part of a larger church media processing system.