Quantum/docs/file-sync.md

File Synchronization System

Overview

The Caddy-RS file synchronization system provides local mirroring and bidirectional sync capabilities using HTTP REST APIs. It enables cloud storage functionality where files are synchronized between a central server and multiple client machines, with the OS treating the sync folder as a native local directory.

Architecture

┌─────────────┐    HTTP API     ┌─────────────┐
│   Server    │ ◄────────────► │   Client    │
│ (Caddy-RS)  │                │ (sync-client)│
└─────────────┘                └─────────────┘
       │                              │
       ▼                              ▼
┌─────────────┐                ┌─────────────┐
│ Server Root │                │ Local Mirror│
│ Directory   │                │ Directory   │
└─────────────┘                └─────────────┘

Components

1. Shared Crate (file-sync/): Core synchronization logic and protocol definitions
2. Server Integration: HTTP endpoints integrated into Caddy-RS proxy
3. Sync Client: Standalone binary for local file mirroring and watching

File Sync Crate Structure

protocol.rs - Communication Protocol

• FileMetadata: File information (path, size, modified time, SHA-256 hash)
• SyncOperation: Create, Update, Delete, Move operations
• SyncRequest/SyncResponse: Bidirectional sync protocol
• ConflictInfo: Conflict detection and resolution strategies

sync.rs - Core Utilities

• calculate_file_hash(): SHA-256 file hashing
• get_file_metadata(): Extract file metadata
• scan_directory(): Recursively scan directory structure
• diff_file_lists(): Generate sync operations between local/remote
• detect_conflicts(): Find conflicting modifications

watcher.rs - File System Monitoring

• Real-time file system event detection using notify crate
• Debounced event processing to avoid rapid-fire changes
• Converts filesystem events to SyncOperation objects
• Watches entire directory tree recursively

client.rs - HTTP Sync Client

• initial_sync(): Downloads complete remote structure
• start_sync_loop(): Periodic bidirectional synchronization
• HTTP operations: upload, download, list, metadata
• Conflict resolution and error handling

server.rs - HTTP Server Handler

• REST API endpoints for file operations
• Security validation (path traversal prevention)
• Client state tracking and conflict detection
• File upload/download with metadata preservation

API Endpoints

/api/list (GET)

Returns list of all files with metadata

[
  {
    "path": "documents/file.txt",
    "size": 1024,
    "modified": "2024-01-01T12:00:00Z",
    "hash": "sha256hash",
    "is_directory": false
  }
]

/api/sync (POST)

Bidirectional synchronization request

{
  "operations": [
    {
      "Create": {
        "metadata": { /* FileMetadata */ }
      }
    }
  ],
  "client_id": "uuid"
}

/api/upload (POST)

Upload file content

• Query param: ?path=relative/file/path
• Body: raw file bytes

/api/download (GET)

Download file content

• Query param: ?path=relative/file/path
• Response: raw file bytes

/api/metadata (GET)

Get file metadata only

• Query param: ?path=relative/file/path
• Response: FileMetadata JSON

Configuration

Server Configuration (caddy.json)

{
  "apps": {
    "http": {
      "servers": {
        "file_sync_server": {
          "listen": [":8080"],
          "routes": [
            {
              "match": [{"matcher": "path", "paths": ["/api/*"]}],
              "handle": [
                {
                  "handler": "file_sync",
                  "root": "./sync-data",
                  "enable_upload": true
                }
              ]
            }
          ]
        }
      }
    }
  }
}

Handler Configuration

• root: Server-side directory to sync
• enable_upload: Whether to allow file uploads (default: false)

Usage

Starting the Server

cargo run -- -c example-sync-config.json

Running Sync Client

# Initial sync and continuous monitoring
cargo run --bin sync-client -- \
  --server http://localhost:8080 \
  --local-path ./local-sync \
  --initial-sync

Client Options

• --server: Server URL to sync with
• --local-path: Local directory to mirror
• --initial-sync: Download all remote files on startup

Synchronization Behavior

Initial Sync

1. Client requests complete file list from server
2. Downloads all files and creates local directory structure
3. Starts file system watcher and periodic sync

Ongoing Sync

1. File Watcher: Detects local changes in real-time
2. Periodic Sync: Every 30 seconds, compares local vs remote
3. Conflict Detection: Identifies files modified on both sides
4. Bidirectional Updates: Applies changes in both directions

Conflict Resolution

When the same file is modified locally and remotely:

• Default strategy: Keep client version
• Alternative: Rename conflicting files
• Future: User-configurable resolution strategies

Security Features

• Path Traversal Prevention: Validates all paths stay within root directory
• SHA-256 Verification: Ensures file integrity during sync
• Client Authentication: Each client has unique identifier
• Access Control: Server validates all file operations

Advantages

vs Network Mounting (SMB/WebDAV/FUSE)

• ✅ Reliability: No network mounting complexity
• ✅ Offline Capability: Works when disconnected
• ✅ Performance: Native local file access
• ✅ Cross-Platform: Works consistently across OS

vs Cloud Services (Dropbox/Drive)

• ✅ Self-Hosted: Full control over data and server
• ✅ No Vendor Lock-in: Standard HTTP + file system
• ✅ Customizable: Configurable sync behavior
• ✅ Integrated: Built into existing Caddy-RS infrastructure

Limitations

• Initial Sync Delay: Large directories take time to download initially
• Storage Requirements: Full mirror requires local disk space
• Sync Latency: 30-second periodic sync interval
• Manual Conflict Resolution: Conflicts require user intervention

Future Enhancements

1. Real-time Sync: WebSocket-based instant synchronization
2. Selective Sync: Choose which files/folders to mirror
3. Compression: Reduce bandwidth usage for large files
4. Delta Sync: Transfer only file differences
5. Web UI: Browser-based file management interface
6. Multi-Server: Sync with multiple servers simultaneously

File Structure

Caddy/
├── file-sync/               # Shared crate
│   ├── src/
│   │   ├── protocol.rs      # API protocol definitions
│   │   ├── sync.rs          # Core sync utilities
│   │   ├── watcher.rs       # File system monitoring
│   │   ├── client.rs        # HTTP sync client
│   │   ├── server.rs        # HTTP server handlers
│   │   └── lib.rs           # Public exports
│   └── Cargo.toml
├── src/
│   ├── bin/
│   │   └── sync-client.rs   # Standalone sync client
│   ├── file_sync.rs         # Caddy integration
│   └── proxy/mod.rs         # Handler integration
├── example-sync-config.json # Sample configuration
└── docs/
    └── file-sync.md         # This document

This file synchronization system provides a robust foundation for cloud storage functionality while maintaining the simplicity and reliability of local file operations.