Quantum/docs/file-sync.md

# File Synchronization System

## Overview

The Caddy-RS file synchronization system provides local mirroring and bidirectional sync capabilities using HTTP REST APIs. It enables cloud storage functionality where files are synchronized between a central server and multiple client machines, with the OS treating the sync folder as a native local directory.

## Architecture

```
┌─────────────┐    HTTP API     ┌─────────────┐
│   Server    │ ◄────────────► │   Client    │
│ (Caddy-RS)  │                │ (sync-client)│
└─────────────┘                └─────────────┘
       │                              │
       ▼                              ▼
┌─────────────┐                ┌─────────────┐
│ Server Root │                │ Local Mirror│
│ Directory   │                │ Directory   │
└─────────────┘                └─────────────┘
```

### Components

1. **Shared Crate (`file-sync/`)**: Core synchronization logic and protocol definitions
2. **Server Integration**: HTTP endpoints integrated into Caddy-RS proxy
3. **Sync Client**: Standalone binary for local file mirroring and watching

## File Sync Crate Structure

### `protocol.rs` - Communication Protocol
- `FileMetadata`: File information (path, size, modified time, SHA-256 hash)
- `SyncOperation`: Create, Update, Delete, Move operations
- `SyncRequest/SyncResponse`: Bidirectional sync protocol
- `ConflictInfo`: Conflict detection and resolution strategies

### `sync.rs` - Core Utilities
- `calculate_file_hash()`: SHA-256 file hashing
- `get_file_metadata()`: Extract file metadata
- `scan_directory()`: Recursively scan directory structure
- `diff_file_lists()`: Generate sync operations between local/remote
- `detect_conflicts()`: Find conflicting modifications

### `watcher.rs` - File System Monitoring
- Real-time file system event detection using `notify` crate
- Debounced event processing to avoid rapid-fire changes
- Converts filesystem events to `SyncOperation` objects
- Watches entire directory tree recursively

### `client.rs` - HTTP Sync Client
- `initial_sync()`: Downloads complete remote structure
- `start_sync_loop()`: Periodic bidirectional synchronization
- HTTP operations: upload, download, list, metadata
- Conflict resolution and error handling

### `server.rs` - HTTP Server Handler
- REST API endpoints for file operations
- Security validation (path traversal prevention)
- Client state tracking and conflict detection
- File upload/download with metadata preservation

## API Endpoints

### `/api/list` (GET)
Returns list of all files with metadata
```json
[
  {
    "path": "documents/file.txt",
    "size": 1024,
    "modified": "2024-01-01T12:00:00Z",
    "hash": "sha256hash",
    "is_directory": false
  }
]
```

### `/api/sync` (POST)
Bidirectional synchronization request
```json
{
  "operations": [
    {
      "Create": {
        "metadata": { /* FileMetadata */ }
      }
    }
  ],
  "client_id": "uuid"
}
```

### `/api/upload` (POST)
Upload file content
- Query param: `?path=relative/file/path`
- Body: raw file bytes

### `/api/download` (GET)
Download file content
- Query param: `?path=relative/file/path`
- Response: raw file bytes

### `/api/metadata` (GET)
Get file metadata only
- Query param: `?path=relative/file/path`
- Response: `FileMetadata` JSON

## Configuration

### Server Configuration (`caddy.json`)
```json
{
  "apps": {
    "http": {
      "servers": {
        "file_sync_server": {
          "listen": [":8080"],
          "routes": [
            {
              "match": [{"matcher": "path", "paths": ["/api/*"]}],
              "handle": [
                {
                  "handler": "file_sync",
                  "root": "./sync-data",
                  "enable_upload": true
                }
              ]
            }
          ]
        }
      }
    }
  }
}
```

### Handler Configuration
- `root`: Server-side directory to sync
- `enable_upload`: Whether to allow file uploads (default: false)

## Usage

### Starting the Server
```bash
cargo run -- -c example-sync-config.json
```

### Running Sync Client
```bash
# Initial sync and continuous monitoring
cargo run --bin sync-client -- \
  --server http://localhost:8080 \
  --local-path ./local-sync \
  --initial-sync
```

### Client Options
- `--server`: Server URL to sync with
- `--local-path`: Local directory to mirror
- `--initial-sync`: Download all remote files on startup

## Synchronization Behavior

### Initial Sync
1. Client requests complete file list from server
2. Downloads all files and creates local directory structure
3. Starts file system watcher and periodic sync

### Ongoing Sync
1. **File Watcher**: Detects local changes in real-time
2. **Periodic Sync**: Every 30 seconds, compares local vs remote
3. **Conflict Detection**: Identifies files modified on both sides
4. **Bidirectional Updates**: Applies changes in both directions

### Conflict Resolution
When the same file is modified locally and remotely:
- Default strategy: Keep client version
- Alternative: Rename conflicting files
- Future: User-configurable resolution strategies

## Security Features

- **Path Traversal Prevention**: Validates all paths stay within root directory
- **SHA-256 Verification**: Ensures file integrity during sync
- **Client Authentication**: Each client has unique identifier
- **Access Control**: Server validates all file operations

## Advantages

### vs Network Mounting (SMB/WebDAV/FUSE)
- ✅ **Reliability**: No network mounting complexity
- ✅ **Offline Capability**: Works when disconnected
- ✅ **Performance**: Native local file access
- ✅ **Cross-Platform**: Works consistently across OS

### vs Cloud Services (Dropbox/Drive)
- ✅ **Self-Hosted**: Full control over data and server
- ✅ **No Vendor Lock-in**: Standard HTTP + file system
- ✅ **Customizable**: Configurable sync behavior
- ✅ **Integrated**: Built into existing Caddy-RS infrastructure

## Limitations

- **Initial Sync Delay**: Large directories take time to download initially
- **Storage Requirements**: Full mirror requires local disk space
- **Sync Latency**: 30-second periodic sync interval
- **Manual Conflict Resolution**: Conflicts require user intervention

## Future Enhancements

1. **Real-time Sync**: WebSocket-based instant synchronization
2. **Selective Sync**: Choose which files/folders to mirror
3. **Compression**: Reduce bandwidth usage for large files
4. **Delta Sync**: Transfer only file differences
5. **Web UI**: Browser-based file management interface
6. **Multi-Server**: Sync with multiple servers simultaneously

## File Structure

```
Caddy/
├── file-sync/               # Shared crate
│   ├── src/
│   │   ├── protocol.rs      # API protocol definitions
│   │   ├── sync.rs          # Core sync utilities
│   │   ├── watcher.rs       # File system monitoring
│   │   ├── client.rs        # HTTP sync client
│   │   ├── server.rs        # HTTP server handlers
│   │   └── lib.rs           # Public exports
│   └── Cargo.toml
├── src/
│   ├── bin/
│   │   └── sync-client.rs   # Standalone sync client
│   ├── file_sync.rs         # Caddy integration
│   └── proxy/mod.rs         # Handler integration
├── example-sync-config.json # Sample configuration
└── docs/
    └── file-sync.md         # This document
```

This file synchronization system provides a robust foundation for cloud storage functionality while maintaining the simplicity and reliability of local file operations.