Quantum/docs/websocket-sync.md

Real-time WebSocket Synchronization

Overview

The Caddy-RS file synchronization system now supports real-time synchronization using WebSocket connections. This eliminates the 30-second polling delay and provides instant file synchronization across all connected clients.

Architecture

┌─────────────────┐    WebSocket    ┌─────────────────┐
│   Client A      │ ◄─────────────► │   Server        │
│                 │                 │                 │
└─────────────────┘                 └─────────────────┘
                                             ▲
                                             │ WebSocket
                                             ▼
                                    ┌─────────────────┐
                                    │   Client B      │
                                    │                 │
                                    └─────────────────┘

Message Flow

1. Client connects → WebSocket handshake → Subscribe to updates
2. File changes → Local watcher detects → Broadcast to all clients
3. All clients → Receive operation → Apply changes immediately

WebSocket Protocol

Connection Endpoint

WS:  ws://localhost:8080/ws
WSS: wss://localhost:8080/ws (when HTTPS is enabled)

Message Types

All messages are JSON-formatted with a type field:

Client → Server Messages

Subscribe

{
  "type": "Subscribe",
  "client_id": "uuid-string"
}

File Operation

{
  "type": "FileOperation",
  "operation": {
    "Create": {
      "metadata": {
        "path": "documents/new-file.txt",
        "size": 1024,
        "modified": "2024-01-21T12:00:00Z",
        "hash": "sha256hash",
        "is_directory": false
      }
    }
  },
  "source_client": "uuid-string"
}

Heartbeat

{
  "type": "Ping"
}

Server → Client Messages

Acknowledgment

{
  "type": "Ack",
  "operation_id": "subscribe"
}

File Operation Broadcast

{
  "type": "FileOperation",
  "operation": {
    "Update": {
      "metadata": {
        "path": "documents/updated-file.txt",
        "size": 2048,
        "modified": "2024-01-21T12:05:00Z",
        "hash": "newhash",
        "is_directory": false
      }
    }
  },
  "source_client": "other-client-uuid"
}

Error

{
  "type": "Error",
  "message": "Error description"
}

Usage

Enhanced Sync Client

Use the new real-time sync client:

cargo run --bin realtime-sync-client -- \
  --server http://localhost:8080 \
  --local-path ./my-realtime-sync \
  --realtime \
  --initial-sync

Command Line Options

• --realtime / -r: Enable WebSocket real-time sync
• --client-id <ID>: Specify client identifier (auto-generated if omitted)
• --server <URL>: Server URL (same as regular client)
• --local-path <PATH>: Local sync directory
• --initial-sync: Download all files on startup

Fallback Mode

If WebSocket connection fails, the client automatically falls back to periodic HTTP sync:

# Without --realtime flag, uses periodic sync
cargo run --bin realtime-sync-client -- \
  --server http://localhost:8080 \
  --local-path ./my-sync

Implementation Details

Server-side WebSocket Support

The WsManager handles:

• Client connection lifecycle
• Message broadcasting to all connected clients
• Heartbeat/keepalive management
• Stale connection cleanup

Client-side WebSocket Support

The WsClient provides:

• Automatic reconnection (future enhancement)
• Message serialization/deserialization
• Heartbeat transmission
• Operation broadcasting

Integration with File System Watcher

The enhanced client combines:

1. File system watcher → Detects local changes
2. WebSocket client → Broadcasts changes to server
3. WebSocket receiver → Applies remote changes locally
4. HTTP client → Fallback for large file transfers

Performance Benefits

Real-time vs Periodic Sync

Aspect	Periodic (30s)	WebSocket Real-time
Latency	Up to 30 seconds	< 1 second
Network Usage	Periodic polling	Event-driven
Server Load	Regular load spikes	Distributed load
Conflict Window	30-second window	Minimal window

Bandwidth Optimization

• Metadata only: WebSocket sends file metadata, not content
• HTTP for content: Large files still use HTTP upload/download
• Event-driven: No unnecessary polling traffic

Security Considerations

Connection Security

• Authentication: Client ID-based identification
• Transport: WSS (WebSocket Secure) for encrypted connections
• Validation: All paths and operations validated server-side

Message Integrity

• JSON Schema: All messages validated against expected format
• Source Filtering: Prevents clients from receiving their own operations
• Error Handling: Graceful degradation on invalid messages

Configuration

Server Configuration

WebSocket endpoint is automatically available when file sync is enabled:

{
  "routes": [
    {
      "match": [{"matcher": "path", "paths": ["/api/*", "/ws"]}],
      "handle": [
        {
          "handler": "file_sync",
          "root": "./sync-data",
          "enable_upload": true
        }
      ]
    }
  ]
}

Client Configuration

Set environment variables for debugging:

# Enable debug logging
RUST_LOG=debug cargo run --bin realtime-sync-client -- \
  --server ws://localhost:8080 \
  --local-path ./test-sync \
  --realtime

Error Handling

Connection Failures

1. Initial connection failure → Fall back to periodic sync
2. Connection drops → Attempt reconnection (future)
3. Message parsing errors → Log and continue

Operation Conflicts

WebSocket reduces conflict probability but doesn't eliminate it:

• Simultaneous edits → Still handled by existing conflict resolution
• Network partitions → Resolved when connection restored
• Client crashes → Other clients continue normally

Monitoring and Debugging

Server-side Logs

INFO WebSocket client abc-123 connected and subscribed
DEBUG Broadcasting file operation: Create { metadata: ... }
INFO Client def-456 disconnected
DEBUG Cleaned up stale WebSocket client: ghi-789

Client-side Logs

INFO Connecting to WebSocket server: ws://localhost:8080/ws
INFO WebSocket client abc-123 connected and subscribed
INFO Received real-time operation: Update { metadata: ... }
DEBUG Received pong from server

Health Checks

Monitor WebSocket health:

• Heartbeat interval: 30 seconds
• Connection timeout: 5 minutes
• Stale cleanup: Automatic every 5 minutes

Future Enhancements

Planned Features

1. Automatic Reconnection

   • Exponential backoff strategy
   • Connection state persistence
   • Offline queue for operations

2. Message Compression

   • Gzip compression for large operations
   • Binary protocol option
   • Batch operation support

3. Advanced Authentication

   • JWT token-based auth
   • Client certificate validation
   • Role-based permissions

4. Scaling Support

   • Redis pub/sub for multi-server
   • Client clustering
   • Load balancing

Integration Points

• Web Interface: Real-time file browser updates
• Metrics: WebSocket connection and message metrics
• Admin API: WebSocket management and monitoring

Troubleshooting

Common Issues

WebSocket connection refused:

Error: Connection refused (os error 61)

• Ensure server is running with file sync enabled
• Check firewall settings for WebSocket port

Messages not received:

DEBUG No local listeners for WebSocket file operations

• Verify client is subscribed with correct client ID
• Check message filtering logic

High memory usage:

WARN WebSocket message queue growing

• Implement message queue limits
• Add backpressure handling

Debug Commands

# Test WebSocket connection manually
wscat -c ws://localhost:8080/ws

# Monitor network traffic
tcpdump -i lo0 port 8080

# Check client connections
curl http://localhost:8080/api/list

---

Status: WebSocket real-time sync implemented and ready for testing!