move-data-nas/README.md

# SyncServer

Self-hosted file sync orchestrator with Wake-on-LAN and web UI. Orchestrates `rsync` over SSH between multiple machines from a single central server.

## Features

- **Web UI**: React-based interface to manage machines, sync pairs, and job history
- **Wake-on-LAN**: Power on remote machines before syncing (UDP magic packet)
- **Rsync over SSH**: Efficient file synchronization with progress tracking
- **Scheduled syncs**: Cron-style scheduler for automated backups
- **Live progress**: Server-Sent Events (SSE) for real-time job status
- **Single binary**: No dynamic dependencies, pure Go with embedded SQLite

## Architecture

```
┌─────────────────────────────────────┐
│         SyncServer (Go binary)        │
│  ┌─────────────────────────────────┐ │
│  │  Web UI (React, embedded)       │ │
│  │  REST API (chi router)          │ │
│  │  Auth (JWT + bcrypt)            │ │
│  │  Scheduler (cron)               │ │
│  │  Sync Engine (rsync runner)     │ │
│  │  WoL (UDP magic packet)         │ │
│  │  SQLite (embedded)              │ │
│  └─────────────────────────────────┘ │
└──────────────┬────────────────────────┘
               │ SSH + UDP
     ┌─────────┴──────────┐
     ▼                   ▼
  Machine A          Machine B
  (rsync+sshd)       (rsync+sshd)
```

## Network Requirements for Wake-on-LAN

**Critical**: The magic packet (WoL) only works when the SyncServer is on the **same L2 network** (same broadcast domain) as the target machine.

### Proxmox LXC Setup

- Use **bridge networking** (`vmbr0`) — NOT NAT
- The LXC must share the same VLAN/LAN as the machines it needs to wake
- No special LXC privileges required (works with unprivileged containers)
- `SO_BROADCAST` works in unprivileged LXC containers

If your LXC is on a different subnet, use the `broadcast_addr` field on the machine to specify the broadcast address of the target machine's subnet (e.g., `192.168.1.255`).

### BIOS/UEFI Requirements on Target Machines

- Enable **Wake-on-LAN** in BIOS/UEFI
- Set to "Power On by PCIe" or "Wake by PCI-E"
- Some motherboards require both PCIe and LAN WoL to be enabled

## Installation

### Quick Start (Development)

```bash
git clone https://github.com/syncserver/syncserver
cd syncserver

# Run with default settings (data in ./data)
SYNCSERVER_ADMIN_USER=admin SYNCSERVER_ADMIN_PASSWORD=secret \
  ./syncserver --addr :8080

# Open http://your-server:8080
```

### Production Install (.deb)

```bash
# Transfer to your Proxmox LXC (Debian/Ubuntu)
scp dist/syncserver_*.deb root@your-lxc:/tmp/

# On the LXC:
dpkg -i /tmp/syncserver_*.deb
# postinst will:
#   - Create syncserver user
#   - Create /var/lib/syncserver/{data,ssh,logs}
#   - Install config at /etc/syncserver/config.yaml
#   - Enable and start the systemd service

# To configure the admin user on first run:
# Set environment variables BEFORE first start (in /etc/syncserver/config.yaml or systemd unit):
```

### Configuration

Edit `/etc/syncserver/config.yaml`:

```yaml
data_dir: /var/lib/syncserver
addr: :8080

auth:
  jwt_secret: "your-secret-here"  # Generate with: openssl rand -hex 32
  jwt_expiry_hours: 24

scheduler:
  timezone: UTC
```

Or use environment variables (override config file):

```bash
SYNCSERVER_ADMIN_USER=admin SYNCSERVER_ADMIN_PASSWORD=secret \
SYNCSERVER_JWT_SECRET=your-secret \
SYNCSERVER_DATA_DIR=/var/lib/syncserver \
/usr/bin/syncserver --config /etc/syncserver/config.yaml
```

## Setup: SSH Keys

### Step 1: Get the Server's Public Key

In the UI, go to **Settings** to copy the server's SSH public key. It looks like:

```
ssh-ed25519 AAAA... syncserver
```

### Step 2: Add the Key to Remote Machines

On each remote machine, add the public key to `~/.ssh/authorized_keys`:

```bash
# As root on the remote machine:
echo "ssh-ed25519 AAAA... syncserver" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
```

Or use `ssh-copy-id`:

```bash
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@remote-machine
# Then replace the key in the remote's authorized_keys with the syncserver key
```

### Step 3: Register Machines in the UI

Go to **Machines** → **Add Machine**:
- Name: e.g., `backup-nas`
- Host: IP or hostname (e.g., `192.168.1.100`)
- SSH Port: `22`
- SSH User: `root` (or your preferred user)
- SSH Key: leave empty to use the server's default key

For **Wake-on-LAN**:
- Enable "Wake-on-LAN"
- Enter the MAC address (e.g., `AA:BB:CC:DD:EE:FF`)
- Optionally set a broadcast address (leave empty for `255.255.255.255`)
- Configure timeout and check interval

## Creating Sync Pairs

Go to **Sync Pairs** → **Add Sync Pair**:

| Field | Description |
|-------|-------------|
| Name | Descriptive name, e.g., `backup-home` |
| Source | Machine + path (or "Local server" for the syncserver itself) |
| Destination | Machine + path |
| Direction | `push` (source→dest), `pull` (dest→source), `mirror` (bidirectional with delete) |
| Rsync Flags | Default: `-aP` (archive, progress). Add `--delete` for mirror mode. |
| Exclude Patterns | One pattern per line, e.g., `*.tmp`, `node_modules/` |

## Scheduling

Add a schedule to any sync pair:

| Cron Expr | Description |
|-----------|-------------|
| `0 2 * * *` | Daily at 2 AM |
| `0 */6 * * *` | Every 6 hours |
| `0 9-17 * * 1-5` | Business hours, weekdays |
| `*/15 * * * *` | Every 15 minutes |

All schedules run in **UTC** by default. Set `scheduler.timezone` in config to change.

## Job States

| State | Description |
|-------|-------------|
| `queued` | Waiting for a worker |
| `waking_up` | Sending WoL magic packet and waiting for machine |
| `running` | rsync in progress |
| `success` | Completed successfully |
| `failed` | Error during sync |
| `cancelled` | User cancelled the job |

## Logs

Job logs are stored at `/var/lib/syncserver/logs/<job_id>.log`.

Application logs go to systemd journal:

```bash
journalctl -u syncserver -f
```

## Upgrade

```bash
# On the LXC:
dpkg -i /tmp/syncserver-new-version.deb
# The service will automatically restart
```

Config at `/etc/syncserver/config.yaml` is preserved (conffile).

## Build from Source

```bash
# Requires: Go 1.22+, Node 18+, npm

./scripts/build.sh
# Output: packaging/debian/usr/bin/syncserver

./scripts/package-deb.sh
# Output: dist/syncserver_VERSION_amd64.deb
```

For ARM64:

```bash
GOARCH=arm64 ./scripts/build.sh
ARCH=arm64 ./scripts/package-deb.sh
```

## Data Storage

```
/var/lib/syncserver/
├── data/
│   └── app.db        # SQLite database (machines, sync_pairs, jobs, etc.)
├── ssh/
│   ├── id_ed25519     # Server's private SSH key
│   ├── id_ed25519.pub # Server's public SSH key
│   └── known_hosts    # Host keys of registered machines
└── logs/
    └── <job_id>.log   # Per-job rsync output logs
```

**Recommended**: Bind mount `/var/lib/syncserver` to persistent storage outside the LXC root filesystem. This way you can recreate the LXC without losing data.

In Proxmox LXC config (`/etc/pve/lxc/<id>.conf`):

```
mp0: /mnt/data/syncserver,mp=/var/lib/syncserver,backup=0
```

## Troubleshooting

### WoL doesn't work
1. Verify the LXC is on the same L2 network as the target
2. Check `ip link` in the LXC shows the bridge interface
3. Try: `tcpdump -i <iface> udp port 9` in the LXC while triggering WoL
4. Verify WoL is enabled in the target machine's BIOS
5. Try sending WoL from another machine on the same subnet to isolate the issue

### SSH connection fails
1. Verify the public key is in `~/.ssh/authorized_keys` on the target
2. Check `sshd` is running on the target machine
3. Try manually: `ssh -i /var/lib/syncserver/ssh/id_ed25519 user@host`
4. Check the target's `sshd_config`: `PubkeyAuthentication yes`, `AuthorizedKeysFile .ssh/authorized_keys`

### rsync fails with "connection unexpectedly closed"
- Usually means SSH key auth failed
- Check the job log for the exact SSH error
- Try running the rsync command manually from the server with `-v` flag

### Machine shows "unknown" status
- Status is updated when a job runs or when you test SSH
- It does not auto-poll; this is by design to avoid network load

## API Reference

All API endpoints require authentication (JWT cookie).

| Method | Path | Description |
|--------|------|-------------|
| POST | `/api/auth/login` | Login with username/password |
| POST | `/api/auth/logout` | Clear session |
| GET | `/api/auth/me` | Current user info |
| GET/POST | `/api/machines` | List/create machines |
| GET/PUT/DELETE | `/api/machines/{id}` | Get/update/delete machine |
| GET/POST | `/api/sync-pairs` | List/create sync pairs |
| GET/PUT/DELETE | `/api/sync-pairs/{id}` | Get/update/delete sync pair |
| POST | `/api/sync-pairs/{id}/run` | Trigger manual sync |
| GET | `/api/jobs` | List jobs |
| POST | `/api/jobs/{id}/cancel` | Cancel a running job |
| GET | `/api/jobs/{id}/log` | Stream job log |
| GET | `/api/jobs/stream` | SSE stream of all job events |
| GET | `/api/settings/pubkey` | Get server's SSH public key |

## License

MIT