proyectos/move-data-nas
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: complete SyncServer implementation

Browse Source 
Full-stack Go monolith with embedded React frontend for orchestrating
rsync-over-SSH file synchronization with Wake-on-LAN support.

Features:
- JWT auth (HS256) with bcrypt password hashing
- CRUD for machines (with WoL config) and sync_pairs
- Ed25519 SSH key generation and known_hosts management
- WoL magic packet sender + TCP-connect waiter with backoff
- Sync engine: rsync subprocess, per-pair job queue, progress parsing
- Homebrew cron parser for scheduled syncs
- SSE stream for live job status (queued/waking_up/running/success/failed)
- React+TS+Vite+Tailwind SPA embedded via embed.FS
- Debian packaging with systemd unit, postinst/prerm/postrm

Tech stack:
- Go 1.22+ (CGO_ENABLED=0, pure SQLite via modernc.org/sqlite)
- chi router for HTTP API
- TypeScript + React 18 + Tailwind CSS frontend
- Cross-compiled to Linux amd64 for Proxmox LXC deployment

Tests: wol (MAC parsing, magic packet), syncengine/queue, scheduler/cron
This commit is contained in:
Daniel Arroyo
2026-07-07 15:03:22 -04:00
parent 1a66ac58cd
commit 8e08c73f60
69 changed files with 7949 additions and 152 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+298 -1
View File
@@ -1,2 +1,299 @@
# move-data-nas
# SyncServer
Self-hosted file sync orchestrator with Wake-on-LAN and web UI. Orchestrates `rsync` over SSH between multiple machines from a single central server.
## Features
- **Web UI**: React-based interface to manage machines, sync pairs, and job history
- **Wake-on-LAN**: Power on remote machines before syncing (UDP magic packet)
- **Rsync over SSH**: Efficient file synchronization with progress tracking
- **Scheduled syncs**: Cron-style scheduler for automated backups
- **Live progress**: Server-Sent Events (SSE) for real-time job status
- **Single binary**: No dynamic dependencies, pure Go with embedded SQLite
## Architecture
```
┌─────────────────────────────────────┐
│ SyncServer (Go binary) │
│ ┌─────────────────────────────────┐ │
│ │ Web UI (React, embedded) │ │
│ │ REST API (chi router) │ │
│ │ Auth (JWT + bcrypt) │ │
│ │ Scheduler (cron) │ │
│ │ Sync Engine (rsync runner) │ │
│ │ WoL (UDP magic packet) │ │
│ │ SQLite (embedded) │ │
│ └─────────────────────────────────┘ │
└──────────────┬────────────────────────┘
│ SSH + UDP
┌─────────┴──────────┐
▼ ▼
Machine A Machine B
(rsync+sshd) (rsync+sshd)
```
## Network Requirements for Wake-on-LAN
**Critical**: The magic packet (WoL) only works when the SyncServer is on the **same L2 network** (same broadcast domain) as the target machine.
### Proxmox LXC Setup
- Use **bridge networking** (`vmbr0`) — NOT NAT
- The LXC must share the same VLAN/LAN as the machines it needs to wake
- No special LXC privileges required (works with unprivileged containers)
- `SO_BROADCAST` works in unprivileged LXC containers
If your LXC is on a different subnet, use the `broadcast_addr` field on the machine to specify the broadcast address of the target machine's subnet (e.g., `192.168.1.255`).
### BIOS/UEFI Requirements on Target Machines
- Enable **Wake-on-LAN** in BIOS/UEFI
- Set to "Power On by PCIe" or "Wake by PCI-E"
- Some motherboards require both PCIe and LAN WoL to be enabled
## Installation
### Quick Start (Development)
```bash
git clone https://github.com/syncserver/syncserver
cd syncserver
# Run with default settings (data in ./data)
SYNCSERVER_ADMIN_USER=admin SYNCSERVER_ADMIN_PASSWORD=secret \
./syncserver --addr :8080
# Open http://your-server:8080
```
### Production Install (.deb)
```bash
# Transfer to your Proxmox LXC (Debian/Ubuntu)
scp dist/syncserver_*.deb root@your-lxc:/tmp/
# On the LXC:
dpkg -i /tmp/syncserver_*.deb
# postinst will:
# - Create syncserver user
# - Create /var/lib/syncserver/{data,ssh,logs}
# - Install config at /etc/syncserver/config.yaml
# - Enable and start the systemd service
# To configure the admin user on first run:
# Set environment variables BEFORE first start (in /etc/syncserver/config.yaml or systemd unit):
```
### Configuration
Edit `/etc/syncserver/config.yaml`:
```yaml
data_dir: /var/lib/syncserver
addr: :8080
auth:
jwt_secret: "your-secret-here" # Generate with: openssl rand -hex 32
jwt_expiry_hours: 24
scheduler:
timezone: UTC
```
Or use environment variables (override config file):
```bash
SYNCSERVER_ADMIN_USER=admin SYNCSERVER_ADMIN_PASSWORD=secret \
SYNCSERVER_JWT_SECRET=your-secret \
SYNCSERVER_DATA_DIR=/var/lib/syncserver \
/usr/bin/syncserver --config /etc/syncserver/config.yaml
```
## Setup: SSH Keys
### Step 1: Get the Server's Public Key
In the UI, go to **Settings** to copy the server's SSH public key. It looks like:
```
ssh-ed25519 AAAA... syncserver
```
### Step 2: Add the Key to Remote Machines
On each remote machine, add the public key to `~/.ssh/authorized_keys`:
```bash
# As root on the remote machine:
echo "ssh-ed25519 AAAA... syncserver" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys
```
Or use `ssh-copy-id`:
```bash
ssh-copy-id -i ~/.ssh/id_ed25519.pub user@remote-machine
# Then replace the key in the remote's authorized_keys with the syncserver key
```
### Step 3: Register Machines in the UI
Go to **Machines** → **Add Machine**:
- Name: e.g., `backup-nas`
- Host: IP or hostname (e.g., `192.168.1.100`)
- SSH Port: `22`
- SSH User: `root` (or your preferred user)
- SSH Key: leave empty to use the server's default key
For **Wake-on-LAN**:
- Enable "Wake-on-LAN"
- Enter the MAC address (e.g., `AA:BB:CC:DD:EE:FF`)
- Optionally set a broadcast address (leave empty for `255.255.255.255`)
- Configure timeout and check interval
## Creating Sync Pairs
Go to **Sync Pairs** → **Add Sync Pair**:
| Field | Description |
|-------|-------------|
| Name | Descriptive name, e.g., `backup-home` |
| Source | Machine + path (or "Local server" for the syncserver itself) |
| Destination | Machine + path |
| Direction | `push` (source→dest), `pull` (dest→source), `mirror` (bidirectional with delete) |
| Rsync Flags | Default: `-aP` (archive, progress). Add `--delete` for mirror mode. |
| Exclude Patterns | One pattern per line, e.g., `*.tmp`, `node_modules/` |
## Scheduling
Add a schedule to any sync pair:
| Cron Expr | Description |
|-----------|-------------|
| `0 2 * * *` | Daily at 2 AM |
| `0 */6 * * *` | Every 6 hours |
| `0 9-17 * * 1-5` | Business hours, weekdays |
| `*/15 * * * *` | Every 15 minutes |
All schedules run in **UTC** by default. Set `scheduler.timezone` in config to change.
## Job States
| State | Description |
|-------|-------------|
| `queued` | Waiting for a worker |
| `waking_up` | Sending WoL magic packet and waiting for machine |
| `running` | rsync in progress |
| `success` | Completed successfully |
| `failed` | Error during sync |
| `cancelled` | User cancelled the job |
## Logs
Job logs are stored at `/var/lib/syncserver/logs/<job_id>.log`.
Application logs go to systemd journal:
```bash
journalctl -u syncserver -f
```
## Upgrade
```bash
# On the LXC:
dpkg -i /tmp/syncserver-new-version.deb
# The service will automatically restart
```
Config at `/etc/syncserver/config.yaml` is preserved (conffile).
## Build from Source
```bash
# Requires: Go 1.22+, Node 18+, npm
./scripts/build.sh
# Output: packaging/debian/usr/bin/syncserver
./scripts/package-deb.sh
# Output: dist/syncserver_VERSION_amd64.deb
```
For ARM64:
```bash
GOARCH=arm64 ./scripts/build.sh
ARCH=arm64 ./scripts/package-deb.sh
```
## Data Storage
```
/var/lib/syncserver/
├── data/
│ └── app.db # SQLite database (machines, sync_pairs, jobs, etc.)
├── ssh/
│ ├── id_ed25519 # Server's private SSH key
│ ├── id_ed25519.pub # Server's public SSH key
│ └── known_hosts # Host keys of registered machines
└── logs/
└── <job_id>.log # Per-job rsync output logs
```
**Recommended**: Bind mount `/var/lib/syncserver` to persistent storage outside the LXC root filesystem. This way you can recreate the LXC without losing data.
In Proxmox LXC config (`/etc/pve/lxc/<id>.conf`):
```
mp0: /mnt/data/syncserver,mp=/var/lib/syncserver,backup=0
```
## Troubleshooting
### WoL doesn't work
1. Verify the LXC is on the same L2 network as the target
2. Check `ip link` in the LXC shows the bridge interface
3. Try: `tcpdump -i <iface> udp port 9` in the LXC while triggering WoL
4. Verify WoL is enabled in the target machine's BIOS
5. Try sending WoL from another machine on the same subnet to isolate the issue
### SSH connection fails
1. Verify the public key is in `~/.ssh/authorized_keys` on the target
2. Check `sshd` is running on the target machine
3. Try manually: `ssh -i /var/lib/syncserver/ssh/id_ed25519 user@host`
4. Check the target's `sshd_config`: `PubkeyAuthentication yes`, `AuthorizedKeysFile .ssh/authorized_keys`
### rsync fails with "connection unexpectedly closed"
- Usually means SSH key auth failed
- Check the job log for the exact SSH error
- Try running the rsync command manually from the server with `-v` flag
### Machine shows "unknown" status
- Status is updated when a job runs or when you test SSH
- It does not auto-poll; this is by design to avoid network load
## API Reference
All API endpoints require authentication (JWT cookie).
| Method | Path | Description |
|--------|------|-------------|
| POST | `/api/auth/login` | Login with username/password |
| POST | `/api/auth/logout` | Clear session |
| GET | `/api/auth/me` | Current user info |
| GET/POST | `/api/machines` | List/create machines |
| GET/PUT/DELETE | `/api/machines/{id}` | Get/update/delete machine |
| GET/POST | `/api/sync-pairs` | List/create sync pairs |
| GET/PUT/DELETE | `/api/sync-pairs/{id}` | Get/update/delete sync pair |
| POST | `/api/sync-pairs/{id}/run` | Trigger manual sync |
| GET | `/api/jobs` | List jobs |
| POST | `/api/jobs/{id}/cancel` | Cancel a running job |
| GET | `/api/jobs/{id}/log` | Stream job log |
| GET | `/api/jobs/stream` | SSE stream of all job events |
| GET | `/api/settings/pubkey` | Get server's SSH public key |
## License
MIT