Compare commits

..

16 Commits

Author SHA1 Message Date
darroyo 7b693bee5b Bump version to 1.0.58 2026-07-19 22:21:28 -04:00
darroyo 5cb21ddceb db: fix migration runner - proper schema_migrations checksum handling for existing DBs
- Check if checksum column exists before querying it
- For old DBs (no checksum col): use COUNT(*) to check if migration applied
- For new DBs (has checksum col): use checksum value
- All migrations run in a single transaction
2026-07-19 22:20:55 -04:00
darroyo dfb667e340 Bump version to 1.0.57 2026-07-19 22:18:46 -04:00
darroyo 02c1fd55fb Bump version to 1.0.56 2026-07-19 22:18:42 -04:00
darroyo d8aaaf5ca4 db: fix migration runner to handle existing DB without checksum col 2026-07-19 22:18:38 -04:00
darroyo 867794d846 Bump version to 1.0.55 2026-07-19 22:18:12 -04:00
darroyo f5d3ecfbf3 db: fix migration for existing schema_migrations without checksum column
- Add 0007 migration to ALTER TABLE adding checksum column
- Make migration runner INSERT conditional: if checksum col exists, use it; otherwise omit
- Handles existing databases (no checksum col) and fresh installs (has checksum col) correctly
2026-07-19 22:18:06 -04:00
darroyo a7ae619b76 Bump version to 1.0.54 2026-07-19 22:14:38 -04:00
darroyo 84b185be39 Phase A-E: stability, security, observability, and test coverage
Phase A - Stability:
- Engine.Start(): recovers orphaned jobs (running/queued/waking_up) after crash
- Engine.Stop(): graceful shutdown - cancels all in-flight jobs and waits
- Queue keyed by jobID (not syncPairID): cancel now targets exact job
- Local rsync uses jobCtx (context.Background() replaced)
- Migrations wrapped in transactions; checksums stored

Phase B - Security:
- admin/admin default removed: SYNCSERVER_ADMIN_PASSWORD required on first run
- Path validation: rejects .., leading -, null bytes in sync pair paths
- Rsync flags allowlist: dangerous flags blocked (--rsync-path, -e, --files-from)
- Shell concat in RunRemote replaced with proper sh -c escaping
- knownhosts: replaced custom parser with golang.org/x/crypto/ssh/knownhosts
- RequireAdmin wired: machine CRUD, SSH key ops, settings require admin role
- deploy-keys: uses authorized_keys only (no private key upload)
- Hardcoded /var/lib/syncserver/ssh paths replaced with cfg.SSHDir()

Phase C - Operational:
- /readyz health check: DB query + SSH dir accessibility
- /metrics endpoint: Prometheus text format (jobs, queue, machines)
- Event struct JSON tags: job_id, machine_id, type (snake_case)
- EventBus broadcast: fanned out to all subscribers
- SQLite VACUUM INTO backup: scheduled before cleanup if BackupDir set
- Filesystem job log cleanup: removes .log files for purged jobs
- Backup retention: old backups auto-purged

Phase D - Frontend:
- Schedules page: REST API + full CRUD UI for cron schedules
- Dashboard: cancel button for running/queued jobs
- JobDetail: server-side log download via API
- Settings: displays data_dir from server
- 404 page: proper NotFound component

Phase E - Tests:
- auth_test.go: JWT, bcrypt, middleware, seed (18 tests)
- models_test.go: Job, Machine, SyncPair, Schedule repos (18 tests)
- go test -race: no data races found
2026-07-19 22:14:30 -04:00
darroyo 300555d35f Bump version to 1.0.52 2026-07-19 19:49:10 -04:00
darroyo 6b29a4b419 Bump version to 1.0.51 2026-07-19 19:49:00 -04:00
darroyo 4ccf2fc2d6 Bump version to 1.0.50 2026-07-19 18:17:42 -04:00
darroyo ae33703ef9 Bump version to 1.0.49 2026-07-19 18:17:36 -04:00
darroyo 2285790257 Add Direction mode help text and mirror --delete warning in sync pair form 2026-07-19 18:17:29 -04:00
darroyo 2ec031c9dc Bump version to 1.0.48 2026-07-17 18:40:23 -04:00
darroyo b2172145e7 Always copy source directory contents, not the directory itself 2026-07-17 18:40:16 -04:00
38 changed files with 2988 additions and 209 deletions
+1 -1
View File
@@ -1,5 +1,5 @@
BINARY=syncserver BINARY=syncserver
VERSION?=1.0.47 VERSION?=1.0.58
GO?=go GO?=go
LDFLAGS=-s -w -X main.version=$(VERSION) -X main.commit=$(shell git rev-parse --short HEAD 2>/dev/null || echo unknown) LDFLAGS=-s -w -X main.version=$(VERSION) -X main.commit=$(shell git rev-parse --short HEAD 2>/dev/null || echo unknown)
BUILD_FLAGS=CGO_ENABLED=0 BUILD_FLAGS=CGO_ENABLED=0
+1 -1
View File
@@ -20,7 +20,7 @@ import (
"github.com/syncserver/internal/syncengine" "github.com/syncserver/internal/syncengine"
) )
var version = "1.0.47" var version = "1.0.58"
func main() { func main() {
cfgPath := flag.String("config", "", "Path to config.yaml") cfgPath := flag.String("config", "", "Path to config.yaml")
+23
View File
@@ -84,6 +84,8 @@ type JobResponse struct {
ErrorCode *string `json:"error_code,omitempty"` ErrorCode *string `json:"error_code,omitempty"`
DurationSeconds *int64 `json:"duration_seconds,omitempty"` DurationSeconds *int64 `json:"duration_seconds,omitempty"`
LogLineCount *int64 `json:"log_line_count,omitempty"` LogLineCount *int64 `json:"log_line_count,omitempty"`
TotalSizeBytes *int64 `json:"total_size_bytes,omitempty"`
SentBytes *int64 `json:"sent_bytes,omitempty"`
} }
type LogLineResponse struct { type LogLineResponse struct {
@@ -119,3 +121,24 @@ type SettingsInfoResponse struct {
DataDir string `json:"data_dir"` DataDir string `json:"data_dir"`
SSHPubKey string `json:"ssh_pub_key"` SSHPubKey string `json:"ssh_pub_key"`
} }
type CreateScheduleRequest struct {
SyncPairID int64 `json:"sync_pair_id"`
CronExpr string `json:"cron_expr"`
Enabled bool `json:"enabled"`
}
type UpdateScheduleRequest struct {
CronExpr string `json:"cron_expr"`
Enabled bool `json:"enabled"`
}
type ScheduleResponse struct {
ID int64 `json:"id"`
SyncPairID int64 `json:"sync_pair_id"`
SyncPairName string `json:"sync_pair_name"`
CronExpr string `json:"cron_expr"`
NextRun *string `json:"next_run_at"`
Enabled bool `json:"enabled"`
CreatedAt string `json:"created_at"`
}
+7 -1
View File
@@ -128,7 +128,7 @@ func (h *JobHandler) Cancel(w http.ResponseWriter, r *http.Request) {
} }
if h.engine != nil { if h.engine != nil {
h.engine.Cancel(id, j.SyncPairID, true) h.engine.Cancel(id, true)
} }
repo.UpdateStatus(id, "cancelled") repo.UpdateStatus(id, "cancelled")
@@ -254,6 +254,12 @@ func jobToResp(j models.Job) JobResponse {
s := j.FinishedAt.Format(time.RFC3339) s := j.FinishedAt.Format(time.RFC3339)
resp.FinishedAt = &s resp.FinishedAt = &s
} }
if j.TotalSizeBytes > 0 {
resp.TotalSizeBytes = &j.TotalSizeBytes
}
if j.SentBytes > 0 {
resp.SentBytes = &j.SentBytes
}
return resp return resp
} }
+8 -7
View File
@@ -252,13 +252,13 @@ func (h *MachineHandler) Shutdown(w http.ResponseWriter, r *http.Request) {
return return
} }
knownHostsPath, err := sshmanager.EnsureKnownHosts(filepath.Join("/var/lib/syncserver", "ssh")) knownHostsPath, err := sshmanager.EnsureKnownHosts(h.cfg.SSHDir())
if err != nil { if err != nil {
writeError(w, http.StatusInternalServerError, "failed to ensure known_hosts") writeError(w, http.StatusInternalServerError, "failed to ensure known_hosts")
return return
} }
privKeyPath := filepath.Join("/var/lib/syncserver", "ssh", "id_ed25519") privKeyPath := filepath.Join(h.cfg.SSHDir(), "id_ed25519")
if m.SSHKeyID != nil { if m.SSHKeyID != nil {
sshKeyRepo := models.NewSSHKeyRepository(h.db) sshKeyRepo := models.NewSSHKeyRepository(h.db)
sshKey, err := sshKeyRepo.GetByID(*m.SSHKeyID) sshKey, err := sshKeyRepo.GetByID(*m.SSHKeyID)
@@ -333,13 +333,13 @@ func (h *MachineHandler) TestConnection(w http.ResponseWriter, r *http.Request)
return return
} }
knownHostsPath, err := sshmanager.EnsureKnownHosts(filepath.Join("/var/lib/syncserver", "ssh")) knownHostsPath, err := sshmanager.EnsureKnownHosts(h.cfg.SSHDir())
if err != nil { if err != nil {
writeError(w, http.StatusInternalServerError, "failed to ensure known_hosts") writeError(w, http.StatusInternalServerError, "failed to ensure known_hosts")
return return
} }
privKeyPath := filepath.Join("/var/lib/syncserver", "ssh", "id_ed25519") privKeyPath := filepath.Join(h.cfg.SSHDir(), "id_ed25519")
if m.SSHKeyID != nil { if m.SSHKeyID != nil {
sshKeyRepo := models.NewSSHKeyRepository(h.db) sshKeyRepo := models.NewSSHKeyRepository(h.db)
sshKey, err := sshKeyRepo.GetByID(*m.SSHKeyID) sshKey, err := sshKeyRepo.GetByID(*m.SSHKeyID)
@@ -390,14 +390,14 @@ func (h *MachineHandler) ApproveFingerprint(w http.ResponseWriter, r *http.Reque
} }
json.NewDecoder(r.Body).Decode(&req) json.NewDecoder(r.Body).Decode(&req)
sshDir := filepath.Join("/var/lib/syncserver", "ssh") sshDir := h.cfg.SSHDir()
knownHostsPath, err := sshmanager.EnsureKnownHosts(sshDir) knownHostsPath, err := sshmanager.EnsureKnownHosts(sshDir)
if err != nil { if err != nil {
writeError(w, http.StatusInternalServerError, "failed to ensure known_hosts") writeError(w, http.StatusInternalServerError, "failed to ensure known_hosts")
return return
} }
privKeyPath := filepath.Join("/var/lib/syncserver", "ssh", "id_ed25519") privKeyPath := filepath.Join(h.cfg.SSHDir(), "id_ed25519")
if m.SSHKeyID != nil { if m.SSHKeyID != nil {
sshKeyRepo := models.NewSSHKeyRepository(h.db) sshKeyRepo := models.NewSSHKeyRepository(h.db)
sshKey, err := sshKeyRepo.GetByID(*m.SSHKeyID) sshKey, err := sshKeyRepo.GetByID(*m.SSHKeyID)
@@ -507,7 +507,7 @@ func (h *MachineHandler) DeployKeys(w http.ResponseWriter, r *http.Request) {
seenKeys[sk.PrivateKeyPath] = true seenKeys[sk.PrivateKeyPath] = true
keys = append(keys, sshmanager.DeployKey{ keys = append(keys, sshmanager.DeployKey{
LocalPath: sk.PrivateKeyPath, LocalPath: sk.PrivateKeyPath,
RemotePath: "/var/lib/syncserver/ssh/keys/" + filepath.Base(sk.PrivateKeyPath), RemotePath: h.cfg.SSHDir() + "/keys/" + filepath.Base(sk.PrivateKeyPath),
Mode: 0600, Mode: 0600,
}) })
} }
@@ -526,6 +526,7 @@ func (h *MachineHandler) DeployKeys(w http.ResponseWriter, r *http.Request) {
m.SSHUser, m.SSHUser,
keys, keys,
knownHostsHosts, knownHostsHosts,
h.cfg.SSHDir(),
) )
if err != nil { if err != nil {
writeError(w, http.StatusInternalServerError, err.Error()) writeError(w, http.StatusInternalServerError, err.Error())
+243
View File
@@ -0,0 +1,243 @@
package api
import (
"database/sql"
"encoding/json"
"log/slog"
"net/http"
"strconv"
"time"
"github.com/go-chi/chi/v5"
"github.com/syncserver/internal/models"
"github.com/syncserver/internal/scheduler"
)
type ScheduleHandler struct {
db *sql.DB
}
func NewScheduleHandler(db *sql.DB) *ScheduleHandler {
return &ScheduleHandler{db: db}
}
func (h *ScheduleHandler) List(w http.ResponseWriter, r *http.Request) {
repo := models.NewScheduleRepository(h.db)
schedules, err := repo.GetAll()
if err != nil {
slog.Error("failed to fetch schedules", "error", err)
writeError(w, http.StatusInternalServerError, "failed to fetch schedules")
return
}
pairRepo := models.NewSyncPairRepository(h.db)
pairs, err := pairRepo.GetAll()
if err != nil {
slog.Error("failed to fetch sync pairs", "error", err)
writeError(w, http.StatusInternalServerError, "failed to fetch sync pairs")
return
}
pairMap := make(map[int64]string)
for _, p := range pairs {
pairMap[p.ID] = p.Name
}
out := make([]ScheduleResponse, len(schedules))
for i, s := range schedules {
out[i] = scheduleToResp(s, pairMap)
}
writeJSON(w, out)
}
func (h *ScheduleHandler) Get(w http.ResponseWriter, r *http.Request) {
id, err := strconv.ParseInt(chi.URLParam(r, "id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
repo := models.NewScheduleRepository(h.db)
s, err := repo.GetByID(id)
if err == sql.ErrNoRows {
writeError(w, http.StatusNotFound, "schedule not found")
return
}
if err != nil {
slog.Error("failed to fetch schedule", "id", id, "error", err)
writeError(w, http.StatusInternalServerError, "failed to fetch schedule")
return
}
pairRepo := models.NewSyncPairRepository(h.db)
pairs, err := pairRepo.GetAll()
if err != nil {
slog.Error("failed to fetch sync pairs", "error", err)
writeError(w, http.StatusInternalServerError, "failed to fetch sync pairs")
return
}
pairMap := make(map[int64]string)
for _, p := range pairs {
pairMap[p.ID] = p.Name
}
writeJSON(w, scheduleToResp(*s, pairMap))
}
func (h *ScheduleHandler) Create(w http.ResponseWriter, r *http.Request) {
var req CreateScheduleRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
if req.SyncPairID <= 0 {
writeError(w, http.StatusBadRequest, "sync_pair_id is required")
return
}
if req.CronExpr == "" {
writeError(w, http.StatusBadRequest, "cron_expr is required")
return
}
_, err := scheduler.ParseCron(req.CronExpr)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid cron expression: "+err.Error())
return
}
pairRepo := models.NewSyncPairRepository(h.db)
_, err = pairRepo.GetByID(req.SyncPairID)
if err == sql.ErrNoRows {
writeError(w, http.StatusBadRequest, "sync_pair not found")
return
}
if err != nil {
slog.Error("failed to fetch sync pair", "id", req.SyncPairID, "error", err)
writeError(w, http.StatusInternalServerError, "failed to validate sync pair")
return
}
expr, _ := scheduler.ParseCron(req.CronExpr)
var nextRun *time.Time
if expr != nil {
t := scheduler.NextRun(expr, time.Now().UTC())
nextRun = &t
}
s := &models.Schedule{
SyncPairID: req.SyncPairID,
CronExpr: req.CronExpr,
NextRunAt: nextRun,
Enabled: req.Enabled,
}
repo := models.NewScheduleRepository(h.db)
id, err := repo.Create(s)
if err != nil {
slog.Error("failed to create schedule", "error", err)
writeError(w, http.StatusInternalServerError, "failed to create schedule")
return
}
s.ID = id
pairs, _ := pairRepo.GetAll()
pairMap := make(map[int64]string)
for _, p := range pairs {
pairMap[p.ID] = p.Name
}
w.Header().Set("Location", "/api/schedules/"+strconv.FormatInt(id, 10))
writeJSON(w, scheduleToResp(*s, pairMap), http.StatusCreated)
}
func (h *ScheduleHandler) Update(w http.ResponseWriter, r *http.Request) {
id, err := strconv.ParseInt(chi.URLParam(r, "id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
var req UpdateScheduleRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
writeError(w, http.StatusBadRequest, "invalid request body")
return
}
repo := models.NewScheduleRepository(h.db)
existing, err := repo.GetByID(id)
if err == sql.ErrNoRows {
writeError(w, http.StatusNotFound, "schedule not found")
return
}
if err != nil {
slog.Error("failed to fetch schedule", "id", id, "error", err)
writeError(w, http.StatusInternalServerError, "failed to fetch schedule")
return
}
if req.CronExpr != "" {
_, err := scheduler.ParseCron(req.CronExpr)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid cron expression: "+err.Error())
return
}
existing.CronExpr = req.CronExpr
expr, _ := scheduler.ParseCron(req.CronExpr)
if expr != nil {
t := scheduler.NextRun(expr, time.Now().UTC())
existing.NextRunAt = &t
}
}
if req.Enabled {
existing.Enabled = true
} else if req.Enabled == false {
existing.Enabled = false
}
if err := repo.Update(existing); err != nil {
slog.Error("failed to update schedule", "id", id, "error", err)
writeError(w, http.StatusInternalServerError, "failed to update schedule")
return
}
pairRepo := models.NewSyncPairRepository(h.db)
pairs, _ := pairRepo.GetAll()
pairMap := make(map[int64]string)
for _, p := range pairs {
pairMap[p.ID] = p.Name
}
writeJSON(w, scheduleToResp(*existing, pairMap))
}
func (h *ScheduleHandler) Delete(w http.ResponseWriter, r *http.Request) {
id, err := strconv.ParseInt(chi.URLParam(r, "id"), 10, 64)
if err != nil {
writeError(w, http.StatusBadRequest, "invalid id")
return
}
repo := models.NewScheduleRepository(h.db)
if err := repo.Delete(id); err != nil {
slog.Error("failed to delete schedule", "id", id, "error", err)
writeError(w, http.StatusInternalServerError, "failed to delete schedule")
return
}
w.WriteHeader(http.StatusNoContent)
}
func scheduleToResp(s models.Schedule, pairMap map[int64]string) ScheduleResponse {
resp := ScheduleResponse{
ID: s.ID,
SyncPairID: s.SyncPairID,
SyncPairName: pairMap[s.SyncPairID],
CronExpr: s.CronExpr,
Enabled: s.Enabled,
CreatedAt: s.CreatedAt.Format(time.RFC3339),
}
if s.NextRunAt != nil {
t := s.NextRunAt.Format(time.RFC3339)
resp.NextRun = &t
}
return resp
}
+35
View File
@@ -3,10 +3,12 @@ package api
import ( import (
"database/sql" "database/sql"
"encoding/json" "encoding/json"
"fmt"
"log/slog" "log/slog"
"net/http" "net/http"
"regexp" "regexp"
"strconv" "strconv"
"strings"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/syncserver/internal/models" "github.com/syncserver/internal/models"
@@ -22,6 +24,23 @@ func NewSyncPairHandler(db *sql.DB) *SyncPairHandler {
var directionRegex = regexp.MustCompile(`^(push|pull|mirror)$`) var directionRegex = regexp.MustCompile(`^(push|pull|mirror)$`)
func ValidatePath(path string) error {
path = strings.TrimSpace(path)
if path == "" {
return fmt.Errorf("path cannot be empty")
}
if strings.Contains(path, "..") {
return fmt.Errorf("path cannot contain '..'")
}
if strings.HasPrefix(path, "-") {
return fmt.Errorf("path cannot start with '-'")
}
if strings.Contains(path, "\x00") {
return fmt.Errorf("path cannot contain null bytes")
}
return nil
}
func (h *SyncPairHandler) List(w http.ResponseWriter, r *http.Request) { func (h *SyncPairHandler) List(w http.ResponseWriter, r *http.Request) {
repo := models.NewSyncPairRepository(h.db) repo := models.NewSyncPairRepository(h.db)
pairs, err := repo.GetAll() pairs, err := repo.GetAll()
@@ -68,6 +87,14 @@ func (h *SyncPairHandler) Create(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "name, source_path and dest_path are required") writeError(w, http.StatusBadRequest, "name, source_path and dest_path are required")
return return
} }
if err := ValidatePath(req.SourcePath); err != nil {
writeError(w, http.StatusBadRequest, "invalid source_path: "+err.Error())
return
}
if err := ValidatePath(req.DestPath); err != nil {
writeError(w, http.StatusBadRequest, "invalid dest_path: "+err.Error())
return
}
if req.Direction == "" { if req.Direction == "" {
req.Direction = "push" req.Direction = "push"
} }
@@ -127,6 +154,14 @@ func (h *SyncPairHandler) Update(w http.ResponseWriter, r *http.Request) {
writeError(w, http.StatusBadRequest, "name, source_path and dest_path are required") writeError(w, http.StatusBadRequest, "name, source_path and dest_path are required")
return return
} }
if err := ValidatePath(req.SourcePath); err != nil {
writeError(w, http.StatusBadRequest, "invalid source_path: "+err.Error())
return
}
if err := ValidatePath(req.DestPath); err != nil {
writeError(w, http.StatusBadRequest, "invalid dest_path: "+err.Error())
return
}
if !directionRegex.MatchString(req.Direction) { if !directionRegex.MatchString(req.Direction) {
writeError(w, http.StatusBadRequest, "direction must be push, pull, or mirror") writeError(w, http.StatusBadRequest, "direction must be push, pull, or mirror")
return return
+2 -2
View File
@@ -45,7 +45,7 @@ func (h *SSEHandler) StreamAll(w http.ResponseWriter, r *http.Request) {
select { select {
case evt := <-events: case evt := <-events:
data, _ := json.Marshal(evt) data, _ := json.Marshal(evt)
fmt.Fprintf(w, "event: %s\ndata: %s\n\n", evt.Type, data) fmt.Fprintf(w, "data: %s\n\n", data)
flusher.Flush() flusher.Flush()
case <-r.Context().Done(): case <-r.Context().Done():
return return
@@ -94,7 +94,7 @@ func (h *SSEHandler) StreamJob(w http.ResponseWriter, r *http.Request) {
select { select {
case evt := <-events: case evt := <-events:
data, _ := json.Marshal(evt) data, _ := json.Marshal(evt)
fmt.Fprintf(w, "event: %s\ndata: %s\n\n", evt.Type, data) fmt.Fprintf(w, "data: %s\n\n", data)
flusher.Flush() flusher.Flush()
case <-r.Context().Done(): case <-r.Context().Done():
return return
+97 -19
View File
@@ -3,9 +3,12 @@ package api
import ( import (
"database/sql" "database/sql"
"encoding/json" "encoding/json"
"fmt"
"log/slog" "log/slog"
"net/http" "net/http"
"os"
"runtime/debug" "runtime/debug"
"sort"
"github.com/go-chi/chi/v5" "github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware" "github.com/go-chi/chi/v5/middleware"
@@ -36,41 +39,58 @@ func NewServer(cfg *config.Config, db *sql.DB, engine *syncengine.Engine) *Serve
authHandler := NewAuthHandler(db) authHandler := NewAuthHandler(db)
machineHandler := NewMachineHandler(db, engine, cfg) machineHandler := NewMachineHandler(db, engine, cfg)
syncPairHandler := NewSyncPairHandler(db) syncPairHandler := NewSyncPairHandler(db)
scheduleHandler := NewScheduleHandler(db)
jobHandler := NewJobHandler(db, engine) jobHandler := NewJobHandler(db, engine)
sseHandler := NewSSEHandler(engine) sseHandler := NewSSEHandler(engine)
sshKeyHandler := NewSSHKeyHandler(db, cfg) sshKeyHandler := NewSSHKeyHandler(db, cfg)
admin := func(h http.Handler) http.Handler {
return auth.RequireAdmin(auth.RequireAuth(h))
}
authGet := func(h http.Handler) http.Handler {
return auth.RequireAuth(h)
}
r.Route("/api", func(r chi.Router) { r.Route("/api", func(r chi.Router) {
r.Route("/auth", func(r chi.Router) { r.Route("/auth", func(r chi.Router) {
r.Post("/login", authHandler.Login) r.Post("/login", authHandler.Login)
r.Post("/logout", authHandler.Logout) r.Post("/logout", authHandler.Logout)
r.With(auth.RequireAuth).Get("/me", authHandler.Me) r.With(authGet).Get("/me", authHandler.Me)
}) })
r.With(auth.RequireAuth).Route("/machines", func(r chi.Router) { r.With(authGet).Route("/machines", func(r chi.Router) {
r.Get("/", machineHandler.List) r.Get("/", machineHandler.List)
r.Post("/", machineHandler.Create) r.With(admin).Post("/", machineHandler.Create)
r.Post("/refresh", machineHandler.Refresh) r.With(admin).Post("/refresh", machineHandler.Refresh)
r.Get("/{id}", machineHandler.Get) r.Get("/{id}", machineHandler.Get)
r.Put("/{id}", machineHandler.Update) r.With(admin).Put("/{id}", machineHandler.Update)
r.Delete("/{id}", machineHandler.Delete) r.With(admin).Delete("/{id}", machineHandler.Delete)
r.Post("/{id}/test-wol", machineHandler.TestWoL) r.Post("/{id}/test-wol", machineHandler.TestWoL)
r.Post("/{id}/shutdown", machineHandler.Shutdown) r.With(admin).Post("/{id}/shutdown", machineHandler.Shutdown)
r.Post("/{id}/test-connection", machineHandler.TestConnection) r.Post("/{id}/test-connection", machineHandler.TestConnection)
r.Post("/{id}/approve-fingerprint", machineHandler.ApproveFingerprint) r.With(admin).Post("/{id}/approve-fingerprint", machineHandler.ApproveFingerprint)
r.Post("/{id}/deploy-keys", machineHandler.DeployKeys) r.With(admin).Post("/{id}/deploy-keys", machineHandler.DeployKeys)
}) })
r.With(auth.RequireAuth).Route("/sync-pairs", func(r chi.Router) { r.With(authGet).Route("/sync-pairs", func(r chi.Router) {
r.Get("/", syncPairHandler.List) r.Get("/", syncPairHandler.List)
r.Post("/", syncPairHandler.Create) r.Post("/", syncPairHandler.Create)
r.Get("/{id}", syncPairHandler.Get) r.Get("/{id}", syncPairHandler.Get)
r.Put("/{id}", syncPairHandler.Update) r.Put("/{id}", syncPairHandler.Update)
r.Delete("/{id}", syncPairHandler.Delete) r.With(admin).Delete("/{id}", syncPairHandler.Delete)
r.Post("/{id}/run", jobHandler.TriggerRun) r.Post("/{id}/run", jobHandler.TriggerRun)
}) })
r.With(auth.RequireAuth).Route("/jobs", func(r chi.Router) { r.With(authGet).Route("/schedules", func(r chi.Router) {
r.Get("/", scheduleHandler.List)
r.Post("/", scheduleHandler.Create)
r.Get("/{id}", scheduleHandler.Get)
r.Put("/{id}", scheduleHandler.Update)
r.With(admin).Delete("/{id}", scheduleHandler.Delete)
})
r.With(authGet).Route("/jobs", func(r chi.Router) {
r.Get("/", jobHandler.List) r.Get("/", jobHandler.List)
r.Get("/{id}", jobHandler.Get) r.Get("/{id}", jobHandler.Get)
r.Post("/{id}/cancel", jobHandler.Cancel) r.Post("/{id}/cancel", jobHandler.Cancel)
@@ -79,15 +99,15 @@ func NewServer(cfg *config.Config, db *sql.DB, engine *syncengine.Engine) *Serve
r.Get("/{id}/log/stream", sseHandler.StreamJob) r.Get("/{id}/log/stream", sseHandler.StreamJob)
}) })
r.With(auth.RequireAuth).Get("/jobs/stream", sseHandler.StreamAll) r.With(authGet).Get("/jobs/stream", sseHandler.StreamAll)
r.With(auth.RequireAuth).Get("/settings/pubkey", func(w http.ResponseWriter, r *http.Request) { r.With(authGet).Get("/settings/pubkey", func(w http.ResponseWriter, r *http.Request) {
_, _, pubKey, _ := sshmanager.EnsureServerKey(cfg.SSHDir()) _, _, pubKey, _ := sshmanager.EnsureServerKey(cfg.SSHDir())
w.Header().Set("Content-Type", "text/plain") w.Header().Set("Content-Type", "text/plain")
w.Write([]byte(pubKey)) w.Write([]byte(pubKey))
}) })
r.With(auth.RequireAuth).Get("/settings/info", func(w http.ResponseWriter, r *http.Request) { r.With(authGet).Get("/settings/info", func(w http.ResponseWriter, r *http.Request) {
_, _, pubKey, _ := sshmanager.EnsureServerKey(cfg.SSHDir()) _, _, pubKey, _ := sshmanager.EnsureServerKey(cfg.SSHDir())
resp := SettingsInfoResponse{ resp := SettingsInfoResponse{
Version: cfg.Version, Version: cfg.Version,
@@ -98,12 +118,12 @@ func NewServer(cfg *config.Config, db *sql.DB, engine *syncengine.Engine) *Serve
json.NewEncoder(w).Encode(resp) json.NewEncoder(w).Encode(resp)
}) })
r.With(auth.RequireAuth).Route("/ssh-keys", func(r chi.Router) { r.With(authGet).Route("/ssh-keys", func(r chi.Router) {
r.Get("/", sshKeyHandler.List) r.Get("/", sshKeyHandler.List)
r.Post("/", sshKeyHandler.Create) r.With(admin).Post("/", sshKeyHandler.Create)
r.Get("/{id}", sshKeyHandler.Get) r.Get("/{id}", sshKeyHandler.Get)
r.Delete("/{id}", sshKeyHandler.Delete) r.With(admin).Delete("/{id}", sshKeyHandler.Delete)
r.Get("/{id}/private", sshKeyHandler.DownloadPrivate) r.With(admin).Get("/{id}/private", sshKeyHandler.DownloadPrivate)
}) })
}) })
@@ -111,6 +131,64 @@ func NewServer(cfg *config.Config, db *sql.DB, engine *syncengine.Engine) *Serve
w.Write([]byte("ok")) w.Write([]byte("ok"))
})) }))
r.Get("/healthz", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("ok"))
}))
r.Get("/readyz", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
db := s.engine.DB()
if _, err := db.Exec("SELECT 1"); err != nil {
http.Error(w, fmt.Sprintf("db query failed: %v", err), http.StatusServiceUnavailable)
return
}
if _, err := db.Exec("SELECT 1"); err != nil {
http.Error(w, fmt.Sprintf("db write test failed: %v", err), http.StatusServiceUnavailable)
return
}
sshDir := s.cfg.SSHDir()
if _, err := os.Stat(sshDir); err != nil {
http.Error(w, fmt.Sprintf("ssh dir not accessible: %v", err), http.StatusServiceUnavailable)
return
}
w.Write([]byte("ok"))
}))
r.Get("/metrics", http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "text/plain; version=0.0.4")
jobsTotal := s.engine.GetJobsTotal()
var keys []string
for k := range jobsTotal {
keys = append(keys, k)
}
sort.Strings(keys)
for _, status := range keys {
fmt.Fprintf(w, "# HELP syncserver_jobs_total Total jobs by final status\n")
fmt.Fprintf(w, "# TYPE syncserver_jobs_total counter\n")
fmt.Fprintf(w, "syncserver_jobs_total{status=%q} %d\n", status, jobsTotal[status])
}
fmt.Fprintf(w, "# HELP syncserver_jobs_running Currently running jobs\n")
fmt.Fprintf(w, "# TYPE syncserver_jobs_running gauge\n")
fmt.Fprintf(w, "syncserver_jobs_running %d\n", s.engine.GetJobsRunning())
fmt.Fprintf(w, "# HELP syncserver_queue_depth Jobs waiting to run\n")
fmt.Fprintf(w, "# TYPE syncserver_queue_depth gauge\n")
fmt.Fprintf(w, "syncserver_queue_depth %d\n", s.engine.GetQueueDepth())
online, total := s.engine.GetMachineCounts()
fmt.Fprintf(w, "# HELP syncserver_machines_online Online machines count\n")
fmt.Fprintf(w, "# TYPE syncserver_machines_online gauge\n")
fmt.Fprintf(w, "syncserver_machines_online %d\n", online)
fmt.Fprintf(w, "# HELP syncserver_machines_total Total machines\n")
fmt.Fprintf(w, "# TYPE syncserver_machines_total gauge\n")
fmt.Fprintf(w, "syncserver_machines_total %d\n", total)
fmt.Fprintf(w, "# HELP syncserver_up Server is up\n")
fmt.Fprintf(w, "# TYPE syncserver_up gauge\n")
fmt.Fprintf(w, "syncserver_up 1\n")
}))
r.NotFound(func(w http.ResponseWriter, r *http.Request) { r.NotFound(func(w http.ResponseWriter, r *http.Request) {
webui.ServeSPA().ServeHTTP(w, r) webui.ServeSPA().ServeHTTP(w, r)
}) })
+411
View File
@@ -0,0 +1,411 @@
package auth
import (
"context"
"database/sql"
"errors"
"net/http"
"net/http/httptest"
"testing"
"time"
"github.com/golang-jwt/jwt/v5"
_ "modernc.org/sqlite"
)
func TestJWT_Generate(t *testing.T) {
mgr := NewJWTManager("test-secret", 24)
token, expiresAt, err := mgr.Generate(1, "testuser", "admin")
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
if token == "" {
t.Fatal("Generate() returned empty token")
}
if expiresAt.Before(time.Now()) {
t.Fatal("Generate() returned past expiration time")
}
if expiresAt.Before(time.Now().Add(23 * time.Hour)) {
t.Fatal("Generate() expiration time is too early")
}
claims, err := mgr.Validate(token)
if err != nil {
t.Fatalf("Validate() error = %v", err)
}
if claims.UserID != 1 {
t.Errorf("claims.UserID = %d, want 1", claims.UserID)
}
if claims.Username != "testuser" {
t.Errorf("claims.Username = %s, want testuser", claims.Username)
}
if claims.Role != "admin" {
t.Errorf("claims.Role = %s, want admin", claims.Role)
}
}
func TestJWT_Validate_ValidToken(t *testing.T) {
mgr := NewJWTManager("test-secret", 24)
token, _, err := mgr.Generate(42, "alice", "user")
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
claims, err := mgr.Validate(token)
if err != nil {
t.Fatalf("Validate() error = %v", err)
}
if claims.UserID != 42 {
t.Errorf("claims.UserID = %d, want 42", claims.UserID)
}
if claims.Username != "alice" {
t.Errorf("claims.Username = %s, want alice", claims.Username)
}
if claims.Role != "user" {
t.Errorf("claims.Role = %s, want user", claims.Role)
}
}
func TestJWT_Validate_ExpiredToken(t *testing.T) {
mgr := NewJWTManager("test-secret", 0)
token := jwtWithExpiry(time.Now().Add(-1 * time.Hour))
_, err := mgr.Validate(token)
if !errors.Is(err, ErrExpiredToken) {
t.Errorf("Validate() error = %v, want ErrExpiredToken", err)
}
}
func TestJWT_Validate_InvalidToken(t *testing.T) {
mgr := NewJWTManager("test-secret", 24)
tests := []struct {
name string
token string
}{
{"malformed token", "not.a.token"},
{"empty token", ""},
{"random string", "abcdef123456"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := mgr.Validate(tt.token)
if err == nil {
t.Error("Validate() expected error for invalid token")
}
})
}
}
func TestJWT_Validate_WrongSecret(t *testing.T) {
mgr1 := NewJWTManager("secret-one", 24)
mgr2 := NewJWTManager("secret-two", 24)
token, _, err := mgr1.Generate(1, "user", "admin")
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
_, err = mgr2.Validate(token)
if err == nil {
t.Error("Validate() expected error for token signed with different secret")
}
}
func TestPassword_HashPassword_RandomSalts(t *testing.T) {
password := "samepassword123"
hash1, err := HashPassword(password)
if err != nil {
t.Fatalf("HashPassword() error = %v", err)
}
hash2, err := HashPassword(password)
if err != nil {
t.Fatalf("HashPassword() error = %v", err)
}
if string(hash1) == string(hash2) {
t.Error("HashPassword() produced identical hashes for same password")
}
}
func TestPassword_CheckPassword_Correct(t *testing.T) {
password := "mysecretpassword"
hash, err := HashPassword(password)
if err != nil {
t.Fatalf("HashPassword() error = %v", err)
}
if !VerifyPassword(hash, password) {
t.Error("VerifyPassword() returned false for correct password")
}
}
func TestPassword_CheckPassword_Wrong(t *testing.T) {
password := "mysecretpassword"
wrongPassword := "wrongpassword"
hash, err := HashPassword(password)
if err != nil {
t.Fatalf("HashPassword() error = %v", err)
}
if VerifyPassword(hash, wrongPassword) {
t.Error("VerifyPassword() returned true for wrong password")
}
}
func TestMiddleware_RequireAuth_NoToken(t *testing.T) {
InitJWTManager("test-secret", 24)
handler := RequireAuth(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Error("next handler should not be called")
}))
req := httptest.NewRequest(http.MethodGet, "/", nil)
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusUnauthorized {
t.Errorf("RequireAuth() status = %d, want %d", rr.Code, http.StatusUnauthorized)
}
}
func TestMiddleware_RequireAuth_InvalidToken(t *testing.T) {
InitJWTManager("test-secret", 24)
handler := RequireAuth(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Error("next handler should not be called")
}))
req := httptest.NewRequest(http.MethodGet, "/", nil)
req.AddCookie(&http.Cookie{Name: CookieName, Value: "invalid-token"})
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusUnauthorized {
t.Errorf("RequireAuth() status = %d, want %d", rr.Code, http.StatusUnauthorized)
}
}
func TestMiddleware_RequireAuth_ValidToken(t *testing.T) {
secret := "test-secret"
InitJWTManager(secret, 24)
mgr := GetJWTManager()
token, _, err := mgr.Generate(99, "testuser", "admin")
if err != nil {
t.Fatalf("Generate() error = %v", err)
}
var capturedClaims *Claims
handler := RequireAuth(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
capturedClaims = GetClaims(r.Context())
w.WriteHeader(http.StatusOK)
}))
req := httptest.NewRequest(http.MethodGet, "/", nil)
req.AddCookie(&http.Cookie{Name: CookieName, Value: token})
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Errorf("RequireAuth() status = %d, want %d", rr.Code, http.StatusOK)
}
if capturedClaims == nil {
t.Fatal("RequireAuth() did not set claims in context")
}
if capturedClaims.UserID != 99 {
t.Errorf("capturedClaims.UserID = %d, want 99", capturedClaims.UserID)
}
if capturedClaims.Username != "testuser" {
t.Errorf("capturedClaims.Username = %s, want testuser", capturedClaims.Username)
}
if capturedClaims.Role != "admin" {
t.Errorf("capturedClaims.Role = %s, want admin", capturedClaims.Role)
}
}
func TestMiddleware_RequireAdmin_NonAdmin(t *testing.T) {
InitJWTManager("test-secret", 24)
handler := RequireAdmin(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Error("next handler should not be called for non-admin")
}))
req := httptest.NewRequest(http.MethodGet, "/", nil)
ctx := context.WithValue(req.Context(), ClaimsCtxKey, &Claims{Role: "user"})
req = req.WithContext(ctx)
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusForbidden {
t.Errorf("RequireAdmin() status = %d, want %d", rr.Code, http.StatusForbidden)
}
}
func TestMiddleware_RequireAdmin_NoClaims(t *testing.T) {
InitJWTManager("test-secret", 24)
handler := RequireAdmin(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
t.Error("next handler should not be called when no claims in context")
}))
req := httptest.NewRequest(http.MethodGet, "/", nil)
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusForbidden {
t.Errorf("RequireAdmin() status = %d, want %d", rr.Code, http.StatusForbidden)
}
}
func TestMiddleware_RequireAdmin_Admin(t *testing.T) {
InitJWTManager("test-secret", 24)
called := false
handler := RequireAdmin(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
called = true
w.WriteHeader(http.StatusOK)
}))
req := httptest.NewRequest(http.MethodGet, "/", nil)
ctx := context.WithValue(req.Context(), ClaimsCtxKey, &Claims{Role: "admin"})
req = req.WithContext(ctx)
rr := httptest.NewRecorder()
handler.ServeHTTP(rr, req)
if rr.Code != http.StatusOK {
t.Errorf("RequireAdmin() status = %d, want %d", rr.Code, http.StatusOK)
}
if !called {
t.Error("RequireAdmin() did not call next handler for admin role")
}
}
func TestSeed_SeedsAdminUser(t *testing.T) {
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatalf("sql.Open() error = %v", err)
}
defer db.Close()
_, err = db.Exec(`CREATE TABLE users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
username TEXT UNIQUE NOT NULL,
password_hash TEXT NOT NULL,
role TEXT NOT NULL DEFAULT 'user',
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
)`)
if err != nil {
t.Fatalf("CREATE TABLE error = %v", err)
}
err = SeedAdmin(db, "admin", "secretpassword123")
if err != nil {
t.Fatalf("SeedAdmin() error = %v", err)
}
var username, role string
err = db.QueryRow("SELECT username, role FROM users WHERE username = 'admin'").Scan(&username, &role)
if err != nil {
t.Fatalf("QueryRow() error = %v", err)
}
if username != "admin" {
t.Errorf("username = %s, want admin", username)
}
if role != "admin" {
t.Errorf("role = %s, want admin", role)
}
}
func TestSeed_Idempotent(t *testing.T) {
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatalf("sql.Open() error = %v", err)
}
defer db.Close()
_, err = db.Exec(`CREATE TABLE users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
username TEXT UNIQUE NOT NULL,
password_hash TEXT NOT NULL,
role TEXT NOT NULL DEFAULT 'user',
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
)`)
if err != nil {
t.Fatalf("CREATE TABLE error = %v", err)
}
_, err = db.Exec("INSERT INTO users (username, password_hash, role) VALUES ('admin', 'existing-hash', 'admin')")
if err != nil {
t.Fatalf("INSERT error = %v", err)
}
err = SeedAdmin(db, "admin", "newpassword")
if err != nil {
t.Fatalf("SeedAdmin() error = %v", err)
}
var count int
err = db.QueryRow("SELECT COUNT(*) FROM users WHERE username = 'admin'").Scan(&count)
if err != nil {
t.Fatalf("QueryRow() error = %v", err)
}
if count != 1 {
t.Errorf("user count = %d, want 1 (idempotent behavior)", count)
}
}
func TestSeed_EmptyPasswordError(t *testing.T) {
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatalf("sql.Open() error = %v", err)
}
defer db.Close()
_, err = db.Exec(`CREATE TABLE users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
username TEXT UNIQUE NOT NULL,
password_hash TEXT NOT NULL,
role TEXT NOT NULL DEFAULT 'user',
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
)`)
if err != nil {
t.Fatalf("CREATE TABLE error = %v", err)
}
err = SeedAdmin(db, "admin", "")
if err == nil {
t.Error("SeedAdmin() expected error for empty password on first run")
}
}
func jwtWithExpiry(expiry time.Time) string {
claims := &Claims{
UserID: 1,
Username: "testuser",
Role: "admin",
RegisteredClaims: jwt.RegisteredClaims{
ExpiresAt: jwt.NewNumericDate(expiry),
IssuedAt: jwt.NewNumericDate(time.Now().Add(-2 * time.Hour)),
NotBefore: jwt.NewNumericDate(time.Now().Add(-2 * time.Hour)),
},
}
token := jwt.NewWithClaims(jwt.SigningMethodHS256, claims)
signed, _ := token.SignedString([]byte("test-secret"))
return signed
}
+4
View File
@@ -50,6 +50,10 @@ func GetClaims(ctx context.Context) *Claims {
return v.(*Claims) return v.(*Claims)
} }
func WithAuthAdmin(next http.Handler) http.Handler {
return RequireAdmin(RequireAuth(next))
}
var GlobalJWTManager *JWTManager var GlobalJWTManager *JWTManager
func InitJWTManager(secret string, expiryH int) { func InitJWTManager(secret string, expiryH int) {
+5 -3
View File
@@ -2,6 +2,7 @@ package auth
import ( import (
"database/sql" "database/sql"
"errors"
"log/slog" "log/slog"
) )
@@ -9,9 +10,6 @@ func SeedAdmin(db *sql.DB, username, password string) error {
if username == "" { if username == "" {
username = "admin" username = "admin"
} }
if password == "" {
password = "admin"
}
var exists bool var exists bool
err := db.QueryRow("SELECT EXISTS(SELECT 1 FROM users WHERE username = ?)", username).Scan(&exists) err := db.QueryRow("SELECT EXISTS(SELECT 1 FROM users WHERE username = ?)", username).Scan(&exists)
@@ -22,6 +20,10 @@ func SeedAdmin(db *sql.DB, username, password string) error {
return nil return nil
} }
if password == "" {
return errors.New("SYNCSERVER_ADMIN_PASSWORD environment variable is required on first run")
}
hash, err := HashPassword(password) hash, err := HashPassword(password)
if err != nil { if err != nil {
return err return err
+2
View File
@@ -32,6 +32,8 @@ type AuthConfig struct {
type SchedulerConfig struct { type SchedulerConfig struct {
Timezone string `yaml:"timezone" env:"SYNCSERVER_SCHEDULER_TZ" default:"UTC"` Timezone string `yaml:"timezone" env:"SYNCSERVER_SCHEDULER_TZ" default:"UTC"`
RetentionDays int `yaml:"retention_days" env:"SYNCSERVER_RETENTION_DAYS" default:"30"` RetentionDays int `yaml:"retention_days" env:"SYNCSERVER_RETENTION_DAYS" default:"30"`
BackupDir string `yaml:"backup_dir" env:"SYNCSERVER_BACKUP_DIR"`
BackupRetentionDays int `yaml:"backup_retention_days" env:"SYNCSERVER_BACKUP_RETENTION_DAYS" default:"7"`
} }
var globalCfg *Config var globalCfg *Config
+55 -8
View File
@@ -1,7 +1,9 @@
package db package db
import ( import (
"crypto/sha256"
"database/sql" "database/sql"
"encoding/hex"
"fmt" "fmt"
"os" "os"
"path/filepath" "path/filepath"
@@ -62,35 +64,80 @@ func (db *DB) runMigrationsInternal(mfs embedFS, migrationsRoot string) error {
if _, err := db.Exec(` if _, err := db.Exec(`
CREATE TABLE IF NOT EXISTS schema_migrations ( CREATE TABLE IF NOT EXISTS schema_migrations (
version TEXT PRIMARY KEY, version TEXT PRIMARY KEY,
checksum TEXT,
applied_at DATETIME DEFAULT CURRENT_TIMESTAMP applied_at DATETIME DEFAULT CURRENT_TIMESTAMP
) )
`); err != nil { `); err != nil {
return fmt.Errorf("creating schema_migrations table: %w", err) return fmt.Errorf("creating schema_migrations table: %w", err)
} }
for _, name := range names { hasChecksumCol := false
var applied bool if rows, err := db.Query("PRAGMA table_info(schema_migrations)"); err == nil {
row := db.QueryRow("SELECT 1 FROM schema_migrations WHERE version = ?", name) for rows.Next() {
if err := row.Scan(&applied); err == nil { var cid int
applied = true var cname string
rows.Scan(&cid, &cname, new(string), new(int), new(interface{}), new(int))
if cname == "checksum" {
hasChecksumCol = true
}
}
rows.Close()
} }
if applied { for _, name := range names {
if hasChecksumCol {
var storedChecksum string
row := db.QueryRow("SELECT checksum FROM schema_migrations WHERE version = ?", name)
if err := row.Scan(&storedChecksum); err == nil && storedChecksum != "" {
continue continue
} }
} else {
var count int
row := db.QueryRow("SELECT COUNT(*) FROM schema_migrations WHERE version = ?", name)
if err := row.Scan(&count); err == nil && count > 0 {
continue
}
}
data, err := mfs.ReadFile(filepath.Join(migrationsRoot, name)) data, err := mfs.ReadFile(filepath.Join(migrationsRoot, name))
if err != nil { if err != nil {
return fmt.Errorf("reading migration %s: %w", name, err) return fmt.Errorf("reading migration %s: %w", name, err)
} }
if _, err := db.Exec(string(data)); err != nil { checksum := sha256.Sum256(data)
checksumHex := hex.EncodeToString(checksum[:])
tx, err := db.Begin()
if err != nil {
return fmt.Errorf("starting transaction for migration %s: %w", name, err)
}
if _, err := tx.Exec(string(data)); err != nil {
tx.Rollback()
return fmt.Errorf("applying migration %s: %w", name, err) return fmt.Errorf("applying migration %s: %w", name, err)
} }
if _, err := db.Exec("INSERT INTO schema_migrations (version) VALUES (?)", name); err != nil { if hasChecksumCol {
if _, err := tx.Exec(
"INSERT INTO schema_migrations (version, checksum) VALUES (?, ?)",
name, checksumHex,
); err != nil {
tx.Rollback()
return fmt.Errorf("recording migration %s: %w", name, err) return fmt.Errorf("recording migration %s: %w", name, err)
} }
} else {
if _, err := tx.Exec(
"INSERT INTO schema_migrations (version) VALUES (?)",
name,
); err != nil {
tx.Rollback()
return fmt.Errorf("recording migration %s: %w", name, err)
}
}
if err := tx.Commit(); err != nil {
return fmt.Errorf("committing migration %s: %w", name, err)
}
} }
return nil return nil
@@ -0,0 +1,4 @@
-- 0006_progress_totals.sql
ALTER TABLE jobs ADD COLUMN total_size_bytes INTEGER DEFAULT 0;
ALTER TABLE jobs ADD COLUMN sent_bytes INTEGER DEFAULT 0;
@@ -0,0 +1,7 @@
-- Migration 0007: Add checksum column to schema_migrations.
-- This migration is idempotent and safe to re-run.
-- For existing databases (pre-1.0.54): ALTER TABLE adds the checksum column.
-- For fresh databases (1.0.54+): 0001_init.sql now creates the table with checksum column.
-- This migration handles the upgrade path only.
ALTER TABLE schema_migrations ADD COLUMN checksum TEXT DEFAULT '';
+70 -6
View File
@@ -2,6 +2,8 @@ package models
import ( import (
"database/sql" "database/sql"
"fmt"
"strings"
"time" "time"
) )
@@ -15,6 +17,8 @@ type Job struct {
LogFile *string `db:"log_file" json:"log_file"` LogFile *string `db:"log_file" json:"log_file"`
ErrorMessage *string `db:"error_message" json:"error_message,omitempty"` ErrorMessage *string `db:"error_message" json:"error_message,omitempty"`
ErrorCode *string `db:"error_code" json:"error_code,omitempty"` ErrorCode *string `db:"error_code" json:"error_code,omitempty"`
TotalSizeBytes int64 `db:"total_size_bytes" json:"total_size_bytes,omitempty"`
SentBytes int64 `db:"sent_bytes" json:"sent_bytes,omitempty"`
CreatedAt time.Time `db:"created_at" json:"created_at"` CreatedAt time.Time `db:"created_at" json:"created_at"`
} }
@@ -43,9 +47,10 @@ func (r *JobRepository) GetByID(id int64) (*Job, error) {
var logFile, errMsg, errCode sql.NullString var logFile, errMsg, errCode sql.NullString
err := r.db.QueryRow(` err := r.db.QueryRow(`
SELECT id, sync_pair_id, trigger_type, status, started_at, finished_at, SELECT id, sync_pair_id, trigger_type, status, started_at, finished_at,
log_file, error_message, error_code, created_at FROM jobs WHERE id = ?`, id).Scan( log_file, error_message, error_code, total_size_bytes, sent_bytes, created_at
FROM jobs WHERE id = ?`, id).Scan(
&j.ID, &j.SyncPairID, &j.TriggerType, &j.Status, &started, &finished, &j.ID, &j.SyncPairID, &j.TriggerType, &j.Status, &started, &finished,
&logFile, &errMsg, &errCode, &j.CreatedAt) &logFile, &errMsg, &errCode, &j.TotalSizeBytes, &j.SentBytes, &j.CreatedAt)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -70,7 +75,7 @@ func (r *JobRepository) GetByID(id int64) (*Job, error) {
func (r *JobRepository) GetAll(limit, offset int) ([]Job, error) { func (r *JobRepository) GetAll(limit, offset int) ([]Job, error) {
rows, err := r.db.Query(` rows, err := r.db.Query(`
SELECT id, sync_pair_id, trigger_type, status, started_at, finished_at, SELECT id, sync_pair_id, trigger_type, status, started_at, finished_at,
log_file, error_message, error_code, created_at log_file, error_message, error_code, total_size_bytes, sent_bytes, created_at
FROM jobs ORDER BY created_at DESC LIMIT ? OFFSET ?`, FROM jobs ORDER BY created_at DESC LIMIT ? OFFSET ?`,
limit, offset) limit, offset)
if err != nil { if err != nil {
@@ -84,7 +89,8 @@ func (r *JobRepository) GetAll(limit, offset int) ([]Job, error) {
var started, finished sql.NullTime var started, finished sql.NullTime
var logFile, errMsg, errCode sql.NullString var logFile, errMsg, errCode sql.NullString
if err := rows.Scan(&j.ID, &j.SyncPairID, &j.TriggerType, &j.Status, if err := rows.Scan(&j.ID, &j.SyncPairID, &j.TriggerType, &j.Status,
&started, &finished, &logFile, &errMsg, &errCode, &j.CreatedAt); err != nil { &started, &finished, &logFile, &errMsg, &errCode,
&j.TotalSizeBytes, &j.SentBytes, &j.CreatedAt); err != nil {
return nil, err return nil, err
} }
if started.Valid { if started.Valid {
@@ -144,11 +150,11 @@ func (r *JobRepository) GetRunningBySyncPair(syncPairID int64) (*Job, error) {
var logFile, errMsg, errCode sql.NullString var logFile, errMsg, errCode sql.NullString
err := r.db.QueryRow(` err := r.db.QueryRow(`
SELECT id, sync_pair_id, trigger_type, status, started_at, finished_at, SELECT id, sync_pair_id, trigger_type, status, started_at, finished_at,
log_file, error_message, error_code, created_at FROM jobs log_file, error_message, error_code, total_size_bytes, sent_bytes, created_at FROM jobs
WHERE sync_pair_id = ? AND status IN ('queued','waking_up','running') WHERE sync_pair_id = ? AND status IN ('queued','waking_up','running')
ORDER BY created_at DESC LIMIT 1`, syncPairID).Scan( ORDER BY created_at DESC LIMIT 1`, syncPairID).Scan(
&j.ID, &j.SyncPairID, &j.TriggerType, &j.Status, &started, &j.ID, &j.SyncPairID, &j.TriggerType, &j.Status, &started,
&j.FinishedAt, &logFile, &errMsg, &errCode, &j.CreatedAt) &j.FinishedAt, &logFile, &errMsg, &errCode, &j.TotalSizeBytes, &j.SentBytes, &j.CreatedAt)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -180,3 +186,61 @@ func (r *JobRepository) DeleteFinishedBefore(before time.Time) (int64, error) {
} }
return res.RowsAffected() return res.RowsAffected()
} }
func (r *JobRepository) SetTotals(id int64, totalSize, sentBytes int64) error {
_, err := r.db.Exec(
"UPDATE jobs SET total_size_bytes = ?, sent_bytes = ? WHERE id = ?",
totalSize, sentBytes, id,
)
return err
}
func (r *JobRepository) GetByStatusAny(statuses []string) ([]Job, error) {
if len(statuses) == 0 {
return nil, nil
}
placeholders := make([]string, len(statuses))
args := make([]interface{}, len(statuses))
for i, s := range statuses {
placeholders[i] = "?"
args[i] = s
}
query := fmt.Sprintf(`
SELECT id, sync_pair_id, trigger_type, status, started_at, finished_at,
log_file, error_message, error_code, total_size_bytes, sent_bytes, created_at
FROM jobs WHERE status IN (%s)`, strings.Join(placeholders, ","))
rows, err := r.db.Query(query, args...)
if err != nil {
return nil, err
}
defer rows.Close()
var jobs []Job
for rows.Next() {
var j Job
var started, finished sql.NullTime
var logFile, errMsg, errCode sql.NullString
if err := rows.Scan(&j.ID, &j.SyncPairID, &j.TriggerType, &j.Status,
&started, &finished, &logFile, &errMsg, &errCode,
&j.TotalSizeBytes, &j.SentBytes, &j.CreatedAt); err != nil {
return nil, err
}
if started.Valid {
j.StartedAt = &started.Time
}
if finished.Valid {
j.FinishedAt = &finished.Time
}
if logFile.Valid {
j.LogFile = &logFile.String
}
if errMsg.Valid {
j.ErrorMessage = &errMsg.String
}
if errCode.Valid {
j.ErrorCode = &errCode.String
}
jobs = append(jobs, j)
}
return jobs, rows.Err()
}
+15 -2
View File
@@ -85,6 +85,19 @@ func (r *JobLogRepository) DeleteBefore(before time.Time) (int64, error) {
return res.RowsAffected() return res.RowsAffected()
} }
func (r *JobLogRepository) TruncateKeepingHeaderTail(jobID int64, head, tail int) error {
_, err := r.db.Exec(`
DELETE FROM job_logs
WHERE job_id = ?
AND id NOT IN (
SELECT id FROM job_logs WHERE job_id = ? ORDER BY id ASC LIMIT ?
UNION ALL
SELECT id FROM job_logs WHERE job_id = ? ORDER BY id DESC LIMIT ?
)`,
jobID, jobID, head, jobID, tail)
return err
}
type JobWithStats struct { type JobWithStats struct {
Job Job
DurationSeconds *int64 `db:"duration_seconds" json:"duration_seconds"` DurationSeconds *int64 `db:"duration_seconds" json:"duration_seconds"`
@@ -125,7 +138,7 @@ func (r *JobLogRepository) GetAllFiltered(limit, offset int, syncPairID *int64,
SELECT SELECT
j.id, j.sync_pair_id, j.trigger_type, j.status, j.id, j.sync_pair_id, j.trigger_type, j.status,
j.started_at, j.finished_at, j.log_file, j.started_at, j.finished_at, j.log_file,
j.error_message, j.error_code, j.created_at, j.error_message, j.error_code, j.total_size_bytes, j.sent_bytes, j.created_at,
CASE WHEN j.finished_at IS NOT NULL AND j.started_at IS NOT NULL CASE WHEN j.finished_at IS NOT NULL AND j.started_at IS NOT NULL
THEN (j.finished_at - j.started_at) ELSE NULL END as duration_seconds, THEN (j.finished_at - j.started_at) ELSE NULL END as duration_seconds,
(SELECT COUNT(*) FROM job_logs WHERE job_id = j.id) as log_line_count (SELECT COUNT(*) FROM job_logs WHERE job_id = j.id) as log_line_count
@@ -148,7 +161,7 @@ func (r *JobLogRepository) GetAllFiltered(limit, offset int, syncPairID *int64,
var durationSeconds sql.NullInt64 var durationSeconds sql.NullInt64
if err := rows.Scan(&j.ID, &j.SyncPairID, &j.TriggerType, &j.Status, if err := rows.Scan(&j.ID, &j.SyncPairID, &j.TriggerType, &j.Status,
&started, &finished, &logFile, &started, &finished, &logFile,
&errMsg, &errCode, &j.CreatedAt, &errMsg, &errCode, &j.TotalSizeBytes, &j.SentBytes, &j.CreatedAt,
&durationSeconds, &j.LogLineCount); err != nil { &durationSeconds, &j.LogLineCount); err != nil {
return nil, 0, err return nil, 0, err
} }
+602
View File
@@ -0,0 +1,602 @@
package models
import (
"database/sql"
"os"
"testing"
"time"
_ "modernc.org/sqlite"
)
func openTestDB(t *testing.T) *sql.DB {
db, err := sql.Open("sqlite", ":memory:")
if err != nil {
t.Fatalf("failed to open in-memory db: %v", err)
}
migrations := []string{initSchema, migration002, migration003, migration004, migration005, migration006}
for _, m := range migrations {
if _, err := db.Exec(m); err != nil {
t.Fatalf("failed to apply migration: %v", err)
}
}
return db
}
const initSchema = `
CREATE TABLE IF NOT EXISTS users (
id INTEGER PRIMARY KEY AUTOINCREMENT,
username TEXT UNIQUE NOT NULL,
password_hash TEXT NOT NULL,
role TEXT NOT NULL DEFAULT 'user',
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS ssh_keys (
id INTEGER PRIMARY KEY AUTOINCREMENT,
label TEXT NOT NULL,
private_key_path TEXT NOT NULL,
public_key TEXT NOT NULL,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS machines (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
host TEXT NOT NULL,
port INTEGER NOT NULL DEFAULT 22,
ssh_user TEXT NOT NULL DEFAULT 'root',
ssh_key_id INTEGER REFERENCES ssh_keys(id),
mac_address TEXT,
wol_enabled INTEGER NOT NULL DEFAULT 0,
broadcast_addr TEXT,
wake_timeout_seconds INTEGER NOT NULL DEFAULT 120,
wake_check_interval_seconds INTEGER NOT NULL DEFAULT 5,
fingerprint_confirmed INTEGER NOT NULL DEFAULT 0,
status TEXT NOT NULL DEFAULT 'unknown',
last_seen_at DATETIME,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS sync_pairs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
name TEXT NOT NULL,
source_machine_id INTEGER REFERENCES machines(id),
source_path TEXT NOT NULL,
dest_machine_id INTEGER REFERENCES machines(id),
dest_path TEXT NOT NULL,
direction TEXT NOT NULL DEFAULT 'push',
rsync_flags TEXT NOT NULL DEFAULT '-aP',
exclude_patterns TEXT NOT NULL DEFAULT '',
enabled INTEGER NOT NULL DEFAULT 1,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS schedules (
id INTEGER PRIMARY KEY AUTOINCREMENT,
sync_pair_id INTEGER NOT NULL REFERENCES sync_pairs(id) ON DELETE CASCADE,
cron_expr TEXT NOT NULL,
next_run_at DATETIME,
enabled INTEGER NOT NULL DEFAULT 1,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS jobs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
sync_pair_id INTEGER NOT NULL REFERENCES sync_pairs(id),
trigger_type TEXT NOT NULL DEFAULT 'manual',
status TEXT NOT NULL DEFAULT 'queued',
started_at DATETIME,
finished_at DATETIME,
log_file TEXT,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS job_logs (
id INTEGER PRIMARY KEY AUTOINCREMENT,
job_id INTEGER NOT NULL REFERENCES jobs(id) ON DELETE CASCADE,
stream TEXT NOT NULL,
content TEXT NOT NULL,
timestamp DATETIME DEFAULT CURRENT_TIMESTAMP
);
CREATE TABLE IF NOT EXISTS sessions (
id TEXT PRIMARY KEY,
user_id INTEGER NOT NULL REFERENCES users(id) ON DELETE CASCADE,
token_hash TEXT NOT NULL,
expires_at DATETIME NOT NULL,
revoked INTEGER NOT NULL DEFAULT 0,
created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
`
const migration002 = `
CREATE INDEX IF NOT EXISTS idx_job_logs_job_id ON job_logs(job_id);
CREATE INDEX IF NOT EXISTS idx_jobs_status_created ON jobs(status, created_at DESC);
CREATE INDEX IF NOT EXISTS idx_jobs_sync_pair_id ON jobs(sync_pair_id);
CREATE TABLE IF NOT EXISTS cleanup_history (
id INTEGER PRIMARY KEY AUTOINCREMENT,
deleted_before DATETIME NOT NULL,
logs_purged INTEGER NOT NULL DEFAULT 0,
jobs_purged INTEGER NOT NULL DEFAULT 0,
executed_at DATETIME DEFAULT CURRENT_TIMESTAMP
);
`
const migration003 = `
ALTER TABLE jobs ADD COLUMN error_message TEXT;
ALTER TABLE jobs ADD COLUMN error_code TEXT;
`
const migration004 = `
ALTER TABLE machines ADD COLUMN host_key_fingerprint TEXT;
`
const migration005 = `
ALTER TABLE machines ADD COLUMN shutdown_command TEXT NOT NULL DEFAULT 'shutdown now';
`
const migration006 = `
ALTER TABLE jobs ADD COLUMN total_size_bytes INTEGER DEFAULT 0;
ALTER TABLE jobs ADD COLUMN sent_bytes INTEGER DEFAULT 0;
`
func TestJobRepository_CreateAndGetByID(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewJobRepository(db)
id, err := repo.Create(1, "manual", "queued")
if err != nil {
t.Fatalf("Create failed: %v", err)
}
job, err := repo.GetByID(id)
if err != nil {
t.Fatalf("GetByID failed: %v", err)
}
if job.ID != id {
t.Errorf("expected ID %d, got %d", id, job.ID)
}
if job.SyncPairID != 1 {
t.Errorf("expected SyncPairID 1, got %d", job.SyncPairID)
}
if job.TriggerType != "manual" {
t.Errorf("expected trigger_type 'manual', got %q", job.TriggerType)
}
if job.Status != "queued" {
t.Errorf("expected status 'queued', got %q", job.Status)
}
}
func TestJobRepository_UpdateStatus_SetsStartedAt(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewJobRepository(db)
id, _ := repo.Create(1, "manual", "queued")
err := repo.UpdateStatus(id, "running")
if err != nil {
t.Fatalf("UpdateStatus failed: %v", err)
}
job, _ := repo.GetByID(id)
if job.StartedAt == nil {
t.Fatal("expected StartedAt to be set for 'running' status")
}
if job.Status != "running" {
t.Errorf("expected status 'running', got %q", job.Status)
}
}
func TestJobRepository_UpdateStatus_SetsFinishedAt(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewJobRepository(db)
id, _ := repo.Create(1, "manual", "queued")
repo.UpdateStatus(id, "running")
err := repo.UpdateStatus(id, "success")
if err != nil {
t.Fatalf("UpdateStatus failed: %v", err)
}
job, _ := repo.GetByID(id)
if job.FinishedAt == nil {
t.Fatal("expected FinishedAt to be set for 'success' status")
}
if job.Status != "success" {
t.Errorf("expected status 'success', got %q", job.Status)
}
}
func TestJobRepository_UpdateStatus_Failed(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewJobRepository(db)
id, _ := repo.Create(1, "manual", "queued")
repo.UpdateStatus(id, "running")
err := repo.UpdateStatus(id, "failed")
if err != nil {
t.Fatalf("UpdateStatus failed: %v", err)
}
job, _ := repo.GetByID(id)
if job.FinishedAt == nil {
t.Fatal("expected FinishedAt to be set for 'failed' status")
}
}
func TestJobRepository_SetError(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewJobRepository(db)
id, _ := repo.Create(1, "manual", "queued")
err := repo.SetError(id, "EIO", "disk read failed")
if err != nil {
t.Fatalf("SetError failed: %v", err)
}
job, _ := repo.GetByID(id)
if job.ErrorCode == nil || *job.ErrorCode != "EIO" {
t.Errorf("expected error_code 'EIO', got %v", job.ErrorCode)
}
if job.ErrorMessage == nil || *job.ErrorMessage != "disk read failed" {
t.Errorf("expected error_message 'disk read failed', got %v", job.ErrorMessage)
}
}
func TestJobRepository_GetByStatusAny(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewJobRepository(db)
_, _ = repo.Create(1, "manual", "queued")
id2, _ := repo.Create(1, "manual", "running")
_, _ = repo.Create(1, "manual", "success")
jobs, err := repo.GetByStatusAny([]string{"running"})
if err != nil {
t.Fatalf("GetByStatusAny failed: %v", err)
}
if len(jobs) != 1 {
t.Fatalf("expected 1 job, got %d", len(jobs))
}
if jobs[0].ID != id2 {
t.Errorf("expected job ID %d, got %d", id2, jobs[0].ID)
}
jobs, _ = repo.GetByStatusAny([]string{"queued", "running"})
if len(jobs) != 2 {
t.Fatalf("expected 2 jobs, got %d", len(jobs))
}
}
func TestJobRepository_DeleteFinishedBefore(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewJobRepository(db)
id1, _ := repo.Create(1, "manual", "queued")
repo.UpdateStatus(id1, "running")
repo.UpdateStatus(id1, "success")
db.Exec("UPDATE jobs SET finished_at = datetime('2020-01-01 00:00:00') WHERE id = ?", id1)
id2, _ := repo.Create(1, "manual", "queued")
repo.UpdateStatus(id2, "running")
repo.UpdateStatus(id2, "success")
cutoff := time.Now()
deleted, err := repo.DeleteFinishedBefore(cutoff)
if err != nil {
t.Fatalf("DeleteFinishedBefore failed: %v", err)
}
if deleted != 1 {
t.Errorf("expected 1 deleted, got %d", deleted)
}
_, err = repo.GetByID(id1)
if err != sql.ErrNoRows {
t.Errorf("expected id1 to be deleted")
}
_, err = repo.GetByID(id2)
if err != nil {
t.Errorf("expected id2 to still exist")
}
}
func TestJobRepository_SetTotals(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewJobRepository(db)
id, _ := repo.Create(1, "manual", "queued")
err := repo.SetTotals(id, 1024, 512)
if err != nil {
t.Fatalf("SetTotals failed: %v", err)
}
job, _ := repo.GetByID(id)
if job.TotalSizeBytes != 1024 {
t.Errorf("expected TotalSizeBytes 1024, got %d", job.TotalSizeBytes)
}
if job.SentBytes != 512 {
t.Errorf("expected SentBytes 512, got %d", job.SentBytes)
}
}
func TestMachineRepository_CreateAndGetByID(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewMachineRepository(db)
mac := "AA:BB:CC:DD:EE:FF"
bcast := "192.168.1.255"
WolEnabled := true
WolTimeout := 300
machine := &Machine{
Name: "test-machine",
Host: "192.168.1.10",
Port: 22,
SSHUser: "admin",
MACAddress: &mac,
WoLEnabled: WolEnabled,
BroadcastAddr: &bcast,
WakeTimeoutSeconds: WolTimeout,
WakeCheckIntervalSeconds: 10,
Status: "unknown",
ShutdownCommand: "shutdown -h now",
}
id, err := repo.Create(machine)
if err != nil {
t.Fatalf("Create failed: %v", err)
}
retrieved, err := repo.GetByID(id)
if err != nil {
t.Fatalf("GetByID failed: %v", err)
}
if retrieved.Name != "test-machine" {
t.Errorf("expected name 'test-machine', got %q", retrieved.Name)
}
if retrieved.Host != "192.168.1.10" {
t.Errorf("expected host '192.168.1.10', got %q", retrieved.Host)
}
if retrieved.MACAddress == nil || *retrieved.MACAddress != mac {
t.Errorf("expected MAC %q, got %v", mac, retrieved.MACAddress)
}
if !retrieved.WoLEnabled {
t.Error("expected WoLEnabled to be true")
}
if retrieved.BroadcastAddr == nil || *retrieved.BroadcastAddr != bcast {
t.Errorf("expected broadcast %q, got %v", bcast, retrieved.BroadcastAddr)
}
if retrieved.WakeTimeoutSeconds != WolTimeout {
t.Errorf("expected wake timeout %d, got %d", WolTimeout, retrieved.WakeTimeoutSeconds)
}
if retrieved.ShutdownCommand != "shutdown -h now" {
t.Errorf("expected shutdown command 'shutdown -h now', got %q", retrieved.ShutdownCommand)
}
}
func TestMachineRepository_UpdateStatus(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewMachineRepository(db)
id, _ := repo.Create(&Machine{Name: "test", Host: "192.168.1.1", Status: "unknown"})
err := repo.UpdateStatus(id, "online")
if err != nil {
t.Fatalf("UpdateStatus failed: %v", err)
}
machine, _ := repo.GetByID(id)
if machine.Status != "online" {
t.Errorf("expected status 'online', got %q", machine.Status)
}
if machine.LastSeenAt == nil {
t.Error("expected LastSeenAt to be set")
}
}
func TestMachineRepository_GetAll(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewMachineRepository(db)
repo.Create(&Machine{Name: "machine-a", Host: "192.168.1.1", Status: "online"})
repo.Create(&Machine{Name: "machine-b", Host: "192.168.1.2", Status: "offline"})
machines, err := repo.GetAll()
if err != nil {
t.Fatalf("GetAll failed: %v", err)
}
if len(machines) != 2 {
t.Errorf("expected 2 machines, got %d", len(machines))
}
}
func TestSyncPairRepository_CreateAndGetByID(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewSyncPairRepository(db)
sp := &SyncPair{
Name: "backup-data",
SourcePath: "/data",
DestPath: "/backup",
Direction: "push",
RsyncFlags: "-aP --delete",
ExcludePatterns: "*.tmp\n*.log",
Enabled: true,
}
id, err := repo.Create(sp)
if err != nil {
t.Fatalf("Create failed: %v", err)
}
retrieved, err := repo.GetByID(id)
if err != nil {
t.Fatalf("GetByID failed: %v", err)
}
if retrieved.Name != "backup-data" {
t.Errorf("expected name 'backup-data', got %q", retrieved.Name)
}
if retrieved.ExcludePatterns != "*.tmp\n*.log" {
t.Errorf("expected exclude patterns '*.tmp\\n*.log', got %q", retrieved.ExcludePatterns)
}
}
func TestSyncPairRepository_ExcludePatternsList(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewSyncPairRepository(db)
sp := &SyncPair{
Name: "test-pair",
SourcePath: "/src",
DestPath: "/dst",
ExcludePatterns: "*.tmp\n *.log\n \n*.bak",
}
id, _ := repo.Create(sp)
retrieved, _ := repo.GetByID(id)
patterns := retrieved.ExcludePatternsList()
if len(patterns) != 3 {
t.Fatalf("expected 3 patterns, got %d: %v", len(patterns), patterns)
}
if patterns[0] != "*.tmp" {
t.Errorf("expected first pattern '*.tmp', got %q", patterns[0])
}
if patterns[1] != "*.log" {
t.Errorf("expected second pattern '*.log', got %q", patterns[1])
}
if patterns[2] != "*.bak" {
t.Errorf("expected third pattern '*.bak', got %q", patterns[2])
}
}
func TestSyncPairRepository_GetAll(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewSyncPairRepository(db)
repo.Create(&SyncPair{Name: "pair-a", SourcePath: "/a", DestPath: "/b"})
repo.Create(&SyncPair{Name: "pair-b", SourcePath: "/c", DestPath: "/d"})
pairs, err := repo.GetAll()
if err != nil {
t.Fatalf("GetAll failed: %v", err)
}
if len(pairs) != 2 {
t.Errorf("expected 2 pairs, got %d", len(pairs))
}
}
func TestScheduleRepository_CreateAndGetByID(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewScheduleRepository(db)
nextRun := time.Now().Add(1 * time.Hour)
sched := &Schedule{
SyncPairID: 1,
CronExpr: "0 0 * * *",
NextRunAt: &nextRun,
Enabled: true,
}
id, err := repo.Create(sched)
if err != nil {
t.Fatalf("Create failed: %v", err)
}
retrieved, err := repo.GetByID(id)
if err != nil {
t.Fatalf("GetByID failed: %v", err)
}
if retrieved.CronExpr != "0 0 * * *" {
t.Errorf("expected cron '0 0 * * *', got %q", retrieved.CronExpr)
}
if !retrieved.Enabled {
t.Error("expected enabled to be true")
}
}
func TestScheduleRepository_UpdateEnabled(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewScheduleRepository(db)
nextRun := time.Now().Add(1 * time.Hour)
id, _ := repo.Create(&Schedule{SyncPairID: 1, CronExpr: "0 0 * * *", NextRunAt: &nextRun, Enabled: true})
sched, _ := repo.GetByID(id)
sched.Enabled = false
err := repo.Update(sched)
if err != nil {
t.Fatalf("Update failed: %v", err)
}
retrieved, _ := repo.GetByID(id)
if retrieved.Enabled {
t.Error("expected enabled to be false after update")
}
}
func TestScheduleRepository_GetEnabledDue(t *testing.T) {
db := openTestDB(t)
defer db.Close()
repo := NewScheduleRepository(db)
pastRun := time.Now().Add(-1 * time.Hour)
futureRun := time.Now().Add(1 * time.Hour)
_, _ = repo.Create(&Schedule{SyncPairID: 1, CronExpr: "0 0 * * *", NextRunAt: &pastRun, Enabled: true})
_, _ = repo.Create(&Schedule{SyncPairID: 2, CronExpr: "0 0 * * *", NextRunAt: &futureRun, Enabled: true})
_, _ = repo.Create(&Schedule{SyncPairID: 3, CronExpr: "0 0 * * *", NextRunAt: &pastRun, Enabled: false})
due, err := repo.GetEnabledDue(time.Now())
if err != nil {
t.Fatalf("GetEnabledDue failed: %v", err)
}
if len(due) != 1 {
t.Errorf("expected 1 due schedule, got %d", len(due))
}
if due[0].SyncPairID != 1 {
t.Errorf("expected sync_pair_id 1, got %d", due[0].SyncPairID)
}
}
func TestMain(m *testing.M) {
os.Exit(m.Run())
}
+82
View File
@@ -3,7 +3,12 @@ package scheduler
import ( import (
"context" "context"
"database/sql" "database/sql"
"fmt"
"log/slog" "log/slog"
"os"
"path/filepath"
"strconv"
"strings"
"sync" "sync"
"time" "time"
@@ -124,6 +129,10 @@ func (s *Scheduler) cleanup() {
} }
before := time.Now().AddDate(0, 0, -retentionDays) before := time.Now().AddDate(0, 0, -retentionDays)
if s.cfg.Scheduler.BackupDir != "" {
s.backupDB(before)
}
logRepo := models.NewJobLogRepository(s.db) logRepo := models.NewJobLogRepository(s.db)
jobRepo := models.NewJobRepository(s.db) jobRepo := models.NewJobRepository(s.db)
@@ -141,5 +150,78 @@ func (s *Scheduler) cleanup() {
if deletedLogs > 0 || deletedJobs > 0 { if deletedLogs > 0 || deletedJobs > 0 {
slog.Info("cleanup: purged old records", "logs_deleted", deletedLogs, "jobs_deleted", deletedJobs, "before", before.Format("2006-01-02")) slog.Info("cleanup: purged old records", "logs_deleted", deletedLogs, "jobs_deleted", deletedJobs, "before", before.Format("2006-01-02"))
s.purgeJobLogFiles(before)
}
s.purgeOldBackups()
}
func (s *Scheduler) backupDB(before time.Time) {
backupDir := s.cfg.Scheduler.BackupDir
if backupDir == "" {
return
}
if err := os.MkdirAll(backupDir, 0700); err != nil {
slog.Error("cleanup: failed to create backup dir", "error", err)
return
}
ts := time.Now().UTC().Format("20060102-150405")
backupPath := filepath.Join(backupDir, fmt.Sprintf("syncserver-%s.db", ts))
if _, err := s.db.Exec(fmt.Sprintf("VACUUM INTO '%s'", backupPath)); err != nil {
slog.Error("cleanup: failed to vacuum into backup", "path", backupPath, "error", err)
return
}
slog.Info("cleanup: database backup created", "path", backupPath)
}
func (s *Scheduler) purgeJobLogFiles(before time.Time) {
logsDir := s.cfg.LogsDir()
entries, err := os.ReadDir(logsDir)
if err != nil {
return
}
for _, entry := range entries {
if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".log") {
continue
}
jobID := strings.TrimSuffix(entry.Name(), ".log")
id, err := strconv.ParseInt(jobID, 10, 64)
if err != nil {
continue
}
jobRepo := models.NewJobRepository(s.db)
job, err := jobRepo.GetByID(id)
if err != nil || job == nil {
os.Remove(filepath.Join(logsDir, entry.Name()))
continue
}
if job.FinishedAt != nil && job.FinishedAt.Before(before) {
os.Remove(filepath.Join(logsDir, entry.Name()))
}
}
}
func (s *Scheduler) purgeOldBackups() {
backupDir := s.cfg.Scheduler.BackupDir
retention := s.cfg.Scheduler.BackupRetentionDays
if backupDir == "" || retention <= 0 {
return
}
cutoff := time.Now().AddDate(0, 0, -retention)
entries, err := os.ReadDir(backupDir)
if err != nil {
return
}
for _, entry := range entries {
if entry.IsDir() || !strings.HasSuffix(entry.Name(), ".db") {
continue
}
info, err := entry.Info()
if err != nil {
continue
}
if info.ModTime().Before(cutoff) {
os.Remove(filepath.Join(backupDir, entry.Name()))
}
} }
} }
+7 -5
View File
@@ -5,7 +5,6 @@ import (
"context" "context"
"fmt" "fmt"
"log/slog" "log/slog"
"net"
"os" "os"
"path/filepath" "path/filepath"
"time" "time"
@@ -25,7 +24,7 @@ type DeployResult struct {
Errors []string `json:"errors"` Errors []string `json:"errors"`
} }
func DeployKeysToMachine(ctx context.Context, serverKeyPath string, host string, port int, user string, keys []DeployKey, knownHostsHosts []string) (*DeployResult, error) { func DeployKeysToMachine(ctx context.Context, serverKeyPath string, host string, port int, user string, keys []DeployKey, knownHostsHosts []string, sshDir string) (*DeployResult, error) {
result := &DeployResult{Success: true, Messages: []string{}, Errors: []string{}} result := &DeployResult{Success: true, Messages: []string{}, Errors: []string{}}
addr := fmt.Sprintf("%s:%d", host, port) addr := fmt.Sprintf("%s:%d", host, port)
@@ -39,8 +38,11 @@ func DeployKeysToMachine(ctx context.Context, serverKeyPath string, host string,
return nil, fmt.Errorf("parsing server key: %w", err) return nil, fmt.Errorf("parsing server key: %w", err)
} }
hostKeyCallback := func(hostname string, remote net.Addr, key ssh.PublicKey) error { knownHostsPath := filepath.Join(sshDir, "known_hosts")
return nil hostKeyCallback, err := NewKnownHostsCallback(knownHostsPath, true)
if err != nil {
slog.Warn("deploy keys: creating host key callback failed, ignoring hosts", "error", err)
hostKeyCallback = ssh.InsecureIgnoreHostKey()
} }
cfg := &ssh.ClientConfig{ cfg := &ssh.ClientConfig{
@@ -60,7 +62,7 @@ func DeployKeysToMachine(ctx context.Context, serverKeyPath string, host string,
} }
defer conn.Close() defer conn.Close()
remoteSSHDir := "/var/lib/syncserver/ssh" remoteSSHDir := sshDir
remoteKeysDir := filepath.Join(remoteSSHDir, "keys") remoteKeysDir := filepath.Join(remoteSSHDir, "keys")
session, err := conn.NewSession() session, err := conn.NewSession()
+20
View File
@@ -8,6 +8,7 @@ import (
"strings" "strings"
"golang.org/x/crypto/ssh" "golang.org/x/crypto/ssh"
"golang.org/x/crypto/ssh/knownhosts"
) )
type KnownHost struct { type KnownHost struct {
@@ -17,6 +18,23 @@ type KnownHost struct {
Fingerprint string Fingerprint string
} }
func NewKnownHostsCallback(knownHostsPath string, strictHostKeyChecking bool) (ssh.HostKeyCallback, error) {
if !strictHostKeyChecking {
return ssh.InsecureIgnoreHostKey(), nil
}
if knownHostsPath == "" {
return ssh.InsecureIgnoreHostKey(), nil
}
_, err := os.Stat(knownHostsPath)
if os.IsNotExist(err) {
return ssh.InsecureIgnoreHostKey(), nil
}
if err != nil {
return nil, err
}
return knownhosts.New(knownHostsPath)
}
func EnsureKnownHosts(sshDir string) (string, error) { func EnsureKnownHosts(sshDir string) (string, error) {
path := filepath.Join(sshDir, "known_hosts") path := filepath.Join(sshDir, "known_hosts")
f, err := os.OpenFile(path, os.O_CREATE|os.O_RDONLY, 0644) f, err := os.OpenFile(path, os.O_CREATE|os.O_RDONLY, 0644)
@@ -27,6 +45,8 @@ func EnsureKnownHosts(sshDir string) (string, error) {
return path, nil return path, nil
} }
// AddKnownHost stores the host key in standard ssh known_hosts format (hostname keytype base64key).
// NOTE: Existing entries in known_hosts may need to be regenerated if they were stored in a different format.
func AddKnownHost(sshDir, host string, port int, keyData []byte) error { func AddKnownHost(sshDir, host string, port int, keyData []byte) error {
path := filepath.Join(sshDir, "known_hosts") path := filepath.Join(sshDir, "known_hosts")
f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644) f, err := os.OpenFile(path, os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0644)
+7 -15
View File
@@ -8,7 +8,6 @@ import (
"fmt" "fmt"
"net" "net"
"os" "os"
"path/filepath"
"strings" "strings"
"time" "time"
@@ -40,24 +39,17 @@ func dialSSH(ctx context.Context, host string, port int, user, privKeyPath, know
var capturedFingerprint string var capturedFingerprint string
var capturedPubKey ssh.PublicKey var capturedPubKey ssh.PublicKey
callback, err := NewKnownHostsCallback(knownHostsPath, strictHostKeyChecking)
if err != nil {
return nil, "", nil, fmt.Errorf("creating host key callback: %w", err)
}
hostKeyCallback := func(hostname string, remote net.Addr, key ssh.PublicKey) error { hostKeyCallback := func(hostname string, remote net.Addr, key ssh.PublicKey) error {
h := sha256.Sum256(key.Marshal()) h := sha256.Sum256(key.Marshal())
capturedFingerprint = "SHA256:" + base64.RawStdEncoding.EncodeToString(h[:]) capturedFingerprint = "SHA256:" + base64.RawStdEncoding.EncodeToString(h[:])
capturedPubKey = key capturedPubKey = key
if strictHostKeyChecking && knownHostsPath != "" { return callback(hostname, remote, key)
kh, err := GetKnownHost(filepath.Dir(knownHostsPath), host, port)
if err != nil {
return fmt.Errorf("checking known_hosts: %w", err)
}
if kh == nil {
return fmt.Errorf("host key not found in known_hosts: %s", hostname)
}
wantFP := kh.Fingerprint
if capturedFingerprint != wantFP {
return fmt.Errorf("host key mismatch: got %s, want %s", capturedFingerprint, wantFP)
}
}
return nil
} }
cfg := &ssh.ClientConfig{ cfg := &ssh.ClientConfig{
+148 -19
View File
@@ -25,16 +25,24 @@ type Engine struct {
mu sync.RWMutex mu sync.RWMutex
stopped bool stopped bool
lastProbeAt atomic.Int64 lastProbeAt atomic.Int64
jobsWG sync.WaitGroup
stopCh chan struct{}
jobsTotal map[string]int64
jobsTotalMu sync.Mutex
} }
type Event struct { type Event struct {
Type string Type string `json:"type"`
JobID int64 JobID int64 `json:"job_id"`
MachineID int64 MachineID int64 `json:"machine_id"`
Key string Key string `json:"key,omitempty"`
Value string Value string `json:"value,omitempty"`
Line string Line string `json:"line,omitempty"`
Stream string Stream string `json:"stream,omitempty"`
Progress *ProgressFields `json:"progress,omitempty"`
TotalBytes int64 `json:"total_bytes,omitempty"`
SentBytes int64 `json:"sent_bytes,omitempty"`
} }
func New(database interface{ SQLDB() *sql.DB }, cfg *config.Config) *Engine { func New(database interface{ SQLDB() *sql.DB }, cfg *config.Config) *Engine {
@@ -43,12 +51,56 @@ func New(database interface{ SQLDB() *sql.DB }, cfg *config.Config) *Engine {
cfg: cfg, cfg: cfg,
queue: NewQueue(), queue: NewQueue(),
eventBus: NewEventBus(200), eventBus: NewEventBus(200),
stopCh: make(chan struct{}),
jobsTotal: map[string]int64{
"success": 0,
"failed": 0,
"cancelled": 0,
},
} }
return e return e
} }
func (e *Engine) Start() {} func (e *Engine) Start() {
func (e *Engine) Stop() {} e.recoverOrphanedJobs()
slog.Info("engine started")
}
func (e *Engine) Stop() {
e.mu.Lock()
if e.stopped {
e.mu.Unlock()
return
}
e.stopped = true
e.mu.Unlock()
close(e.stopCh)
runningIDs := e.queue.RunningJobs()
for _, id := range runningIDs {
e.queue.Cancel(id, false)
}
e.jobsWG.Wait()
slog.Info("engine stopped")
}
func (e *Engine) recoverOrphanedJobs() {
jobRepo := models.NewJobRepository(e.db)
jobs, err := jobRepo.GetByStatusAny([]string{"queued", "waking_up", "running"})
if err != nil {
slog.Warn("failed to recover orphaned jobs", "error", err)
return
}
for _, j := range jobs {
slog.Warn("recovered orphaned job, marking as failed",
"job_id", j.ID, "pair_id", j.SyncPairID, "status", j.Status)
jobRepo.UpdateStatus(j.ID, "failed")
jobRepo.SetError(j.ID, "crash_recovery",
fmt.Sprintf("job was %s when server shut down unexpectedly", j.Status))
}
}
func (e *Engine) SubscribeJob(jobID int64) (chan Event, func()) { func (e *Engine) SubscribeJob(jobID int64) (chan Event, func()) {
return e.eventBus.Subscribe(jobID) return e.eventBus.Subscribe(jobID)
@@ -81,9 +133,12 @@ func (e *Engine) wakeMachine(ctx context.Context, m *models.Machine) {
} }
func (e *Engine) Run(ctx context.Context, jobID int64, pairID int64) error { func (e *Engine) Run(ctx context.Context, jobID int64, pairID int64) error {
if e.queue.IsRunning(pairID) { if e.queue.IsRunning(jobID) {
existingJobID, _ := e.queue.GetJobID(pairID) return fmt.Errorf("%w: job %d is already running", ErrAlreadyRunning, jobID)
return fmt.Errorf("%w: job %d is already running", ErrAlreadyRunning, existingJobID) }
if existingJobID, exists := e.queue.GetByPair(pairID); exists {
return fmt.Errorf("%w: job %d is already running for this sync pair", ErrAlreadyRunning, existingJobID)
} }
jobCtx, cancel := context.WithCancel(ctx) jobCtx, cancel := context.WithCancel(ctx)
@@ -96,7 +151,10 @@ func (e *Engine) Run(ctx context.Context, jobID int64, pairID int64) error {
if enqueueErr != nil { if enqueueErr != nil {
return enqueueErr return enqueueErr
} }
defer e.queue.Dequeue(pairID) defer e.queue.Dequeue(jobID)
e.jobsWG.Add(1)
defer e.jobsWG.Done()
pairRepo := models.NewSyncPairRepository(e.db) pairRepo := models.NewSyncPairRepository(e.db)
pair, err := pairRepo.GetByID(pairID) pair, err := pairRepo.GetByID(pairID)
@@ -243,7 +301,17 @@ func (e *Engine) Run(ctx context.Context, jobID int64, pairID int64) error {
} }
} }
var lastFileName string
onLine := func(stream, line string) { onLine := func(stream, line string) {
if stream == "stdout" && isProgressOnlyLine(line) {
if p := parseProgressFields(line); p != nil {
e.emit(Event{Type: "progress", JobID: jobID, Line: line, Stream: stream, Progress: p, Value: lastFileName})
}
return
}
if stream == "stdout" && isFileNameLine(line) {
lastFileName = line
}
f, _ := os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0644) f, _ := os.OpenFile(logPath, os.O_APPEND|os.O_WRONLY, 0644)
if f != nil { if f != nil {
fmt.Fprintln(f, line) fmt.Fprintln(f, line)
@@ -286,6 +354,11 @@ func (e *Engine) Run(ctx context.Context, jobID int64, pairID int64) error {
} }
flush() flush()
stats := result.Stats
if stats != nil && stats.TotalSize > 0 {
e.emit(Event{Type: "progress_total", JobID: jobID, TotalBytes: stats.TotalSize, SentBytes: stats.SentBytes})
}
if err != nil { if err != nil {
if jobCtx.Err() != nil { if jobCtx.Err() != nil {
code := "cancelled_shutdown" code := "cancelled_shutdown"
@@ -297,6 +370,7 @@ func (e *Engine) Run(ctx context.Context, jobID int64, pairID int64) error {
e.setJobError(jobID, code, msg) e.setJobError(jobID, code, msg)
e.setJobStatus(jobID, "cancelled") e.setJobStatus(jobID, "cancelled")
e.emit(Event{Type: "status", JobID: jobID, Key: "status", Value: "cancelled", Line: msg}) e.emit(Event{Type: "status", JobID: jobID, Key: "status", Value: "cancelled", Line: msg})
e.persistAndClose(jobID)
return jobCtx.Err() return jobCtx.Err()
} }
e.setJobStatus(jobID, "failed") e.setJobStatus(jobID, "failed")
@@ -315,11 +389,15 @@ func (e *Engine) Run(ctx context.Context, jobID int64, pairID int64) error {
e.setJobStatus(jobID, "failed") e.setJobStatus(jobID, "failed")
e.setJobError(jobID, errCode, errMsg) e.setJobError(jobID, errCode, errMsg)
e.emit(Event{Type: "status", JobID: jobID, Key: "status", Value: "failed", Line: errMsg}) e.emit(Event{Type: "status", JobID: jobID, Key: "status", Value: "failed", Line: errMsg})
e.persistAndClose(jobID)
return fmt.Errorf("rsync exited with code %d: %s", result.ExitCode, result.Stderr) return fmt.Errorf("rsync exited with code %d: %s", result.ExitCode, result.Stderr)
} }
e.setJobStatus(jobID, "success") e.setJobStatus(jobID, "success")
e.emit(Event{Type: "status", JobID: jobID, Key: "status", Value: "success"}) e.emit(Event{Type: "status", JobID: jobID, Key: "status", Value: "success"})
if stats != nil {
e.persistJobTotals(jobID, stats)
}
slog.Info("job completed", "job_id", jobID, "pair", pair.Name) slog.Info("job completed", "job_id", jobID, "pair", pair.Name)
e.persistAndClose(jobID) e.persistAndClose(jobID)
return nil return nil
@@ -327,6 +405,23 @@ func (e *Engine) Run(ctx context.Context, jobID int64, pairID int64) error {
func (e *Engine) persistAndClose(jobID int64) { func (e *Engine) persistAndClose(jobID int64) {
e.eventBus.CloseJobChannels(jobID) e.eventBus.CloseJobChannels(jobID)
logRepo := models.NewJobLogRepository(e.db)
count, err := logRepo.CountByJobID(jobID)
if err == nil && count > 2000 {
if truncateErr := logRepo.TruncateKeepingHeaderTail(jobID, 50, 100); truncateErr != nil {
slog.Warn("failed to truncate job logs", "job_id", jobID, "error", truncateErr)
}
}
}
func (e *Engine) persistJobTotals(jobID int64, stats *RsyncStats) {
if stats == nil {
return
}
jobRepo := models.NewJobRepository(e.db)
if err := jobRepo.SetTotals(jobID, stats.TotalSize, stats.SentBytes); err != nil {
slog.Warn("failed to persist job totals", "job_id", jobID, "error", err)
}
} }
func (e *Engine) resolveSSHKey(machine *models.Machine) (string, error) { func (e *Engine) resolveSSHKey(machine *models.Machine) (string, error) {
@@ -341,17 +436,18 @@ func (e *Engine) resolveSSHKey(machine *models.Machine) (string, error) {
return sshKey.PrivateKeyPath, nil return sshKey.PrivateKeyPath, nil
} }
func (e *Engine) Cancel(jobID int64, syncPairID int64, byUser bool) bool { func (e *Engine) Cancel(jobID int64, byUser bool) bool {
if e.queue.IsRunning(syncPairID) { return e.queue.Cancel(jobID, byUser)
e.queue.Cancel(syncPairID, byUser)
return true
}
return false
} }
func (e *Engine) setJobStatus(jobID int64, status string) { func (e *Engine) setJobStatus(jobID int64, status string) {
jobRepo := models.NewJobRepository(e.db) jobRepo := models.NewJobRepository(e.db)
jobRepo.UpdateStatus(jobID, status) jobRepo.UpdateStatus(jobID, status)
if status == "success" || status == "failed" || status == "cancelled" {
e.jobsTotalMu.Lock()
e.jobsTotal[status]++
e.jobsTotalMu.Unlock()
}
} }
func (e *Engine) setJobLogFile(jobID int64, path string) { func (e *Engine) setJobLogFile(jobID int64, path string) {
@@ -455,3 +551,36 @@ func (e *Engine) ProbeAllMachines() {
} }
wg.Wait() wg.Wait()
} }
func (e *Engine) GetJobsTotal() map[string]int64 {
e.jobsTotalMu.Lock()
defer e.jobsTotalMu.Unlock()
return e.jobsTotal
}
func (e *Engine) GetJobsRunning() int64 {
return int64(len(e.queue.RunningJobs()))
}
func (e *Engine) GetQueueDepth() int64 {
return int64(e.queue.Len())
}
func (e *Engine) GetMachineCounts() (online, total int64) {
machineRepo := models.NewMachineRepository(e.db)
ms, err := machineRepo.GetAll()
if err != nil {
return 0, 0
}
for _, m := range ms {
total++
if m.Status == "online" {
online++
}
}
return online, total
}
func (e *Engine) DB() *sql.DB {
return e.db
}
+6 -28
View File
@@ -8,20 +8,17 @@ import (
type EventBus struct { type EventBus struct {
subscribers map[int64]map[chan Event]struct{} subscribers map[int64]map[chan Event]struct{}
mu sync.RWMutex mu sync.RWMutex
global chan Event
bufferSize int bufferSize int
globalSubs []globalSub globalSubs []globalSub
} }
type globalSub struct { type globalSub struct {
ch chan Event ch chan Event
done chan struct{}
} }
func NewEventBus(bufferSize int) *EventBus { func NewEventBus(bufferSize int) *EventBus {
return &EventBus{ return &EventBus{
subscribers: make(map[int64]map[chan Event]struct{}), subscribers: make(map[int64]map[chan Event]struct{}),
global: make(chan Event, bufferSize),
bufferSize: bufferSize, bufferSize: bufferSize,
globalSubs: nil, globalSubs: nil,
} }
@@ -51,32 +48,10 @@ func (eb *EventBus) Subscribe(jobID int64) (chan Event, func()) {
func (eb *EventBus) SubscribeGlobal() (chan Event, func()) { func (eb *EventBus) SubscribeGlobal() (chan Event, func()) {
ch := make(chan Event, eb.bufferSize) ch := make(chan Event, eb.bufferSize)
done := make(chan struct{})
eb.mu.Lock() eb.mu.Lock()
eb.globalSubs = append(eb.globalSubs, globalSub{ch: ch, done: done}) eb.globalSubs = append(eb.globalSubs, globalSub{ch: ch})
eb.mu.Unlock() eb.mu.Unlock()
go func() {
defer func() {
if r := recover(); r != nil {
slog.Error("SubscribeGlobal goroutine panicked", "reason", r)
}
close(ch)
}()
for {
select {
case evt := <-eb.global:
select {
case ch <- evt:
default:
slog.Warn("global event subscriber buffer full, dropping event", "type", evt.Type)
}
case <-done:
return
}
}
}()
return ch, func() { return ch, func() {
close(done)
eb.mu.Lock() eb.mu.Lock()
for i, s := range eb.globalSubs { for i, s := range eb.globalSubs {
if s.ch == ch { if s.ch == ch {
@@ -85,6 +60,7 @@ func (eb *EventBus) SubscribeGlobal() (chan Event, func()) {
} }
} }
eb.mu.Unlock() eb.mu.Unlock()
close(ch)
} }
} }
@@ -102,10 +78,12 @@ func (eb *EventBus) Publish(evt Event) {
} }
} }
for _, sub := range eb.globalSubs {
select { select {
case eb.global <- evt: case sub.ch <- evt:
default: default:
slog.Warn("global event bus full, dropping event", "type", evt.Type) slog.Warn("global event subscriber buffer full, dropping event", "type", evt.Type)
}
} }
} }
+95
View File
@@ -24,6 +24,15 @@ type ProgressLine struct {
XferedBytes int64 XferedBytes int64
} }
type ProgressFields struct {
FileBytes int64 `json:"file_bytes"`
Pct int `json:"pct"`
SpeedBps int64 `json:"speed_bps"`
EtaSeconds int `json:"eta_seconds"`
XfrDone int `json:"xfr_done"`
XfrTotal int `json:"xfr_total"`
}
var ( var (
progressRegex = regexp.MustCompile(`\s*([\d,]+)\s+([\d,]+)\s+([\d%]+)\s*`) progressRegex = regexp.MustCompile(`\s*([\d,]+)\s+([\d,]+)\s+([\d%]+)\s*`)
sentRegex = regexp.MustCompile(`sent\s+([\d,]+)\s+bytes`) sentRegex = regexp.MustCompile(`sent\s+([\d,]+)\s+bytes`)
@@ -32,6 +41,24 @@ var (
filesRegex = regexp.MustCompile(`Number of files: ([\d,]+)`) filesRegex = regexp.MustCompile(`Number of files: ([\d,]+)`)
) )
var perFileProgressRegex = regexp.MustCompile(
`^\s*(\d{1,3}(?:,\d{3})+)\s+(\d+)%\s+(\d+\.\d+)([kMG])B/s\s+(\d+:\d{2}:\d{2})(.*)`,
)
var xfrRegex = regexp.MustCompile(`xfr#(\d+).*to-chk=(\d+)/(\d+)`)
func parseXfrSuffix(suffix string) (done, total int) {
m := xfrRegex.FindStringSubmatch(suffix)
if m == nil {
return 0, 0
}
done, _ = strconv.Atoi(m[1])
t, _ := strconv.Atoi(m[2])
_ = t
total, _ = strconv.Atoi(m[3])
return done, total
}
func ParseProgressLine(line string) *ProgressLine { func ParseProgressLine(line string) *ProgressLine {
if strings.Contains(line, "files to consider") || strings.Contains(line, "files...") { if strings.Contains(line, "files to consider") || strings.Contains(line, "files...") {
return &ProgressLine{Phase: "scanning"} return &ProgressLine{Phase: "scanning"}
@@ -74,3 +101,71 @@ func ParseFinalStats(output string) *RsyncStats {
} }
return stats return stats
} }
func isProgressOnlyLine(line string) bool {
return perFileProgressRegex.MatchString(line)
}
func parseProgressFields(line string) *ProgressFields {
m := perFileProgressRegex.FindStringSubmatch(line)
if m == nil {
return nil
}
bytes, _ := strconv.ParseInt(strings.ReplaceAll(m[1], ",", ""), 10, 64)
pct, _ := strconv.Atoi(m[2])
speed, _ := strconv.ParseFloat(m[3], 64)
unit := m[4]
eta := m[5]
suffix := m[6]
speedBps := int64(speed * 1e6)
switch unit {
case "k", "K":
speedBps = int64(speed * 1e3)
case "m", "M":
speedBps = int64(speed * 1e6)
case "g", "G":
speedBps = int64(speed * 1e9)
}
etaSecs := 0
parts := strings.Split(eta, ":")
if len(parts) == 3 {
h, _ := strconv.Atoi(parts[0])
m, _ := strconv.Atoi(parts[1])
s, _ := strconv.Atoi(parts[2])
etaSecs = h*3600 + m*60 + s
}
pf := &ProgressFields{
FileBytes: bytes,
Pct: pct,
SpeedBps: speedBps,
EtaSeconds: etaSecs,
}
if suffix != "" {
done, total := parseXfrSuffix(suffix)
pf.XfrDone = done
pf.XfrTotal = total
}
return pf
}
func isFileNameLine(line string) bool {
if line == "" || strings.TrimSpace(line) == "" {
return false
}
if strings.Contains(line, "sending incremental file list") ||
strings.Contains(line, "building file list") ||
strings.Contains(line, "cannot open") ||
strings.Contains(line, "skipping non-regular") ||
strings.HasPrefix(line, "sent ") ||
strings.HasPrefix(line, "total ") ||
strings.HasPrefix(line, "Number of files:") ||
strings.Contains(line, "bytes received") {
return false
}
return !isProgressOnlyLine(line)
}
+124
View File
@@ -0,0 +1,124 @@
package syncengine
import "testing"
func TestIsProgressOnlyLine(t *testing.T) {
cases := []struct {
name string
line string
expect bool
}{
{"per-file progress 0%", " 32,768 0% 0.00kB/s 0:00:00", true},
{"per-file progress 7%", " 2,260,893,696 7% 51.14MB/s 0:08:45", true},
{"per-file progress with xfr suffix", " 67,141,632 0% 32.02MB/s 0:15:06 (xfr#1, to-chk=4/10)", true},
{"filename line", "Dragon Ball Sleeping Princess in Devil's Castle (1987)/", false},
{"sending incremental file list header", "sending incremental file list", false},
{"sent bytes stats", "sent 123,456 bytes received 789 bytes 12.34kB/s", false},
{"total size stats", "total size is 999,999,999 speedup is 1.23", false},
{"Number of files stats", "Number of files: 10", false},
{"building file list", "building file list ...", false},
{"empty line", "", false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := isProgressOnlyLine(tc.line)
if got != tc.expect {
t.Errorf("isProgressOnlyLine(%q) = %v, want %v", tc.line, got, tc.expect)
}
})
}
}
func TestParseProgressFields(t *testing.T) {
cases := []struct {
name string
line string
wantPct int
wantSpeedBps int64
wantEtaSeconds int
wantXfrDone int
wantXfrTotal int
}{
{
name: "progress 0% with kB/s",
line: " 32,768 0% 0.00kB/s 0:00:00",
wantPct: 0,
wantSpeedBps: 0,
wantEtaSeconds: 0,
},
{
name: "progress 7% with MB/s",
line: " 2,260,893,696 7% 51.14MB/s 0:08:45",
wantPct: 7,
wantSpeedBps: 51_140_000,
wantEtaSeconds: 8*60 + 45,
},
{
name: "progress with xfr suffix",
line: " 67,141,632 0% 32.02MB/s 0:15:06 (xfr#1, to-chk=4/10)",
wantPct: 0,
wantSpeedBps: 32_020_000,
wantEtaSeconds: 15*60 + 6,
wantXfrDone: 1,
wantXfrTotal: 10,
},
{
name: "progress with GB/s",
line: " 1,234,567,890 50% 1.23GB/s 0:01:30",
wantPct: 50,
wantSpeedBps: 1_230_000_000,
wantEtaSeconds: 1*60 + 30,
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
p := parseProgressFields(tc.line)
if p == nil {
t.Fatalf("parseProgressFields(%q) returned nil, want non-nil", tc.line)
}
if p.Pct != tc.wantPct {
t.Errorf("pct = %d, want %d", p.Pct, tc.wantPct)
}
if p.SpeedBps != tc.wantSpeedBps {
t.Errorf("speedBps = %d, want %d", p.SpeedBps, tc.wantSpeedBps)
}
if p.EtaSeconds != tc.wantEtaSeconds {
t.Errorf("etaSeconds = %d, want %d", p.EtaSeconds, tc.wantEtaSeconds)
}
if tc.wantXfrTotal > 0 && p.XfrDone != tc.wantXfrDone {
t.Errorf("xfrDone = %d, want %d", p.XfrDone, tc.wantXfrDone)
}
if tc.wantXfrTotal > 0 && p.XfrTotal != tc.wantXfrTotal {
t.Errorf("xfrTotal = %d, want %d", p.XfrTotal, tc.wantXfrTotal)
}
})
}
}
func TestIsFileNameLine(t *testing.T) {
cases := []struct {
name string
line string
expect bool
}{
{"directory path", "Dragon Ball Sleeping Princess in Devil's Castle (1987)/", true},
{"file path", "Dragon Ball Sleeping Princess in Devil's Castle (1987)/Dragon Ball...WEBDL-2160p.mkv", true},
{"sending incremental file list header", "sending incremental file list", false},
{"sent stats", "sent 12,345 bytes received 1,234 bytes", false},
{"total size stats", "total size is 999,999,999", false},
{"Number of files", "Number of files: 10", false},
{"progress line", " 2,260,893,696 7% 51.14MB/s 0:08:45", false},
{"empty", "", false},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
got := isFileNameLine(tc.line)
if got != tc.expect {
t.Errorf("isFileNameLine(%q) = %v, want %v", tc.line, got, tc.expect)
}
})
}
}
+44 -19
View File
@@ -14,59 +14,84 @@ type Queue struct {
type RunInfo struct { type RunInfo struct {
JobID int64 JobID int64
SyncPairID int64
Cancel func() Cancel func()
ByUser bool CancelledBy bool
} }
func NewQueue() *Queue { func NewQueue() *Queue {
return &Queue{runs: make(map[int64]*RunInfo)} return &Queue{runs: make(map[int64]*RunInfo)}
} }
func (q *Queue) Enqueue(syncPairID, jobID int64, cancel func()) error { func (q *Queue) Enqueue(syncPairID, jobID int64, cancelFn func()) error {
q.mu.Lock() q.mu.Lock()
defer q.mu.Unlock() defer q.mu.Unlock()
if _, exists := q.runs[syncPairID]; exists { if _, exists := q.runs[jobID]; exists {
return ErrAlreadyRunning return ErrAlreadyRunning
} }
q.runs[syncPairID] = &RunInfo{JobID: jobID, Cancel: cancel} for _, info := range q.runs {
if info.SyncPairID == syncPairID {
return ErrAlreadyRunning
}
}
q.runs[jobID] = &RunInfo{JobID: jobID, SyncPairID: syncPairID, Cancel: cancelFn}
return nil return nil
} }
func (q *Queue) Dequeue(syncPairID int64) { func (q *Queue) Dequeue(jobID int64) {
q.mu.Lock() q.mu.Lock()
defer q.mu.Unlock() defer q.mu.Unlock()
delete(q.runs, syncPairID) delete(q.runs, jobID)
} }
func (q *Queue) IsRunning(syncPairID int64) bool { func (q *Queue) IsRunning(jobID int64) bool {
q.mu.Lock() q.mu.Lock()
defer q.mu.Unlock() defer q.mu.Unlock()
_, exists := q.runs[syncPairID] _, exists := q.runs[jobID]
return exists return exists
} }
func (q *Queue) GetJobID(syncPairID int64) (int64, bool) { func (q *Queue) GetByPair(syncPairID int64) (jobID int64, exists bool) {
q.mu.Lock() q.mu.Lock()
defer q.mu.Unlock() defer q.mu.Unlock()
info, exists := q.runs[syncPairID] for _, info := range q.runs {
if !exists { if info.SyncPairID == syncPairID {
return 0, false
}
return info.JobID, true return info.JobID, true
} }
}
return 0, false
}
func (q *Queue) Cancel(syncPairID int64, byUser bool) { func (q *Queue) Cancel(jobID int64, byUser bool) bool {
q.mu.Lock() q.mu.Lock()
defer q.mu.Unlock() defer q.mu.Unlock()
if info, exists := q.runs[syncPairID]; exists && info.Cancel != nil { if info, exists := q.runs[jobID]; exists && info.Cancel != nil {
info.ByUser = byUser info.CancelledBy = byUser
info.Cancel() info.Cancel()
return true
} }
return false
} }
func (q *Queue) IsCancelledByUser(syncPairID int64) bool { func (q *Queue) IsCancelledByUser(jobID int64) bool {
q.mu.Lock() q.mu.Lock()
defer q.mu.Unlock() defer q.mu.Unlock()
info, exists := q.runs[syncPairID] info, exists := q.runs[jobID]
return exists && info.ByUser return exists && info.CancelledBy
}
func (q *Queue) RunningJobs() []int64 {
q.mu.Lock()
defer q.mu.Unlock()
ids := make([]int64, 0, len(q.runs))
for id := range q.runs {
ids = append(ids, id)
}
return ids
}
func (q *Queue) Len() int {
q.mu.Lock()
defer q.mu.Unlock()
return len(q.runs)
} }
+15 -13
View File
@@ -7,7 +7,7 @@ import (
func TestQueue(t *testing.T) { func TestQueue(t *testing.T) {
q := NewQueue() q := NewQueue()
if q.IsRunning(1) { if q.IsRunning(100) {
t.Error("queue should be empty") t.Error("queue should be empty")
} }
@@ -16,35 +16,37 @@ func TestQueue(t *testing.T) {
err := q.Enqueue(1, 100, cancel) err := q.Enqueue(1, 100, cancel)
if err != nil { if err != nil {
t.Errorf("Enqueue(1) unexpected error: %v", err) t.Errorf("Enqueue(1, 100) unexpected error: %v", err)
} }
if !q.IsRunning(1) { if !q.IsRunning(100) {
t.Error("queue should contain syncPair 1") t.Error("queue should contain job 100")
} }
jobID, ok := q.GetJobID(1) jobID, ok := q.GetByPair(1)
if !ok || jobID != 100 { if !ok || jobID != 100 {
t.Errorf("GetJobID(1) = %d, %v, want 100, true", jobID, ok) t.Errorf("GetByPair(1) = %d, %v, want 100, true", jobID, ok)
} }
err = q.Enqueue(1, 200, nil) err = q.Enqueue(1, 200, nil)
if err != ErrAlreadyRunning { if err != ErrAlreadyRunning {
t.Errorf("Enqueue(1) again = %v, want ErrAlreadyRunning", err) t.Errorf("Enqueue(1, 200) = %v, want ErrAlreadyRunning", err)
} }
q.Cancel(1, true) ok = q.Cancel(100, true)
if !ok {
t.Error("Cancel(100) should return true")
}
if !cancelCalled { if !cancelCalled {
t.Error("Cancel should have called the cancel func") t.Error("Cancel should have called the cancel func")
} }
if !q.IsCancelledByUser(1) { if !q.IsCancelledByUser(100) {
t.Error("IsCancelledByUser should return true after Cancel(1, true)") t.Error("IsCancelledByUser should return true after Cancel(100, true)")
} }
q.Dequeue(1) q.Dequeue(100)
q.Dequeue(1) if q.IsRunning(100) {
if q.IsRunning(1) {
t.Error("queue should be empty after Dequeue") t.Error("queue should be empty after Dequeue")
} }
} }
+132 -9
View File
@@ -4,11 +4,97 @@ import (
"context" "context"
"fmt" "fmt"
"io" "io"
"log/slog"
"os/exec" "os/exec"
"path/filepath" "path/filepath"
"strconv"
"strings" "strings"
) )
var allowedRsyncFlags = map[string]bool{
"-v": true,
"-vv": true,
"-q": true,
"-h": true,
"-P": true,
"-n": true,
"-z": true,
"-c": true,
"-u": true,
"-W": true,
"-i": true,
"-a": true,
"-r": true,
"-l": true,
"-t": true,
"-p": true,
"-g": true,
"-o": true,
"-D": true,
"--verbose": true,
"--quiet": true,
"--help": true,
"--partial": true,
"--partial-dir": true,
"--delay-updates": true,
"--delete": true,
"--delete-before": true,
"--delete-after": true,
"--delete-excluded": true,
"--exclude": true,
"--exclude-from": true,
"--dry-run": true,
"--compress": true,
"--skip-compress": true,
"--whole-file": true,
"--checksum": true,
"--update": true,
"--existing": true,
"--ignore-existing": true,
"--remove-source-files": true,
"--chmod": true,
"--owner": true,
"--group": true,
"--perms": true,
"--executability": true,
"--acls": true,
"--xattrs": true,
"--numeric-ids": true,
"--fake-super": true,
"--bwlimit": true,
"--max-size": true,
"--min-size": true,
"--append": true,
"--append-verify": true,
"--itemize-changes": true,
}
var blockedRsyncFlags = map[string]bool{
"--rsync-path": true,
"-e": true,
"--files-from": true,
"--read-batch": true,
"--write-batch": true,
"--log-file": true,
}
func isSafeRsyncFlag(flag string) bool {
if allowedRsyncFlags[flag] {
return true
}
safePrefixes := []string{
"-a", "-v", "-z", "-P", "-n", "-c", "-u", "-W", "-i",
"--exclude=", "--chmod=", "--bwlimit=", "--max-size=", "--min-size=",
"--partial-dir=", "--skip-compress=",
}
for _, p := range safePrefixes {
if strings.HasPrefix(flag, p) {
return true
}
}
return false
}
type RsyncResult struct { type RsyncResult struct {
ExitCode int ExitCode int
Stdout string Stdout string
@@ -40,13 +126,13 @@ func NewRsyncRunner(sshDir, privKey string) *RsyncRunner {
} }
func (r *RsyncRunner) Run(ctx context.Context, pair *SyncPairConfig, onLine func(stream string, line string)) (*RsyncResult, error) { func (r *RsyncRunner) Run(ctx context.Context, pair *SyncPairConfig, onLine func(stream string, line string)) (*RsyncResult, error) {
cmd := r.buildRsyncCmd(pair) cmd := r.buildRsyncCmd(ctx, pair)
return r.runCmd(ctx, cmd, onLine) return r.runCmd(ctx, cmd, onLine)
} }
func (r *RsyncRunner) buildRsyncCmd(pair *SyncPairConfig) *exec.Cmd { func (r *RsyncRunner) buildRsyncCmd(ctx context.Context, pair *SyncPairConfig) *exec.Cmd {
args := r.buildArgs(pair) args := r.buildArgs(pair)
cmd := exec.CommandContext(context.Background(), "rsync", args...) cmd := exec.CommandContext(ctx, "rsync", args...)
if r.privKey != "" { if r.privKey != "" {
sshCmd := fmt.Sprintf("ssh -i %s -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=%s", sshCmd := fmt.Sprintf("ssh -i %s -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=%s",
r.privKey, strings.TrimRight(r.sshDir, "/")+"/known_hosts") r.privKey, strings.TrimRight(r.sshDir, "/")+"/known_hosts")
@@ -59,7 +145,19 @@ func (r *RsyncRunner) buildArgs(pair *SyncPairConfig) []string {
var args []string var args []string
flags := strings.Fields(pair.RsyncFlags) flags := strings.Fields(pair.RsyncFlags)
args = append(args, flags...) for _, flag := range flags {
if strings.HasPrefix(flag, "-") {
if blockedRsyncFlags[flag] {
slog.Warn("blocked dangerous rsync flag", "flag", flag)
continue
}
if !isSafeRsyncFlag(flag) {
slog.Warn("disallowed rsync flag", "flag", flag)
continue
}
}
args = append(args, flag)
}
for _, pattern := range pair.ExcludePatterns { for _, pattern := range pair.ExcludePatterns {
args = append(args, "--exclude="+pattern) args = append(args, "--exclude="+pattern)
@@ -69,15 +167,29 @@ func (r *RsyncRunner) buildArgs(pair *SyncPairConfig) []string {
args = append(args, "--delete") args = append(args, "--delete")
} }
args = append(args, "--")
src := ensureDirSlash(pair.Source)
if pair.Direction == "pull" { if pair.Direction == "pull" {
args = append(args, pair.Dest, pair.Source) args = append(args, pair.Dest, src)
} else { } else {
args = append(args, pair.Source, pair.Dest) args = append(args, src, pair.Dest)
} }
return args return args
} }
// ensureDirSlash guarantees the source path is treated by rsync as a
// directory whose contents are copied, regardless of whether the user
// supplied a trailing slash. This avoids the common foot-gun where
// "rsync host:/path/series /dest/" creates /dest/series/<contents> nested
// inside an extra "series" subdirectory.
func ensureDirSlash(p string) string {
if strings.HasSuffix(p, "/") {
return p
}
return p + "/"
}
type MachineKeys struct { type MachineKeys struct {
Host string Host string
Port int Port int
@@ -100,12 +212,23 @@ func (r *RsyncRunner) RunRemote(ctx context.Context, pair *SyncPairConfig, src *
innerSSH := fmt.Sprintf("ssh -i %s -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=%s", innerSSH := fmt.Sprintf("ssh -i %s -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=%s",
destKey, filepath.Join(r.sshDir, "known_hosts")) destKey, filepath.Join(r.sshDir, "known_hosts"))
rsyncFlags := strings.Join(args[:len(args)-2], " ") rsyncFlags := args[:len(args)-2]
sourcePath := args[len(args)-2] sourcePath := args[len(args)-2]
destPath := args[len(args)-1] destPath := args[len(args)-1]
remoteCmd := fmt.Sprintf("rsync %s -e %q %s %s", var rsyncCmd []string
rsyncFlags, innerSSH, sourcePath, destPath) rsyncCmd = append(rsyncCmd, "rsync")
rsyncCmd = append(rsyncCmd, "-e")
rsyncCmd = append(rsyncCmd, innerSSH)
rsyncCmd = append(rsyncCmd, rsyncFlags...)
rsyncCmd = append(rsyncCmd, sourcePath, destPath)
remoteCmd := "rsync"
for _, arg := range rsyncFlags {
remoteCmd += " " + strconv.Quote(arg)
}
remoteCmd += " -e " + strconv.Quote(innerSSH) + " " + strconv.Quote(sourcePath) + " " + strconv.Quote(destPath)
remoteCmd = "sh -c " + strconv.Quote(remoteCmd)
sshArgs := []string{ sshArgs := []string{
"-i", src.PrivKey, "-i", src.PrivKey,
+82 -7
View File
@@ -1,6 +1,7 @@
package syncengine package syncengine
import ( import (
"context"
"strings" "strings"
"testing" "testing"
) )
@@ -40,8 +41,8 @@ func TestBuildArgs_Push(t *testing.T) {
if args[0] != "-aP" { if args[0] != "-aP" {
t.Errorf("first flag = %q, want %q", args[0], "-aP") t.Errorf("first flag = %q, want %q", args[0], "-aP")
} }
if args[len(args)-2] != "/local/src" { if args[len(args)-2] != "/local/src/" {
t.Errorf("source = %q, want %q", args[len(args)-2], "/local/src") t.Errorf("source = %q, want %q (auto-appended trailing slash)", args[len(args)-2], "/local/src/")
} }
if args[len(args)-1] != "admin@10.5.0.144:/remote/dst" { if args[len(args)-1] != "admin@10.5.0.144:/remote/dst" {
t.Errorf("dest = %q, want %q", args[len(args)-1], "admin@10.5.0.144:/remote/dst") t.Errorf("dest = %q, want %q", args[len(args)-1], "admin@10.5.0.144:/remote/dst")
@@ -61,8 +62,8 @@ func TestBuildArgs_Pull(t *testing.T) {
if args[len(args)-2] != "/local/dst" { if args[len(args)-2] != "/local/dst" {
t.Errorf("pull: second-to-last (dest) = %q, want %q", args[len(args)-2], "/local/dst") t.Errorf("pull: second-to-last (dest) = %q, want %q", args[len(args)-2], "/local/dst")
} }
if args[len(args)-1] != "admin@10.5.0.144:/remote/src" { if args[len(args)-1] != "admin@10.5.0.144:/remote/src/" {
t.Errorf("pull: last (source) = %q, want %q", args[len(args)-1], "admin@10.5.0.144:/remote/src") t.Errorf("pull: last (source) = %q, want %q (auto-appended trailing slash)", args[len(args)-1], "admin@10.5.0.144:/remote/src/")
} }
} }
@@ -162,8 +163,8 @@ func TestRunRemote_PushSrcAndDestStayAsIs(t *testing.T) {
} }
srcArg, dstArg := tc.build() srcArg, dstArg := tc.build()
if srcArg != "/share/homes/admin/media" { if srcArg != "/share/homes/admin/media/" {
t.Errorf("push src = %q, want raw path '/share/homes/admin/media' (no user@host: prefix added)", srcArg) t.Errorf("push src = %q, want '/share/homes/admin/media/' (auto-appended trailing slash, no user@host: prefix added)", srcArg)
} }
if dstArg != "/share/media/peliculas" { if dstArg != "/share/media/peliculas" {
t.Errorf("push dst = %q, want raw path '/share/media/peliculas'", dstArg) t.Errorf("push dst = %q, want raw path '/share/media/peliculas'", dstArg)
@@ -193,6 +194,80 @@ func TestRunRemote_PullSrcAndDestStayAsIs(t *testing.T) {
} }
} }
func TestBuildArgs_AutoAppendsTrailingSlashToSource(t *testing.T) {
cases := []struct {
name string
source string
dest string
direction string
wantSrc string
}{
{
name: "push, source without trailing slash",
source: "/mnt/storage/multimedia/series",
dest: "/share/media/series",
direction: "push",
wantSrc: "/mnt/storage/multimedia/series/",
},
{
name: "push, source already has trailing slash (idempotent)",
source: "/mnt/storage/multimedia/series/",
dest: "/share/media/series",
direction: "push",
wantSrc: "/mnt/storage/multimedia/series/",
},
{
name: "push, remote source without trailing slash",
source: "admin@baby-nas:/mnt/storage/multimedia/series",
dest: "/share/media/series",
direction: "push",
wantSrc: "admin@baby-nas:/mnt/storage/multimedia/series/",
},
{
name: "pull, source without trailing slash",
source: "admin@baby-nas:/mnt/storage/multimedia/series",
dest: "/share/media/series",
direction: "pull",
wantSrc: "admin@baby-nas:/mnt/storage/multimedia/series/",
},
{
name: "mirror, source without trailing slash",
source: "/mnt/storage/multimedia/series",
dest: "admin@10.5.0.144:/share/media/series",
direction: "mirror",
wantSrc: "/mnt/storage/multimedia/series/",
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
runner := NewRsyncRunner("/tmp/ssh", "")
pair := &SyncPairConfig{
Source: tc.source,
Dest: tc.dest,
Direction: tc.direction,
}
args := runner.buildArgs(pair)
var gotSrc, gotDest string
if tc.direction == "pull" {
gotDest = args[len(args)-2]
gotSrc = args[len(args)-1]
} else {
gotSrc = args[len(args)-2]
gotDest = args[len(args)-1]
}
if gotSrc != tc.wantSrc {
t.Errorf("source = %q, want %q (dest must never be touched)", gotSrc, tc.wantSrc)
}
if gotDest != tc.dest {
t.Errorf("dest = %q, want %q (dest must never be normalized)", gotDest, tc.dest)
}
})
}
}
func countOccurrences(s, substr string) int { func countOccurrences(s, substr string) int {
return strings.Count(s, substr) return strings.Count(s, substr)
} }
@@ -207,7 +282,7 @@ func TestRun_FlagsPreservedWithPrivKey(t *testing.T) {
ExcludePatterns: []string{}, ExcludePatterns: []string{},
} }
cmd := runner.buildRsyncCmd(pair) cmd := runner.buildRsyncCmd(context.Background(), pair)
if cmd.Args[0] != "rsync" { if cmd.Args[0] != "rsync" {
t.Errorf("cmd.Args[0] = %q, want 'rsync'", cmd.Args[0]) t.Errorf("cmd.Args[0] = %q, want 'rsync'", cmd.Args[0])
+22 -1
View File
@@ -6,6 +6,7 @@ import {
NavLink, NavLink,
Outlet, Outlet,
useNavigate, useNavigate,
Link,
} from 'react-router-dom'; } from 'react-router-dom';
import { useState, useEffect } from 'react'; import { useState, useEffect } from 'react';
import { Toaster } from 'sonner'; import { Toaster } from 'sonner';
@@ -20,6 +21,8 @@ import {
LogOut, LogOut,
Menu, Menu,
X, X,
Clock,
ArrowLeft,
} from 'lucide-react'; } from 'lucide-react';
import { ErrorBoundary } from './components/ErrorBoundary'; import { ErrorBoundary } from './components/ErrorBoundary';
import { Spinner } from './components/ui/Spinner'; import { Spinner } from './components/ui/Spinner';
@@ -34,11 +37,13 @@ import JobHistory from './pages/JobHistory';
import JobDetail from './pages/JobDetail'; import JobDetail from './pages/JobDetail';
import SettingsPage from './pages/Settings'; import SettingsPage from './pages/Settings';
import SSHKeys from './pages/SSHKeys'; import SSHKeys from './pages/SSHKeys';
import Schedules from './pages/Schedules';
const navItems = [ const navItems = [
{ to: '/', label: 'Dashboard', icon: LayoutDashboard }, { to: '/', label: 'Dashboard', icon: LayoutDashboard },
{ to: '/machines', label: 'Machines', icon: Server }, { to: '/machines', label: 'Machines', icon: Server },
{ to: '/sync-pairs', label: 'Sync Pairs', icon: GitCompare }, { to: '/sync-pairs', label: 'Sync Pairs', icon: GitCompare },
{ to: '/schedules', label: 'Schedules', icon: Clock },
{ to: '/jobs', label: 'Jobs', icon: History }, { to: '/jobs', label: 'Jobs', icon: History },
{ to: '/ssh-keys', label: 'SSH Keys', icon: Key }, { to: '/ssh-keys', label: 'SSH Keys', icon: Key },
{ to: '/settings', label: 'Settings', icon: SettingsIcon }, { to: '/settings', label: 'Settings', icon: SettingsIcon },
@@ -217,6 +222,21 @@ function Layout() {
); );
} }
function NotFound() {
return (
<div className="flex flex-col items-center justify-center min-h-[50vh] gap-3">
<p className="text-2xl font-bold text-fg">404</p>
<p className="text-fg-muted">Page not found</p>
<Button variant="secondary" asChild>
<Link to="/">
<ArrowLeft className="h-4 w-4" />
Back to Dashboard
</Link>
</Button>
</div>
);
}
export default function App() { export default function App() {
return ( return (
<BrowserRouter> <BrowserRouter>
@@ -233,12 +253,13 @@ export default function App() {
<Route path="/" element={<Dashboard />} /> <Route path="/" element={<Dashboard />} />
<Route path="/machines" element={<Machines />} /> <Route path="/machines" element={<Machines />} />
<Route path="/sync-pairs" element={<SyncPairs />} /> <Route path="/sync-pairs" element={<SyncPairs />} />
<Route path="/schedules" element={<Schedules />} />
<Route path="/jobs" element={<JobHistory />} /> <Route path="/jobs" element={<JobHistory />} />
<Route path="/jobs/:id" element={<JobDetail />} /> <Route path="/jobs/:id" element={<JobDetail />} />
<Route path="/ssh-keys" element={<SSHKeys />} /> <Route path="/ssh-keys" element={<SSHKeys />} />
<Route path="/settings" element={<SettingsPage />} /> <Route path="/settings" element={<SettingsPage />} />
</Route> </Route>
<Route path="*" element={<Navigate to="/" />} /> <Route path="*" element={<NotFound />} />
</Routes> </Routes>
</ErrorBoundary> </ErrorBoundary>
</BrowserRouter> </BrowserRouter>
+10
View File
@@ -143,6 +143,16 @@ export interface ShutdownResponse {
error?: string; error?: string;
} }
export interface Schedule {
id: number;
sync_pair_id: number;
sync_pair_name: string;
cron_expr: string;
next_run_at: string | null;
enabled: boolean;
created_at: string;
}
export async function shutdownMachine(machineId: number): Promise<ShutdownResponse> { export async function shutdownMachine(machineId: number): Promise<ShutdownResponse> {
return api<ShutdownResponse>(`/api/machines/${machineId}/shutdown`, { return api<ShutdownResponse>(`/api/machines/${machineId}/shutdown`, {
method: 'POST', method: 'POST',
+27 -1
View File
@@ -1,6 +1,6 @@
import { useEffect, useState } from 'react'; import { useEffect, useState } from 'react';
import { Link } from 'react-router-dom'; import { Link } from 'react-router-dom';
import { Server, Activity, HardDrive, Clock, Plus } from 'lucide-react'; import { Server, Activity, HardDrive, Clock, Plus, XCircle } from 'lucide-react';
import { api, Machine, Job, SyncPair } from '../api/client'; import { api, Machine, Job, SyncPair } from '../api/client';
import { Badge } from '@/components/ui/Badge'; import { Badge } from '@/components/ui/Badge';
import { Button } from '@/components/ui/Button'; import { Button } from '@/components/ui/Button';
@@ -12,6 +12,7 @@ import { Skeleton } from '@/components/ui/Skeleton';
import { statusVariant, statusLabel } from '@/lib/status'; import { statusVariant, statusLabel } from '@/lib/status';
import { formatRelativeTime } from '@/lib/utils'; import { formatRelativeTime } from '@/lib/utils';
import { subscribeMachineStatus } from '@/lib/sse'; import { subscribeMachineStatus } from '@/lib/sse';
import { toast } from 'sonner';
export default function Dashboard() { export default function Dashboard() {
const [machines, setMachines] = useState<Machine[]>([]); const [machines, setMachines] = useState<Machine[]>([]);
@@ -48,6 +49,17 @@ export default function Dashboard() {
const pairName = (id: number) => pairs.find(p => p.id === id)?.name ?? `Pair ${id}`; const pairName = (id: number) => pairs.find(p => p.id === id)?.name ?? `Pair ${id}`;
async function cancelJob(jobId: number) {
try {
await api(`/api/jobs/${jobId}/cancel`, { method: 'POST' });
toast.success('Job cancelled');
const j = await api<Job[]>(`/api/jobs?limit=5`);
setJobs(j);
} catch (e: unknown) {
toast.error((e as Error).message);
}
}
const online = machines.filter(m => m.status.startsWith('online')).length; const online = machines.filter(m => m.status.startsWith('online')).length;
const todayJobs = jobs.filter(j => { const todayJobs = jobs.filter(j => {
if (!j.started_at) return false; if (!j.started_at) return false;
@@ -156,6 +168,7 @@ export default function Dashboard() {
<TableHead>Sync Pair</TableHead> <TableHead>Sync Pair</TableHead>
<TableHead>Status</TableHead> <TableHead>Status</TableHead>
<TableHead>Started</TableHead> <TableHead>Started</TableHead>
<TableHead className="w-16">Actions</TableHead>
</TableRow> </TableRow>
</TableHeader> </TableHeader>
<TableBody> <TableBody>
@@ -178,6 +191,19 @@ export default function Dashboard() {
<TableCell className="text-fg-muted text-xs"> <TableCell className="text-fg-muted text-xs">
{j.started_at ? formatRelativeTime(j.started_at) : '-'} {j.started_at ? formatRelativeTime(j.started_at) : '-'}
</TableCell> </TableCell>
<TableCell>
{['queued', 'waking_up', 'running'].includes(j.status) && (
<Button
variant="ghost"
size="icon-sm"
onClick={() => cancelJob(j.id)}
className="text-rose-400 hover:text-rose-300 hover:bg-rose-500/10"
title="Cancel job"
>
<XCircle className="h-3.5 w-3.5" />
</Button>
)}
</TableCell>
</TableRow> </TableRow>
))} ))}
</TableBody> </TableBody>
+155 -3
View File
@@ -1,6 +1,6 @@
import { useEffect, useState, useRef, useCallback } from 'react'; import { useEffect, useState, useRef, useCallback } from 'react';
import { useParams, Link } from 'react-router-dom'; import { useParams, Link } from 'react-router-dom';
import { api } from '../api/client'; import { api, apiRaw } from '../api/client';
import type { Job, LogLine, SyncPair } from '../api/client'; import type { Job, LogLine, SyncPair } from '../api/client';
import { Badge } from '@/components/ui/Badge'; import { Badge } from '@/components/ui/Badge';
import { Button } from '@/components/ui/Button'; import { Button } from '@/components/ui/Button';
@@ -28,18 +28,32 @@ import {
Terminal, Terminal,
AlertCircle, AlertCircle,
Ban, Ban,
ChevronDown,
Activity,
} from 'lucide-react'; } from 'lucide-react';
import { toast } from 'sonner'; import { toast } from 'sonner';
import { statusVariant, statusLabel, getErrorCodeInfo } from '@/lib/status'; import { statusVariant, statusLabel, getErrorCodeInfo } from '@/lib/status';
import { formatDuration } from '@/lib/utils'; import { formatDuration } from '@/lib/utils';
import { cn } from '@/lib/utils'; import { cn } from '@/lib/utils';
interface SSEProgress {
file_bytes: number;
pct: number;
speed_bps: number;
eta_seconds: number;
xfr_done: number;
xfr_total: number;
}
interface SSEEvent { interface SSEEvent {
type: string; type: string;
job_id: number; job_id: number;
status?: string; status?: string;
line?: string; line?: string;
stream?: string; stream?: string;
progress?: SSEProgress;
totalBytes?: number;
sentBytes?: number;
} }
export default function JobDetail() { export default function JobDetail() {
@@ -56,6 +70,10 @@ export default function JobDetail() {
const [cancelModal, setCancelModal] = useState(false); const [cancelModal, setCancelModal] = useState(false);
const [cancelReason, setCancelReason] = useState(''); const [cancelReason, setCancelReason] = useState('');
const [errorModal, setErrorModal] = useState(false); const [errorModal, setErrorModal] = useState(false);
const [progress, setProgress] = useState<SSEProgress | null>(null);
const [finalTotals, setFinalTotals] = useState<{ totalBytes: number; sentBytes: number } | null>(null);
const [hasMoreLogs, setHasMoreLogs] = useState(false);
const [loadingMore, setLoadingMore] = useState(false);
useEffect(() => { useEffect(() => {
loadJob(); loadJob();
@@ -78,6 +96,13 @@ export default function JobDetail() {
return updated; return updated;
}); });
} }
if (evt.type === 'progress' && evt.progress) {
setProgress(evt.progress);
}
if (evt.type === 'progress_total' && evt.totalBytes !== undefined && evt.sentBytes !== undefined) {
setFinalTotals({ totalBytes: evt.totalBytes, sentBytes: evt.sentBytes });
setProgress(null);
}
}; };
} }
return () => esRef.current?.close(); return () => esRef.current?.close();
@@ -109,8 +134,10 @@ export default function JobDetail() {
)) ?? []; )) ?? [];
if (offset === 0) { if (offset === 0) {
setLogs(ls); setLogs(ls);
setHasMoreLogs(ls.length === 1000);
} else { } else {
setLogs(prev => [...prev, ...ls]); setLogs(prev => [...prev, ...ls]);
setHasMoreLogs(ls.length === 1000);
} }
} catch {} } catch {}
} }
@@ -130,12 +157,33 @@ export default function JobDetail() {
} }
} }
function downloadLog() { async function downloadLog() {
try {
const resp = await apiRaw(`/api/jobs/${id}/log/download`);
if (resp.ok && resp.body) {
const blob = await resp.blob();
const url = URL.createObjectURL(blob);
const a = document.createElement('a');
a.href = url;
a.download = `job-${id}.log`;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(url);
} else {
fallbackDownload();
}
} catch {
fallbackDownload();
}
}
function fallbackDownload() {
const allLines = [ const allLines = [
...logs.map(l => `[${l.timestamp}] [${l.stream}] ${l.content}`), ...logs.map(l => `[${l.timestamp}] [${l.stream}] ${l.content}`),
...liveLines.map(l => `[LIVE] [${l.stream}] ${l.text}`), ...liveLines.map(l => `[LIVE] [${l.stream}] ${l.text}`),
]; ];
const blob = new Blob([allLines.join('\n')], { type: 'text/plain' }); const blob = new Blob(allLines as string[], { type: 'text/plain' });
const url = URL.createObjectURL(blob); const url = URL.createObjectURL(blob);
const a = document.createElement('a'); const a = document.createElement('a');
a.href = url; a.href = url;
@@ -321,6 +369,10 @@ export default function JobDetail() {
</div> </div>
)} )}
{(progress || finalTotals) && (
<TransferProgress progress={progress} finalTotals={finalTotals} />
)}
<Card className="flex flex-col min-h-0"> <Card className="flex flex-col min-h-0">
<div className="flex items-center justify-between px-4 py-3 border-b border-border shrink-0"> <div className="flex items-center justify-between px-4 py-3 border-b border-border shrink-0">
<div className="flex items-center gap-2"> <div className="flex items-center gap-2">
@@ -378,6 +430,23 @@ export default function JobDetail() {
</> </>
)} )}
<div ref={logEndRef} /> <div ref={logEndRef} />
{hasMoreLogs && (
<div className="flex justify-center py-2">
<Button
variant="secondary"
size="sm"
onClick={async () => {
setLoadingMore(true);
await loadLogs(logs.length);
setLoadingMore(false);
}}
disabled={loadingMore}
>
<ChevronDown className="h-4 w-4" />
{loadingMore ? 'Loading...' : 'Load more'}
</Button>
</div>
)}
</div> </div>
</Card> </Card>
@@ -452,3 +521,86 @@ function LogLine({
</div> </div>
); );
} }
function formatBytes(bytes: number): string {
if (bytes === 0) return '0 B';
const units = ['B', 'kB', 'MB', 'GB', 'TB'];
const i = Math.floor(Math.log(bytes) / Math.log(1024));
return `${(bytes / Math.pow(1024, i)).toFixed(1)} ${units[i]}`;
}
function formatSpeed(bps: number): string {
if (bps === 0) return '0 B/s';
const units = ['B/s', 'kB/s', 'MB/s', 'GB/s'];
const i = Math.floor(Math.log(bps) / Math.log(1000));
return `${(bps / Math.pow(1000, i)).toFixed(1)} ${units[i]}`;
}
function TransferProgress({
progress,
finalTotals,
}: {
progress: SSEProgress | null;
finalTotals: { totalBytes: number; sentBytes: number } | null;
}) {
const globalPct = finalTotals && finalTotals.totalBytes > 0
? Math.round((finalTotals.sentBytes / finalTotals.totalBytes) * 100)
: null;
return (
<Card>
<div className="p-4 space-y-3">
<div className="flex items-center gap-2">
<Activity className="h-4 w-4 text-accent" />
<span className="text-xs font-semibold text-fg-muted uppercase tracking-wider">
Transfer Progress
</span>
</div>
{progress && (
<>
<div className="space-y-1.5">
<div className="flex justify-between text-xs text-fg-muted">
<span>Current file</span>
<span>{progress.pct}%</span>
</div>
<div className="h-2 bg-border rounded-full overflow-hidden">
<div
className="h-full bg-accent transition-all duration-300 rounded-full"
style={{ width: `${progress.pct}%` }}
/>
</div>
</div>
<div className="flex justify-between text-xs text-fg-muted">
<span className="font-mono">
xfr#{(progress.xfr_done).toLocaleString()}/{progress.xfr_total > 0 ? progress.xfr_total.toLocaleString() : '?'}
</span>
<span className="font-mono">
{formatSpeed(progress.speed_bps)}
</span>
<span className="font-mono">
ETA {progress.eta_seconds > 0 ? `${Math.floor(progress.eta_seconds / 60)}m ${progress.eta_seconds % 60}s` : '-'}
</span>
</div>
</>
)}
{finalTotals && (
<div className="space-y-1.5">
<div className="flex justify-between text-xs text-fg-muted">
<span>Total transferred</span>
<span>{globalPct}% {formatBytes(finalTotals.sentBytes)} / {formatBytes(finalTotals.totalBytes)}</span>
</div>
<div className="h-2 bg-border rounded-full overflow-hidden">
<div
className="h-full bg-emerald-500 transition-all duration-300 rounded-full"
style={{ width: `${globalPct ?? 0}%` }}
/>
</div>
</div>
)}
</div>
</Card>
);
}
+334
View File
@@ -0,0 +1,334 @@
import { useEffect, useState } from 'react';
import { api, SyncPair, Schedule } from '../api/client';
import { Button } from '@/components/ui/Button';
import { Input } from '@/components/ui/Input';
import { Label } from '@/components/ui/Label';
import { Select, SelectValue, SelectTrigger, SelectContent, SelectItem } from '@/components/ui/Select';
import {
Modal,
ModalContent,
ModalHeader,
ModalTitle,
ModalDescription,
ModalBody,
ModalFooter,
} from '@/components/ui/Modal';
import { PageHeader } from '@/components/ui/PageHeader';
import {
Table,
TableHeader,
TableBody,
TableHead,
TableRow,
TableCell,
} from '@/components/ui/Table';
import { EmptyState } from '@/components/ui/EmptyState';
import { Badge } from '@/components/ui/Badge';
import { Card } from '@/components/ui/Card';
import { Trash2, Plus, Clock, Pencil, AlertTriangle } from 'lucide-react';
import { toast } from 'sonner';
import { cn } from '@/lib/utils';
type ScheduleForm = {
id: number | undefined;
sync_pair_id: number | null;
cron_expr: string;
enabled: boolean;
};
const defaultForm: ScheduleForm = {
id: undefined,
sync_pair_id: null,
cron_expr: '',
enabled: true,
};
export default function Schedules() {
const [schedules, setSchedules] = useState<Schedule[]>([]);
const [syncPairs, setSyncPairs] = useState<SyncPair[]>([]);
const [modalOpen, setModalOpen] = useState(false);
const [deleteId, setDeleteId] = useState<number | null>(null);
const [form, setForm] = useState<ScheduleForm>(defaultForm);
const [loading, setLoading] = useState(false);
useEffect(() => {
load();
}, []);
async function load() {
try {
const [s, p] = await Promise.all([
api<Schedule[]>('/api/schedules'),
api<SyncPair[]>('/api/sync-pairs'),
]);
setSchedules(s);
setSyncPairs(p);
} catch {}
}
function openCreate() {
setForm(defaultForm);
setModalOpen(true);
}
async function handleSubmit(e: React.FormEvent) {
e.preventDefault();
if (!form.sync_pair_id) {
toast.error('Sync pair is required');
return;
}
if (!form.cron_expr.trim()) {
toast.error('Cron expression is required');
return;
}
setLoading(true);
try {
await api(form.id ? `/api/schedules/${form.id}` : '/api/schedules', {
method: form.id ? 'PUT' : 'POST',
body: {
sync_pair_id: form.sync_pair_id,
cron_expr: form.cron_expr,
enabled: form.enabled,
},
});
setModalOpen(false);
toast.success(form.id ? 'Schedule updated' : 'Schedule created');
load();
} catch (e: unknown) {
toast.error((e as Error).message);
} finally {
setLoading(false);
}
}
async function handleToggleEnabled(schedule: Schedule) {
try {
await api(`/api/schedules/${schedule.id}`, {
method: 'PUT',
body: {
cron_expr: schedule.cron_expr,
enabled: !schedule.enabled,
},
});
toast.success(`Schedule ${schedule.enabled ? 'disabled' : 'enabled'}`);
load();
} catch (e: unknown) {
toast.error((e as Error).message);
}
}
async function handleDelete() {
if (deleteId === null) return;
try {
await api(`/api/schedules/${deleteId}`, { method: 'DELETE' });
toast.success('Schedule deleted');
setDeleteId(null);
load();
} catch (e: unknown) {
toast.error((e as Error).message);
}
}
function syncPairName(id: number) {
const p = syncPairs.find(p => p.id === id);
return p ? p.name : `Sync Pair ${id}`;
}
function formatNextRun(nextRun: string | null) {
if (!nextRun) return 'Not scheduled';
const d = new Date(nextRun);
return d.toLocaleString();
}
return (
<div className="space-y-6">
<PageHeader
title="Schedules"
description="Automate sync pair execution with cron-based scheduling"
actions={
<Button onClick={openCreate} size="sm">
<Plus className="h-4 w-4" />
Add Schedule
</Button>
}
/>
<Card>
<div className="p-0">
{schedules.length === 0 ? (
<EmptyState
icon={<Clock className="h-5 w-5" />}
title="No schedules"
description="Create a schedule to automate sync pair execution"
action={
<Button onClick={openCreate} size="sm">
<Plus className="h-4 w-4" />
Add Schedule
</Button>
}
/>
) : (
<Table>
<TableHeader>
<TableRow>
<TableHead>Sync Pair</TableHead>
<TableHead>Cron Expression</TableHead>
<TableHead>Next Run</TableHead>
<TableHead>Status</TableHead>
<TableHead className="w-28">Actions</TableHead>
</TableRow>
</TableHeader>
<TableBody>
{schedules.map(s => (
<TableRow key={s.id}>
<TableCell className="font-medium">{syncPairName(s.sync_pair_id)}</TableCell>
<TableCell>
<code className="text-xs bg-surface-raised px-2 py-1 rounded font-mono">
{s.cron_expr}
</code>
</TableCell>
<TableCell className="text-fg-muted text-sm">
{formatNextRun(s.next_run_at)}
</TableCell>
<TableCell>
<button
onClick={() => handleToggleEnabled(s)}
className={cn(
'relative inline-flex h-5 w-9 items-center rounded-full transition-colors focus:outline-none focus:ring-2 focus:ring-accent/40 focus:ring-offset-2 focus:ring-offset-canvas',
s.enabled ? 'bg-emerald-500/20' : 'bg-surface-raised'
)}
>
<span
className={cn(
'inline-block h-3.5 w-3.5 transform rounded-full bg-fg-muted transition-transform',
s.enabled ? 'translate-x-4 bg-emerald-400' : 'translate-x-1 bg-fg-subtle'
)}
/>
</button>
</TableCell>
<TableCell>
<div className="flex items-center gap-1">
<Button
variant="ghost"
size="icon-sm"
onClick={() => setDeleteId(s.id)}
className="text-rose-400 hover:text-rose-300 hover:bg-rose-500/10"
title="Delete"
>
<Trash2 className="h-3.5 w-3.5" />
</Button>
</div>
</TableCell>
</TableRow>
))}
</TableBody>
</Table>
)}
</div>
</Card>
<Modal open={modalOpen} onOpenChange={setModalOpen}>
<ModalContent size="md">
<ModalHeader>
<ModalTitle>Add Schedule</ModalTitle>
<ModalDescription>
Schedule automated sync pair execution using cron syntax.
Format: &quot;minute hour day-of-month month day-of-week&quot;
</ModalDescription>
</ModalHeader>
<form onSubmit={handleSubmit}>
<ModalBody className="space-y-4">
<div className="space-y-1.5">
<Label htmlFor="sch-sync-pair" required>
Sync Pair
</Label>
<Select
value={form.sync_pair_id?.toString() ?? ''}
onValueChange={v => setForm({ ...form, sync_pair_id: Number(v) })}
>
<SelectTrigger id="sch-sync-pair">
<SelectValue placeholder="Select a sync pair" />
</SelectTrigger>
<SelectContent>
{syncPairs.map(p => (
<SelectItem key={p.id} value={p.id.toString()}>
{p.name}
</SelectItem>
))}
</SelectContent>
</Select>
</div>
<div className="space-y-1.5">
<Label htmlFor="sch-cron" required>
Cron Expression
</Label>
<Input
id="sch-cron"
placeholder="0 2 * * *"
value={form.cron_expr}
onChange={e => setForm({ ...form, cron_expr: e.target.value })}
className="font-mono"
/>
<p className="text-xs text-fg-subtle">
Format: &quot;m h dom mon dow&quot; (5 fields, no seconds)
<br />
Examples: &quot;0 2 * * *&quot; (daily at 2am), &quot;0 */6 * * *&quot; (every 6 hours)
</p>
</div>
<div
className={cn(
'flex items-center gap-2 rounded-card p-3 transition-colors',
form.enabled
? 'bg-accent/5 border border-accent/20'
: 'bg-surface-raised border border-border'
)}
>
<input
type="checkbox"
id="sch-enabled"
checked={form.enabled}
onChange={e => setForm({ ...form, enabled: e.target.checked })}
className="h-4 w-4 rounded border-border accent-accent"
/>
<Label htmlFor="sch-enabled" className="cursor-pointer mb-0">
Enable this schedule
</Label>
</div>
</ModalBody>
<ModalFooter>
<Button
type="button"
variant="secondary"
onClick={() => setModalOpen(false)}
>
Cancel
</Button>
<Button type="submit" loading={loading}>
Add Schedule
</Button>
</ModalFooter>
</form>
</ModalContent>
</Modal>
<Modal open={deleteId !== null} onOpenChange={v => !v && setDeleteId(null)}>
<ModalContent size="sm">
<ModalHeader>
<ModalTitle>Delete Schedule</ModalTitle>
<ModalDescription>
Are you sure you want to delete this schedule?
</ModalDescription>
</ModalHeader>
<ModalFooter>
<Button variant="secondary" onClick={() => setDeleteId(null)}>
Cancel
</Button>
<Button variant="danger-solid" onClick={handleDelete}>
Delete
</Button>
</ModalFooter>
</ModalContent>
</Modal>
</div>
);
}
+21 -2
View File
@@ -3,16 +3,21 @@ import { CopyButton } from '@/components/ui/CopyButton';
import { Button } from '@/components/ui/Button'; import { Button } from '@/components/ui/Button';
import { PageHeader } from '@/components/ui/PageHeader'; import { PageHeader } from '@/components/ui/PageHeader';
import { Card, CardHeader, CardTitle, CardBody } from '@/components/ui/Card'; import { Card, CardHeader, CardTitle, CardBody } from '@/components/ui/Card';
import { Key, Download, Terminal } from 'lucide-react'; import { Key, Download, Terminal, HardDrive } from 'lucide-react';
import { api, SettingsInfo } from '../api/client';
export default function Settings() { export default function Settings() {
const [pubKey, setPubKey] = useState(''); const [pubKey, setPubKey] = useState('');
const [dataDir, setDataDir] = useState('');
const [loading, setLoading] = useState(true); const [loading, setLoading] = useState(true);
useEffect(() => { useEffect(() => {
Promise.all([
fetch('/api/settings/pubkey', { credentials: 'include' }) fetch('/api/settings/pubkey', { credentials: 'include' })
.then(r => (r.ok ? r.text() : '')) .then(r => (r.ok ? r.text() : ''))
.then(t => setPubKey(t)) .then(t => setPubKey(t)),
api<SettingsInfo>('/api/settings/info').then(info => setDataDir(info.data_dir)),
])
.catch(() => {}) .catch(() => {})
.finally(() => setLoading(false)); .finally(() => setLoading(false));
}, []); }, []);
@@ -75,6 +80,20 @@ export default function Settings() {
</CardBody> </CardBody>
</Card> </Card>
<Card>
<CardHeader>
<div className="flex items-center gap-2">
<div className="rounded-card bg-accent/10 p-1">
<HardDrive className="h-4 w-4 text-accent" />
</div>
<CardTitle>Data Directory</CardTitle>
</div>
</CardHeader>
<CardBody>
<div className="text-sm font-mono text-fg">{dataDir || '-'}</div>
</CardBody>
</Card>
<Card> <Card>
<CardHeader> <CardHeader>
<div className="flex items-center gap-2"> <div className="flex items-center gap-2">
+31 -3
View File
@@ -26,7 +26,7 @@ import {
import { EmptyState } from '@/components/ui/EmptyState'; import { EmptyState } from '@/components/ui/EmptyState';
import { Badge } from '@/components/ui/Badge'; import { Badge } from '@/components/ui/Badge';
import { Card } from '@/components/ui/Card'; import { Card } from '@/components/ui/Card';
import { Play, Trash2, Plus, GitCompare, ArrowRight, ArrowLeft, Pencil } from 'lucide-react'; import { Play, Trash2, Plus, GitCompare, ArrowRight, ArrowLeft, Pencil, AlertTriangle } from 'lucide-react';
import { toast } from 'sonner'; import { toast } from 'sonner';
import { cn } from '@/lib/utils'; import { cn } from '@/lib/utils';
@@ -295,8 +295,8 @@ export default function SyncPairs() {
<ModalTitle>{form.id ? 'Edit Sync Pair' : 'Add Sync Pair'}</ModalTitle> <ModalTitle>{form.id ? 'Edit Sync Pair' : 'Add Sync Pair'}</ModalTitle>
<ModalDescription> <ModalDescription>
{form.id {form.id
? 'Update the configuration for this sync pair' ? 'Update this sync pair. Direction: push (src→dst), pull (dst←src), or mirror (push + --delete).'
: 'Define a new source and destination for data syncing'} : 'Define source and destination for data syncing. Pick a direction: push (src→dst), pull (dst←src), or mirror (push + --delete).'}
</ModalDescription> </ModalDescription>
</ModalHeader> </ModalHeader>
<form onSubmit={handleSubmit}> <form onSubmit={handleSubmit}>
@@ -375,6 +375,9 @@ export default function SyncPairs() {
setForm({ ...form, source_path: e.target.value }) setForm({ ...form, source_path: e.target.value })
} }
/> />
<p className="text-xs text-fg-subtle">
Directory on the source machine. Its <span className="font-medium">contents</span> will be copied into the destination. Trailing <code className="font-mono">/</code> is optional.
</p>
</div> </div>
<div className="space-y-1.5"> <div className="space-y-1.5">
<Label htmlFor="sp-dest-path" required> <Label htmlFor="sp-dest-path" required>
@@ -388,6 +391,9 @@ export default function SyncPairs() {
setForm({ ...form, dest_path: e.target.value }) setForm({ ...form, dest_path: e.target.value })
} }
/> />
<p className="text-xs text-fg-subtle">
Directory on the destination machine where the source contents will land.
</p>
</div> </div>
</div> </div>
<div className="grid grid-cols-2 gap-3"> <div className="grid grid-cols-2 gap-3">
@@ -408,6 +414,12 @@ export default function SyncPairs() {
<SelectItem value="mirror">Mirror</SelectItem> <SelectItem value="mirror">Mirror</SelectItem>
</SelectContent> </SelectContent>
</Select> </Select>
<p className="text-xs text-fg-subtle">
<span className="font-medium">Push</span> copies source destination.{' '}
<span className="font-medium">Pull</span> reverses it (dest source).{' '}
<span className="font-medium">Mirror</span> is like push but adds{' '}
<code className="font-mono">--delete</code> (see warning).
</p>
</div> </div>
<div className="space-y-1.5"> <div className="space-y-1.5">
<Label htmlFor="sp-rsync-flags">Rsync Flags</Label> <Label htmlFor="sp-rsync-flags">Rsync Flags</Label>
@@ -421,6 +433,22 @@ export default function SyncPairs() {
/> />
</div> </div>
</div> </div>
{form.direction === 'mirror' && (
<div
role="alert"
className="flex items-start gap-2 rounded-card p-3 bg-amber-500/10 border border-amber-500/30"
>
<AlertTriangle className="h-4 w-4 text-amber-400 shrink-0 mt-0.5" />
<div className="text-xs text-amber-200 leading-relaxed">
<span className="font-semibold">Mirror will delete files.</span>{' '}
With <code className="font-mono">--delete</code>, any file in the
destination that doesn&apos;t exist in the source is permanently
removed. Double-check both paths before saving a wrong
destination (e.g. <code className="font-mono">/</code> or{' '}
<code className="font-mono">/home</code>) can wipe data.
</div>
</div>
)}
<div className="space-y-1.5"> <div className="space-y-1.5">
<Label htmlFor="sp-exclude">Exclude Patterns</Label> <Label htmlFor="sp-exclude">Exclude Patterns</Label>
<Textarea <Textarea