Skip to content

Commit 5549afd

Browse files
committed
Implement SQL Server
1 parent 37915e4 commit 5549afd

33 files changed

Lines changed: 1799 additions & 86 deletions

‎README.md‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ The service runs as LocalSystem, keeps its state (`agent.yaml`, caches, logs) in
6060

6161
### Database backups
6262

63-
`fallbck-agent` backs up **PostgreSQL** (`pg_dump` per database + `pg_dumpall --globals-only`, and/or `pg_basebackup` with streamed WAL), **MySQL/MariaDB** (`mysqldump`/`mariadb-dump --single-transaction` with the binlog position, one `--all-databases` file or one file per database, optionally the binary logs written since the previous run) and **MongoDB** (`mongodump --oplog`, restorable with `--oplogReplay`) on every host where the vendor client tools are installed. The agent detects the tools at start and every ten minutes and reports them with its heartbeat (`db_tools`, shown as a *DB tools* badge on the Agents page); database jobs can then be assigned to it. Nothing else changes: same enrollment, same service, same data directory as the filesystem backups.
63+
`fallbck-agent` backs up **PostgreSQL** (`pg_dump` per database + `pg_dumpall --globals-only`, and/or `pg_basebackup` with streamed WAL), **MySQL/MariaDB** (`mysqldump`/`mariadb-dump --single-transaction` with the binlog position, one `--all-databases` file or one file per database, optionally the binary logs written since the previous run) and **MongoDB** (`mongodump --oplog`, restorable with `--oplogReplay`) on every host where the vendor client tools are installed. The agent detects the tools at start and every ten minutes and reports the engines it can back up with its heartbeat (`db_engines`, shown as a *DB: postgres, mssql* badge on the Agents page; `mssql` is always among them); a database job can be assigned to an agent that reports the job's engine. Nothing else changes: same enrollment, same service, same data directory as the filesystem backups.
6464

6565
```bash
6666
apt install fallbck-agent postgresql-client mariadb-client # or the tools for the engines you back up
@@ -73,6 +73,12 @@ Database jobs are created in the Web UI (job type *Database*: engine, host, user
7373

7474
Restore: `proxmox-backup-client restore <snapshot> db.pxar <dir>` (or download the manifest blob first) and follow `restore.commands` in `fallbck-db.json`. The `oci.fi/fallbck/agent:tools` image (target `tools` of `Dockerfile.agent`) ships the client tools; the Linux packages rely on the distribution's `postgresql-client`, `mariadb-client`/`mysql-client` (binary-log fetching additionally needs `mysqlbinlog`/`mariadb-binlog`, which Alpine only ships in the `mariadb` package) and `mongodb-database-tools`. `pg_dump` must be at least as new as the server; the agent looks in the usual per-version directories (`/usr/lib/postgresql/*/bin`, `/usr/pgsql-*/bin`, `/usr/libexec/postgresql*`) or `db_backup.tools.postgres_bindir`.
7575

76+
**Microsoft SQL Server** is backed up natively, with no client tools: the agent runs `BACKUP DATABASE` / `BACKUP LOG ... TO DISK` over its built-in driver, SQL Server writes the files into a directory on its own host, and the agent moves them into its spool, uploads them as `db.pxar` and deletes them. Every `fallbck-agent` therefore reports the `mssql` engine; the usual setup is the agent installed on the SQL Server host itself (Windows service or Linux package). Job options: **mode** `auto` (a full backup when none is held, when the chain broke or when the last full is older than *full every N days*, a differential otherwise), `full`, `differential` or `log` (transaction log only, as its own job on a short schedule); *also back up the log* after each full/differential for databases in the FULL or BULK_LOGGED recovery model; native compression (off by default: PBS deduplicates uncompressed backups far better, and Express cannot compress); copy-only. `master` is always backed up in full, `tempdb` never, offline databases and snapshots are skipped with a note.
77+
78+
The agent keeps the chain it holds per database (last full, last differential, the log backups since) in its job state and compares it with `msdb` before every run: when the server's newest full backup is not the one FallBCK holds — another tool took one, or a run failed after the backup — the next differential becomes a full (*chain reset*), and a log backup that does not start where the previous held one ended is reported as a *log chain gap*. The run page lists every backup set with its LSNs and the reason for its type, and the restore commands give the chain in order: `RESTORE DATABASE ... WITH NORECOVERY` for the full and the newest differential, `RESTORE LOG` for the logs, then `WITH RECOVERY` (add `STOPAT` for a point in time).
79+
80+
Where the files land: by default the instance's **default backup path** (`InstanceDefaultBackupPath`), which the SQL Server service can write and an agent on the same host can read. A job may name another directory; when SQL Server runs elsewhere or in a container, give both views of the same directory: `backup_dir` as SQL Server sees it and `agent_dir` as the agent sees it (UNC share, bind mount). A probe file verifies the two match before any backup runs. Authentication is a SQL login (password on the job or in a credential profile) or, with the user left blank on Windows, integrated authentication as the agent's service account (LocalSystem needs a login with backup rights, e.g. `db_backupoperator` in every database or `sysadmin`). `host\\instance` selects a named instance. SQL Server jobs cannot run on a Kubernetes agent, because the backup file would be written inside the SQL Server pod.
81+
7682
### Kubernetes agent
7783

7884
`fallbck-k8s-agent` runs **inside** the target cluster (agent type `kubernetes`, one identity per install) and backs up **all API resources** plus **PVC data**: with CSI snapshots it takes a `VolumeSnapshot`, clones it to a temporary PVC and uploads the files; without snapshot support it mounts the live volume read-only (crash-consistent). Install with the Helm chart:

‎STRUCTURE.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -62,7 +62,7 @@ Database jobs (`job_type: database`, `db_config`) follow the same path with `fal
6262
| `pkg/agent/runtime` | shared agent daemon skeleton (config + self-enrollment, gateway connections, worker, heartbeat/token rotation) |
6363
| `pkg/agent/backup` | agent backup engine on top of gopbs (pxar upload, blobs) |
6464
| `pkg/dbbackup` | database job spec, credential profiles, manifest (shared by server, agent and config) |
65-
| `pkg/agent/dbbackup` | database engines (PostgreSQL, MySQL/MariaDB, MongoDB): tool detection, prechecks, dumps, spool, state |
65+
| `pkg/agent/dbbackup` | database engines (PostgreSQL, MySQL/MariaDB, MongoDB by their client tools; Microsoft SQL Server natively over the built-in driver with full/differential/log chains): tool detection, prechecks, dumps, spool, state |
6666
| `cmd/fallbck-k8s-agent` | in-cluster Kubernetes agent (Helm-installed; `worker` mode uploads one volume, `db-worker` mode dumps one database on the engine image) |
6767
| `pkg/k8sbackup` | kubernetes job spec, discovery plan, run manifest (shared) |
6868
| `pkg/pvebackup` | Proxmox VE host job spec and run manifest (shared) |

‎cmd/fallbck-agent/main.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -121,13 +121,13 @@ func main() {
121121
}
122122
logging.ApplyConfigLogLevel(cfg)
123123
// Database jobs run on this agent too when the host has the client tools:
124-
// the probe feeds the db_tools flag in heartbeats and gates the jobs server-side
124+
// the probe feeds the db_engines list in heartbeats, which gates the jobs server-side
125125
dbCfg := cfg.DBBackup
126126
dbCfg.ApplyDefaults(cfg.DataDir)
127127
if err := dbrun.CleanSpool(dbCfg); err != nil {
128128
slog.Warn("Could not clean the spool directory", "dir", dbCfg.SpoolDir, "error", err)
129129
}
130-
agent.DBTools = runtime.DBToolsProbe(dbCfg, 10*time.Minute)
130+
agent.DBEngines = runtime.DBEnginesProbe(dbCfg, 10*time.Minute)
131131
register := func(w worker.Worker, conns *runtime.Conns, store *runtime.TokenStore) error {
132132
acts := activities.NewAgentActivities()
133133
acts.Tunnel = conns.Tunnel

‎deploy/agent.yaml.example‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,3 +41,7 @@ token_expires_at: 2027-08-25T00:00:00Z
4141
# engine: mysql
4242
# user: backup
4343
# password: "s3cret"
44+
# mssql: # SQL Server: a SQL login; leave the job's user
45+
# engine: mssql # blank instead for Windows integrated authentication
46+
# user: fallbck_backup
47+
# password_file: /etc/fallbck/mssql-backup.pass

‎go.mod‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,8 +10,10 @@ require (
1010
github.com/go-chi/chi/v5 v5.3.2
1111
github.com/go-sql-driver/mysql v1.9.0
1212
github.com/golang-jwt/jwt/v5 v5.3.1
13+
github.com/golang-sql/sqlexp v0.1.0
1314
github.com/google/uuid v1.6.0
1415
github.com/jackc/pgx/v5 v5.10.0
16+
github.com/microsoft/go-mssqldb v1.11.0
1517
github.com/mxk/go-vss v1.2.1
1618
github.com/osshield/gopbs v0.3.0-rc.2
1719
github.com/robfig/cron/v3 v3.0.1
@@ -114,6 +116,7 @@ require (
114116
github.com/gocql/gocql v1.7.0 // indirect
115117
github.com/gogo/protobuf v1.3.2 // indirect
116118
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
119+
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
117120
github.com/golang/mock v1.6.0 // indirect
118121
github.com/golang/snappy v0.0.4 // indirect
119122
github.com/gomarkdown/markdown v0.0.0-20260411013819-759bbc3e3207 // indirect

‎go.sum‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,18 @@ dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7
2828
filippo.io/edwards25519 v1.1.0/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
2929
filippo.io/edwards25519 v1.1.1 h1:YpjwWWlNmGIDyXOn8zLzqiD+9TyIlPhGFG96P39uBpw=
3030
filippo.io/edwards25519 v1.1.1/go.mod h1:BxyFTGdWcka3PhytdK4V28tE5sGfRvvvRV7EaN4VDT4=
31+
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.0 h1:4gRPBpN1f6xt88yi4WR26m7XaD9OlWtVT6bWPdGUIok=
32+
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.23.0/go.mod h1:G7QVLxw1j1JVyrO1MA95S8m8HStaaleDZYTcfGgjB2o=
33+
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.0 h1:CU4+EJeJi3TKYWEcYuSdWsjzw0nVsK/H0MSQOiPcymU=
34+
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.14.0/go.mod h1:q0+UTSRvShwUCrR/s5HtyInYphN7Wvxb7snFM3u+SLA=
35+
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0 h1:fhqpLE3UEXi9lPaBRpQ6XuRW0nU7hgg4zlmZZa+a9q4=
36+
github.com/Azure/azure-sdk-for-go/sdk/internal v1.12.0/go.mod h1:7dCRMLwisfRH3dBupKeNCioWYUZ4SS09Z14H+7i8ZoY=
37+
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.5.0 h1:MaKvxE6D0KkjOg6Wd9M00iqP5PR0kUxCfiezes4JweM=
38+
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/azkeys v1.5.0/go.mod h1:i2h9fsTFKZorh8RdV2IcSUf/Qj98GlTkrTvUbX/s8as=
39+
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.2.0 h1:nCYfgcSyHZXJI8J0IWE5MsCGlb2xp9fJiXyxWgmOFg4=
40+
github.com/Azure/azure-sdk-for-go/sdk/security/keyvault/internal v1.2.0/go.mod h1:ucUjca2JtSZboY8IoUqyQyuuXvwbMBVwFOm0vdQPNhA=
41+
github.com/AzureAD/microsoft-authentication-library-for-go v1.7.2 h1:RHK7bS+HQMslb1sZpAokUt+zTVmue0hKSs2C791hhzU=
42+
github.com/AzureAD/microsoft-authentication-library-for-go v1.7.2/go.mod h1:HKpQxkWaGLJ+D/5H8QRpyQXA1eKjxkFlOMwck5+33Jk=
3143
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
3244
github.com/BurntSushi/toml v1.4.0/go.mod h1:ukJfTF/6rtPPRCnwkur4qwRxa8vTRFBF0uk2lLoLwho=
3345
github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
@@ -217,6 +229,10 @@ github.com/golang-jwt/jwt/v4 v4.5.2 h1:YtQM7lnr8iZ+j5q71MGKkNw9Mn7AjHM68uc9g5fXe
217229
github.com/golang-jwt/jwt/v4 v4.5.2/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
218230
github.com/golang-jwt/jwt/v5 v5.3.1 h1:kYf81DTWFe7t+1VvL7eS+jKFVWaUnK9cB1qbwn63YCY=
219231
github.com/golang-jwt/jwt/v5 v5.3.1/go.mod h1:fxCRLWMO43lRc8nhHWY6LGqRcf+1gQWArsqaEUEa5bE=
232+
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 h1:au07oEsX2xN0ktxqI+Sida1w446QrXBRJ0nee3SNZlA=
233+
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9/go.mod h1:8vg3r2VgvsThLBIFL93Qb5yWzgyZWhEmBwUJWevAkK0=
234+
github.com/golang-sql/sqlexp v0.1.0 h1:ZCD6MBpcuOVfGVqsEmY5/4FtYiKz6tSyUv9LPEDei6A=
235+
github.com/golang-sql/sqlexp v0.1.0/go.mod h1:J4ad9Vo8ZCWQ2GMrC4UCQy1JpCbwU9m3EOqtpKwwwHI=
220236
github.com/golang/freetype v0.0.0-20170609003504-e2365dfdc4a0/go.mod h1:E/TSTwGwJL78qG/PmXZO1EjYhfJinVAhrmmHX6Z8B9k=
221237
github.com/golang/mock v1.5.0/go.mod h1:CWnOUgYIOo4TcNZ0wHX3YZCqsaM1I1Jvs6v3mP3KVu8=
222238
github.com/golang/mock v1.6.0 h1:ErTB+efbowRARo13NNdxyJji2egdxLGQhRaY+DUumQc=
@@ -309,6 +325,8 @@ github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWE
309325
github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
310326
github.com/mattn/go-sqlite3 v1.14.22 h1:2gZY6PC6kBnID23Tichd1K+Z0oS6nE/XwU+Vz/5o4kU=
311327
github.com/mattn/go-sqlite3 v1.14.22/go.mod h1:Uh1q+B4BYcTPb+yiD3kU8Ct7aC0hY9fxUwlHK0RXw+Y=
328+
github.com/microsoft/go-mssqldb v1.11.0 h1:YbDqolEjGH9hBfvKzONTf5/dbl9RKXmizMJE93lVxNs=
329+
github.com/microsoft/go-mssqldb v1.11.0/go.mod h1:goQLDOPlMN/l1REhnNPElMoY/yX+fUWn1+7UoFJPH9Y=
312330
github.com/mitchellh/copystructure v1.2.0 h1:vpKXTN4ewci03Vljg/q9QvCGUDttBOGBIa15WveJJGw=
313331
github.com/mitchellh/copystructure v1.2.0/go.mod h1:qLl+cE2AmVv+CoeAwDPye/v+N2HKCj9FbZEVFJRxO9s=
314332
github.com/mitchellh/mapstructure v1.5.0 h1:jeMsZIYE/09sWLaz43PL7Gy6RuMjD2eJVyuac5Z2hdY=
@@ -337,6 +355,8 @@ github.com/opentracing/opentracing-go v1.2.0 h1:uEJPy/1a5RIPAJ0Ov+OIO8OxWu77jEv+
337355
github.com/opentracing/opentracing-go v1.2.0/go.mod h1:GxEUsuufX4nBwe+T+Wl9TAgYrxe9dPLANfrWvHYVTgc=
338356
github.com/osshield/gopbs v0.3.0-rc.2 h1:RRsiRp9f3nwj4nGW/GSKlfAX/OFmAusEhS6zzdOWm1M=
339357
github.com/osshield/gopbs v0.3.0-rc.2/go.mod h1:8sLWqNW3jyE3t5otnzaqqP4hpI/swnYziMmoLl/7qtM=
358+
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
359+
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU=
340360
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
341361
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
342362
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=

‎pkg/agent/dbbackup/engine.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,9 @@ type DumpRequest struct {
101101
// SpoolDir exists and is empty on entry; SecretDir (0700) holds credential files
102102
SpoolDir string
103103
SecretDir string
104+
// Snapshot is the PBS snapshot this run writes (type/id/time); engines that keep
105+
// a chain record it so later runs can point back at it
106+
Snapshot string
104107
// PrevState is the last successful run's state (nil on the first run)
105108
PrevState *dbspec.JobState
106109
Progress ProgressFunc
@@ -122,6 +125,7 @@ type EngineResult struct {
122125
Postgres *dbspec.PostgresManifest
123126
MySQL *dbspec.MySQLManifest
124127
MongoDB *dbspec.MongoManifest
128+
MSSQL *dbspec.MSSQLManifest
125129
Restore dbspec.Restore
126130
// State is the engine part of the job state saved after a successful upload
127131
State *dbspec.JobStateRecord
@@ -173,6 +177,7 @@ var engines = map[string]Engine{
173177
dbspec.EnginePostgres: postgresEngine{},
174178
dbspec.EngineMySQL: mysqlEngine{},
175179
dbspec.EngineMongoDB: mongoEngine{},
180+
dbspec.EngineMSSQL: mssqlEngine{},
176181
}
177182

178183
// Lookup returns the engine for a canonical engine name

0 commit comments

Comments
 (0)