VolSync Backup Strategy¶
Overview¶
VolSync provides Kubernetes-native persistent volume replication and backup, preventing data loss scenarios like the current cluster rebuild situation.
Architecture¶
Backup Flow¶
text
Kubernetes PVs (emerald/fuji) → VolSync → Restic → Apollo NFS → unRAID Storagetext
```text
Integration Points¶
- Source: Kubernetes persistent volumes (VM-FAST/VM-BULK storage)
- Transport: VolSync with restic backend
- Destination: Apollo "Freezer" pool (dedicated backup storage)
- Retention: Configurable snapshot policies
Implementation Plan¶
Phase 1: VolSync Installation¶
```bash
```bash
Install VolSync operator¶
kubectl apply -f https://github.com/backube/volsync/releases/latest/download/volsync.yaml
Verify installation¶
kubectl get pods -n volsync-system ```text
```text
Phase 2: Apollo NFS Configuration¶
Create dedicated backup shares on apollo "Freezer" pool¶
```text
text
/mnt/disks/Freezer/k8s-backups/
├── volsync-repo/ # Restic repository
├── vaultwarden/ # VaultWarden backups
├── wordpress/ # WordPress backups
├── configs/ # Config backups
└── cluster-state/ # Cluster configuration backupstext
```text
Freezer pool advantages¶
- Dedicated backup storage: 16x 600GB SAS drives in MD1220
- Separate from main array: Backup isolation from media storage
- Enterprise drives: High reliability for critical backup data
- ~9.6TB usable: Substantial capacity for comprehensive backups
Phase 3: Restic Repository Setup¶
```yaml
```yaml apiVersion: v1 kind: Secret metadata: name: restic-secret namespace: volsync-system type: Opaque stringData: RESTIC_REPOSITORY: "/mnt/nfs/volsync-repo" RESTIC_PASSWORD: "secure-backup-password"
apiVersion: volsync.backube/v1alpha1 kind: ReplicationDestination metadata: name: apollo-backup-dest namespace: volsync-system spec: trigger: schedule: "0 /6 * * " # Every 6 hours restic: repository: restic-secret retain: hourly: 24 daily: 7 weekly: 4 monthly: 3 ```text
```text
Service-Specific Backup Configurations¶
VaultWarden Backup (Critical)¶
```yaml
yaml
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
name: vaultwarden-backup
namespace: vaultwarden
spec:
sourcePVC: vaultwarden-data
trigger:
schedule: "0 */2 * * *" # Every 2 hours
restic:
repository: restic-secret
retain:
hourly: 48 # 2 days of hourly
daily: 14 # 2 weeks of daily
weekly: 8 # 2 months of weekly
monthly: 6 # 6 months of monthly
copyMethod: Snapshottext
```text
WordPress Database Backup¶
```yaml
yaml
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
name: wordpress-db-backup
namespace: wordpress
spec:
sourcePVC: wordpress-mysql-data
trigger:
schedule: "0 */4 * * *" # Every 4 hours
restic:
repository: restic-secret
retain:
hourly: 24
daily: 30
weekly: 12
monthly: 12
copyMethod: Snapshottext
```text
Media System Configs¶
```yaml
yaml
apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
name: media-configs-backup
namespace: media
spec:
sourcePVC: radarr-config
trigger:
schedule: "0 2 * * *" # Daily at 2 AM
restic:
repository: restic-secret
retain:
daily: 7
weekly: 4
monthly: 3text
```text
Recovery Procedures¶
Disaster Recovery Workflow¶
- Deploy new cluster (current situation)
- Install VolSync operator
- Configure restic repository access to apollo
- Create ReplicationDestination resources
- Restore specific PVs as needed
VaultWarden Recovery Example¶
```yaml
yaml
apiVersion: volsync.backube/v1alpha1
kind: ReplicationDestination
metadata:
name: vaultwarden-restore
namespace: vaultwarden
spec:
trigger:
manual: restore-20240921 # Trigger restore
restic:
repository: restic-secret
destinationPVC: vaultwarden-data-restored
capacity: 10Gitext
```text
Recovery Commands¶
```bash
```bash
List available snapshots¶
kubectl exec -it restic-pod -- restic -r /path/to/repo snapshots
Restore specific snapshot¶
kubectl apply -f vaultwarden-restore.yaml
Verify restored data¶
kubectl exec -it vaultwarden-pod -- ls -la /data ```text
```text
Monitoring and Alerting¶
Backup Health Monitoring¶
```yaml
```yaml
Prometheus monitoring for VolSync¶
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: volsync-metrics spec: selector: matchLabels: app.kubernetes.io/name: volsync endpoints:
- port: metrics
```text
```text
Alert Rules¶
```yaml
```yaml groups:
- name: volsync-alerts
rules:
-
alert: VolSyncBackupFailed
expr: volsync_replication_source_last_sync_status != 1
for: 1h annotations: summary: "VolSync backup failed for {{ $labels.name }}"
-
alert: VolSyncBackupStale
expr: time() - volsync_replication_source_last_sync_time > 86400
for: 30m annotations: summary: "VolSync backup stale for {{ $labels.name }}" ```text
```text
Apollo Integration¶
NFS Mount Configuration¶
```yaml
```yaml apiVersion: v1 kind: PersistentVolume metadata: name: apollo-backup-storage spec: capacity: storage: 10Ti accessModes:
- ReadWriteMany
nfs:
server: apollo.local
path: /mnt/user/k8s-backups
mountOptions:
- nfsvers=4.1
- rsize=1048576
- wsize=1048576
- hard
- intr
```text
```text
Multi-Tier Backup Strategy (3-2-1 Rule)¶
Tier 1: Production Storage¶
- Location: emerald/fuji (VM-FAST/VM-BULK)
- Purpose: Live application data
- Protection: RAID redundancy
Tier 2: Local Backup (Apollo Freezer)¶
- Location: Apollo Freezer pool (16x 600GB SAS)
- Purpose: Fast local recovery and frequent snapshots
- Retention: Hourly/daily/weekly snapshots
- Capacity: ~9.6TB dedicated backup storage
Tier 3: Offsite Backup (Backblaze B2)¶
- Location: Cloud storage (geographically separate)
- Purpose: Disaster recovery and long-term retention
- Data selection: Critical data only (cost optimization)
- Retention: Monthly snapshots, extended retention
Performance Considerations¶
Backup Timing¶
- Off-peak hours: Schedule during low-usage periods
- Staggered backups: Avoid multiple concurrent backups
- Resource limits: Prevent backup jobs from impacting applications
Network Impact¶
- Bandwidth management: Monitor network utilization during backups
- Compression: Restic provides built-in compression
- Deduplication: Restic deduplicates across snapshots
Security¶
Encryption¶
- At rest: Restic encrypts all backup data
- In transit: NFS over secure network (VLAN 103)
- Key management: Secure storage of restic passwords
Access Control¶
- RBAC: Limit VolSync operator permissions
- Network policies: Restrict backup traffic to appropriate VLANs
- Apollo access: Secure NFS export configuration