dotfiles_arch/ansible/BACKUP_RESTORE.md

# Backup and Restore Guide

Complete guide for backing up and restoring your Nextcloud Stack.

## Automated Backups

### What Gets Backed Up

- **Daily (3:00 AM)**:
  - PostgreSQL database dump
  
- **Weekly (Sundays, 3:00 AM)**:
  - Nextcloud data volume
  - Configuration files

- **Retention**: 30 days

### Backup Locations

```
/opt/nextcloud-stack/backups/
├── database/
│   └── nextcloud_db_YYYYMMDD_HHMMSS.sql.gz
├── volumes/
│   ├── nextcloud_data_YYYYMMDD_HHMMSS.tar.gz
│   └── configs_YYYYMMDD_HHMMSS.tar.gz
└── backup.log
```

### Manual Backup

```bash
ssh user@server
/opt/nextcloud-stack/backup.sh
```

### Check Backup Status

```bash
# View backup log
tail -f /opt/nextcloud-stack/backups/backup.log

# List backups
ls -lh /opt/nextcloud-stack/backups/database/
ls -lh /opt/nextcloud-stack/backups/volumes/

# Check disk usage
du -sh /opt/nextcloud-stack/backups/
```

---

## Restore Procedures

### 1. Restore Database Only

**When to use:**
- Database corruption
- Accidental data deletion
- Need to revert to earlier state

**Steps:**

```bash
# 1. Stop Nextcloud (keep database running)
cd /opt/nextcloud-stack
docker compose stop next

# 2. List available backups
ls -lh backups/database/

# 3. Choose backup to restore
BACKUP_FILE="backups/database/nextcloud_db_20260216_030000.sql.gz"

# 4. Drop existing database (CAUTION!)
docker exec next-db psql -U nextcloud -c "DROP DATABASE nextcloud;"
docker exec next-db psql -U nextcloud -c "CREATE DATABASE nextcloud;"

# 5. Restore from backup
gunzip < $BACKUP_FILE | docker exec -i next-db psql -U nextcloud -d nextcloud

# 6. Restart Nextcloud
docker compose start next

# 7. Run Nextcloud upgrade (if needed)
docker exec -u www-data next php occ upgrade

# 8. Disable maintenance mode
docker exec -u www-data next php occ maintenance:mode --off
```

### 2. Restore Data Volumes

**When to use:**
- Lost files
- Corrupted data directory
- Complete system failure

**Steps:**

```bash
# 1. Stop all services
cd /opt/nextcloud-stack
docker compose down

# 2. List available backups
ls -lh backups/volumes/

# 3. Choose backup
BACKUP_FILE="backups/volumes/nextcloud_data_20260216_030000.tar.gz"

# 4. Restore volume
# Note: This requires accessing the Docker volume directory
sudo tar -xzf $BACKUP_FILE -C /var/lib/docker/volumes/nextcloud_nextcloud_data/_data/

# 5. Restore configs (optional)
CONFIG_BACKUP="backups/volumes/configs_20260216_030000.tar.gz"
sudo tar -xzf $CONFIG_BACKUP -C /opt/nextcloud-stack/configs/

# 6. Fix permissions
sudo chown -R www-data:www-data /var/lib/docker/volumes/nextcloud_nextcloud_data/_data/

# 7. Start services
docker compose up -d

# 8. Run Nextcloud file scan
docker exec -u www-data next php occ files:scan --all
```

### 3. Complete System Restore

**When to use:**
- Disaster recovery
- Migration to new server
- Complete system failure

**Steps:**

```bash
# 1. On new server, install base system
# Run Ansible playbooks 01-04 (up to Tailscale)
ansible-playbook playbooks/01-preflight-checks.yml --ask-vault-pass
ansible-playbook playbooks/02-system-setup.yml --ask-vault-pass
ansible-playbook playbooks/03-docker-setup.yml --ask-vault-pass
ansible-playbook playbooks/04-tailscale-setup.yml --ask-vault-pass

# 2. Create deployment directory structure
ssh user@new-server
sudo mkdir -p /opt/nextcloud-stack/backups/{database,volumes}

# 3. Copy backups from old server to new server
# On your local machine:
scp -r old-server:/opt/nextcloud-stack/backups/* user@new-server:/opt/nextcloud-stack/backups/

# 4. Deploy stack (creates volumes and containers)
ansible-playbook playbooks/05-deploy-stack.yml --ask-vault-pass

# 5. Stop services on new server
ssh user@new-server
cd /opt/nextcloud-stack
docker compose down

# 6. Restore database
BACKUP_FILE="backups/database/nextcloud_db_20260216_030000.sql.gz"
docker compose up -d next-db next-redis
sleep 10
gunzip < $BACKUP_FILE | docker exec -i next-db psql -U nextcloud -d nextcloud

# 7. Restore data volumes
VOLUME_BACKUP="backups/volumes/nextcloud_data_20260216_030000.tar.gz"
sudo tar -xzf $VOLUME_BACKUP -C /var/lib/docker/volumes/nextcloud_nextcloud_data/_data/

# 8. Restore configs
CONFIG_BACKUP="backups/volumes/configs_20260216_030000.tar.gz"
sudo tar -xzf $CONFIG_BACKUP -C /opt/nextcloud-stack/configs/

# 9. Fix permissions
sudo chown -R www-data:www-data /var/lib/docker/volumes/nextcloud_nextcloud_data/_data/

# 10. Start all services
docker compose up -d

# 11. Run Nextcloud maintenance
docker exec -u www-data next php occ maintenance:mode --off
docker exec -u www-data next php occ files:scan --all
docker exec -u www-data next php occ db:add-missing-indices

# 12. Continue with remaining playbooks
exit
ansible-playbook playbooks/06-configure-caddy.yml --ask-vault-pass
ansible-playbook playbooks/07-setup-backups.yml --ask-vault-pass
ansible-playbook playbooks/08-post-deployment.yml --ask-vault-pass
```

---

## Off-Site Backups with rclone

### Initial Setup

```bash
ssh user@server

# Configure rclone
rclone config

# Example: Setup Backblaze B2
# Follow prompts:
# - Choose: New remote
# - Name: b2backup
# - Storage: Backblaze B2
# - Enter account ID and application key
```

### Sync Backups to Remote

```bash
# Test sync (dry run)
rclone sync /opt/nextcloud-stack/backups/ b2backup:nextcloud-backups --dry-run

# Actual sync
rclone sync /opt/nextcloud-stack/backups/ b2backup:nextcloud-backups
```

### Automated Off-Site Backups

Add to `/opt/nextcloud-stack/backup.sh` (after the cleanup section):

```bash
# Off-site backup (if rclone configured)
if command -v rclone &> /dev/null && rclone listremotes | grep -q "b2backup"; then
    log "Syncing backups to off-site storage..."
    rclone sync $BACKUP_DIR b2backup:nextcloud-backups --log-file=$LOG_FILE
    log "Off-site sync completed"
fi
```

### Restore from Off-Site Backup

```bash
# List remote backups
rclone ls b2backup:nextcloud-backups/database/

# Download specific backup
rclone copy b2backup:nextcloud-backups/database/nextcloud_db_20260216_030000.sql.gz /opt/nextcloud-stack/backups/database/

# Download all backups
rclone sync b2backup:nextcloud-backups /opt/nextcloud-stack/backups/
```

---

## Backup Verification

### Test Database Backup Integrity

```bash
# 1. Extract backup
gunzip -c backups/database/nextcloud_db_20260216_030000.sql.gz > /tmp/test_restore.sql

# 2. Check for errors
grep -i "error" /tmp/test_restore.sql

# 3. Verify size (should be reasonable)
ls -lh /tmp/test_restore.sql

# 4. Cleanup
rm /tmp/test_restore.sql
```

### Test Volume Backup Integrity

```bash
# Test tar archive
tar -tzf backups/volumes/nextcloud_data_20260216_030000.tar.gz | head -20

# Check for errors
tar -tzf backups/volumes/nextcloud_data_20260216_030000.tar.gz > /dev/null && echo "Archive OK"
```

---

## Migration to New Server

**Complete migration guide:**

1. **Prepare new server:**
   - Same or newer Ubuntu version
   - Run Ansible setup playbooks (01-04)

2. **Backup old server:**
   ```bash
   /opt/nextcloud-stack/backup.sh
   ```

3. **Copy backups to new server:**
   ```bash
   rsync -avz old-server:/opt/nextcloud-stack/backups/ new-server:/opt/nextcloud-stack/backups/
   ```

4. **Update DNS:**
   - Point all domains to new server IP
   - Wait for propagation

5. **Restore on new server:**
   - Follow "Complete System Restore" steps above

6. **Verify:**
   - Test all services
   - Check Nextcloud functionality
   - Verify SSL certificates

7. **Decommission old server:**
   - Only after confirming new server works
   - Keep final backup from old server

---

## Backup Best Practices

### DO:
- ✅ Test restores regularly (monthly)
- ✅ Keep backups off-site (rclone to cloud storage)
- ✅ Verify backup integrity
- ✅ Monitor backup log for errors
- ✅ Document restore procedures
- ✅ Keep multiple backup generations

### DON'T:
- ❌ Store backups only on same server
- ❌ Assume backups work without testing
- ❌ Delete old backups without verification
- ❌ Skip database backups
- ❌ Ignore backup failure notifications

---

## Backup Schedule Customization

Edit `/opt/nextcloud-stack/backup.sh` to change:

- Backup frequency
- Retention period
- What gets backed up
- Compression settings

Edit cron job:
```bash
sudo crontab -e

# Change from 3:00 AM to 2:00 AM
0 2 * * * /opt/nextcloud-stack/backup.sh >> /opt/nextcloud-stack/backups/backup.log 2>&1
```

---

## Troubleshooting Backups

### Backup script fails

**Check logs:**
```bash
tail -100 /opt/nextcloud-stack/backups/backup.log
```

**Common issues:**
- Disk space full
- Docker container not running
- Permission denied

**Solutions:**
```bash
# Check disk space
df -h

# Check containers
docker ps

# Fix permissions
sudo chown -R root:root /opt/nextcloud-stack/backup.sh
sudo chmod +x /opt/nextcloud-stack/backup.sh
```

### Database backup empty

**Verify database is running:**
```bash
docker exec next-db pg_isready -U nextcloud
```

**Test manual backup:**
```bash
docker exec next-db pg_dump -U nextcloud nextcloud > /tmp/test.sql
ls -lh /tmp/test.sql
```

---

## Emergency Recovery

If all backups are lost and you need to start fresh:

```bash
# 1. Remove everything
ansible-playbook playbooks/99-rollback.yml --ask-vault-pass
cd /opt/nextcloud-stack
docker compose down -v

# 2. Redeploy from scratch
ansible-playbook playbooks/site.yml --ask-vault-pass

# 3. Reconfigure Nextcloud manually
# Login and set up users, apps, etc.
```

---

**Remember:** Backups are worthless unless you've tested restoring from them!

**Last Updated:** 2026-02-16