Self-hosting gives you complete control over your data and services. But with great power comes great responsibility — if your Nextcloud, Jellyfin, or Vaultwarden instance is misconfigured or exposed, you become an easy target. Security breaches in home servers are real: exposed ports get hit by scanners within minutes of going online.

This guide covers the practical security layers every self-hoster should have in place in 2026 — from firewalls and HTTPS to authentication layers and intrusion prevention. You don’t need to implement everything at once, but each layer you add makes your setup significantly harder to compromise.

💡 This article contains affiliate links. If you buy through them, we earn a small commission at no extra cost to you. Learn more.

Layer 1: Start With a Firewall

The first rule of self-hosting security: only expose what you absolutely need to expose. A firewall is your first line of defense.

UFW (Uncomplicated Firewall) on Ubuntu/Debian

UFW is the easiest way to manage iptables rules on Debian-based systems:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

# Install UFW
sudo apt install ufw

# Default: deny all incoming, allow all outgoing
sudo ufw default deny incoming
sudo ufw default allow outgoing

# Allow SSH (do this BEFORE enabling UFW or you'll lock yourself out)
sudo ufw allow ssh

# Allow HTTP and HTTPS for your reverse proxy
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

# Enable UFW
sudo ufw enable

# Check status
sudo ufw status verbose

Never open ports like 8080, 8443, or custom ports directly to the internet. Route everything through a reverse proxy on ports 80/443 instead.

Docker and UFW — A Hidden Gotcha

Docker bypasses UFW by default by modifying iptables directly. A container with a published port (-p 8080:8080) will be accessible from the internet even if UFW blocks it.

Fix this by editing /etc/docker/daemon.json:

1
2
3

{
  "iptables": false
}

Then use your reverse proxy to route traffic instead of publishing ports directly. Or use a dedicated Docker firewall tool like docker-ufw-proxy.

Alternatively, bind container ports to localhost only:

1
2

ports:
  - "127.0.0.1:8080:8080"  # Only accessible on localhost

Layer 2: Reverse Proxy + Automatic HTTPS

A reverse proxy sits in front of all your services and handles:

  • TLS termination (HTTPS)
  • Routing by domain name
  • Authentication headers
  • Rate limiting

The most popular choices are Traefik, Nginx Proxy Manager, and Caddy. For a full breakdown, see our Traefik vs Caddy vs Nginx Proxy Manager comparison.

Caddy — Simplest Way to Get HTTPS

Caddy automatically handles Let’s Encrypt certificates with zero configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# docker-compose.yml
services:
  caddy:
    image: caddy:2-alpine
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - proxy

networks:
  proxy:
    external: true

volumes:
  caddy_data:
  caddy_config:

1
2
3
4
5
6
7
8

# Caddyfile
nextcloud.yourdomain.com {
    reverse_proxy nextcloud:80
}

jellyfin.yourdomain.com {
    reverse_proxy jellyfin:8096
}

Caddy will automatically request and renew Let’s Encrypt certificates. That’s it.

Never skip HTTPS — even on your home network. Credentials sent over HTTP can be intercepted, and modern browsers increasingly distrust HTTP sites.

Layer 3: Authentication — Don’t Expose Services Bare

Some apps (like Nextcloud and Vaultwarden) have built-in login screens. Others (like Prometheus or Grafana without proper setup) don’t. And even for apps with logins, you might want an additional authentication layer.

Basic Auth for Quick Protection

For services that need simple protection, add HTTP Basic Auth at the reverse proxy level.

Caddy:

1
2
3
4
5
6

prometheus.yourdomain.com {
    basicauth {
        admin $2a$14$Zkx19XLiW6VYouLHR5NmfOFU0z2GTNmpkT/5qqR7hx4IjWJPDhjvG
    }
    reverse_proxy prometheus:9090
}

Generate the hash with: caddy hash-password --plaintext 'yourpassword'

Nginx:

1
2
3
4
5

location / {
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/.htpasswd;
    proxy_pass http://localhost:9090;
}

Authelia — Single Sign-On for Your Homelab

For a more complete solution, Authelia provides SSO, two-factor authentication, and access control policies for all your services from one place.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

# docker-compose.yml
services:
  authelia:
    image: authelia/authelia:latest
    container_name: authelia
    restart: unless-stopped
    volumes:
      - ./authelia:/config
    environment:
      - TZ=Europe/Berlin
    networks:
      - proxy

networks:
  proxy:
    external: true

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# authelia/configuration.yml
server:
  host: 0.0.0.0
  port: 9091

log:
  level: info

jwt_secret: a_very_long_random_string_here
default_redirection_url: https://auth.yourdomain.com

authentication_backend:
  file:
    path: /config/users_database.yml
    password:
      algorithm: argon2id

access_control:
  default_policy: deny
  rules:
    - domain: "jellyfin.yourdomain.com"
      policy: bypass  # Has its own auth
    - domain: "prometheus.yourdomain.com"
      policy: two_factor
    - domain: "*.yourdomain.com"
      policy: one_factor

session:
  name: authelia_session
  secret: another_long_random_string
  expiration: 3600
  inactivity: 300
  domain: yourdomain.com

storage:
  local:
    path: /config/db.sqlite3

notifier:
  filesystem:
    filename: /config/notification.txt

Then configure your reverse proxy to forward requests through Authelia for authentication. Authelia supports TOTP (Google Authenticator), WebAuthn, and push notifications.

Layer 4: Keep Everything Updated

Outdated software is the most common attack vector. A Docker container running a year-old image may have dozens of known CVEs.

Watchtower — Automated Container Updates

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# docker-compose.yml
services:
  watchtower:
    image: containrrr/watchtower:latest
    container_name: watchtower
    restart: unless-stopped
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_SCHEDULE=0 0 4 * * *  # Run at 4 AM daily
      - WATCHTOWER_NOTIFICATIONS=email
      - WATCHTOWER_NOTIFICATION_EMAIL_FROM=alerts@yourdomain.com
      - WATCHTOWER_NOTIFICATION_EMAIL_TO=you@example.com
      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER=smtp.example.com
      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PORT=587
      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_USER=you@example.com
      - WATCHTOWER_NOTIFICATION_EMAIL_SERVER_PASSWORD=yourpassword

For production setups, use --monitor-only mode and update manually after review:

1
2

environment:
  - WATCHTOWER_MONITOR_ONLY=true  # Only notify, don't auto-update

Also keep your host OS updated:

1
2
3

# Set up unattended upgrades for security patches
sudo apt install unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades

Layer 5: Fail2Ban — Block Brute Force Attacks

Fail2Ban monitors log files and automatically bans IPs that show malicious signs (too many failed logins, scanning activity, etc.).

1

sudo apt install fail2ban

SSH Protection (Built-In)

Fail2Ban includes SSH protection out of the box. Enable it:

1
2
3
4
5
6
7
8
9

# /etc/fail2ban/jail.local
[sshd]
enabled = true
port = ssh
filter = sshd
logpath = /var/log/auth.log
maxretry = 5
bantime = 3600
findtime = 600

Nginx/Caddy Log Protection

Create a custom filter for your reverse proxy:

1
2
3
4

# /etc/fail2ban/filter.d/nginx-401.conf
[Definition]
failregex = ^<HOST> .* "(GET|POST|HEAD).*" 401
ignoreregex =

1
2
3
4
5
6
7
8

# /etc/fail2ban/jail.local
[nginx-401]
enabled = true
filter = nginx-401
logpath = /var/log/nginx/access.log
maxretry = 10
bantime = 86400  # 24 hours
findtime = 3600

Authelia Fail2Ban Filter

1
2
3
4

# /etc/fail2ban/filter.d/authelia.conf
[Definition]
failregex = ^.*Unsuccessful (1FA|TOTP|Duo|U2F) authentication attempt by user .*remote_ip="?<HOST>"?.*$
ignoreregex =

1
2
3
4
5
6
7
8

# /etc/fail2ban/jail.local
[authelia]
enabled = true
filter = authelia
logpath = /path/to/authelia/notification.txt
maxretry = 3
bantime = 86400
findtime = 600

Restart Fail2Ban after changes:

1
2

sudo systemctl restart fail2ban
sudo fail2ban-client status  # Check active jails

Layer 6: Network Segmentation with Docker Networks

Don’t let all your containers talk to each other freely. Use Docker networks to isolate services.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

# docker-compose.yml
services:
  nextcloud:
    image: nextcloud:latest
    networks:
      - proxy      # Can receive traffic from reverse proxy
      - nextcloud  # Internal network for DB communication

  nextcloud-db:
    image: mariadb:latest
    networks:
      - nextcloud  # Only accessible from nextcloud container

  caddy:
    image: caddy:latest
    networks:
      - proxy      # Frontend network

networks:
  proxy:
    external: true   # Shared across stacks for reverse proxy access
  nextcloud:
    internal: true   # No external access, isolated backend

By using internal: true on the database network, your MariaDB or PostgreSQL instances have no internet access whatsoever — they can only communicate with the containers in the same network.

Layer 7: VPN Access for Internal Services

For services you don’t want exposed to the internet at all (Home Assistant, internal dashboards, NAS management), use a VPN. This is the most secure option — nothing is exposed publicly.

Tailscale — Easiest Zero-Config VPN

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

services:
  tailscale:
    image: tailscale/tailscale:latest
    container_name: tailscale
    hostname: homelab
    environment:
      - TS_AUTHKEY=tskey-auth-xxxxxxxxxxxxxxxx
      - TS_EXTRA_ARGS=--advertise-exit-node
      - TS_STATE_DIR=/var/lib/tailscale
    volumes:
      - tailscale_state:/var/lib/tailscale
      - /dev/net/tun:/dev/net/tun
    cap_add:
      - NET_ADMIN
      - SYS_MODULE
    restart: unless-stopped

volumes:
  tailscale_state:

Once connected, you access services via their Tailscale IP (e.g., http://100.64.x.x:8096) — no port forwarding, no exposure to the internet. Tailscale also supports MagicDNS so you can use hostnames like homelab.tailnet-name.ts.net.

For a full comparison, see our Tailscale vs WireGuard guide.

Layer 8: Secrets Management — No Plaintext Passwords

Never store passwords directly in docker-compose.yml files committed to Git. Use Docker secrets or environment files.

.env Files

1
2
3

# .env (gitignored)
DB_PASSWORD=super_secret_password_here
REDIS_PASSWORD=another_secret

1
2
3
4
5

# docker-compose.yml
services:
  nextcloud:
    environment:
      - MYSQL_PASSWORD=${DB_PASSWORD}

Add .env to .gitignore:

1

echo ".env" >> .gitignore

Docker Secrets (Swarm Mode)

For production deployments using Docker Swarm:

1

echo "super_secret_password" | docker secret create db_password -

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

services:
  db:
    image: mariadb
    secrets:
      - db_password
    environment:
      - MYSQL_PASSWORD_FILE=/run/secrets/db_password

secrets:
  db_password:
    external: true

Layer 9: Security Headers

Add security headers at your reverse proxy level to protect against XSS, clickjacking, and other browser-based attacks.

Caddy Security Headers

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

(security_headers) {
    header {
        Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
        X-Content-Type-Options "nosniff"
        X-Frame-Options "SAMEORIGIN"
        X-XSS-Protection "1; mode=block"
        Referrer-Policy "strict-origin-when-cross-origin"
        Permissions-Policy "geolocation=(), microphone=(), camera=()"
        -Server
    }
}

nextcloud.yourdomain.com {
    import security_headers
    reverse_proxy nextcloud:80
}

Test your headers at securityheaders.com.

Layer 10: Monitoring and Alerting

You need to know when something goes wrong. Set up monitoring to detect:

  • Service downtime
  • Failed login attempts
  • High resource usage
  • Unexpected network traffic

Uptime Kuma is perfect for service availability monitoring. See our Uptime Kuma setup guide for a complete walkthrough.

For log aggregation and anomaly detection, consider:

  • Grafana + Loki — full log aggregation stack
  • Graylog — centralized log management
  • CrowdSec — community-powered intrusion prevention

Basic Log Review

At minimum, check these logs regularly:

1
2
3
4
5
6
7
8
9

# Failed SSH logins
grep "Failed password" /var/log/auth.log | tail -20

# Fail2Ban bans
sudo fail2ban-client status sshd

# Docker container logs
docker logs --tail=100 caddy
docker logs --tail=100 authelia

Security Checklist

Here’s your quick reference for securing a new self-hosted service:

  • Firewall configured (UFW) — deny all incoming by default
  • Docker ports bound to 127.0.0.1 or internal networks only
  • Reverse proxy in place (Caddy/Traefik/Nginx)
  • HTTPS enabled with valid certificate (Let’s Encrypt)
  • Authentication layer added (service login + optional Authelia/basic auth)
  • Strong, unique passwords (use Vaultwarden)
  • No secrets in docker-compose.yml files (use .env files, gitignored)
  • Watchtower set up for update notifications
  • Fail2Ban enabled for SSH and web services
  • Internal services accessible via VPN only (Tailscale/WireGuard)
  • Security headers configured
  • Uptime monitoring in place
  • Regular backups verified (see our Homelab Backup Guide)

Conclusion

Self-hosting security doesn’t have to be complicated, but it does require intentional layering. Start with the basics: firewall, HTTPS, and strong passwords. Then add Fail2Ban and keep software updated. When you’re comfortable, layer in Authelia SSO or VPN-only access for sensitive services.

The goal isn’t perfect security — it’s raising the cost of attack high enough that attackers move on to easier targets. With these layers in place, your homelab will be more secure than most services people trust with their data every day.

Which security layer do you struggle with most? Let us know in the comments — or share your setup with the community.