Network

Web application firewall and honeypot

Every server's proxy runs a WAF in front of the apps. It blocks attacks, banned addresses and those threat feeds list. It applies each app's rate limits, country rules and restricted paths, and challenges suspicious visitors to prove they run a browser. It answers people probing for secrets with decoys, and bans them on every server. It also acts on what the traffic intel finds (Traffic intelligence): floods, brute-forced logins, path enumeration and botnet campaigns. The panel's Firewall page shows what it did and manages the bans, each app's mode and policy, and the settings.

How it runs

  • Changes to modes, policies and bans apply on every server within a few seconds, without restarting the proxy.
  • Fails open. If the WAF is disabled or fails, requests go through untouched. ACME challenges are never touched.
  • Upgrades. A stackd release that changes the WAF restarts the proxy once on each server, with a few seconds offline.

The panel

The panel's own host (from STACKD_PUBLIC_URL) is protected too, unless Protect the panel is off (Settings):

  • Bans apply, but not those for floods, brute-forced logins, path enumeration and swarms: those can catch an office or a shared address, and locking admins out is worse than letting a scanner see a login page. Bans by hand, for critical attacks, decoy logins and canaries do apply.
  • Blocking feeds apply.
  • Logins are limited to 10 POSTs a minute per address.
  • Critical rules run on the path and headers only. Admins save scripts, configs and queries that look like attacks, so bodies and queries are never read, high-severity rules don't apply, and nobody is challenged. Probes for secrets get decoys and bans as on an app.

The panel also limits failed logins itself, whether or not they came through a proxy: after 5 wrong passwords from one address for one account, or 20 from one address for any, its next attempts wait 30 seconds, doubling with each failure up to 15 minutes (429 with Retry-After). Counting per address and account means nobody can lock someone else out by failing their password.

The allowlist wins here too: keep your office or VPN on it. Locked out anyway, an admin can lift the ban from a server's console with DELETE /api/waf/bans?ip=…, or turn the panel's protection off.

Modes

Each app has a mode, protect by default (PUT /api/apps/{id}/waf):

Mode Critical attacks High-severity attacks Banned addresses Decoys
off — — — —
monitor logged logged logged —
protect blocked (403) logged refused (403) yes
strict blocked blocked refused yes

Requests for a host no app has are let through untouched, but for the panel's own (above). The exception is a request for a bare address, or with no host at all. Only scanners send those, so they all get decoys (the trap).

What a request goes through

In this order; the first that refuses it answers.

  1. The challenge's answer (/.well-known/stackd-challenge, below).
  2. Bans and blocking feeds: refused (denied).
  3. Countries and networks the app's policy blocks (geo), and restricted paths from anyone not allowed (restricted): 403.
  4. Rate limits (login_rate, rate_limit): 429.
  5. The rules, below.
  6. The challenge, for the visitors it applies to.
  7. The conditional decoys (/wp-login.php and the like, on a 404).

Rules

The rules read the path, the query, the headers, and the body of forms, JSON, plain text and XML (its first 32 KiB, put back for the app), and of multipart forms the fields and file names, not the files' contents (in the first 64 KiB). Everything is decoded first (double encoding, %uXXXX, HTML entities, JSON escapes, SQL comments, odd whitespace), so encoding doesn't hide an attack.

Words that real text sometimes has (information_schema, powershell, onload=) only count two at a time, so a search for "powershell tips" isn't blocked in strict mode.

Rule Severity What
canary critical The request carries the canary from a decoy's fake secrets, anywhere: query, headers (Authorization…), body.
log4shell critical ${jndi: and its obfuscations, anywhere in the request, every header included.
traversal critical ../ in a path, ../../ and the encodings anywhere, /etc/passwd, win.ini, null bytes, …
php_wrapper critical php://, phar://, expect://, file:///, …
scanner_ua critical sqlmap, nikto, nuclei, zgrab, masscan, wpscan, …
probe critical Paths only ever asked for to steal: .env, .git, .aws, .ssh, key files, config backups, phpunit's eval-stdin.php, cgi-bin, …
ssrf high Cloud metadata addresses in the query or body, plain or encoded.
sqli high union select, tautologies (' or 1=1, ") or ("a"="a), sleep(5), benchmark(, …
xss high <script, a tag with an event handler (<x onfoo=), javascript:alert, …
cmdi high ;wget , |sh, $(curl, /dev/tcp/, …
bad_method high TRACE, TRACK, DEBUG, CONNECT.
oversized high A form, JSON, text or XML body over 32 KiB: what's past that isn't read, and padding is how an attack hides there. Strict mode refuses it; an exception lets an app that takes big bodies have them (/api/import → oversized). Multipart forms are exempt: files make them long.

App policies

Besides its mode, each app has a policy (PUT /api/apps/{id}/waf/policy, the Policy button on the Apps tab). In protect and strict it's applied; in monitor what it would refuse is only logged.

Setting Default What
rate_limit 0 (none) Requests a minute per address. Each server counts its own, so an app on 3 servers lets an address through up to 3 times this. Over it: 429 (throttled).
path_limits — Tighter limits under path prefixes (/api/search at 20 a minute), counted apart from rate_limit; the first prefix that matches applies. Over it: 429 (rule rate_limit, the path in the detail).
login_rate_limit 30 POSTs a minute per address to paths with login, signin, auth, session, token or password in them.
challenge auto Who's challenged: off, auto or always (below).
security_headers off Add X-Content-Type-Options: nosniff, X-Frame-Options: SAMEORIGIN, Referrer-Policy and HSTS (over HTTPS) to responses lacking them. Websockets and event streams are left alone.
block_countries, block_asns — Refused with a 403 (geo).
challenge_countries — Challenged, unless challenge is off.
restricted — Path prefixes only some addresses or ranges may reach (an admin area): 403 for anyone else (restricted).
exceptions — Path prefixes where some rules (or all) don't apply, for false positives such as an editor saving HTML (/admin/editor → xss). Exceptions can name the attack rules and oversized, rate_limit (path limits too), login_rate, challenge and restricted.

Rate limits count an address by what a ban would hold: an IPv4 address, or an IPv6 address's /64, so a visitor can't get a fresh allowance from each address of its network.

Country and network rules need the GeoIP databases (Traffic intelligence). A server that hasn't downloaded them yet lets those visitors through.

The challenge

A page that makes the browser find a proof of work (a number whose SHA-256, with the puzzle, starts with four zero hex digits: a fraction of a second) and trade it at /.well-known/stackd-challenge for a stackd_clearance cookie, signed and bound to the visitor's address, valid 12 hours. Bots that don't run scripts never get one; those that do pay for every address they use.

Who's challenged, unless the app's challenge is off:

  • visitors the traffic intel classed suspicious or botnet in the last hour (only on page loads, so a page already open keeps working);
  • requests matching an active botnet campaign (same app, method, normalized path and user agent as one the intel saw in the last 10 minutes);
  • addresses in a feed set to challenge (Tor, say), and countries the policy challenges;
  • everyone, when the challenge is always: for an app under attack. Search engines and API clients can't pass it.

Requests that aren't page loads (API calls, form posts) and lack a clearance are refused with a 403, since there's nowhere to show the challenge. Plain HTTP is never challenged: it's redirected to HTTPS first. Event: challenged.

Honeypot

  • Decoys for probes. A probe gets a convincing fake instead of a 403: a Laravel .env, a .git/config, AWS credentials, a key file. All their "secrets" carry the fleet's canary, so a later request using one is caught (rule canary), followed by a token of that decoy's own (token=… in the event's detail). When a secret is used, the control plane finds the decoy with that token and says whom it was served to and when.
  • Paths a real app may serve (/wp-login.php, /wp-admin, /xmlrpc.php, /phpmyadmin, /administrator, /actuator, …). These go to the app first. Only when the app answers 404 is its response replaced by a decoy login page, so a real WordPress keeps working.
  • Captured logins. A login submitted to a decoy is recorded as captured. Only the user name and the password's length are kept, never the password.
  • No caching. Decoys are sent with Cache-Control: no-store.

Bans

With automatic bans on, the control plane bans an address on every server for:

  • Critical attacks: 24 hours, then 3 days, 7 days and 30 days for repeat offenders.
  • Captured logins and canary use: 7 days, then 30.
  • Five high-severity attacks within 10 minutes: 1 hour, then 24 hours, 7 days and 30 days.

With act on the traffic intel on too (the default), it bans for 1 hour, then 24 hours, 7 days and 30 days, what the intel found in a minute:

  • floods (over 300 requests a minute on a browser user agent, 600 otherwise),
  • brute-forced logins (10 or more failed logins),
  • path enumeration (60 errors on 40 or more paths).

Search and AI crawlers and monitors are never banned for these. Only attacks logged or blocked count toward the five high-severity ones; being throttled or challenged doesn't.

With automatic bans on, what a mail server bans (see Mail) is banned on every server too, for 1 hour, then 24 hours, 7 days and 30 days.

Swarms. Scanners spread over a hosting provider's block send a request or two from each address, too few for their own bans to stop anything. When three or more addresses of one network (a /24, or a /48 for IPv6) make critical attacks within 24 hours, the whole network is banned (rule swarm), for 24 hours, then 3 days, 7 days and 30 days. A swarm spread over a cloud's many networks (one address per /24) isn't caught this way: banning a cloud's ranges would refuse everyone else on them too. Its probes still get decoys.

An IPv6 address is banned by its /64, which whoever has one address usually has all of. Strikes are remembered for 30 days after a ban runs out. The proxy also bans a critical attacker (its /64 for IPv6) on its own server for an hour right away, before the control plane's list arrives.

Some addresses are never banned, by hand or automatically:

  • the allowlist (Settings), which is never blocked at all;
  • the servers' own addresses;
  • private, loopback, CGNAT and mesh ranges, and Cloudflare's ranges (banning one of those would ban everyone behind it).

Admins can also ban an address or a CIDR by hand (POST /api/waf/bans), for a while or until removed, and lift any ban (DELETE /api/waf/bans?ip=).

GET /api/waf counts the active bans (totals) and lists the newest 100. The rest page newest first at GET /api/waf/bans?limit=&before= (next is the cursor of the following page; since= refreshes the top of a list already paged through). POST /api/waf/bans/lookup with {"ips": [...]} (at most 500) says which of those addresses a ban covers, and which ban.

Network bans

With Drop bans at the network on (Settings, the default), each server also drops new connections from banned addresses and blocking feeds in its kernel: before TCP and TLS are done, so a flood of them costs the proxy nothing, and on every port Docker publishes (databases' too), not only the proxy's.

  • Only bans that apply to the panel go there (critical attacks, decoys, canaries, by hand). Floods, brute force, enumeration and swarms stay in the proxy, which can tell the panel's host from an app's. The mail servers' bans (see Mail) are dropped on the mail ports only.
  • On the host itself, it covers the proxy's ports and the mail ports (25, 465, 587, 143, 993, 995 and 4190), on every server whether a mail server listens there or not.
  • It applies to every app, in monitor and off modes too: the kernel doesn't know which app a connection is for.
  • The allowlist, the servers' own addresses, private, loopback, CGNAT, mesh and Cloudflare's ranges are cut out of what's dropped.
  • SSH and the host's other services are left alone: a ban that caught an admin doesn't lock them out of the server.
  • Changes are applied at most every 5 seconds, and expired bans are removed within a minute. On a server without nft, the proxy still refuses them all; the Exposure tab shows the error.

Host firewall

Off by default (Settings, Host firewall). Docker's published ports go around ufw, and the host's own services (SSH, anything installed by hand) are as the server had them. On, each server drops what comes in unless it is:

  • a reply, from loopback, the mesh (stackd0) or a Docker bridge,
  • ICMP and ICMPv6, DHCP replies,
  • for the proxy's ports (80 and 443) or the mesh's UDP port,
  • SSH (ssh_ports, 22 by default): from anyone, or with Only allowed addresses (ssh_restricted) from ssh_allow and the grants in force only, nobody else even when both are empty,
  • for open_tcp and open_udp,
  • for a published port from its allowlist (Docker's userland proxy, IPv6 without ip6tables),
  • from the allowlist or the servers' own addresses.

SSH for a while. POST /api/waf/ssh-grants with {"ip": …, "hours": …} (an address or CIDR, the caller's own when left out; 1 to 168 hours, 24 by default) lets SSH in from it on every server until then, and turns Only allowed addresses on: everyone else is denied SSH, also once the grant runs out, until the next one. The Settings tab offers it for your own address. A grant for an address already granted replaces it. DELETE /api/waf/ssh-grants?ip= ends one now; GET lists those in force with your address as the panel sees it. A grant ends on time on every server (within a minute), even when the panel is unreachable. Grants need the host firewall on (409 otherwise), and saving the settings keeps them as they are.

What Docker publishes is forwarded, not input, and keeps its own rules (external database access, Databases). A rule that shuts too much out never cuts a server off from the panel, where it can be put right. An SSH allowlist without your address locks you out of SSH: the allowlist always gets in.

Mail

A mail service's servers (Mail services) take SMTP, submission, IMAP, POP3 and ManageSieve straight from the internet, on the host's network: the proxy and its WAF never see those connections. The firewall protects them this way:

  • In the kernel. With Drop bans at the network on, every ban that applies to the panel and every blocking feed (Spamhaus DROP, by default) is dropped on the mail ports before TCP is done, as it is on 80 and 443.
  • The mail servers' own bans. Stalwart counts, per address and across its cluster, failed logins (over IMAP, POP3, submission, ManageSieve and JMAP), abuse (relaying, unknown recipients, errors), connections held open doing nothing, and its web paths scanned for exploits. Over a rate, it blocks the address on the service for a day. Every such ban shows as an event (rules mail_auth, mail_abuse, mail_loiter, mail_scan; action blocked; the account tried, if any, in detail). With automatic bans on, the control plane bans the address on every server: 1 hour, then 24 hours, 7 days and 30 days. Those bans are dropped in the kernel on the mail ports (a mail service's other nodes and every other service included) and refused by the apps' WAF, but not on the panel: a phone with an old password at the office would otherwise lock its admins out.
  • Stalwart's settings. stackd sets Stalwart's ban periods to a day (Stalwart's default is forever: stackd's own bans take the repeat offenders), and keeps the allowlist and the servers' own addresses from being banned, within a minute of a change. The rates are Stalwart's: 100 failed logins, 35 abuses, 150 idle connections or 30 scans a day.
  • Lifting a ban (DELETE /api/waf/bans?ip=) lifts the mail services' own blocks inside it too, on the services with a node up.

Exposure

The Exposure tab (GET /api/waf/exposure) lists, for each server, every port it listens on or publishes, checked every minute:

  • how far each reaches by the address it's bound to: the internet (every address, or a public one), the mesh, Docker's containers, or the server itself (loopback);
  • for those reaching the internet, what stands in front: the proxy and its WAF, an allowlist, the mesh only, the host firewall (blocked, or SSH's allowlist), a mail server's own bans and the firewall's (Mail + bans: Stalwart on a mail port, open to anyone by design), or nothing (open to anyone, listed first). A DHCP client's port and the mesh's WireGuard port aren't counted as open: the one takes only replies to the server's own requests, the other answers only peers with the mesh's keys;
  • whose it is: the app, database or panel replica whose container publishes it, or the process listening (SSH held by systemd, as recent Ubuntu has it, shows as such);
  • a service on every IPv4 and every IPv6 address alike is one row (0.0.0.0, ::);
  • whether the server drops bans in its kernel and runs the host firewall, and why not when either fails.

An app without a domain is published on a loopback port only, so it isn't reachable from outside.

Threat feeds

Lists of addresses known for abuse, refreshed every 6 hours. Each is off, challenge or block (Settings):

Feed Default What
spamhaus block Spamhaus DROP (IPv4 and IPv6): networks hijacked or run by criminals.
tor off Tor exit nodes.
firehol off FireHOL level 1: attacks, malware and abuse seen in the last days.

Entries holding private, loopback, CGNAT or other reserved addresses (feeds list bogons) and very wide ones are dropped. A feed that fails to refresh keeps its last list; the Settings tab shows the error. The allowlist and the servers' own addresses win over the feeds. Event: rule feed, action denied.

Alerts

With a webhook set (Settings, https:// only), the control plane posts JSON {"text": …, "content": …}, which Slack, Mattermost and Discord take, when:

  • a decoy captures a login,
  • someone uses a decoy's secret,
  • an app has 500 requests stopped within 5 minutes.

At most one alert an hour for each address (or app). POST /api/waf/alerts/test sends a test one.

They also go to the alert channels that take firewall alerts (Alerts).

Retention

What Kept
Firewall events 7 days (up to 8)
Bans, with their strikes 30 days after they run out
Visitors the traffic intel has challenged An hour after last seen