title: "Your Fiber Is Under a Shovel: Why Local Isn't a Preference, It's a Survival Strategy"
slug: "your-fiber-is-under-a-shovel"
category: "homelab"
tags:
- homelab
- networking
- infrastructure
- local-first
author: "Matt"
excerpt: "Your gigabit fiber is one backyard project away from becoming a very expensive paperweight. Here's what happens when you realize the internet is not a utility — it's a chain of custody."
published_at: "2026-05-07"
reading_time: 7
cover_image: "/api/blog/static/images/posts/neighbors-shovel/hero-wired.svg"
It was a Tuesday afternoon. I was halfway through debugging a Tailscale ACL when the calendar widget on my phone went grey. Not "loading slowly" grey. Just… nothing.
I assumed a Docker container had crashed. I SSH'd into the Beelink. Containers were fine. DNS was resolving. I pinged google.com — 15ms. Then, on a hunch, I pinged the upstream gateway. Nothing.
The fiber was down.
I stepped outside. Our neighbor was standing in his backyard holding a shovel, staring at a small trench in the ground. He looked up, saw me, and shrugged. "Didn't know there was a cable there."
That's the moment it clicked. Every layer of abstraction I'd built — the reverse proxy, the Docker networks, the Cloudflare tunnel, the Tailscale mesh — none of it matters when the copper (or glass) in the ground gets severed by someone planting a hydrangea.
The Chain of Failure
{{ '/api/blog/static/images/posts/neighbors-shovel/chain-of-failure.svg' | svg_inline }}
This SVG isn't cute. It's an accurate map of what happened:
ISP Central Office → Buried Conduit → Demarc → Your Router → You
Four links. One shovel. Zero redundancy.
The thing about buried infrastructure is that you don't own it, you can't see it, and you can't fix it. You're renting a piece of glass in a bundle of other people's glass, and the only thing standing between you and a total blackout is a polite agreement with your ISP that they'll try to fix it within 48 business hours.
The Lie We Tell Ourselves
Here's the narrative we buy into:
- "Gigabit fiber is reliable."
- "My ISP has 99.9% uptime SLA."
- "I have a UPS, so my infrastructure is solid."
All of these are technically true and practically irrelevant.
99.9% uptime means 8.7 hours of downtime per year. That's fine for Netflix. It's not fine for:
- Your smart home locks refusing to accept local commands because the cloud can't authenticate
- Your family calendar not rendering because Radicale's DNS expired and no one cached it
- Your kid screaming that Bluey won't play — and you can't explain "DNS resolution failure" to a four-year-old
- Your calendar widget going grey because it never even tried to fall back to local data
The lie isn't that the internet goes down. The lie is that downtime is rare enough to not plan for.
The Neighbor Test
Here's a question worth asking yourself:
If your neighbor (or a backhoe, or a gopher, or a bored teenager with a trenching tool) severed your incoming fiber right now, would your family notice within 60 seconds?
If the answer is yes, your infrastructure is fragile.
If the answer is "yes, but they'd notice because the dashboard they check is still working," you're in the top 1% of home operators.
The goal isn't to hide the outage. The goal is to make the outage irrelevant to the people who live there.
Building the Fallback
This is where the project shifts from "internet infrastructure" to "local infrastructure." Every service you depend on should have an opinion about what happens when the WAN disappears.
{{ '/api/blog/static/images/posts/neighbors-shovel/degraded-mode-arch.svg' | svg_inline }}
What you're looking at is a degraded-mode architecture — every service has a known state when the internet drops, and that state is designed, not discovered in the moment.
1. DNS That Caches Aggressively
This is the single highest-impact change you can make. A Pi-hole or AdGuard instance running locally — even on a Raspberry Pi — will cache DNS responses. After a week of use, your cache hit rate should be 60-80%. When the WAN drops, those cached records keep everything working.
The gotcha: Make sure your DHCP hands out the local resolver as primary, not your router's DNS forwarder. A Ubiquiti Dream Machine will happily forward to 8.8.8.8 even when the WAN is down, and your devices will wait 30 seconds for timeout. Put 192.168.1.x (your Pi-hole) as the only DNS server in DHCP.
2. NTP That Doesn't Drift
When NTP servers are unreachable, system clocks drift. A 3-minute drift after 48 hours is normal. That means:
- Cron jobs fire at the wrong time
- CalDAV conflicts appear when sync resumes
- Logs are impossible to correlate
Solution: Run a local NTP server (chrony is fine) configured with a local stratum. Even without WAN, your network clock stays within milliseconds of itself. External sync is a bonus, not a requirement.
3. CalDAV That Works Offline
If your family calendar depends on iCloud or Google Calendar, you're outsourcing an essential family utility to someone else's SLA. Radicale (or Baikal, or Xandikos) running locally means the calendar is always readable.
The design constraint: Your client apps need to work with it. Apple Calendar on iOS talks to CalDAV fine. The HoffDesk dashboard renders from Radicale directly. When the WAN drops, the data is still there because the database never left the building.
4. Media That Doesn't Phone Home
Jellyfin + local NAS = streaming that works when the internet doesn't. This one is obvious, but it's worth testing: turn off your WAN and try to play something from Jellyfin. If it fails because of some OAuth dance or remote auth, fix it.
5. The Test (Not the Hope)
Here's the most important part: test your failover monthly.
I set a cron job on the first of every month. It:
1. Disables the WAN interface on the router
2. Waits 60 seconds
3. Runs a health check against local services
4. Re-enables WAN
The results go into a Telegram message. If any local service fails the health check, I know about it before my neighbor picks up a shovel again.
What You Actually Need
You don't need 99.999% uptime. You don't need redundant fiber from two different ISPs. You don't need a data center in your garage.
You need your family to not notice when the internet goes away.
That's a different engineering problem entirely — and a much more rewarding one. It shifts the conversation from "how do I keep the WAN up" to "how do I design services that treat the WAN as optional?"
The shovel in the ground wasn't a failure of my infrastructure. It was a design constraint I hadn't accounted for.
Now I have. And I test for it.
This post is part of my Home Lab series. If you're building your own local-first infrastructure, I'd love to hear what your biggest "oh no" moment was. You can find me on Mastodon or in the HoffDesk Discord.