A few days ago, a client’s data center (well, actually a server room) "vanished" overnight. My monitoring showed that all devices were unreachable. Not even the ISP routers responded, so I assumed a sudden connectivity drop. The strange part? Not even via 4G.
I then suspected a power failure, but the UPS should have sent an alert.
The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
To make a long story short: the company deals in gold and precious metals. They have an underground bunker with two-meter thick walls. They were targeted by a professional gang. They used a tactic seen in similar hits: they identify the main power line, tamper with it at night, and send a massive voltage spike through it.
The goal is to fry all alarm and surveillance systems. Even if battery-backed, they rarely survive a surge like that. Thieves count on the fact that during holidays, owners are away and fried systems can't send alerts. Monitoring companies often have reduced staff and might not notice the "silence" immediately.
That is exactly what happened here. But there is a "but": they didn't account for my Uptime Kuma instance monitoring their MikroTik router, installed just weeks ago. Since it is an external check, it flagged the lack of response from all IPs without needing an internal alert to be triggered from the inside.
The team rushed to the site and found the mess. Luckily, they found an emergency electrical crew to bypass the damage and restore the cameras and alarms. They swapped the fried server UPS with a spare and everything came back up.
The police warned that the chances of the crew returning the next night to "finish" the job were high, though seeing the systems back online would likely make them move on. They also warned that thieves sometimes break in just to destroy servers to wipe any video evidence.
Nothing happened in the end. But in the meantime, I had to sync all their data off-site (thankfully they have dual 1Gbps FTTH), set up an emergency cluster, and ensure everything was redundant.
Never rely only on internal monitoring. Never.
reshared this
advokatt
in reply to Stefano Marinelli • • •EnigmaRotor
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to EnigmaRotor • • •EnigmaRotor
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to EnigmaRotor • • •EnigmaRotor
in reply to Stefano Marinelli • • •Marios Efstathiou
in reply to EnigmaRotor • • •@EnigmaRotor
I am quite keen to look into Uptime Kuma. Our current monitor is antiquated.
On a side note, you guys are hilarious! I genuinely had a good laugh at your comments.
EnigmaRotor
in reply to Marios Efstathiou • • •Stefano Marinelli
in reply to EnigmaRotor • • •Stefano Marinelli
in reply to Marios Efstathiou • • •Installing Uptime-Kuma in a FreeBSD Jail
Stefano Marinelli (IT Notes)Marios Efstathiou
in reply to Stefano Marinelli • • •Saupreiss #Präparat500 🗽
in reply to Stefano Marinelli • • •That sounds like a hell of a yell for an official port…
@marios @EnigmaRotor
Dianora (Diane Bruce)
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Dianora (Diane Bruce) • • •EnigmaRotor
in reply to Stefano Marinelli • • •Also note that Stefano’s story does not mention (yet) an overexcited systemd. I don’t see plagiarism in this. Let’s expect an hommage at some point, in pure Hans Zimmer style. /* insert big brass sound sample here */
Michael W Lucas :flan_on_fire:
in reply to Stefano Marinelli • • •Spiritual guide? Your spirits are gonna go somewhere pretty dang weird, sir.
Dianora (Diane Bruce)
in reply to Michael W Lucas :flan_on_fire: • • •Michael W Lucas :flan_on_fire:
in reply to Dianora (Diane Bruce) • • •beer is too normal...
Tim Chase
in reply to Michael W Lucas :flan_on_fire: • • •@mwl
beer? clearly the conversation involves *spirits* not beer… 😛
@Dianora @EnigmaRotor @stefano
EnigmaRotor
in reply to Tim Chase • • •Stefano Marinelli
in reply to EnigmaRotor • • •EnigmaRotor
in reply to Stefano Marinelli • • •Ollivier Robert 🇺🇦😷🌈
in reply to Dianora (Diane Bruce) • • •Dianora (Diane Bruce)
in reply to Ollivier Robert 🇺🇦😷🌈 • • •Stefano Marinelli
in reply to Dianora (Diane Bruce) • • •EnigmaRotor
in reply to Stefano Marinelli • • •EnigmaRotor
in reply to EnigmaRotor • • •Stefano Marinelli
in reply to EnigmaRotor • • •EnigmaRotor
in reply to Stefano Marinelli • • •You know the trade offs of emulation, this could lead to a suboptimal experience… thus I might get rid of the icecream and focus on the bottle. (From simplicity comes user satisfaction)
#drunkCipherMachine #veryRandomNumberGenerator
Stefano Marinelli
in reply to EnigmaRotor • • •EnigmaRotor
in reply to EnigmaRotor • • •EnigmaRotor
in reply to EnigmaRotor • • •I wish this article would be available in English too, yet it is available in Italian. (For some reason)
nom vernaculaire
Contributeurs aux projets Wikimedia (Fondation Wikimedia, Inc.)Stefano Marinelli
in reply to EnigmaRotor • • •EnigmaRotor
in reply to Stefano Marinelli • • •#alps
Elena Rossini ⁂
in reply to EnigmaRotor • • •Stefano Marinelli
in reply to Elena Rossini ⁂ • • •ozoned
in reply to Elena Rossini ⁂ • • •Elena Rossini ⁂
in reply to ozoned • • •Stefano Marinelli
in reply to ozoned • • •EnigmaRotor
in reply to Elena Rossini ⁂ • • •Johan Sköld
in reply to Stefano Marinelli • • •that advice also applies to monitoring scheduled backup jobs (or any other automated process). I use a service that emails me if I don't hit a specific URL roughly every 24 hours, and I hit that at the end of my backup job if it was successful.
Better than finding out the hard way at some point in the future that something happened with my backup job, preventing it from running for the last month.
Stefano Marinelli reshared this.
Stefano Marinelli
in reply to Johan Sköld • • •James Seward
in reply to Johan Sköld • • •Johan Sköld
in reply to James Seward • • •Stefano Marinelli
in reply to Johan Sköld • • •mkj
in reply to Stefano Marinelli • • •Sounds like a case of either good design or *very* good luck too that the UPS took the brunt of it.
We can't protect against everything, but we *can* have an idea for what to do when the unimagined happens.
Stefano Marinelli
in reply to mkj • • •Ricardo Martín :bsdhead:
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Ricardo Martín :bsdhead: • • •mkj
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to mkj • • •pedernal
in reply to Stefano Marinelli • • •Ricardo Martín :bsdhead:
in reply to Stefano Marinelli • • •Utarg of Utarg 🔬🇪🇺🇸🇪🇬🇧🇺🇦
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Utarg of Utarg 🔬🇪🇺🇸🇪🇬🇧🇺🇦 • • •Ricardo Martín :bsdhead:
in reply to Stefano Marinelli • • •@stefano @toxy
Utarg of Utarg 🔬🇪🇺🇸🇪🇬🇧🇺🇦
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Utarg of Utarg 🔬🇪🇺🇸🇪🇬🇧🇺🇦 • • •Elena ``of Valhalla''
in reply to Stefano Marinelli • •like this
Stefano Marinelli, Luca Sironi (fantasma edition), Simon Willcocks, Tom, Leonardo Taccari, johnn_bunyan, J. Steven York e kasperd like this.
Stefano Marinelli
in reply to Elena ``of Valhalla'' • • •Elena ``of Valhalla'' likes this.
Luca Sironi (fantasma edition)
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Luca Sironi (fantasma edition) • • •Wulfy—Speaker to the machines
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Wulfy—Speaker to the machines • • •Kev
in reply to Stefano Marinelli • • •Bojan Landekić
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Bojan Landekić • • •Bojan Landekić
in reply to Stefano Marinelli • • •Elena Rossini ⁂
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Elena Rossini ⁂ • • •@_elena Thank you! Sure, I will 👍
But, to be honest, I don't think any of those stories will ever be a film.
The big, most scary one is yet to come, anyway...
Bob Tregilus
in reply to Stefano Marinelli • • •I don't know, you told this short story like a pro. Starts out, ya, data center suddenly goes dark over the holidays. UPS fails, kinda of ya, ya , still interesting then you introduce the gold, two-meter thick walls, professional thieves, wow, that's some drama! Although, I wonder how they were able to send such a massive power surge down the lines and why the bus mains didn't blow before the equipment was damaged? Looking forward to your next tale!
@_elena
Stefano Marinelli
in reply to Bob Tregilus • • •I'm just relaying what I was told and what I know about the company, for which I've been providing some services for many years. The details came directly from their internal manager and, honestly, I didn't have much interest in digging deeper into the technical specifics of the incident.
My focus was simply making sure their servers were back up and running and that their data was safe. Everything else, electrical infrastructure, physical security, and similar aspects, is outside of my scope and handled by other people.
Bob Tregilus
in reply to Stefano Marinelli • • •Indeed. Still would be interesting to find out about the details of the infrastructure failure and how they pulled it off. Sounds like a good story for a documentary, especially if this is something that has happened in the past.
@_elena
Stefano Marinelli
in reply to Bob Tregilus • • •Ian Campbell 🏴
in reply to Stefano Marinelli • • •This is such a good, if niche, example of "paying attention to the fundamentals and the alerts covers all sorts of things you'd never imagine happening."
Thanks for sharing.
Stefano Marinelli
in reply to Ian Campbell 🏴 • • •stux⚡️
in reply to Stefano Marinelli • • •Great job!
This is why is always run up time on different servers in other places!
Perfect!
Stefano Marinelli
in reply to stux⚡️ • • •Hugo Mills
in reply to Stefano Marinelli • • •About 15 years ago, the place I worked had a supercomputer. One night, the aircon in the machine room failed. The machine kept computing, and the temperature rose. It rose *quite a lot*.
Sadly, the first thing to fail from the heat was the core switch for the room. You know, the one that handles all of the network for everything in the room. Including the temperature alerts.
It was finally spotted about 8am when the security patrol wondered why the door shutters were so hot.
Stefano Marinelli
in reply to Hugo Mills • • •Hugo Mills
in reply to Stefano Marinelli • • •Fortunately, the only thing that did fail after the aircon was the switch. (And a pair of ear muffs which had been hanging on a metal rail -- they'd melted).
The fire brigade turned up, checked everything, and ran some big positive pressure fans to get airflow through the room from one door to the other to cool everything down.
Lasse Leegaard
in reply to Stefano Marinelli • • •Stefano Marinelli reshared this.
Stefano Marinelli
in reply to Lasse Leegaard • • •rasteri
in reply to Stefano Marinelli • • •penguin42
in reply to Stefano Marinelli • • •Space flip-flops
in reply to Stefano Marinelli • • •First off, as a Ukrainian, I know that powerlines can survive "the spikes" by just cutting the power at the very input. No damage to equipment behind the input electric circuit breaker, nope. You just get damaged input.
Next, I used to work in a bank. And here we had a clear requirement for data storage center: more than one power input -- is a must.
Space flip-flops
in reply to Space flip-flops • • •Fourth, if there is a power spike and cut off, it won't go unnoticed by those who control power lines. They will be the first on site to see what happened.
Stefano Marinelli
in reply to Space flip-flops • • •Conny Nasch
in reply to Stefano Marinelli • • •gbsills
in reply to Stefano Marinelli • • •William Weber Berrutti
in reply to Stefano Marinelli • • •The Gaffer
in reply to Stefano Marinelli • • •This immediately brought to mind coming into the office after a holiday weekend in 2005 and finding “my” computer room dark. I found our infrastructure manager, who told me that they had an unexpected power outage over the weekend. Confused, I said “But how is that possible? We have multiple feeds and a huge uninterruptible power supply!”
I will never forget his response, delivered in his thick Scottish brogue: “Yes, we do. But it doesn’t do much good when the UPS catches fire.” 😳
javensbukan
in reply to The Gaffer • • •@thegaffer That reminds me of an incident that happened at work. We have multiple sources of electricity and generators, but none of that matters if the room with the UPS and power controller where all the power sources meet floods from an overflowing toilet a floor above 🙃😅
Whoopsie daisy!
I just finished bypassing all the network switches in the closets from that circuit when they managed to bypass it and catastrophe averted.
That was a fun night! /s
Stefano Marinelli
in reply to javensbukan • • •Pedro Bufulin
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Pedro Bufulin • • •indyradio
in reply to Stefano Marinelli • • •Since 9/11 there are a few new spooky things, and one is modulating the power with pulses
Pedro Bufulin
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Pedro Bufulin • • •indyradio
in reply to Stefano Marinelli • • •You have shared many useful items in a thoughtful way. I appreciate it, and am glad to let you know. 😀
Stefano Marinelli
in reply to indyradio • • •Jim
in reply to Stefano Marinelli • • •itthinx
in reply to Stefano Marinelli • • •Wokebloke for Democracy
in reply to Stefano Marinelli • • •Ondrej Zizka
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Ondrej Zizka • • •joseph
in reply to Stefano Marinelli • • •And while not relying on internal monitoring make sure your external monitoring doesn't share anything with the monitored systems:
Different ISP, different cloud provider if in the cloud, no shared infra at any level
Sharquaydius
in reply to Stefano Marinelli • • •Dan 🌻
in reply to Stefano Marinelli • • •The true horror part of this story:
> The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
Home for the holidays, sick, serious family issue?? Who cares! You know what's more important?? Keeping that data center up and running!
Glory to sacrificing yourself for the system!!
Or maybe get someone else next time.
Stefano Marinelli
in reply to Dan 🌻 • • •miki
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to miki • • •lorenzo
in reply to Stefano Marinelli • • •Andreas (82MHz)
in reply to lorenzo • • •I think Stefano, the mild mannered barista of the BSD Cafe who posts pictures of sunsets and from his walks in nature is just a cover, and in reality he is a tough-as-nails secret military agent who's chasing cybercriminals around the globe.
See also his comment to my blog post about "just telling people to call the Barista" to make them crap their pants... this Barista has a secret! 🕵️
Stefano Marinelli
in reply to Andreas (82MHz) • • •lorenzo
in reply to Stefano Marinelli • • •💙
CC: @82mhz@bsd.cafe
James Scholes
in reply to Stefano Marinelli • • •xinqu
in reply to Stefano Marinelli • • •Uriel Fanelli
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Uriel Fanelli • • •Uriel Fanelli
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to Uriel Fanelli • • •Tionisla
in reply to Stefano Marinelli • • •wow, cool story and well done! 👍
And yes sometimes the truth is really better than fiction (thinking about about something a while back I was part in in my job that could have been easily from a badly scripted reality TV show. Can't go into details because of nda 🙈 )
Stefano Marinelli
in reply to Tionisla • • •Tionisla
in reply to Stefano Marinelli • • •Dianora (Diane Bruce)
in reply to Stefano Marinelli • • •You know this but it bears repeating!
Stefano Marinelli reshared this.
Stefano Marinelli
in reply to Dianora (Diane Bruce) • • •The Psychotic Network Ferret
in reply to Stefano Marinelli • • •Stefano Marinelli reshared this.
Peter Sommerlad
in reply to Stefano Marinelli • • •J. Steven York
in reply to Stefano Marinelli • • •Stefano Marinelli reshared this.
Stefano Marinelli
in reply to J. Steven York • • •zako
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to zako • • •₢ætures :plural_heart: :therian: :ms_nonbinary_flag:
in reply to Stefano Marinelli • • •Krafting
in reply to Stefano Marinelli • • •tkr
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to tkr • • •Sir Dr Rusty o the Isle 🖤💛❤️
in reply to Stefano Marinelli • • •Martin Seeger
in reply to Stefano Marinelli • • •ffffennek
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to ffffennek • • •Paul Wilde 😺 (snac2 acct)
in reply to ffffennek • • •Really, if the primary point of call was out of action, it would be up to the business itself to arrange alternatives, allowing the sick person to stay out of action.
Stefano Marinelli
in reply to Paul Wilde 😺 (snac2 acct) • • •@paul @fennek I am familiar with that organization - and I know that the person (the one who was home sick, even if I didn't know he was home sick) has a deep sense of loyalty, but he is not reckless. If he hadn't been well enough, he wouldn't have gone. I even offered to go in his place myself. It is a healthy environment, not "that typical company" that exploits its employees. For obvious reasons, I cannot disclose details (and I work with several similar companies in different areas), but I can guarantee that everyone acted with the utmost respect for human decency. Fortunately, not all businesses operate like malicious entities that only think about harming their employees and collaborators.
I always strive to distance myself from such organizations, as they do not align with my outlook on life and the world.
Elena ``of Valhalla'' likes this.
ffffennek
in reply to Stefano Marinelli • • •Stefano Marinelli
in reply to ffffennek • • •db
in reply to Stefano Marinelli • • •db
seuros
in reply to Stefano Marinelli • • •Damn you Stefano.
You just spoiled a future Netflix movie.
Instead of watching in 2027 : `The Power Surge Heist`... we will have `The Uptime` with Stefano as sysadmin.
Following you so i can keep up with all the movies i will be missing.
devolute
in reply to Stefano Marinelli • • •I'm upset this didn't turn into a story of how the police laid a trap.
Uptime Kuma is dope tho.