The Day Our Software Update Took Down a City
Before cloud, before remote updates, before anyone called them MSPs — we were carrying servers onto planes and learning about exponential growth the hard way.
In the early days, the product that would later be known as ConnectWise PSA was a fully on-premise offering. In the earliest days, we would visit the office of an MSP (they weren't called that at the time) and we would bring a server along with us.*
It was just easier that way. Our installation was a buzzword bingo card of early 2000s technology. We required Windows 2000 Server, along with SQL 2000, as well as a license to use ColdFusion which — despite years of effort — we just couldn't seem to completely eliminate. The server needed IIS with classic ASP, and Component Services and dozens of settings, any number of which could fail with nothing more than a hexadecimal error code. BINGO!
As the years would progress, that list of requirements would always seem to grow, not shrink.
The initial implementations would take a team of two, usually our CEO and a technical person. In the earliest implementations, we would actually purchase a seat on the plane for the server (this was before 9/11).
I was the technical sidekick for one such implementation in New York City. I remember everyone screaming at each other, and it seemed that was just the way they communicated in New York. By the end of the implementation, our CEO had fully adapted to that style of communication and was screaming along with everyone else. I peered at them from behind the monitor with a mixture of bemusement and concern.
Desperate to avoid a repeat of any of this, I soon recommended we start doing remote installs with Terminal Services. Of course, remote installs meant remote transfers, and remote updates.
ConnectWise was growing exponentially. I learned a few things about exponential growth in those days. It would sneak up on you. Things would feel slow, and then suddenly you were swamped.
This happened twice with our remote updates — both in dramatic fashion. I'll tell you the first story here:
For a couple of years, we were happily sending out updates with no problems, from our mighty office T1. The first sign of trouble was an update where there was some background mumbling about how the internet was a little slow. Hmm.
Fast forward a few weeks, and I released the update on a Friday before leaving for the weekend. When I returned:
CEO: You took down Vero Beach
Me: The city? What does that mean?
We had been hosting the city of Vero Beach, Florida's website and our updates had taken them — and our entire internet connection — down for the weekend.
I had no idea this was the beginning of a multi-year journey to what would become known as the cloud.
In the really early days, we would install updates with floppy disks, but that's a story for another time.