Tell ‘Em Before They Ask…or Else!


If you ask most IT Directors what single system they support that carries with it the highest level of stress, you’ll likely get a pretty uniform response – their e-mail system.
This point was driven home like a wooden stake in the heart of the vampire last week when Microsoft’s Cloud e-mail offering, Exchange Online – part of their Office 365 suite – decided to go tango uniform for more than 5 hours.

Most people that make a decision to move their e-mail operations do so for one of two reasons. The statistically more likely reason is that properly implementing and operating e-mail systems involves a great deal of both capital investment and operating expense. Like it or not, e-mail is mission critical – the vast majority of business transactions today take place and are documented by the organization’s e-mail system. E-mail downtime equals paralysis, pure and simple.

Designing in and then implementing the proper degree of hardware and software fault tolerance is staggeringly expensive, and employing the people that understand and can effectively manage that complex infrastructure is only slightly less so. In startup or small organizations making the necessary expenditures can be prohibitively expensive, from both the capital investment and operating expense perspectives.

The second reason is almost a logical outgrowth of the first – uptime and availability is a function of both investment and expertise – and smaller organization should be able to exploit that advanced expertise and larger, leveraged investments. At least in theory, the major providers of cloud e-mail services – Microsoft, Google, et al – should be masters of that discipline. Looked at dispassionately, Microsoft should be able to design and operate a multi-site fault tolerant service that can take a direct hit, lose a site, and no one will ever notice.

Operative concept here is should.

One of my favorite writers on the subject of engineering is Kevin Cameron, who is a mechanical engineer by trade. Kevin is a large, versatile mind, and his thoughts on design principles and practice have applicability far beyond the internal combustion subjects about which he writes.

One of his recent columns was examining the systems integration issues being encountered by new forms of multinational production, especially those being seen in the commercial aircraft industry. Kevin concluded that the types of systems now being designed are so staggering complex, and their components have such an extraordinary range of interactions with each other, that engineers no longer have the ability to predict the types of system failures that can occur – essentially, that systems have become so complex, that when one component failure starts to cascade to other system components, that the collapse of the system is a completely unpredicted event that may cause large portions of the systems to have to be redesigned to accommodate it.

Kevin might have been talking about the new Boeing Dreamliner, but he just as well might have been talking about any complex cloud infrastructure – both Exchange Online and Amazon Web Services have both experienced widespread system outages that their architects would likely have told you were impossible before they occurred.

Everything built by the hand of man can and will break. The odds of such failures – if proper precautions have been taken – may be infinitesimally low, but they will occur.

Good IT service providers are fully prepared to communicate definitive information to their customers when stuff under their control, and on which their customers depend, inevitably breaks.

My employer’s IT Service contracts define extremely specific notification procedures and methods to be used in the event of a failure with the potential to impact the customer’s business. Those commitments list specific individuals to be contacted, the times they will be provided with initial notification and updates, and the specific communications methods that will be used to make those notifications.

When people entrust you with their business, the most important thing they want to know is that you, as an IT Service Provider, understand the disruption and problems that a service outage is causing. They want to know that you are aware of the scope of a problem, that you have a plan to address it and are working that plan, and your best estimates as to when service is going to be restored.

Contrast that best practices approach with what occurred during the Microsoft Exchange Online outage. Astoundingly, with untold tens of thousands of folks depending on the service, Microsoft didn’t appear to have an out-of-band method already defined to proactively communicate with their users.

“I know… we’ll just send out an e-mail…..oh….waitasecond…….”

When they did finally determine a method to communicate with their customers – several hours into the event – we were treated to the unseemly situation of Microsoft resorting to the use of Twitter … one of the biggest technology companies in the world using someone else’s application….to get the word out. This event and the way that Microsoft shared information with their mission-critical customers made them look unprepared and frankly, like rank amateurs.

Everyone makes mistakes. It’s how one responds to those mistakes that demonstrates the quality and character of an organization.

If you go to the market to purchase any sort of IT Service, you should rightfully expect that your service provider will have predetermined and failsafe methods already defined to notify you of any issue that has the potential to impact your business. You should also expect that those methods and escalation paths should be documented and part of your agreement with them.

A service provider that doesn’t think that proactive communication around outages and return to service isn’t a critical part of what they should be doing for you is either guilty of extreme hubris, delusional, or maybe both.


The Fact-Free Universe


Some days one just feels out of sorts.

You can’t exactly put your finger on what’s wrong, but something clearly is.

Cause and effect don’t seem to be working. The comforting normal logic that usually holds everything together seems to have taken some ‘paid-time-off’.

Then it hits you.

People are saying stuff, but none of it is making any sense. Lips are moving, but nothing useful is implanting in your brain.

Folks are repeating the same nonsense – over and over – in the fond hope that if they repeat it enough times, it will somehow become true.

Welcome to the fact-free universe.

Things didn’t used to be this way.

The objectively observable – what we used to call facts – used to underlie our entire intellectual lives.

It didn’t matter what the sphere of discourse was. Public Policy. Science. Technology. And most assuredly, Commerce. Commerce, above all things, was firmly rooted in the most objective of facts, the dollar.

But then something happened.

It likely started in politics, where the distinction between ‘Facts’ and various grades of untruth, ranging from ‘the excusable spin’, to ‘the misrepresentation’, to ‘the inexcusable fabrication’ all the way through ‘the jaw dropping whopper’, were just a little less clear cut.

Things generally deteriorated when the internet, media and new media collaborated to create an entire ecosystem for reinforcing statements which were not based in fact. Repetition was essentially substituted for objectively observable truth – “Say something enough times and people will falsely conclude that it is true”.

Soon, that system was spun all the way up and working well enough that people saying stuff that was so far out there – so far removed from fact – became just another everyday occurrence. That it became something that people stopped paying attention to.

And that my friend, was most assuredly not a good thing.

Because with Politics, at least the fact free universe was confined within a cosmically small proportioned box.

But then, it got out.

And like the Manhattan Project’s bright flash in the desert of Alamogordo, a chain reaction began that we’re not entirely confident will ever actually stop.

Lately, I’ve been seeing it showing up in IT Service Business contexts, and I’m not sure I have the toolset to deal with it effectively.

The signs are pretty easy to recognize.

There will be an very very expensive suit. Inside the suit is either an attorney who has never been present in previous conversations, or a consultant from one of the 10 firms that make their bullion by negotiating-IT-Service-deals-by-proxy.

The job of the-person-in-the-suit will be to say things on behalf of his client that his client would struggle mightily to say without busting out laughing.

Folks that have sat across the table from me always inevitably remark on my ability to remain calm, civil, and tell people that they are completely whack-a-doodle in a way that somehow escapes their notice and that they inexplicably end up feeling good about.

It’s a talent.

But folks that will sidle up to me and smilingly say stuff that has nothing to do with the world in which I live and work have lately caused me to take and extra breath or three before I speak.

“I’m sorry, but you cannot ask me to agree to service penalties of two million dollars on a service contract worth one hundred thousand dollars.”

“I’m sorry, but we cannot agree to guarantee end-to-end availability on a network where we only provide 10% of the total connection path and devices.”

“I’m sorry, but I cannot agree to a master agreement where we agree in advance to the types and amounts of lawsuits to be filed against us for various forms of service degradation.”

“I’m sorry, but we cannot agree to a 2 hour onsite hardware repair commitment for your ‘field office’ in the hills on the border between Pakistan and Afghanistan.”

Calm. Civil. Based in fact.

If the person on the other side of the table doesn’t engage on these terms then, and wants to stay out in the twilight zone, it’s a walk away.

I always say precisely what I will do, and work to ensure that I and my companies do exactly that.

I’ve spoken before about the need for absolute transparency in conducting IT Service business. That quality of communications is completely symmetrical – it’s a two way street. If both parties to the transaction aren’t equally committed to being clear and unambiguous, then the conversation is just a monologue.

An organization that will resort to these kinds of tactics is simply not going to be a trustworthy business partner – not because they will use untruths to gain advantage, but because they may have squandered their ability to differentiate the truth from something less. One can’t have any assurances that customers will perceive value and that suppliers will make money in an environment devoid of objective facts – performance, commitments, measured timings. Some organizations have gone so far out into the fact free universe, that they can’t even find their way back to the objective world in which the rest of us live and work

Hardware is for the Weak

Me, I love hardware.

Almost doesn’t even matter what kind of hardware it is.

I used to have an Apple Lisa that was one of the earlier pre-production units. The entire system looked handbuilt and like it was 100% focused on the guy that might have to someday service it. All of the subsystems – power, memory, processor, floppy drives (yeah…) — were encased in aluminum billet modular sub-chassis, secured by Dzus quarter turn fasteners like a formula race car or bike, and the entire system could be disassembled in about 40 seconds with no tools. It had Steve and Woz’ signatures cast into the interior of the back cover.


I have had several very early Compaq rack mount ProLiants that were almost as pretty.

My stereo speakers are a set of vintage Yamaha studio monitors. Apart from their dead flat response curves, if one pulls the drivers out of the cabinets one finds the commonality between Yamaha’s music and motorcycle businesses – the driver frames, which are normally stamped steel – are, in these speakers, machined from a cube of solid aluminum billet, with all of the lathe marks clearly still visible.


Folks that know me well know I’m a bit too enthusiastic about the motorcycles that BMW makes. Apart from the elegance and economy of their older designs, if one opens one of their engines up – especially one that has been through 100,000 miles on the road – one sees material choices and machining that meet or exceed NASA standards. My K bike – which has a motor that was simply a miniaturized version of BMW’s class leading early 80s F1 powerplant – looks like a brand new engine that has never been run after more than 80,000 miles – components that were polished or peened during manufacture look new – camshafts and valve buckets have no scuffing or wear marks.


And that’s just the tip of the hardware iceberg. We could talk about titanium oxide plated pocketknives, modern database engine servers with 20 way symmetric processors and acres of DRAM, aluminum bicycle frames with welds that look like jewelry, anything made of Damascus steel, or the intricate machining and solid heft of a well-made firearm.

But we won’t.

Because I’ve come to the belated conclusion that hardware is for the weak. The material world that you and I grew up in is an artifact of the past. Its dead weight that holding us back and keeping us from seeing the real source of power in the modern universe. Power that is nearly without limit, but comes free from mass, free from the need to be maintained, and is completely devoid of any physical manifestation at all.

That power is software, and more specifically it is the software that stands behind newly emerging software defined networks.

You’ve been told, likely a thousand times, that some technology will change the nature of the technology universe. I could tell you that this is different, and you wouldn’t buy it. So let’s skip the hype and cut right to the substance.

Network devices, when compared to their cousins on the systems side of the house, have always been more tightly tied to the hardware on which they run compared to general purpose servers. The three levels of a network device – the data plane, which forward packets; the control plane, which directs traffic on the data plane; and the applications plane, which provides higher level functions like IP telephony, quality of service and security functions – are tightly tied to the hardware on which they run. Network devices – at least ones beyond the basic commodity level – are built around their application-specific integrated circuits (ASICs), and the circuit boards and boxes in which they live.

You want a router? You get a box.

Want a network switch? You get a box.

Want firewalls, quality of service appliances or secure access endpoints, and you might need three more.

You want a better any of the above? You need to get rid of the old box and get a new box.

And those boxes, their maintenance, their management, and all the racking and stacking, unstacking and re-racking, all cost money. Lots of money.

I’ll freely disclose that I work for a company whose legacy is in managed network services. We’re expert in the discipline of keeping all these boxes playing together like a well-tuned orchestra, and its been a very good business for us. Standing up new networks requires large capital expenditures, and a team to manage that network – whether in-house or through a service provider – requires large operating expenses. The more types of traffic one drives through those pipes – data, voice, video, i.e. the higher the degree of convergence – the larger those capital expenditures and operating expenses get.

And when those devices get to the end of their useful lives – whether through increased demand or through age – the entire cycle of investment and expenditure repeats.

Designing, implementing and managing networks is a dynamic, risky and expensive endeavor. If you’ve done it for long enough you are bound to think that there has to be a better way.

And now there is.

I was out grazing the Internet when this story got stuck between my teeth —–39-s-global-cloud-services-show-what-software-based-networks-can-do.html .

My interest is not entirely academic, as my afore-not-exactly-mentioned employer is also a wholly owned-subsidiary of the same corporate parent, so this services roll-out is by a sister company.

The initial suite of features that are currently in production: Firewalls, Application Acceleration, and Secure Access Control represent a pretty substantial percentage of the overall market for network technology and managed network services. So the market is definitely there if the technology does what it claims.

And what it claims to do is staggering.

Good engineering has reworked those three layers that make up network services, and deployed code that can be run remotely and virtualized on somebody else’s shared utility computing cloud.

Want to deploy a new firewall?

No box.

Go to a service portal and provision one. Or, if you’re a managed customer, the provider does it for you.

10 minutes later you’ve got a firewall.

You didn’t make a capital investment for a new appliance. And as you run it, you also avoid the maintenance contract, licensing, support and potentially even some staffing costs that used to come with that firewall. The notion of a ‘technology refresh’ – the cyclical reinvestment that also came with that firewall is rendered meaningless.

Whenever any feature needs to be changed, whether for bug fixes, security patches, or feature upgrades, the virtual machine code can be updated. The provider tests it, determines it works, then shuts down your virtual firewall and brings up a new one.

10 minutes again. Still no box.

You just pay your bill.

Imagine how much one could save out of your total budget for a firewall.

Now imagine your network has 300 firewalls.

And 300 application acceleration appliances.

And few hundred more secure remote endpoints.

See where this is going?

The kind of changes that eliminating capital expenditures and lowering operational expenses of this magnitude make to a business has a direct impact on their financial results.

So if the technology works, this is what good business technical people will do. This technology will completely change the nature of the technology universe.

Now no one will start ripping out running networks to be the first to do this.

But for startups or organizations deploying new locations, this will look smarterfastercheaper.

For companies whose business involves sale and support of traditional networks this looks like one of those pivotal opportunities. To either get on the software defined networking technology train, or risk being run over by it.

The Fantastic Four vs. Auditman


Admit it.

When you were about 9 years old, you know you wanted to be Superman.

Ok. Maybe it wasn’t Superman. Perhaps it was Spiderman. Or The Flash. Or Johnny Storm, the Human Torch.

But as a little kid, with no filter of maturity or responsibility or self-consciousness to interfere with those innermost longings, you know you wanted to be a superhero.

How could you not?

Who wouldn’t want to fly, to bend time and space, to control the elements and physics itself?

Who wouldn’t want to make the impossible actual?

Of course you did.

You’re not 9 anymore, though, and humming the Superman theme song under your breath isn’t going to help with the marching orders you just received at work.

Superhero powers will be required… especially the compression of time and space one.

Startup companies and anyone that works on software projects know exactly how this goes. A customer project due date or a ship date gets established, usually in a complete vacuum of any understanding about what needs to occur to actually meet the commitment. Then, good people, with better intentions, and really understanding family members, burn the midnight oil, then burn the 2 a.m. oil, then watch the sun come up. And repeat. Sometimes for weeks at a time.

These superheroes come up against a seemingly impossible problem, and just guts it out by throwing themselves at it, hoping that the flash of insight needed to get through it will occur, just when it needs to, and just in time to save the day.

You and I, we idolize these people. How many times have we sat through a team meeting where after a successful delivery a glowing management type has gushed over the accomplishments of one of his engineers? We respect their effort, their determination, their results, their superpowers.

They deserve our respect, and we give it to them.

But is this really how it’s supposed to be?

Inevitably there are people that will say that this IS how it’s supposed to be – that without the hard, cold pressure of a cruel deadline, that no one ever accomplished anything. That perspective, and the barely organized chaos that it engenders, make it possible for the same dysfunctional pattern to recur again and again.

It’s like we want to see the flashing capes, see Mr. Fantastic stretching his mind and his body to solve the unsolvable, not see the Invisible Woman as she manipulates things we cannot perceive, see The Thing crush, see the Torch melting weaponry that was thought to be invincible. We want to see the full superheroic arsenal – intellect, physicality, raw power, excitement, explosions – on full display in the achievement of our technologic goals.

It only takes a minute’s introspection, however, to really understand that in the world of technology and technology service, we shouldn’t be craving this kind of excitement, when what we really need is predictable, utterly undramatic boredom. When we contract with a partner to complete a technology implementation, we don’t want Mr. Fantastic, we want Auditman – easily identified by his green eyeshade and readers.

Repeatability and predictability – boredom – is brought about through expert and technically informed project planning, mature processes and disciplined execution. If a technology or service partner is regaling you with battle tales of finishing code or implementing network services in three day long battles under withering enemy fire from disgruntled aliens, what they’re sending you is a coded message – that they hadn’t done this before, or they didn’t understand what it would take and they didn’t leave themselves enough time.

If your provider is telling you that their last technology deployment connected 3,000 sites, that every one was completed when it was supposed to be, that no-one ever so much as got the slightest bit of raised blood pressure, no one was ever late for dinner, and everyone got to bed on time, that’s the kind of war story you should find exciting.

We all need our heros.

Just save them for the comics.


All comic book characters are the property of their respective publishers. My Thanks to Segal and Shuster and to Stan Lee, who taught me that not every lesson from books comes from schoolbooks. And my apologies to the accounting profession for even suggesting they are all boring.