Blog

On Ships, and Shipping Incrementally

Photo courtesy of NASA Archives

This is the U.S. Navy’s MQ-8C Fire Scout. It’s an unmanned drone, intended to provide recon and offense for ships at sea.

The MQ-8C Fire Scout. Photo courtesy US Navy
The MQ-8C Fire Scout. Photo courtesy US Navy

There are two weird things about it:

  1. It isn't what one imagines when one thinks “drone”; it basically looks like a normal helicopter with the windows painted over.
  2. It didn't go way over schedule or budget, like so many modern military projects.

These things are related!

The MQ-8C airframe is sourced from the Bell 407, a best-selling manned helicopter design since 1996, which itself is based on the Bell 206, which in turn was based on the YOH-4, which first flew in 1962. This basic design has a rich heritage, with thousands of copies and millions of flight-hours behind it.

Similarly, the robotic UAV system was sourced from another existing design - the MQ-8B Fire Scout, which has been in production for 20 years.

The Navy - uncharacteristically - took an incremental approach to the MQ-8C. They relied on proven technology, made some modest changes, and within a few short years had a new, capable system deployed in the field.

Compare this to the money pits of recent Navy infamy: the Zumwalt Destroyer, the F-35 fighter jet, the Gerald Ford-class Carrier, or the Littoral Combat Ship. Combined, these have run billions of dollars and decades over plan, in large part because they tried to make “great leaps”, advancing many parts of their stack all at once. It wasn’t enough to introduce 1 or 2 new technologies into a new version - the whole platform was to be built anew, from the frames to the sensors to the power systems to the weapons.

In the tar pits

The incentives to approach development as a "big bang" are strong. Planners only work on so many projects in their careers, and have all these cool ideas, and it's someone else's money... why not try and push it all at once?

While appealing, this approach is doomed to fail. Each upgraded sub-system has some natural risk of delay, and the more one crams into a new platform, the higher the likelihood that the whole thing gets bogged down by the "long pole in the tent". And so we wind up with a brand-new destroyer getting towed through the Panama Canal.

There are clear analogs in the software world, which is also marred by delayed projects and vaporware, especially around re-writes. In The Mythical Man-Month, Fred Brooks coined the term "Second System Effect" to describe the tendency to over-scope successor projects, leading to their (often indefinite) delays.

Despite the book's presence on the shelves of generations of product and engineering managers, this trap - the figurative tar pit from the cover - is still easy to fall into. Some of us at Hex have been snared in the past. It's exactly what happened to me with a failed re-write a few years ago.

The cover of The Mythical Man Month has a tar pit for a reason
The cover of The Mythical Man Month has a tar pit for a reason

We hated our legacy backend layer, had big ideas for a new one, and were so excited for everything we could build on top of it. Our team had a vision, and to us that meant diving straight into the new thing, guns blazing. Anything less would have felt like a compromise.

The first couple months of development went great: lots of progress, the new system performed great in isolation, and we were already developing new frontend features to take advantage of it.

In reality, we were speeding toward a brick wall. Our plan required a big bang cut-over between the old and new backends, and we kept finding new gaps that we needed to make the transition 100%. We were so close, but after months of grinding we had to pull the plug because it just wasn't going to work.

Through this, I learned (the hard way) that the only path around the 2.0 tar pit is incrementalism: replacing the previous platform through small, deliberate steps versus one fell swoop.

Incrementalism in Action

This excellent article on rearchitecting explores incrementalism in depth, framing Rebecca Parsons's concept of "evolutionary architecture" as "one that supports incremental, guided change across multiple dimensions."

The piece also references Martin Fowler's classic analogy of a strangler fig: "gradually create a new system around the edges of the old, letting it grow slowly over several years until the old system is strangled." While a bit macabre, it resonates deeply.

The circle of life
The circle of life

The WAFL (Well-Architected, Functionally Limited) approach from CircleCI could also be a helpful framework, especially for frontend redesigns where an in situ, piece-by-piece system replacement might not be feasible.

Shopify's recent post on their big storefront re-write serves as an excellent example of incrementalism:

After our success with the password page, we tackled the most frequently accessed storefront pages on the platform (product pages, collection pages, etc). Diff by diff, endpoint by endpoint, we slowly increased the parity rate between the legacy and new implementations.

By placing a new system alongside their legacy system and slowly cutting cutting traffic over, they built confidence and avoided a risky big-bang transition. Incrementalism FTW!

Slow is Fast

More than any specific method or technique, the first step to avoiding "second system" failures is internalizing that an incremental approach doesn't mean a lack of ambition. It's okay to take things step by step. Focusing on small, shippable chunks can demonstrate progress to yourselves, and others. It may not be as sexy, but will feel better for everyone involved, especially versus holding breath for a singular drop that may never come.

And now back to the Navy: I'm not a weapons system designer 1, but sometimes I wonder how the F-35 would have turned out if they had pursued an incremental path, like the MQ-8C: take all the advanced avionics, and first get them working in an F-18 (or vice versa: put an F-18’s older systems in a new, stealthy airframe), and then iterate from there. While the first version wouldn't be as capable on paper, I bet it could have been deployed years earlier and still reach the ultimate destination faster – without lighting billions of dollars on fire.

While software is cheaper to develop than a warship 2, the U.S. military's mistakes are close kin to those committed by many development teams. And, like good product managers, perhaps our nation's Admiralty need to internalize that the fastest way to get to the ultimate destination is through small steps, not giant leaps.


  1. I would love to hear from you if you are

  2. usually, at least, despite the military's best efforts

At Hex, we're taking an incremental approach to building software, and making mostly new mistakes along the way.
Sign up for the beta