I was playing Santa this year (as always), giving away beautifully packed gifts to my loved ones. During that blissful moment, I happily discovered that my little brother (he’s almost two meters tall now… but you know) gave me one of the books that were hiding on my I want to read bookshelf for some time.

That was - obviously - The Phoenix Project, although in my native, Polish language. I really think it has a cool cover, just take a look below!

First impressions - spoilers ahead

Lately, I’m looking for ways to improve internal processes and teams efficiency across the company, so the statement below the title saying “help your business win” caught my attention briskly. I hoped that Gene Kim, Kevin Behr, and George Spafford will explain this difficult to grasp concept, and maybe I’ll start to embrace devOps as I’ve heard from other guys who recommended the book.

Reading the back cover, I’ve realized quickly, that situation of the main book hero, is quite… familiar. Obviously my story is not that intensive, but that’s something for another post.

Bill is a tech-guy who was just promoted (to his surprise) to a way more responsible position in a fictional, big company called Parts Unlimited. The reason was simple - two of his previous superiors have quit on the same day… Now his main objective is to fix all of the issues and problems gathered through months or years that stop critical project of the company - called Phoenix - from launching into the sky! Great, good luck!

I recall these good ol’ days when I was just transforming cups of coffee into code pieces that I was proud of. Now, there are days when I need to react swiftly to plenty of small disasters, and prioritize my actions since there is not enough time in the day to do all of that crap.

Because of that - I felt immediately that I need to read this book and see how Bill handles the pressure and what’s his thinking process!

It’s not a manual

The book is a novel, and I really like such way of presenting tech concept. Don’t we all just need a good story? Stories are easier to remember, they catch attention and show plenty of context, that manuals don’t. It’s one of the reasons why good stories are one of the best openings for tech talks and presentations in general.

However, that brings one of the issues - for many readers, lack of easily findable recap or a summary of Bill’s actions will cause at least two issues:

• people won’t easily notice the essence, the moral,
• readers can’t treat this as a manual and won’t take this book from the shelf for a quick recap of their knowledge.

Bill’s story is engaging

I really enjoyed it. It looks exactly as I’ve expected - when we think, that situation is already bad, authors happily show us that it may be much worse than that. Life of a high-level manager is not easy - Bill is being hit by all kind of problems, some seem to be clearly technical, other relate to bad communication or different business priorities, but most of the times their origin is hard to trace and they are just outcomes of decisions made long ago.

It’s fun to read how Bill solves these issues, obviously not entirely by himself - but how he manages to find out where these problems came from. It’s like a small IT detective story, where we hope that main hero will finally solve the riddle and piece all clues together.

However, it’s naive

I need to say, I was surprised how small improvements that Bill had introduced created huge impact on the business. It’s like the Parts Unlimited company was hidden behind some kind of a technical iceberg for millennia, and they’ve discovered Kanban board just now. It was quite unbelievable, how tangled the mess is and how they, as a company, could grow so big in that deeply and chaotic environment.

Especially, it was not easy for me to believe that in the end, the huge company was able to:

• transition into the cloud,
• move from old ways of working to agile,
• create an environment with multiple deployments every week…

They achieved all of that in just a few weeks, when everything was about to collapse… I know, I know - dramatic stories like these are cool and easy to remember. That’s the point! I guess we shouldn’t be too harsh about it and ignore these little details. I know that for some readers it will ruin the whole book - so keep that in mind.

Types of work

During that solving process, Bill encounters different problems which are worth further discussion.

One of the first things Bill finds out is that he cannot clearly see what people are actually working on! In agile world, maybe we would call it as a lack of transparency in viewing internal company processes.

The most obvious type of work are business projects. These are basically the main company projects that everyone is aware of, they link directly to company business, and important people say that these “bring value”.

• That may be - for example, website that sells your company products online. They almost always have their own names and the funny thing is, that usually management thinks that this is the only type of work that even exists!

The second type of work are internal projects. These are usually big improvements that the company accepts, but these are less managed and not treated as strictly as business projects. Because of that, most of the time we don’t even know when these end or who’s even working on it… Why these are important? Because they help you build business projects! If you never improve your production line, how can you expect business to deliver products quickly and easily? If you play computer games, treat this as a utility upgrade.

• It may be application accessible only in internal network, that shows all current business projects with their workers, statuses, versions, and deployment statuses.

The third type of work is tricky - those are changes and updates. What do I mean? Well, I didn’t find this to be greatly explained, but basically I understand this as all overhead connected with previous two types of work. Usually, we can treat this as a support kind of work, or routine work. Unfortunately, it’s not usually easy to plan, likes to show up quite unexpectedly, and can be small, but urgent.

• Today you should be working on an important business project but you’ve just received a call from your boss, that new vice president was just recruited and you need to invite this guy to all internal services. It used to be 30 minutes like 2 years ago, but now it takes 4 hours to do that, and you need to do it right now, which causes a delay in the business project work.

Alright! Now - the last type of work - the unplanned work. This is unfortunately the area that no one likes - incidents, bugs, recovering from a different types of disasters that happily struck us when we open our mailbox with a fresh cup of coffee, and naive, optimistic mindset that this day will be different. The previous kind of work tend to generate this type - of unplanned, unforeseen digesters of time. However, we need to work with these, since they are most of the time extremely urgent and interrupt everything else.

• Example - it’s Friday, 5 PM, you should pack your stuff and go home, hug your wife, but instead you receive a note that the production system went down. If it’s not up next hour, you may pack your stuff anyways…

Here’s a great image summary of planning possibilities I’ve found on the web: ¹

And here’s my own interpretation of inner connections between types of work:

Work in progress problem

One of the small things that were eye-opening for me was a visualization of how wait time corresponds to resource utilization.

It was not that obvious for me, that great engineer can be a process bottleneck. The reason is, that everyone wants this person to work on their project, or do any other technical thing, since we all know, that professional will do it best. However, because of that, it blocks everyone else!

By analyzing this graph you can quickly deduce, that:

• If you’re utilizing your engineering team above 90%, any new, urgent type of work that shows up, will be completed later on!

That’s quite important to comprehend. So how to avoid this? Make sure not to have too many work in progress items at the same time.

The Three ways

The next concept - the sacred three ways - are being emphasized a few times thanks to conversations between Bill and mysterious devops zen, called Erik. I need to say - I didn’t buy the concept completely. However, let me explain the ways first.

The First Way is all about the flow. Basically - to enhance the whole system, we need to maximize the flow between Development - which are people responsible for building our application - and Operations, guys responsibile for deploying it to our complicated production environment and keeping it healthy. How we can achieve this?

• Through continuous integration and deployment,
• By limiting work in progress (see next chapter),
• By limiting the number of known defects we pass down the stream,
• By basically automating whatever we can in our process.
  
  
  
  
  
  

The Second Way is related to feedback. We need to know as soon as possible if we have any issues or problems in our system. When we have the feedback, it will quickly allow Development team to improve the system. How to get to this point?

• Create automated test suites,
• Monitor your build status,
• Keep an eye on production system metrics like memory or CPU usage or any other variables.
  
  
  
  
  
  
  
  

The Third Way is the final part, where we continue to experiment and learn. We not only have tools that allow us to improve rapidly, but we have the mindset and culture in the company, that emphasizes how important it is to think about a system as a whole. How to approach this?

• Constantly delegate time to work on non-functional requirements,
• Take risks and experiment to learn from failure Chaos Monkeys,
• Create a company culture that embraces resiliency - every fault makes us stronger.
  
  
  
  
  
  
  
  

The book describes these ways like it’s some kind of hard to find mystery, but I don’t think that’s the case. Maybe besides the third way which nicely introduces chaos engineering - and that was a new concept to me. However - the rest? Quite obvious things, to be honest. It’s hard for me to believe that huge company that works on a single product with 200 onboarded engineers is not using any continuous integration these days, or development team does not understand that if they create a feature which eats all instance memory (they’ve detected it and it’s a known issue), and they still pass it to operations, this problem will bite them later on right in the ass. In other words, I think that just spotting these issues is not enough.

The problem is, that engineers and companies still do all kinds of decision mistakes despite the fact they know what’s the right thing to do.

It’s like eating badly - you know you shouldn’t do this, it’s bad for your health. You don’t need anyone to explain that fact to you… or do you? Sometimes we need to revisit the old problems once again and remind ourselves what’s important. And here’s the challenge - doing things right is simply not easy.

• For us, engineers, because we need to push ourselves more and think about another human being who works with us.
• For business, who needs to place IT tooling into its priorities and treat this as an investment, or company will stay behind in the delivery race.

Summary

It was a cool book to read and I’ve learned something out of it. I wouldn’t say it was so influential for me like The Pragmatic Programmer for example, but it was decent enough that I can say - you definitely won’t waste your time reading it. And because of the way how it was written, it’s so easy to read you can do it during travel, which was a great plus for me.

The Book

The Pheonix Project - it’s not affiliate link!

Other Resources