“Like matter and antimatter, in the presence of unplanned work, all planned work ignites with incandescent fury, incinerating everything around it.” ~ Gene Kim, The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win1
The book makes the case for proper planning, rigor, and discipline as an antidote to getting out of the impossible quagmire of technical debt, tightly coupled systems and unachievable product deadlines. The world is moving fast and successful companies usually grow much faster than anyone in the company can envision, usually in unanticipated directions. Of course, it is always better to have a messy system that serves customers than a clean system that does not have customers. A road full of potholes can serve the purpose of taking someone from one point to another but it is a toil, harmful for the vehicles and the people, and it only gets worse with time leading to despair. Work has to be joyful and a smooth build and release pipeline should be the last of the worries for engineers.
Was reminded of this quote from Dave Thomas’s2 talk on “Agile is Dead (Long live Agility)”3 -
“99% of software projects looked like this [picture of revolutionaries with guns], valiant people manning the barricades, do-or-die, mostly die. Chaos, smoke, gunfire, death - that was your average software project, and clearly that’s not sustainable. It’s fun and you get to wear cool uniforms but it’s not sustainable.”
Through engaging story telling, “The Phoenix Project” takes the reader from visualizing a situation of complete disarray where the company is failiing financially, not able to launch things on time and people are overworked, to seeing an ideal state where everthing, including the unpredictable things, are predictably executed. Most of the situations that the characters go through will evoke a sense of déjà vu for folks who have been active in Software development in the last couple of decades, especially in ‘online consumer’ companies. Many of the situations and dialogues might seem forced and a bit artificial. But overall, it is a good read and recommended reading for most engineers. It is a fast read and almost unputdownable.
I look back to a time when downloading code from clearcase repository, doing a build and starting the server could take almost a day sometimes. Now everything is so fast that it seems magical. The ability to clone a random repository from github and have it up an running in minutes is nothing short of extraordinary. The story underscores the importance of incredible feeling that one gets when things go smooth everytime and things get done as per expectations.
Inspired from ‘The Goal’4
For those who have read Goldratt - The Goal, this book will seem very familiar. In fact, the structure is quite the same and the authors do give credit to “Sensei Goldratt” in a few places. When you read the beginning, you know how it will end but it is the journey that makes it fascinating. I’ve been involved in multiple large scale software migrations, most of them being backend services that handled billions of API calls per day and had huge blast radii. It is usually not the question of what the end goal is but about handling situations that arise during the migration. It is like you’re on one bank of a river and want to get to the other bank. You get on the bridge and realize midway (or earlier) that it is incomplete. You’ve got to figure out how to bridge the gap in the bridge.
Bill Palmer, VP of IT, who is the central character, figures out how to get out of many difficult situations by thinking through questions raised by Dr. Erik, who is similar to the Jonah, the ‘Yoda’ like person from ‘The Goal’. He comes up with the four types of work just by the prodding questions from Dr. Erik. In the same way, Bill graps the concept of Goldratt’s “Theory of Constraints” and works out that a brilliant engineer like Brent who is called for every important thing and is always overworked, is actually one of the bottlenecks. Using TOC, the team identifies bottlenecks in the system and introduces continuous integration and deployement thereby improving the flow of work.
Here are some concepts from the book.
The Four Types of Work
- Business Projects - Much needed for the business to survive, driven by market forces and business leadership within the company.
- Internal Projects - Also called as “Platform projects”, these are technical in nature and involve upgrades, rearchitectures, patches, etc. The technology landscape is changing so fast that no adopting the right technology can have huge impacts to business deliverables. Say - not having Disaster Recovery (DR) capability and being unavailable for customers, or exposing customer data due to a security breach, could destroy the business.
- Changes - Bug fixes, updates, improvements generated by first two types of work. Changes can be anticipated and handled usina a process like Kanban.
- Unplanned Work - It is invisible force that drags everything else down. If something that is done has to be redone because the implementation was faulty, that is usually not accounted for. The quote at the beginning of this post is said in this context. “Like matter and antimatter, in the presence of unplanned work, all planned work ignites with incandescent fury, incinerating everything around it.” Phew!
The Three Ways
At the end of the novel is a section fully dedicated to explaining the Three Ways in detail. It is taken from the follow-on book ‘The DevOps Handbook’.
- The First Way — Principles of Flow - Continuously improve the flow of work. Make work visible, limit work-in-progress (lean principle), reduce batch sizes, reduce hand-offs, remove bottlenecks and waste.
- The Second Way — Principles of Feedback - ‘Fail fast’ and build new knowledge fast, move quality closer to the source. Build a strong feedback loop by shortening the bridge between operations (or customer support) and development.
- The Third Way — Principles of Continuous Learning - Improve daily work and reduce toil, embrace a learning culture, try out new things, learn from failures.
Notable Quotes
“Until code is in production, no value is actually being generated, because it’s merely WIP stuck in the system.”
“Left unchecked, technical debt will ensure that the only work that gets done is unplanned work!”
“Remember, outcomes are what matter — not the process, not controls, or, for that matter, what work you complete.”
“Improving daily work is even more important than doing daily work.”
“Being able to take needless work out of the system is more important than being able to put more work into the system.”
“Features are always a gamble. If you’re lucky, ten percent will get the desired benefits.”
“Any improvements made anywhere besides the bottleneck are an illusion.”
“To tell the truth is an act of love. To withhold the truth is an act of hate. Or worse, apathy.”
“Practice creates habits, and habits create mastery of any process or skill.”
“You get what you design for.”
“Every work center is made up of four things: the machine, the man, the method, and the measures.”
“There should be absolutely no way that the Dev and QA environments don’t match the production environment.”
“It’s not the upfront capital that kills you, it’s the operations and maintenance on the back end.”
“But problems, like dog poop left in the rain, rarely get better just by ignoring them.”
“IT is not merely a department. Instead, it’s pervasive, like electricity. It’s a skill, like being able to read or do math.”
Footnotes
-
Kim, G., Behr, K., & Spafford, K. (2014). The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win. IT Revolution. ↩
-
{{< youtube a-BOSpxYJ9M >}} ↩
-
Goldratt, E. M., & Cox, J. (2016). The Goal: A Process of Ongoing Improvement. Routledge. ↩