“Teams that are best at DevOps and continuous delivery can release software thousands of times more quickly and with a small fraction of the defects.”

The above quote is from Dave Farley’s review¹ of “The State of DevOps” report². It rang so true that I tweeted it. Martin Fowler responded to the original Tweet from Dave asking if there was a transcript available. So, I downloaded the transcript from YouTube, sanitized it a bit as I was reading through it. It is posted below. I found reading through the transcript very useful.

Transcript

For the past nine years, the state of DevOps report has given us data from tens of thousands of organizations developing software. This data has given us valuable insight into what works and what doesn’t. The reports have produced consistent findings that show things like -

Teams that are best at DevOps and continuous delivery can release software thousands of times more quickly and with a small fraction of the defects. These teams have better job satisfaction, better work-life balance scores, and better retention of their employees. The organizations that work this way are more successful and make more money.

And these practices work at all different kinds of organizations in all different kinds of industries at all different kinds of scales and with virtually any tech stack.

As far as I am concerned, the state of DevOps report is the only source for such extensive data for our industry. We need these tools. I was very much looking forward to the 2020 version of the state of DevOps report and what new insights it might add. But to be honest, I was a little bit disappointed, let me explain why.

Hi, I’m Dave Farley of Continuous Delivery. Welcome to my channel. If you haven’t already, please do hit subscribe and hit the bell notification icon so we can keep you informed of new episodes when they come out every Wednesday.

I’ve been a longtime supporter of the state of DevOps report, and I hope that the authors of the report will treat my feedback as in the spirit in which it’s intended. I see myself as an ally in the same cause, and I want to support them, but I think there are some things to talk about in this year’s report.

This is the ninth report, and overall over those nine years, they’ve conducted over 35,000 surveys. They’ve also this year donated forty-five thousand dollars to charities supporting vulnerable communities in the face of covid19, so I applaud them for that too. This year’s report was probably a little bit down on previous versions with only 2,400 participants, but that’s probably also related to covid, I think. The model that underpins the state of DevOps report described in the Accelerate³ book is important.

The measures of stability and throughput give us, for the first time, tools that tell us something important.

Stability measures the quality of what we produce, and throughput measures the efficiency with which we can produce software at that quality.

These measures give us the ability to evaluate the impact of almost any technical procedural, or organizational change. They won’t tell us if our product ideas are good. We need other kinds of measures for that but for nearly everything else. We can evaluate - “does this move the dial on either stability and throughput for any kind of change”? That’s a fantastic resource.

The state of DevOps report is the first, the only analysis of industry performance that is, as far as I know, based on a scientifically rational approach to measuring the performance of software organizations. It has found some really quite important things. There’s no trade-off between speed and quality. Teams with high scores exceed performance of teams that don’t, often by orders of magnitude on a variety of different measures. Organizations with high scores in stability and throughput make more money than organizations that don’t.

Anyone who has watched any of my videos here will know that I’m a fan of science. I don’t claim or practice a rigorous scientific approach to software development, but I do strongly believe that the approach that I recommend is strongly informed by scientific thinking. That’s what I mean when I say things like continuous delivery is engineering for software, applying that kind of scientific rationalism to solving problems in software.

As a hobbyist, I’m an avid reader and follower of science and engineering, and I’m particularly interested in physics. I just love that stuff. So I confess that I have a physicist’s disdain for lesser disciplines in science. Analyzing human practices like software development is sociology, and it’s hard, realistically impossible, to do controlled experiments in sociology. As a result, sociology is even more prone to common human failings like confirmation bias, and so in many ways, scientific rigor is maybe even more important than in disciplines like physics, where you can have a more constrained controlled experiment.

What has all this got to do with the state of DevOps report? Well, it is and always has been sociology, and despite that, I’ve used the results to help me make my case. There’s always been a bit of me that has worried a little bit about that that the authors were at risk of starting from a conclusion and then seeking evidence to support their conclusion. That’s the route to bad science. In science, we usually do need a hypothesis; we need to have some kind of idea that we’re heading towards. But we have to be very, very careful to collect the evidence that disproves our ideas as assiduously as we collect the evidence that supports them.

This year’s report feels to me purely subjectively to be much closer to that line and maybe slightly over it than previous reports. I agree with many of the conclusions, but that should make me walk more wary, not less. It feels a little bit less detached from the results than previous efforts to my mind. The focus of the whole report is on three topics. Actually, more like two and a half because the last one security is he’s probably slightly less deeply covered than the first two.

The three, the three topics are Platforms, Change Management, and Security. The gross summary of the findings - the ’executive executive’ summary, is platforms are good, change management is usually done badly or inefficiently, and teams that embrace DevOps do better at security. And those are probably what viewers of my channel would expect to see from something like the state of DevOps report.

Platforms

So let’s start with platforms and what the report has to say about that. I think that I might do a diff, a separate video on the idea of platforms. But a quick summary of my thinking. Personally, I’m a bit of a skeptic of the current descriptions and use of and promotion of the idea of platforms and platform teams. That’s not because I think that the ideas are wrong. in particular, I love the work of Matt Skelton and Manuel Pais in Team Topologies⁴, and one of the team types that they call out is a platform team. But, all of the successful teams that I have worked with have ended up with a platform of some kind. My problem is that it’s the “ended up with” that’s important.

I think that the idea of “platform” is best seen as a kind of emergent property of good design rather than an organizing principle in its own right. The vast majority of platforms and platform teams that I see are not something that you would hope for. Most platforms are seemingly a random collection of stuff that’s leftover when you’ve identified all the stuff that you know where to put it. And that’s not how you create a good platform or a good platform team. But I have the suspicion that that is what most people most organizations hear when you say the word platform. I think that a good platform is what happens when you focus first and most on the separation of concerns as a driving principle for good design. If I can kind of abstract away the accidental complexity of my system so that I’m able to focus more on the essential complexity, that’s going to be a good thing, and I’m going to be able to move more quickly when I’m solving problems in the essential complexity that is of real business value. So the ideas of platforms maybe a little bit uneasy.

The first part of this year’s State of DevOps report is so focused on platforms that it almost feels as though they’re trying to sell us something. It doesn’t feel like an objective assessment of software architecture, for example, which might have given possibly different answers. As I said, there’s stuff that I agree with; there’s a lot of stuff that I disagree with. One of the statements in the report is - “you have to make a platform a compelling option”. You don’t force a platform on its consumers. Instead, you build a platform that’s so good that people want to come and use it. I agree with that hundred percent. That’s absolutely how you do a good job of these things. But the report also says things like this. The biggest mistake we see is organizations treating and funding platform development as a project. I think that’s too subjective. I think that’s just one route to success, one possible route to success, and not the only one.

The best platforms that I have ever been involved with or ever seen weren’t built by a separate team. They were built and maintained by the Stream Teams, as Matt and Manuel would call them. The teams that are focused on delivering business value and they built the platform features as they found that they needed them. This may not scale as well, but you get a much better platform with this approach. The smaller approach also gives you fewer organizational dependency problems the team can move forward, not waiting on features that it requires from a platform managed by somebody else. And so, it’s contextual. It depends. There are multiple ways of getting to a good end, and the report doesn’t talk about any of those things.

The report also says things like this. The more a respondent agreed with these statements about their platform teams, the higher their score for a product mindset. Why? What’s that got to do with anything? Why does the score and the product mindset matter? Where’s the value in that? Where’s the evidence that that’s an important outcome?

Change Management

The next section in the doc in this year’s document, in this year’s report, is on change management and reinforces some findings from earlier reports. It’s nicely summarized in this year’s report in these words.

“the most effective change management is achieved by a high degree of test and deploy automation allowing employees more scope to influence change management and a DevOps process and culture.”

It goes on to say orthodox approval processes for change management are nine times more likely to have high levels of inefficiency. Employee involvement is five times more likely to have a highly effective change management strategy. Good stuff, interesting stuff.

The biggest challenge for effective change management, as reported in this year’s document, is poor test coverage, organizational mindset, and coupled application architecture. If you’d asked me, I would have guessed those things. This feels a little bit more like motherhood and apple pie. But having data show that most mothers are good is probably not is better than not. It feels that this may be a more profitable area for deeper exploration to me, though. Looking into the detail of those things or how do organizations that are successful improve their test strategy or/and change management processes. Those kinds of ideas digging a bit more deeply into those might be an interesting thing to find out in next year’s report.

Security

The last section talks about security. Some interesting findings that reinforce existing findings from previous reports rather than expose new thinking. It would be my summary overall. But there’s still some interesting data here. Organizations with full security integration, which I think means kind of building, you know DevSecOps if you want to use that horrible term but taking the continuous delivery approach and evaluating your security behaviors within your deployment pipeline and process. If you do that kind of thing, 45% of those organizations can address a critical volume vulnerability within a working day. In comparison, in organizations that don’t have that kind of forced security integration, only 25% of organizations can do the same thing.

And then the report says things that just irritate me a bit. for DevOps principles to spread further though, people who care about the movement need to extend their empathy and passion beyond the teams that are closest to them and learn to collaborate with teams whose functions are further away. Well, I can’t disagree with that as a sentiment, but this is more kind of motherhood and apple pie, and it’s not really based on data or evidence. This is based on the assumption of the authors of the report. My problem here is that I think that this stuff is more important than that. I believe that the evidence from the previous state of DevOps report tells us something quite profound.

We can see for the first time in software and measure the kind of impact that we see in other disciplines only when we apply scientific engineering style thinking. We have here the basis of a true engineering discipline for software development. We can go faster and produce better software, have better work-life balance, and make more money if we do these things. Those are the measured results of previous year’s versions of the state of DevOps reports. This is or should be more than just a social cause. I don’t disagree that we need to reach out and apply this thinking more broadly. We do. But the focus of the state of DevOps report is to give us better evidence, not fond wishes. One of the other quotes from the report is this - perhaps a few years from now, the term DevOps will sound quaint. To be honest, this did make me smile because I’ve thought for some time that DevOps sounded rather quaint and a bit wrong thank you very much for watching.

Image Credit - https://puppet.com/resources/report/2020-state-of-devops-report/