The Rest of the Company Can’t Handle Agile

Medium for some reason thinks I continue to want to read about the demise of Agile project management methodologies for making software. My daily email has headline after headline by Product Managers and Engineering Leads writing why Agile—by which they usually actually mean Scrum, a framework of Agile principles turned into a repeatable, teachable methodology—is over.

(I am wondering if, for balance, some other designer who is heavily into Waterfall is getting a ton of headlines about why Agile is the best thing ever and will never die.)

The articles always discuss the shortfalls the authors think comes with Scrum with its short cycles and loads of coordination, but I have barely ever seen discussions of what I consider the actually most painful problem with Agile: the rest of the company can’t handle it. To put it as short as I can:

All our best methodologies to create successful software successfully—Agile, Lean UX, Outcome-oriented delivery—say that every software team needs to be in constant learning and adaptation mode, which means that delivery is now in an R&D model. But R&D is fundamentally unpredictable and most companies are not set up to live that way.

It becomes impossible to plan marketing for a launch, to budget commercials, to have an announcement for a conference, to set up your suppliers, to keep relationships with your partners, to project the ROI to finance and stakeholders, or in any way to hit your internal targets when every week or two the direction, delivery time, or the projected features of the digital product can change.

Inside anything but a small start-up, functions and departments depend on each-other and end up depending on what features and forms are on the website, and yet the software industry at all levels advocates not committing to both a time and scope of work beyond 4 weeks because predictions that include both time and scope beyond 4 weeks overwhelmingly turned out to be wrong in the last 80 years. That’s a really tough message to accept when you depend on that software. Every other discipline can deliver to time and scope, but somehow software can’t? Indeed it can’t, actually, but to truly internalize that you have to have been a software engineer who repeatedly has blown past deadlines you committed to with full confidence just a few months ago.

(Software, but see also residential bathroom remodels. I have never heard of one of those being on time and budget and on-plan either.)

It’s difficult for agencies

Now try being a digital agency, whose model depends on promising clients certain functionality by a certain date. The dance I have seen account managers do with clients to first tell them user and market research may uncover things they absolutely do not want to hear, and then that delivery of what they agree to make might have a large margin of error and thus cost (and never downwards), has been a sight to behold. But in the end, companies use agencies to lower their risk and so they will insist the statements of work eventually specify time and scope of what will be delivered — because who wants to spend money and not know what they are getting when? Most agencies will thus sign everyone in their ranks up to doing fake Agile: everything made in little chunks but with a fixed end date, leading always to overtime.

The best way I ever saw this managed when I worked agency-side was how an account manager, after a lot of discussion with the client and a number of smaller engagements where we had proven ourselves, negotiated that the client would only purchase weeks of time of a certain team. We got away from billing for finished pieces, features, screens, separate deliverables; instead we agreed together on what outcome the client was after, settled on an approach, and would follow design and delivery for it, with a lot of touchpoints to exchange progress and feedback and agree on course-corrections as we learned. We continued to work hard to keep the trust that allowed us to stay within this model that got us away from endless negotiations about milestones and estimates, while staying true to the Agile principle of committing long-term only to time or only to scope, not both—in this case, time.

And businesses that have been around for a while

Inside companies that make their software in-house, this boundary between the Agile and the more traditional parts can get especially painful if it that company has a centralized software group to make the websites and apps that other divisions depend on. This is the model you see most often in companies older than the Internet, who had to learn how to make software later; the company tries to control the unpredictability of creating software by keeping it isolated and concentrated in one place.

This boundary between the Agile software delivery side and the departments relying on features is usually a beleaguered Product Manager who is constantly trying to figure out what the priorities really are this month while also trying not to over-promise anything. They have to handle increasingly pressing questions from department heads about why this thing they need is not live on the website yet and when are we enabling this new product category in the CMS, in between having to convince stakeholders about what the latest tests and data uncovered, to suddenly have management believe your findings (without crediting you) and change direction and expecting results yesterday while the budget stays the same because budgets are set for the year. The Product Manager feel they aren’t really in charge of their product, the other departments feel they have no control over their future, and the C-suite doesn’t understand why everything is so slow and everyone is so defeated and hires another COO to clean things up.

The solution: pushing the Agile boundary up and out

I honestly believe that the only solution long-term to this is the opposite of having a specific IT department, but instead to push software-creation deeper into the company. As I evangelized inside one of the place set up this dysfunctional: “Do you think the business group inside Facebook that manages the friendslist writes up their feature priorities for the year and then submits them to a some central programming group? Sitting there, just hoping their features get prioritized over the needs of the calendaring business group, perhaps shouting louder on every call with the central programmers to get what they need? Of course not, inside Facebook everyone whose outcomes depend on what is on the website gets to program that little piece of the website with their own team — last I heard 7000 teams could push changes to the website. Do you think Spotify’s playlist recommender is a separate business division begging for time from the central Spotify programming teams? Of course not, the Playlist business group makes their own Playlist features. Yet here we are in this platform division of [this company I was working at] trying to juggle a backlog of stories listing what 5 different internal product divisions need from our web platform and asking the VP above them to please set the priority so we can exist without always having at least 4 knives in our back.”

It takes tons of coordination to keep a large digital service coherent when so many teams can push things to the web and apps, but we have structures and procedures for that, and at least it lets all the teams in the company chart their own course. But yes, it does mean a lot more people throughout such a legacy business have to learn about making software. It means that a lot of people who signed up 20 years ago to do, say, production chemistry or business admin, now due to their successful careers in management need to lead software efforts for their division and learn very quickly what difference is between an MVP and an MVT. And lot of people are instead more comfortable leaving that to a separate software group and then bitching at them.

A stunning amount of workplace friction, if not outright toxicity, comes directly from this misalignment between traditional corporate structures and Agile ways of working. It remains to be seen whether AI-assisted software engineering is going to change the unpredictability of making software or the unpredictability of what users will use; all signs are that AI will deliver more of everything per cycle — more screens, more prototypes, more code — but no AI prompting will compensate for not knowing what the market wants or what bugs hide in back-end integrations with your new code. It will remain necessary for software system creators to show their work often, to users, to stakeholders, in order to get feedback whether they’re on the right path. You can call it something else than Agile if you want, but the principles won’t change, and thus neither will the need to do the hard work inside the rest of the company to align everyone around it.

The Map Is Not The Territory And A Wish List Is Not A Map (1)

“Where have all the task decompositions gone?” I was talking to a very experienced Head of UX about the state of our vocation when she asked that. I had to agree I have not seen one in years either. A task decomposition is when you take a task and divide it into steps, and then divide those into smaller steps, until you reach some granularity that makes sense for why you are doing this, like making screens or coordinating robot movements.

We used to do them all the time in UX, mostly when the field was still called HCI, to make sure we understood what the human was doing before we taught the computer to help them with it. There were many notation systems for them, and you could write PhD’s on comparing these notation systems and then inventing new ones.

Also something I haven’t seen in a decade is a specification full of descriptions of features that a system SHOULD and MUST and COULD have. The were called Functional Requirements, and while they often tried not to impose a view on how the system should look to users, you could tell how desperate the writer was trying to convey their needs in something else than fuzzy human language when they invariably started to use Word’s shapes tool to mock up screens—and then write the word SUGGESTION underneath so as to not offend their designers.

TDs and FRs are a relic from the waterfall period when you did a lot of design and understanding up front to make sure you were making the right thing for people before you committed the programming resources to make it. They were intrinsically incomplete in the same way a map always leaves things out of its description of the actual terrain, and expensive to make, and of limited use because:

Humans can not accurately describe what they want out of a software system until it exists.

Bit of an issue.

Rule nr 1 is and was true all the time. You’d computerize a workflow of paper files in a shop or local government and at the end it would turn out that there were all these exceptions being made by clerks and admins using different color pens or writing in the margins that all the workers understood, but nobody above them working with IT did. The exception would be so important you’d have to retool the whole thing down to the tables in the database, and the project would be late and expensive.

When Agile originally said to deliver value frequently, it wasn’t to unlock money from customers cycle after cycle—that wasn’t even really possible until we started putting everything on the instantly monetizable web. It wasn’t for investors either, they will happily wait years for a return if the projected return is big enough. Agile wants frequent releases so you can show the results to humans fast and get feedback and then correct, instead of finding out when you deliver the whole thing after two years that rule nr 1 above always holds. It’s only around the time Lean Startup came along that every iteration wasn’t just to correct the course but also had to deliver some new mini-feature every time.

So if you want to replace Agile Scrum or Kanban with something, you have to deal with the fact that 40 years of trying to first find out how people work, and then making wish lists in all kinds of notations of how that work is to be done by computers, never was really successful and often a total failure.

Still, adding functionality bit by bit as you explore what is needed comes with things you should be aware of:

The resulting system is kludged together cycle after cycle, unless you take some choice time between cycles to refactor huge chunks. This is why every seven years a software team wants to just start over, they can’t take doing archeology in all those cycles of hacks anymore and don’t feel they can add any more functionality without watching the tower of hacks fall over.
It’s actually not faster than Waterfall. It just decreases the risk of ending up with ~~garbage~~ a sub-optimal product-market fit.

But, but, but, if ~~wish lists~~ specifications didn’t work because making software in itself changes the work the software is supposed to help out with, what about prototypes? That worked, right? Yes, with a list of caveats including that Agile actually doesn’t know when to use them and that AI-derived UX is deeply changing that game in the last 6 months, but I’ll discuss that in part 2.

The Two Rules Of Software Creation From Which Every Problem Derives

Scrum has been having a bad time for the last ten years, and thus so has Agile. My favorite article on this is truly exhaustive about all the problems we have encountered in the last two decades trying to deliver software of any kind using these methodologies. (I know nothing about the writer, this could totally be a Milkshake Duck experience; some algorithm just recommended this insanely long post to me one day and I went “Uh huh. Uh huh. Uh huh. Uncharitable but I can see where they are coming from. Baby and bathwater but yeah. Uh huh.” It’s been on my open tabs for 4 months now waiting for me to write this.)

Thing is, I remember the before times. Waterfall, short or long. I delivered projects in those systems. In fact, the first time I encountered Agile in a company, I was new and young and thus stupid enough to not ask what that was about and thus very confused until I looked up where it came from. I remained very confused how you get from the statements in the Agile manifesto (seriously, check them out again) to the rituals of Scrum like stand-ups, retros, pointing and everything else that makes programmers so angry they only get to program 50% of their day and have to talk to other people otherwise, until I actually did some work in Agile Scrum and understood what it was trying to do.

None of the critics are offering real alternatives, just modifications of Scrum to fit Design or Product management in. I don’t think anyone wants to go back to Waterfall, but they can’t really explain why not.

I can. With two rules (which may become more as we discuss them).

It’s the two rules that actually are behind every statement in the agile manifesto. The manifesto unfortunately doesn’t name them really; the people behind it were so steeped in the problems of software delivery—and what they thought would fix it—that they posited their statements without saying why each of these things are necessary to deliver good software. (Unfortunately, necessary but not enough for success, but that we found out in the next decades.)

They are

Humans can not accurately describe what they want out of a software system until it exists.
Humans can not accurately predict how long any software effort will take beyond four weeks. And after 2 weeks it is already dicey.

That’s it. Every other problem that you have to solve in software delivery rolls out of these two major issues, I think. I may be wrong, you may need a third or fourth.

Am I going to spend time proving these two correct? No. There’s enough literature out there, most of it documenting failed software efforts in Waterfall in the 60s, 70s, 80s, 90s, to support them, and I am not going to go over that. What I will, in the next few days, is go over their implications, how they led to current Agile practices, and how they can not be ignored when you want to make things better.