Why software dev is so hard, Part 1, Part 2

Testing in production is really expensive

As discussed before, decades of experience teaches us we have to show users what they say they want so we can find out what will actually work for them, and we have to do this often. So often, that we created a whole inventory of ways to find out, from ‘painted door’ for concept tests to A/B tests for more tangible experiences, to interviews with prototypes before making things, and then validation testing after making things, in small conversations for insights or large numerical samples. But as this article by Judd Antin posits, what this led to is a lot of research that tells you about features, and very few answers to the big questions of what will actually make a change in people’s lives that they are willing to pay for.

For a decade a whole industry told all of software creation to “build test measure” but forgot to give a real founded answer to build what? Everyone could see that it actually meant “throw stuff you think will work at the wall and hope something sticks”, with stuff being whatever the poor Product Manager or HIPPO could come up with from either their hunches or customer service or competitor features, while at the same time we were telling ourselves it was ever so scientific and valid. You couldn’t but feel cognitive dissonance looking at the process. Meanwhile the design process didn’t even fit software development properly anyway, see Part 2.

It is then not surprising that many Product Managers end up frustrated and thinking that they could get the answers they really need by “getting out of the building” and “talking to a few users.” They feel the real gap in their knowledge is how their users really live with their products, not whether the OK button is green enough. (“Getting out of the building” without a rigorous agenda and user selection doesn’t work, BTW: just talking to whichever target person you end up with will just enable enormous amounts of confirmation bias.)

Don’t take just my word for it; Pavel Samsonov here discusses with other examples how cyclical design and development ends up unsatisfying. Many comments on the article are beautiful examples of copium: repeating Ur Doing It Wrong without engaging with the main problem that everything about making experiences for users, from deciding what to make to how to make it, chafes for every role in the team, and has for the last 15 years.

I don’t agree with the article, and others like it, that the solution is to redefine delivering value every cycle not as making a feature or capability, but to mean learning some lesson. The problem with that is you are still using your development team to ‘try something’ over and over, but this time with the explicit assumption they will throw 75% away. A full-fledged, production-code releasing dev team is a very expensive place to learn what not to build. Among other things, it puts a large psychological toll on designers and developers who are pouring blood, sweat, and tears on stuff to then have it be thrown away in the name of ‘learning’, over and over. You also can’t explain it to the budget people who will see the throw-away as waste.

This cycle was and will remain necessary for a while in organizations terrified of Big Design Up Front or that don’t have a separate deep research capability, but it will feel uncomfortable, and the team will slowly move back from releasing-to-learn to releasing-features-to-hit-KPIs. You don’t build careers by publicly admitting you ‘are going to waste’ 75% of your dev efforts.

But the article does point to a way out: what if your dev team didn’t have to create and release experiment after experiment to find out what to build for real? What if the design and research group could test alternatives much faster, and not just one-page A/B tests, but whole realistic funnels, whole new concepts, many at the same time, without needing dev resources?

Enter AI and no-code

In case you haven’t followed UX for the last 20 years, one of the questions we always debated was whether UX designers should be able to code, and with code we always seemed to mean HTML/CSS/JS (which then implied that UX was only about web pages). The pro-arguments included that designers who could code wouldn’t be wasting everyone’s time by designing impossible pages, or, on the extreme end of the arguments, that designers who actually coded their designs would speed up time to delivery so much because there wouldn’t be a whole specification (first Photoshop, then Sketch, now Figma) stage. There’s a whole set of assumptions in that statement that I don’t have time to unpack, even from experience because I used to be such a designer-developer, but the rarity of user-experience designers who code their own designs into production says enough about how realistic that idea was.

But creating front-end web code has become a lot easier.

• In the same amount of time it takes to become ace at Figma, you can become an expert in Webflow, a system to visually make web-pages. I can now output a static HTML/CSS front end of acceptable code quality, ready for a JS programmer to add connectivity and motion to, as fast as I can wireframe. This means I can explore about three different directions in a day, and thus three different funnels in a week, ready to go live in a sandbox for qual and quant testing.
  
• I recently asked vo.dev, a promp-based (so an LLM under the hood) site-builder, to make me a portfolio site for a very specific kind of UX candidate. It knocked the full code for 3 pages out in 5 minutes, ready for me to change with CSS styling cues.
  
• UIzard (also seems LLM based) is like having a (really dumb) UX Production Assistant on call, that you have to explain a lot to, but will allow you to iterate your ideas inside the tool, prompt after prompt.
  
• I know one high-powered web researcher who is turning around the deepest knowledge-representation experiments, experiments sponsored by the biggest data warehouses in the world, whose outcomes could change whole paradigms, twenty times faster than he used to because he can now add a menu to a web-page to select or transform stored data as easy and fast as he can ask Claude. (The interfaces look super basic and are definitely not production-stable, but for that experimentation that is irrelevant.)

It’s now just a matter of time until the AI UX generators find their way into the no-code graphical HTML tools, with full round trips of intention between the AI and the human. Pretty soon we will be

• describing personas and tasks and creative directions to our UX tools,
• see screens appear as designs but also code, for us to then move and re-color and warp and remix,
• to then be submitted to stakeholders whose comments can be absorbed into the designs real-time,
• to then output live code directly into our A/B sandboxes or crowd-sourced user-testing systems or interview workflows,
• after which the results and statistics get pushed back into our tools so we can tweak the designs.

This is way beyond an AI plugin in Figma, but a live continuous dialog between design, development, and testing, mediated through language, iteration, integration, and direct manipulation like in our current design tools.

The tools themselves will not innovate, their designs will be bland, but they will inform. LLM-based tool only regurgitate what they already know, they are retrospective. ChatGPT will never tell you to query ChatGPT, because ChatGPT’s corpus doesn’t include ChatGPT—but v0 does know everything about what features and paths we have been grouping together in our interfaces. The very first time I used v0 I didn’t just become more productive, but also more complete: its output included some flourishes and ideas I had not considered yet but upon inspection were actually baseline for what I was doing.

(Yeah, there are issues here: LLMs are in the same category of resource-greedy as all crypto-currency, and the big LLMs were all created by stealing the intellectual property of everybody who ever published anything on the web. This can’t be discounted. But if the latest Chinese efforts bear fruit and you no longer need the complete energy output of Wichita and the daily production of Dasani to get a mock-up web-page, plus we factor in that as UX designers we were all copying each-other’s ideas already anyway, then using them for UX design actually could become ethical.)

If a design team can prototype at full fidelity quickly, a lot of current ways we organize designing software can change.

• We can show users many things even better than we used to, and get many comments, keeping us on track. Rule 1 is taken into account.
  
• Design will outrun dev even more than it actually already does in many places, except that now design needs that speed so they can thoroughly vet what they are making at whole new levels.
  
• Product and UX as a discipline will have to get even better at finding hidden ideas and needs from all sources like interviews, customer support, comments, observation, stakeholder knowledge, and at getting quickly to statistical validity for choosing a direction. The main art of UX will go back to being the glue and shepherd this whole design process together to working conclusions, not decide on color wheels.
  
• What you get out of the AI box will be so middle-of-the-road, that making the designs stand out for brands that need it, or for innovative new functions, will require real sweat.
  
• Content Designers will have to push harder than ever for Product to please, please, please start with content first, and let them innovate on content first, or all content will be forced into the same 10 formats. However, a team with strong content designers that takes the time to design some different content formats first will be able to test very fast which of those formats gets the best results for their specific brand and goal.
  
• Design gets a lot closer to being research. Design can experiment more to learn lessons faster, without eating up dev time on prototypes or experiments that will be thrown away.
  
• Dev can now focus on robustly delivering the ‘winner’ but will have to take the design output and run every line through their own translators to integrate them into existing production code. And they will have to fight to be allowed to do it because what the AI / no code / Design cycle produced for testing will look good enough to release. There will be production UXers involved to keep all touch points coherent.
  
• Which will then organize developers and designers into parallel tracks that are less dependent on each other and allow both disciplines to operate at their own speed, increasing comfort and quality.
  
• Hand-off between stages will be even more of a trip. The AIs will help us manage the design systems, the code repositories, link the tokens and and variables directly, drop templates into the CMS for instant use, be helpful in all kinds of ways, yet somehow unpredictably fall flat on their face serving some users complete garbage in ways we can’t even imagine right now. We have to check all systems with intense reviews and QA before anything goes live.
  
• We will need to know our users better than we ever have before we even start to design something, or we will flood the zone with so many bad ideas our users will run away from our product or brand at the speed of light. Y’all have no idea how many bad A/B experiments y’all were spared just because Optimizely had the built-in bottleneck of requiring JS programmers to really work. UX Research had better get really good at answering the big questions about our customers and their issues to keep us on track.

Yeah, design is absolutely in flux with the lay-offs and the re-orgs and the reckonings, but not in the way you think it is right now. How we work is about to be reorganized. The new designers are very ready to use no-code and AI tools to make the current wireframe jockeys look like chumps. Get ready.

In the last two posts I described two of the ways that I have seen Agile development and UX mix: Sprint Ahead, and a Two Streams model. Two streams is arguably a form of sprint ahead, but one in which the design and development sprints are not locked in their time periods. However you structure User Experience in the development process, though, it does have to come first, it has to help define what development is going to make. Even if you have a separate research department, even if the UXers are completely embedded and blending with developers, UX guides what is made and will be developed and visually designed by interpreting research.

That creates the problem of how to transition what UX specifies to development. It is in the handover that the friction appears. What can the development team handle in the time allotted? And what happens when a hypothesis gets disproven after a sprint?

Handing over: Simple / Better / Delightful

It can be sketches and conversations. Annotated comps. Discussions sitting together. Prototypes. But somehow the concept has to go from UX to developer, even when the work is done in the same sprint by a blended team. In a recent experience, the UX work I did would feed into development cycles, but because of ever-changing circumstances, decisions about what could be put in production could change from sprint to sprint, even while the finished designs were all ready and waiting.

The team soon settled on a handover structure that dealt with this:  I would design a Simple / Better / Delightful solution, and introduce them with a thorough background from testing, research, and stakeholder input.

• The Simple alternative would be the minimum to reach the product goal, usually a very concentrated intervention on the existing product
• Better would be close to state-of-the-art, perhaps also requiring changes to other areas to create a smoother flow with this new feature
• Delightful would be all about creating a unified flow with nice micro-interactions, that would involve significant investment in smarter interactions and perhaps dropping previous product decisions.

While this sounds like designing everything in triplicate, it actually usually meant that I would start with Better, tone it down for Simple, and scale it up with what could almost be called flights of fancy for Delightful. Because of the way Agile design decomposes the work into small issues, this was all very manageable–I wasn’t designing a complete flow all the time, but features in context.

By exploring the solution space this way deeply, come decision making time the decision-maker, like the UX designer or Product Manager, can be very flexible to only go with what they have resources for to put in production. They can show the stakeholders variations together with their cost, and get buy-in for informed decisions.

And Then Everything Changes

As described previously in Sprint Ahead, for some substantial pieces of work I was lucky enough to get budget to do real concept research before the actual design sprints, allowing the design team to test some serious assumptions of what users wanted. This can be seen as expensive, but for major e-commerce it is still cheaper than producing a bad flow that users can’t understand. You do want to avoid getting to a place where you have designed the whole product as a testable prototype before design actually starts: that’s simply about not wasting money. Focus on testing only assumptions, pieces of the flow you have real doubts about. My personal heuristic on top of that is to test to disprove, and do so by testing wild concepts: conversations with users about inappropriate or magical interfaces have often guided me to the heart of their needs.

Currently in our modern Agile world, the Lean methodology is helping speed up product delivery and cut costs by doing all testing and research almost during development: little piece-meal cycles of adding and subtracting features. One thing I’ve never read is how you go from MVP to some sort of MVP+ to maybe a full-featured delightful product: the shops I see trying to get to MVP with whatever permutation of Lean they call Lean basically just keep going with the same process to get to delightful, and I am really not sure that is appropriate; it doesn’t feel like you ever really pay off the design debt incurred from only always adding little pieces together.

However, if you do insist, you will get to the point where you prove or disprove some major assumption about how your product is going to work. A recent case was making a product for which we had a tough time getting to the right specific users until we had something concrete to show, and we had to work mostly with proxies: people who knew the space and possible users really well. Any meeting could (and often did!) yield completely new fundamental insights about the user’s conceptual model. While we were building. This meant not only incurring a huge design debt, but having to pay it on the spot.

Design is unable to keep the development pipeline full, or keep it from outright mutiny, if you have to scrap everything every 3 weeks, no matter how you have structured the cooperation with sprint ahead or two streams or blending all together. It is too wasteful, and angering. Once I realized the team could end up in this situation, my mitigation (I first typed in “solution”, and it is so not) was to defer fundamental design decisions as far as possible, or opt for the most flexible solutions at all time even if they were a little more complex than what seemed self-evident, using my experience to intuit what decision could be fundamental.

An example would be in the Information Architecture: first we thought we would have four major areas of activity, so the suggestion arose to organize them as tabs. I held off strongly because tabs really do not scale on a screen very well if areas are added and could not be secure yet that users would never need to see information from both areas at the same time. I instead opted for collapsible pods that could be added vertically. A few months later we were up to 6 areas of activity, with a seventh on the horizon, and where there was significant difference in the amount of activity in each area. Tabs would indeed have been inappropriate.

In all honesty, the way I managed this designing for possible futures was by mentally, and in sketches, constantly preparing what v2.0 and v3.0 of this product might look like based on what I could extrapolate from what I knew when making v1.0. Also, every time I added something, I had to ask myself, what if this is not true later on, and have an alternative ready. I have to say that working initially on this product as a web app in a responsive design was very helpful, since responsive design done mobile first has to be so constrained in what constructs are used anyway.

In the end the process felt less like putting features together into coherent flows, but more a sculpting by chiseling away hazy possible digital futures down to a product in current reality. There were days it felt like a high-wire act, and I had to do emotional management to make sure I saw every new requirement as a fun challenge instead of a blow to the work already done. I do regret I was only around to take the product to a Beta, I really want to see what the first contact with reality, the first real tests would take it to.

In the first article I described combining User Experience design with Agile timelines by having the design team be 1 sprint ahead for a specific project. But at another engagement, the development team had a steady weekly sprint rhythm, and plenty of backlog for an already live product that needed new features and continuous improvement. Slotting design and prototyping efforts into the weekly sprints made no sense, as this pre-development work was very separate from the development team, and would not impact weekly releases. Doing user research, doing user testing, looking at comparators and competitors, trying alternatives, making a prototype, deciding on tests, workshopping and all just doesn’t fit weekly schedules, nor need it.

Our Product Manager, already brilliantly straightening out the backlog and pointing process, suggested creating 2 streams: one for design, and one for development. Each stream would have its own Jira board, but design would be kanban, and development would continue for weekly sprints. She wrote the design stories based on management, and product ideas, and prioritized them along business and user priorities, with my input on structuring the work. I would pull the top one or two design stories and put them in flight.

The design stories would be worked on with whatever tools of the User Centered Design cycle as required to finish. When a story had been explored, prototypes, tested, discussed, workshopped, the results would then be turned into by us into stories on the development board, with the designs and justification attached. The product design team, by that time, would have worked so close in the process I would not even have to present the designs for grooming or sprint planning, as the Product Manager would know all the nuances. This way, the design stories could take the time they needed, but I did have a responsibility to stay timely and keep the development pipeline filled.

Designing in the Browsers and Boundaries

One issue that we did not fully work out during the engagement was how to fit in Designing in the Browser. The visual designer I worked with on the green stream of work had also become an accomplished coder, and when we designed together he would create the designs directly in CSS & HTML, bypassing any comps stage. This means that the resulting design was guaranteed to be implementable–we pretty much had just coded it during the design phase, save for JS wiring–but some of the resources for the gray stream had been put into action before stakeholders had seen the results. The end result was delivered faster in the end by being able to skip the comps stage, but there was a certain unease that committing resources to coding had happened too early and could have ended up wasted. Unfortunately, we never got to the point where we could prove or disprove that designing in code was as cheap as whipping up a quick Sketch comp for people to approve or steer.

Next: Handing over Minimal / More / Complete

One of the issues that arose mixing Agile and UX in two streams like this was the amount of fine-tuning that had to be done after exploring and settling on a design, when handing it over to the development team and their ever-changing circumstances within a dynamic environment. Another issue was how, because the Agile process explores requirements as it builds systems, having a coherent experience as a goal could create major design debt in a moment when a fundamental assumption of the design suddenly got challenged by the result of a sprint or exploration with the client. I will address how these got managed in the next post.

So much has already been discussed about merging the seemingly unmergable: Agile, a software practice based on not even knowing what you will deliver when you start, but delivering it in small, validated, increments, and User Experience Design, which, like every design practice, is about delivering a complete, though-out, lovely, all-encompassing solution. Endless discussions in Agile about which practices to forget and how to convince your client to trust you without a set endpoint, and case-study after mock-up in UX circles about how micro-interactions are a place to shine and create cohesion between the end-to-end service experience. All I have to add to this topic are actual experiences of doing UX in Agile environments.

Agile always proposes People Over Procedures, so of course there isn’t one good recipe for making good UX happen in an Agile team. Here is my first example of a project that I thought went well.

Sprint 0 / Sprint Ahead

This was an agency project, in which the client did indeed have a specific goal in mind. While Agile classically does not like to have an end point truly fully specced and specified before you start, you can’t just tell a client “Give us your money and we’ll see what we will deliver at the end.” It requires a level of trust you just will not get at the start of a relationship. (It’s also counter-intuitive to any other transaction ever: give me your money and let’s see how much cheese I can give you. Give me your money and I will let you know when your bathroom is finished and what is in it. Give me your money and I may give you that much of a massage, or not. Yes, every project has unknowns and cost-overruns, but it is only in software that we so have no clue what we are doing that we cooked up a methodology where we explicitly state we are not quite sure what’s going on, but we’ll all do our best. And then try to sell it to clients. And it is considered best practice based on the bad results of previous methodologies. Software is truly terrible to make.)

This project had very high commercial value, and thus a lot of heavy weight stakeholders. While Agile is supposed to come with taking risks in releases and fixing them quickly, the client would rather first use the considerable star power they had hired to do Digital to give us in the agency good feedback on everything we did. We therefore made the following choices:

• The Design team consisted of a blend of digital graphic designers and user experience designers. Design would be one sprint ahead of Development. This required the Coding & QA team to trust us in Design not to finish a sprint with a design they could not possibly implement. We tried to manage this risk by including them in the scoping and pointing of Design sprints, and by giving them their own pointing and scoping process in their sprints. Looking back I would say that took a really big leap of faith from Development, and I am very grateful they did it, and that Design honored that leap.
• Sprints were three weeks, with a demo to stakeholders at the end of every week. The demos at the end the weeks were: sketches, which the stakeholders would help narrow down, for week one, further worked out half-comps for week two that could be further critiqued by the stakeholders, and fully finished and annotated comps in week 3 that the stakeholders had better sign off on at that point. Delivery management very quickly learned how to elicit the right feedback at the right time to keep sprints within their boundaries and not have to do emergency work after the last demo of a sprint.
• Before the first real sprint, Design was given a sprint to experiment, concept and research. While the graphic designers spent that time concepting and exploring the brand, we in UX explored various aspects of the funnel by sketching the extremes of how clients might make selections on various platforms, mocking them up in Axure, and formally user testing them. By involving the client in this phase, we created a shared understanding of the truisms and of what was actually false about retailing in this sector, and learned to work together. It is true that basically the client paid to receive knowledge they probably already had in their customer experience divisions, but it was only by exploring this area together, and in new ways from our fresh eyes, that we together could push the state of the art further with confidence. Erika Hall wrote a great article about how this works.
  
• We had also negotiated a week of international user testing before the last sprint so we could do major course correction if necessary. This did put significant pressure on the first two sprints to deliver something that could be quickly be made interactive to be tested (and yes, being responsible for that to happen did give me some bad nights) but it totally paid off when we could have really productive discussions about commercial realities vs user satisfaction regarding the amount of steps in the funnel, now with actual data instead of the guesswork and intuition during previous design.
  (Although it pains me that even after this success, too many people in the agency considered user-testing so negotiable with clients such that I kept having to put my foot down again and again.)

The result was a high-quality front-end design and implementation that could be dropped in and wired up by the client to their back-ends.

Because Development was at the story-pointing meetings for Design, they had their say early and could help us create something they could deliver. Still, there were some moments were Design, and specifically UX, was called over to help out with something that simply could not be implemented as designed in the time available, and required re-concepting on the spot. But hey, real-time design is always my favorite.

This was a very specific structure for a new effort, with no Business-As-Usual to work from, and heavy on the stakeholders and experts. It went well enough that we stayed the course for the second brand we were contracted to design an experience for.

In my next post, I will discuss Agile and UX working together on a continuously developed project, and then a post on how Agile requires UXers to think in versions way far ahead.