Why software dev is so hard, Part 1, Part 2

Testing in production is really expensive

As discussed before, decades of experience teaches us we have to show users what they say they want so we can find out what will actually work for them, and we have to do this often. So often, that we created a whole inventory of ways to find out, from ‘painted door’ for concept tests to A/B tests for more tangible experiences, to interviews with prototypes before making things, and then validation testing after making things, in small conversations for insights or large numerical samples. But as this article by Judd Antin posits, what this led to is a lot of research that tells you about features, and very few answers to the big questions of what will actually make a change in people’s lives that they are willing to pay for.

For a decade a whole industry told all of software creation to “build test measure” but forgot to give a real founded answer to build what? Everyone could see that it actually meant “throw stuff you think will work at the wall and hope something sticks”, with stuff being whatever the poor Product Manager or HIPPO could come up with from either their hunches or customer service or competitor features, while at the same time we were telling ourselves it was ever so scientific and valid. You couldn’t but feel cognitive dissonance looking at the process. Meanwhile the design process didn’t even fit software development properly anyway, see Part 2.

It is then not surprising that many Product Managers end up frustrated and thinking that they could get the answers they really need by “getting out of the building” and “talking to a few users.” They feel the real gap in their knowledge is how their users really live with their products, not whether the OK button is green enough. (“Getting out of the building” without a rigorous agenda and user selection doesn’t work, BTW: just talking to whichever target person you end up with will just enable enormous amounts of confirmation bias.)

Don’t take just my word for it; Pavel Samsonov here discusses with other examples how cyclical design and development ends up unsatisfying. Many comments on the article are beautiful examples of copium: repeating Ur Doing It Wrong without engaging with the main problem that everything about making experiences for users, from deciding what to make to how to make it, chafes for every role in the team, and has for the last 15 years.

I don’t agree with the article, and others like it, that the solution is to redefine delivering value every cycle not as making a feature or capability, but to mean learning some lesson. The problem with that is you are still using your development team to ‘try something’ over and over, but this time with the explicit assumption they will throw 75% away. A full-fledged, production-code releasing dev team is a very expensive place to learn what not to build. Among other things, it puts a large psychological toll on designers and developers who are pouring blood, sweat, and tears on stuff to then have it be thrown away in the name of ‘learning’, over and over. You also can’t explain it to the budget people who will see the throw-away as waste.

This cycle was and will remain necessary for a while in organizations terrified of Big Design Up Front or that don’t have a separate deep research capability, but it will feel uncomfortable, and the team will slowly move back from releasing-to-learn to releasing-features-to-hit-KPIs. You don’t build careers by publicly admitting you ‘are going to waste’ 75% of your dev efforts.

But the article does point to a way out: what if your dev team didn’t have to create and release experiment after experiment to find out what to build for real? What if the design and research group could test alternatives much faster, and not just one-page A/B tests, but whole realistic funnels, whole new concepts, many at the same time, without needing dev resources?

Enter AI and no-code

In case you haven’t followed UX for the last 20 years, one of the questions we always debated was whether UX designers should be able to code, and with code we always seemed to mean HTML/CSS/JS (which then implied that UX was only about web pages). The pro-arguments included that designers who could code wouldn’t be wasting everyone’s time by designing impossible pages, or, on the extreme end of the arguments, that designers who actually coded their designs would speed up time to delivery so much because there wouldn’t be a whole specification (first Photoshop, then Sketch, now Figma) stage. There’s a whole set of assumptions in that statement that I don’t have time to unpack, even from experience because I used to be such a designer-developer, but the rarity of user-experience designers who code their own designs into production says enough about how realistic that idea was.

But creating front-end web code has become a lot easier.

• In the same amount of time it takes to become ace at Figma, you can become an expert in Webflow, a system to visually make web-pages. I can now output a static HTML/CSS front end of acceptable code quality, ready for a JS programmer to add connectivity and motion to, as fast as I can wireframe. This means I can explore about three different directions in a day, and thus three different funnels in a week, ready to go live in a sandbox for qual and quant testing.
  
• I recently asked vo.dev, a promp-based (so an LLM under the hood) site-builder, to make me a portfolio site for a very specific kind of UX candidate. It knocked the full code for 3 pages out in 5 minutes, ready for me to change with CSS styling cues.
  
• UIzard (also seems LLM based) is like having a (really dumb) UX Production Assistant on call, that you have to explain a lot to, but will allow you to iterate your ideas inside the tool, prompt after prompt.
  
• I know one high-powered web researcher who is turning around the deepest knowledge-representation experiments, experiments sponsored by the biggest data warehouses in the world, whose outcomes could change whole paradigms, twenty times faster than he used to because he can now add a menu to a web-page to select or transform stored data as easy and fast as he can ask Claude. (The interfaces look super basic and are definitely not production-stable, but for that experimentation that is irrelevant.)

It’s now just a matter of time until the AI UX generators find their way into the no-code graphical HTML tools, with full round trips of intention between the AI and the human. Pretty soon we will be

• describing personas and tasks and creative directions to our UX tools,
• see screens appear as designs but also code, for us to then move and re-color and warp and remix,
• to then be submitted to stakeholders whose comments can be absorbed into the designs real-time,
• to then output live code directly into our A/B sandboxes or crowd-sourced user-testing systems or interview workflows,
• after which the results and statistics get pushed back into our tools so we can tweak the designs.

This is way beyond an AI plugin in Figma, but a live continuous dialog between design, development, and testing, mediated through language, iteration, integration, and direct manipulation like in our current design tools.

The tools themselves will not innovate, their designs will be bland, but they will inform. LLM-based tool only regurgitate what they already know, they are retrospective. ChatGPT will never tell you to query ChatGPT, because ChatGPT’s corpus doesn’t include ChatGPT—but v0 does know everything about what features and paths we have been grouping together in our interfaces. The very first time I used v0 I didn’t just become more productive, but also more complete: its output included some flourishes and ideas I had not considered yet but upon inspection were actually baseline for what I was doing.

(Yeah, there are issues here: LLMs are in the same category of resource-greedy as all crypto-currency, and the big LLMs were all created by stealing the intellectual property of everybody who ever published anything on the web. This can’t be discounted. But if the latest Chinese efforts bear fruit and you no longer need the complete energy output of Wichita and the daily production of Dasani to get a mock-up web-page, plus we factor in that as UX designers we were all copying each-other’s ideas already anyway, then using them for UX design actually could become ethical.)

If a design team can prototype at full fidelity quickly, a lot of current ways we organize designing software can change.

• We can show users many things even better than we used to, and get many comments, keeping us on track. Rule 1 is taken into account.
  
• Design will outrun dev even more than it actually already does in many places, except that now design needs that speed so they can thoroughly vet what they are making at whole new levels.
  
• Product and UX as a discipline will have to get even better at finding hidden ideas and needs from all sources like interviews, customer support, comments, observation, stakeholder knowledge, and at getting quickly to statistical validity for choosing a direction. The main art of UX will go back to being the glue and shepherd this whole design process together to working conclusions, not decide on color wheels.
  
• What you get out of the AI box will be so middle-of-the-road, that making the designs stand out for brands that need it, or for innovative new functions, will require real sweat.
  
• Content Designers will have to push harder than ever for Product to please, please, please start with content first, and let them innovate on content first, or all content will be forced into the same 10 formats. However, a team with strong content designers that takes the time to design some different content formats first will be able to test very fast which of those formats gets the best results for their specific brand and goal.
  
• Design gets a lot closer to being research. Design can experiment more to learn lessons faster, without eating up dev time on prototypes or experiments that will be thrown away.
  
• Dev can now focus on robustly delivering the ‘winner’ but will have to take the design output and run every line through their own translators to integrate them into existing production code. And they will have to fight to be allowed to do it because what the AI / no code / Design cycle produced for testing will look good enough to release. There will be production UXers involved to keep all touch points coherent.
  
• Which will then organize developers and designers into parallel tracks that are less dependent on each other and allow both disciplines to operate at their own speed, increasing comfort and quality.
  
• Hand-off between stages will be even more of a trip. The AIs will help us manage the design systems, the code repositories, link the tokens and and variables directly, drop templates into the CMS for instant use, be helpful in all kinds of ways, yet somehow unpredictably fall flat on their face serving some users complete garbage in ways we can’t even imagine right now. We have to check all systems with intense reviews and QA before anything goes live.
  
• We will need to know our users better than we ever have before we even start to design something, or we will flood the zone with so many bad ideas our users will run away from our product or brand at the speed of light. Y’all have no idea how many bad A/B experiments y’all were spared just because Optimizely had the built-in bottleneck of requiring JS programmers to really work. UX Research had better get really good at answering the big questions about our customers and their issues to keep us on track.

Yeah, design is absolutely in flux with the lay-offs and the re-orgs and the reckonings, but not in the way you think it is right now. How we work is about to be reorganized. The new designers are very ready to use no-code and AI tools to make the current wireframe jockeys look like chumps. Get ready.