All models are wrong, some are useful.
Introduction
The following is based on my experience building software systems at various scales for more than a decade. In that time, I’ve gone from a lone wolf in a university research lab, to an early engineer in a successful healthcare analytics startup, back to a small HR start up, through acquisition to a large HR software company. Most of that experience was working alongside 10-20 software and data engineers, though my current position puts that number north of 200. My experiences don’t encompass the whole of developing software products, and I am notably lacking experiences in the realm of 70-80 engineers or at companies with >10k employees. Therefore, there will be holes. However, there are holes in all of our lived experiences because time is a finite resource. The best we can do is try to make sense of what we’ve seen.
While this work draws inspiration from the book King, Warrior, Magician, Lover, an introduction to the Jungian archetypes of mature masculinity, software engineering is genderless. I firmly believe that the current bias in the field towards introverted males is a product of far greater systemic educational and cultural biases relating to how parents select, often at too-early ages, the activities and fields we think our children will best excel. In that sense, the current demographics in software engineering is a reflection of our educational system 10-20 years prior. My ongoing hope is the choices we’re currently making with our own children are an improvement over those made by our parents and that our offspring will make even better choices when they come of age.
Archetypes and their usages
Archetypes are consistently recurring motifs, patterns, or symbols found in works from seemingly unrelated sources. These motifs are found throughout literature and the arts. For example, if a gun is on display in the first act of a play, odds are pretty good it will be used in the final act. Sometimes we’re instantly aware of these motifs in the art we consume, while at other times it takes years of study to decipher the underlying principles in a collection of work. Such efforts of study are often rewarded with Ph.Ds.
From the lens of software engineering, archetypes are our base classes or low-level modules. They encode some of the most basic functionality about our system, logic that is invariably linked to the domain at hand. Changing these layers results in profound effects through the system, not all of which can be considered good. These are the explicit and implicit assumptions about how the system does or should behave in the real world.
Understanding the archetypes within a field is essential to develop expertise. By knowing the archetypes we can more effectively critique a work and by employing them in our own art, we’re building on well-defined concepts with broad understanding. Artists employing archetypes don’t have to reinvent the wheel and can instead focus on the important details to push the art form further. Some of the most influential artists choose to push the archetypes themselves, work akin to the principal software engineer or architect operating at a level that raises the abilities of the entire engineering organization.
Modeling Archetypes and their shadows
I will use a triangle to model the following archetypes, and each will follow this structure:
Every archetype has a name which denotes the corresponding behavior or energy through which a SWE may choose to work. When healthy, channeling the archetype helps the SWE to produce work that maximally satisfies the goal of the archetype. However, all archetypes contain shadow versions of themselves. This occurs when either too much (+) or too little (-) of the underlying energy is channeled. SWEs out of balance with a particular archetype will produce work that is less than their true potential.
It is very, very difficult to directly compare the archetypal balance between any two engineers because every SWE is doing exactly the best work they’re capable of at any given time with the tools at their disposal. The more interesting approach is to think about a specific engineer in the present and how they can improve their craft in the future. The power of these 4 archetypes rests in their ability to show us how we can improve and regain balance in our work.
How do we make use of these archetypes? Each and every effort we make as SWEs towards our projects, products, and applications channels some weighted average of the 4 archetypes. As an example, when we close our email, turn off Slack notifications, and go heads down in deep work to solve a particular issue, we are nearly fully channeling the Solver archetype to 100%. However, the other archetypes are still present within us, though highly muted. One helpful analogy is a mixing board with 4 channels. At any given time, every archetype is tuned to some volume, which gates the amount of that archetypes energy flowing into the work. When all the channels are turned down, we are not writing software or contributing any energy to our systems. When everything is turned to the max, we’re burning hot with energy and likely working at an unsustainable pace, though we may accomplish great things in a short amount of time. This is risky behavior but it may be productive in extremely short bursts. Leadership that tries to get this performance and behavior out of their teams for more than a few days on end will toast their people to a crisp. I have been there, mostly on the receiving end, but from the management side at times as well.
Another helpful analogy is the board of a company. The four archetypes sit on the board, ever present in our internal conversation about what we’re trying to accomplish. When healthy, each voice is heard, but their advice is not necessarily followed. We are the chairperson of our board and we get final say in how a particular line of work is executed.
Given the model of the archetype and our understanding of how we can harness their power at any given time, it is time to introduce the 4 archetypes of software engineering: Solver, Planner, Scribe, and Prophet.
Solver
The Solver is the quintessential energy of all engineering disciplines, and the default archetype we think about when we consider the craft. At our core, engineers solve problems. Sometimes the problem is that we’ve clearly defined how a software module should behave, and the solution lies in correctly identifying and implementing the necessary interfaces. At other times, we are triaging a bug that only exists in production and cannot be replicated locally. In this case, we are solving for the difference between our mental model of how things should behave and how they are actually behaving in reality. Above all, when solving in balance, we produce the exact solution necessary to the business problem at hand, with nothing more and nothing less.
In the + shadow, known as Rube Goldberg, we over-solve problems. This takes the form of over-engineering code to the point of unnecessary complexity. Such efforts reduce the balance of the software system and those working on it because the author now becomes one of the few people who can help to maintain the overly complicated code. Furthermore, Rube Goldberg will manifest emotions such as anger or resentment when others do not consider such complications necessary. This creates strife amongst the team, further pushing everyone involved out of balance. The Rube Goldberg may go so far as to split the team between “believers” and “non-believers”. This line of thinking and needs to addressed by management as early as possible. Split-brain is a dangerous state for distributed systems and teams alike.
In the - shadow (EZ Bake Oven Chef), we underengineer solutions. My favorite term of art for this is “yoloing it to production”. Here, we do not consider enough of the edge cases inherent in the domain and produce work that is susceptible to bugs, instability, and lacks features. Not every line of code is created equal and some are more important than others, but the EZ Bake Oven Chef considers them all of equal value, which is to say, not much. They are focused on delivering an issue, except their work is devoid of value due to the inadequate effort. This particular shadow is ever present in startups, especially when product is focused on delivering features before the competition. Bringing a solution to market before others typically doesn’t matter in the long run, especially if such a rushed product cannot sustain actual customer usage. The true difficulty we face as balanced Solvers is being able to communicate to product stakeholders that we slowed down to completely solve a particular issue so that it doesn’t resurface in the future as a bug or feature request.
Planner
The healthy Planner removes uncertainty, ambiguity, complexity, and chaos by understanding how to efficiently accomplish the work ahead. This work takes the shape of many forms, but the main effort this archetype contributes is to break down a project specification into a series of achievable steps, each providing value on their own. The healthy planner understands the ratio between a task’s value and effort. The healthy Planner will necessarily grok the aspects of the project that are of highest value and will seek to prioritize them quickly and reduce the work to be done to deliver the bare core of the effort at hand. The Planner knows that correctly segmenting the work and planning the route between where the product is and where it needs to be involves maintaining deep and accurate mental models of the systems at play. By leveraging these models, the healthy Planner always finds the shortest path to achieving the project goals with the minimal amount of effort (but not any less).
The + shadow of the Planner, the Suit, attempts to break down the work too much and will invariably spread the inherent complexity of the domain across too many subsystems. The Suit will always honor the Software Development Lifecycle (SDLC) of the organization, even to the detriment of the project. There is such a thing as too much planning, and doing so minimizes the art of software engineering. To plan too much is to not honor what we’re going to learn along the way in developing a solution and strict adherence to a plan can lead us to solutions falling into a local minima that are not the best global solution. There are simply some things we cannot know before starting work and the more flexible we can keep our plans, the better place we’ll be at the end.
There is also too little planning, this persona is the Jackson Pollock. These throw-paint-at-the-wall efforts feel good at the moment, because we think we’re making progress and getting paint on the canvas. But with too little planning, we’re just blindly cutting a path through the domain, and we can’t be sure that we’re going in the right direction. In practice, less-than-minimal planning leads to unwieldy Pull Requests (PRs) because the author never knew when to stop and take account of their surroundings. Large PRs inevitably don’t get the review they deserve because it’s far easier to critique a 20-line PR than it is a 500-line behemoth. We thus add noise to the system through unverified assumptions, uncaught edge cases, and progress-for-the-sake-of-progress. Many SDLC processes exists to prevent under-planning. Use them to your benefit, but be careful in assuming that over- or under-planning will prevent the wrong solution.
Scribe
Communicating the state of the work to any audience falls squarely on the shoulders of the Scribe. Scribes understand the importance of translating our understanding of the work from technical ideas into words accessible to designers, product managers, and business stakeholders. This work builds alignment among non-technical colleagues and gives them confidence that SWEs are building what everyone thinks they are. Scribes raise the flag when dilemmas inevitably occur and then push for conversations that determine the best path forward. This is especially important work when software engineers cannot make the best choice on their own.
Scribes communicate through different mediums, always tailoring their message to the audience at hand. Often times this means writing things down into documentation, as they understand the written word scales their work across space and time. Scribes especially seek contributions from different perspectives, without which their shared knowledge can become one dimensional. Scribes also understand the importance of synchronous communication in meetings. Good meetings align disparate viewpoints and accomplish work in an hour that would take a week of asynchronous follow-ups. Scribes mediate such meetings and share the key takeaways to those who couldn’t be present.
The + shadow (the Typesetter, because. everything. just. has. to. be. perfect.) of the Scribe archetype leaves the team with too much documentation, not simplified to its essence. While such deep translations appear worthwhile, they are inherently brittle and cannot keep up with progress unless a large amount of energy is expended. After a while in this loop, the typesetter leading documentation efforts resents the other team members that don’t care as much about maintaining said documents. The issue isn’t the other team members, it’s that we’re trying to document things that move too quickly and don’t need to be written down (e.g., code). We must understand that some ideas change quickly and some more slowly. These slowly-changing ideas tend to represent the distilled version of the domain and they won’t change day to day. These are the ideas that new team members need to understand quickly and efficiently. This must be recorded for posterity.
The - shadow of the Scribe, the Orator, does not understand the value of writing down ideas for others and will instead use that time and energy to push the work ahead, thereby exacerbating the problem. They are happy to talk about their ideas but resist writing them down, possibly because they think something will be lost in translation, which is true but also necessary. Such behavior leads to low bus factors, tribal knowledge, and systems with a single owner. This person becomes discouraged that nobody else can help them to triage bugs or contribute new features to their fiefdom. In reality, everyone wants to help but nobody understands where to begin and thus are more hesitant. By not writing down the why, the Orator seals their fate that they’re the sole owner of a subsystem. It is not a good place to live. If you find yourself in this place, the next best thing to do (short of pausing to work on documentation) is to work a bug or feature while pairing with a teammate. Talk about the nouns and verbs of the system, what things work together, and what pieces are not connected. The more experts can convey their mental models to others, the more the team can evolve to a greater understanding. This work has an exponential return on investment because it raises everyone’s ability to contribute.
Prophet
Finally, the Prophet understands how things are going to change in the future. Prophets predict the sociotechnical failure modes in the current system and work to share this knowledge with product managers and stakeholders who control future work. Prophets are intimately aware of the tradeoffs made during building the current system and can delineate between load-bearing technical debt and minor debt that likely won’t have long-term consequences. Prophets seek to understand the dimensions along which a system is susceptible to poor behavior. For example, a prophet will understand that 10x traffic to a system isn’t inherently bad, but that the background processing layer is too chatty and will be overwhelmed with the increase in work. Given this understanding, prophets inform necessary teammates about the danger of not addressing these issues, especially when the system is currently behaving as expected.
The + shadow of the Prophet, the Zealot, turns their predictions negative, downplaying the work it took to get the system where it currently is and over-hyping how much a complete redesign or rebuild will fix all the problems. Often times, the Zealot will reach for a new technology that the team may or may not have experience with, assuming it will bring order to the chaos because they’ve read about how it helped larger organizations. The Zealot is neglecting the context in which the larger organization operates and downplays the added work in supporting and operating a new and/or more complex tech stack. Rebuilds are almost never worth it as they ignore the blood and sweat poured into the current version. The healthy Prophet find ways to intentionally refactor the current system into a better state.
The - shadow prophet, Cassandra, thinks that nothing will change in the current landscape, purposefully ignores changes they see, or is otherwise unable to communicate the need for refactors and maintenance. They might be happy with the status quo, possibly because it makes them less combative to product stakeholders that are only interested in pushing new features. This desire for fake harmony is disingenuous and doesn’t honor the ever-changing landscape of technology, the domain, and reality. Competition, customers, and technology are always changing and a team with a Cassandra will not be prepared to evolve their system in the future. Such a system will become difficult to fix and/or add features because the necessary refactors were ignored or unseen. Time can easily turn what was once a feature into a bug and not accounting for that leaves the team and product in a worse state to compete in the market.
In summary
Many of the shadow behaviors lead to similar negative states. Our software is difficult to grow and maintain, with single owners of a subsystem, and stakeholders are unclear of what we’re accomplishing. This leads to emotional toil, burnout, and negative consequences for all involved. People leave, further reducing bus factors. These teams are out of balance with the 4 archetypes and over a long enough time period, everything falls apart and the software is unable to compete in the market.
But just as easily as we fall into these negative energies, we can pop ourselves out and alter our direction. Change might not happen overnight, but our work is a marathon, not a sprint. Even minor positive behavior adds up over time, making it easier for others to find balance and deliver their best work. Positive feedback loops are created and fostered, allowing the team to accomplish things previously not possible.
My hope is that by putting a name to these archetypes, we can better understand the nature of our work and build cohesive teams that are able to sustainably ship valuable software while having lives outside of work. Everyone benefits when we’re free to do great work.