Creating software is tough. Let me list some of the problems in our industry…

• Projects run over-budget
• Projects run over time
• Software uses calculations or memory inefficiently
• Software is of a low quality
• Software fails to meet the requirements it was developed to meet
• Projects become unmanageable and code is difficult to maintain
• Software often never finishes development

Would you agree these are common? 🤔 That’s what scientists concluded back in 1968, at a NATO Software Engineering Conference in Germany. The entire situation was concluded as “Software Crisis”!

That was the first bigger event when the “Software Engineering” term was used, originally coined by Margaret Hamilton famous for her involvement in the Apollo missions.

The report of the conference is still available. It has 136 pages and some of them try to explain the nature of “Software Engineering”, together with pieces of advice, best practices and approaches to solve the problems listed above. The idea was to introduce a more mature approach to a profession which was brand new, so it could grow and become more effective like the rest of the older engineering branches. It’s an interesting read - some ideas are still valid.

56 years have passed and we still have the same problems. Of course, the scale and complexity of our projects are different than it was back then, so I wouldn’t say it was a failure - but certainly, there is still a lot of work to be done.

How software engineering is viewed today?

I wouldn’t say there is one, unified perception of Software Engineering like in other branches.

For some people, creating software has nothing to do with engineering. The most common reasons relate to the fact that it’s not physical and doesn’t create repeatable, tangible objects on a mass scale. Often software development will be compared more to art or… craft. This is why Manifesto for Software Craftsmanship was born and signed by over 23k people in less than 24 hours since its release. Till today, a lot of software developers see themselves as craftsmen, meaning most often they take pride in the quality of the work they produce, and they see that programming fits craft as an idea. We produce unique systems, tailored to the user’s needs and we cannot easily automate the entire process (yet? 😅) or just throw more bodies at it to solve it. Additionally, mentoring is very important in our industry, which matches nicely the idea of apprentice and masters.

For others, connotation with engineering is negative. Usually relates to bureaucracy and inefficient processes, often learned in university and not used in practice. Reasons are different: they are either outdated or simply not useful in the real-life context. DHH said in one of the interviews, that he wouldn’t call himself an engineer, sees himself more as a “software writer”. Glenn Vandenburg made a great speech about this problem.

For a majority (in my eyes, at least in Poland) the “Software Engineer” term is neutral, just a synonym for a “Software Developer” role. If you see a job offer with any of these titles, you’d probably assume this is the same role.

Finally, some people see it in positive light. For example in Canada 🇨🇦, it is illegal to call yourself a Software Engineer, unless you have an engineer university title! The reasoning is that it carries high professional and ethical standards - the public places a high degree of trust, and accountability in such individuals. As such, software engineers are elite profession, superior to programmers.

Misconception around production and design engineering

We tend to compare software engineering with typical engineering often through the lenses of mass production of physical products.

In typical engineering, once a product is designed, there is a lot of effort in creating it and distributing it to the end users. Often this is the biggest problem and if we cannot do it cost-effectively, the product won’t succeed on the market.

As Dave Farley suggests, in the software world our closest reassemble would be deployment and distribution of written software. Most of the time this means making a copy, which takes fractions of seconds, or making it accessible on the web. Compared to the cost of producing the software in the first place, cost of building is marginal effort.

However, if you look at how innovative, physical products are being built for the first time, like:

• New types of spacecraft
• A first smartphone and similar devices
• A personalized, new type of intelligent arm prothesis

They belong to engineering disciplines, but before they are going to be mass-produced and optimized (if ever) majority of effort happens on the design part.

To succeed in such a project we will have to excel in learning quickly and adapting incoming products, assuring it fits the purpose. Often this will lead to prototypes, user tests, modelling, surveying and so on. At this point most probably you’ll notice already that this IS familiar with our job. It’s called design engineering.

If you are not convinced that this proves that software engineering is real engineering, I can recommend to you another talk from Hillel Wayne who interviewed engineers having experience working in both worlds - “real” engineering and software engineering.

Dave’s definition of Software Engineering and craft

The definition from the book goes like this:

”Software engineering is the application of an empirical, scientific approach to finding efficient, economic solutions to practical problems in software.”

Every word has a meaning here. To fully understand what this definition adds on top of a common understanding of our job, I’ll go through a couple of examples and assume that we are already journeymen or master-level craftsmen. In my eyes that means the following:

• We’ve completed plenty of projects
• We have fantastic programming skills
• We have an awareness of best practices
• We have mentorship experience
• We have a growth mindset to keep on improving

Woah - you’re a powerhouse! To call yourself a software engineer, you’ll have to channel that power on a specific approach - it’s possible to be at the same time a great craftsman, but bad engineer. Let’s see how.

Empirical approach

It was said before that we have to excel in learning. The best approach for learning that humanity has discovered so far is scientific method. The Wikipedia picture placed below tells the story perfectly.

It’s nothing new in the industry - we have been speaking for years about the importance of success criteria, data-driven decisions or focusing on the problem instead of solutions. However, as senior craftsmen, we are especially vulnerable to falling into… mind traps! 🤯

The first of the traps is called by Dave as “I know this error”. When compared to inexperienced apprentices, we’ve seen a lot of errors. This gives us the benefit of making swift, snap decisions - it makes us fast - a blessing! However, that may also lead to a trap - false assumptions - a curse.

Our brain places a lot of biological optimisations, coming from our old days when instinct and quick decision-making would often save our lives. We are not optimized for long, energy-burning analytical thinking which is often needed in our craft. That often requires taking a step back. It breaks the flow and seems inefficient.

This is why, especially as a senior you have to watch out even more because the baggage of past experiences will often suggest to you that past solutions apply in the current context as well, while it doesn’t have to be the truth. 80% of the time it will help you, while 20% may elude you, make you sound arrogant or waste extra time to roll things back. Watch out!

Additionally, we can sometimes adjust reality to match our opinion. Of course, not consciously. We’d like to think that we are selecting the best tool for a given job, but this is not always the case. Assuming that you have a lot of experience under your belt and a growth mindset, it will also mean that you want to experience the new - to learn. This is why so often we are victims of hype-driven engineering.

Ideally, we should gather data to prove that the usage of new technology makes sense in the current context, but because we have a strong bias to test the new tool, we won’t objectively gather the data. That’s because the new tool may be the option we are secretly counting on! 🤞 Maybe if we would assume “I’m probably wrong… what data could prove it?” mindset, it could help us be more objective.

One of the great examples of being more empirical is simply running more experiments. Spikes, PoC, prototypes, and proofs within data. Dave also suggests that if we are following the TDD approach, then our spec can also be treated as experiments. Empirical approach also means working not only incrementally, but also iteratively.

Incremental approach means slicing problem into smaller problems and building it step by step. This is how you can deliver the project in optimal time, especially if you map how each step combines into the larger picture and find ways to make it more parallel. The problem is that it works like this only if you know upfront exactly what you want to build - we often sort of know this for small epics, but we don’t know it for bigger projects or problems.

Iterative focuses on building the entire thing in a simple way to gain feedback so you can discover what the end state should be… because you don’t know upfront. However, building things that way introduces cost - because it has to work end-to-end. The next iteration may remove something you’ve built just for the sake of making it executable end-to-end or just to verify your assumption. In the eyes of many developers, this may look like a waste in delivery. But it allows us to learn, and this often is a lower cost than building based on assumptions. We are too costly professionals to live those things up to the chance.

The approach suggested by Dave is to work both incrementally and iteratively at the same time. I understand this as slicing a big project into iterations and working on each iteration incrementally.

Finally, to learn quickly we have to have quick feedback loops on multiple levels. We’ve covered the importance of this topic in DevEx article. In short, the faster (ideally automatic) feedback loops you have in place, the lower the cost in time spent and people sanity you’ll have to pay to fix issues in code. Important - good configuration of IDE, local tests, type checking, pair programming or ChatGPT assistant may be the fastest loop you can get.

Pragmatic approach

The quote below, famous on Reddit, made me think strongly about our craft and engineering approach:

“Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands.”

Your initial attitude to this quote may be negative - in the end, who wants to build something that “barely stands”? It implies bad quality, sloppy work, and a lack of craftsman responsibility and skills. Seems unsafe too!

But, on the other hand… do think that civil engineers would risk people’s lives? Certainly not - they are signing with their name under the plan. If the bridge collapses, they will have difficulty in getting the next job… and it’s not only if the bridge can sustain people going through it, there are different weather conditions, critical loads in case of heavy transportation and so on.

If you think about it, building something of good quality without budget constraints is easy. You have to have a skilled engineer to know how to design something in a way where, despite limitations (time, budget, physics and everything else) they will build a product which does its job with a quality level appropriate for its function.

In our world, this is especially interesting. The dangerous bridge with holes and hanging pieces - alert us immediately. If we see a bridge which was 10x bigger and more costly than it was needed, some of us would notice and ask questions, maybe we could consider it an art/monument in some cases if that was the purpose. But code? Nobody sees if it’s under or over-engineered besides software developers.

I won’t cover the situation when tech debt and under-engineering kill us. This was covered multiple times and happens all the time. Additionally, we assume that you are a skilful craftsman and you won’t let such a situation emerge because your internal quality standard won’t let it happen to some extent.

It can cause you to apply high standards too often - even when a project or feature doesn’t require it yet at this phase. As a good craftsman, you’re keeping the project consistent with the rest of the codebase, quality is high, and apprentices can learn and grow. Apprentices may even expect you to keep those standards, otherwise they will be looking for a new master. Additionally, if product vision is blurry, and requirements unclear, then software developers may protect against difficult changes in the future “out of safety”, knowing that they will be blamed as not professional if such change would take a lot of time to do… knowing that we are working iteratively and everything is prone to change, it seems smart to invest a bit of time to make future changes as easy as possible. This is good advice unless that investment doesn’t pay off in the long term. I also think that if this is mainly out of our fear (how we are being seen), this is not the most pragmatic incentive.

Unfortunately, as Marty Cagan says - “The first such truth is that at least half of our ideas are just not going to work”. He said that in the context of product development, but it’s equally true in the case of tech initiatives, future-proofing and other assumptions we make on how product will evolve in the future. If our ideas are wrong, we invest time into writing software which won’t be used, and if we have more of it (and more complex), it’s more difficult to maintain and keep healthy. Some definitions of over-engineering go as far as to say that “it is every code that solves problems you don’t currently have”.

As good, senior craftsmen we will be especially prone to keep our high standards and professionalism, which often means following best industry practices, but for good engineers keeping this standard high always may be considered ineffective usage of resources, or simply - won’t be pragmatic. Have you ever seen an engineering field which is not pragmatic? It cannot be - one doesn’t exist without another.

There is also less heroic reason. Boredom 😊. If you spent enough time with one company but your growth mindset is not satisfied, you’re prone to “inventing” different, more difficult problems for yourself. Watch out, that’s another challenge for experienced developers…

Summary

I’m twisted into thinking about many aspects of life through the lenses of RPG players. In this case, I’d say that if we are building a character and investing development points, software engineers require craftsmanship. It is necessary to have high skills to produce good quality software. It’s fundamental. However, they cannot on invest all points there.

As a result, it’s perfectly normal to find amazing, world-class craftsmen, masters of their skill, who are not good software engineers, because they lack empirical approach and/or pragmatism. As Dave says in his book, craft is great, but it’s simply not enough.

It’s an interesting topic and it’s also difficult. Exploring the next levels of craftsmanship and what’s possible is engaging, and stimulating. It’s often what gives us the most fun out of our job. It’s interesting that as soon as we start creating our product and we become founders, suddenly it’s less important than the product meeting its purpose.

Good engineering and pragmatism are working out the attitude which places the software purpose in front, despite it’s not always the most fun or growing thing to do.