Dr. Neil's Notes
General > Projects
Underestimating the Size of the Work
Introduction
A common problem in many projects is underestimating the amount of work required to reach the desired results. There are many reasons for this that often combine to compound the problem.
The High Level View
Sometimes known as the coastline effect, this issue relates to seeing a high level view of the problem and missing the individual details of each issue. Consider flying at 80,000 feet over an island, you get the rough outline of the island, it looks fairly simple, you can draw it and it looks about right. As you descend you see more detail. Once you stand on the shoreline and realize you have to consider every grain of sand to 100% accurately represent the outline of the island, the level of complexity has increased to a point where completion feels impossible. Building products, especially innovative products, can be a lot like this. As you get closer to each part of the problem, the size of the task becomes clearer.
Demo to Delivery Distance
Often a pitch for a new product idea will be accompanied by a demo. The demonstration of how the product will look, feel, and work, helps to sell the idea, and raise funding, to enable the product to be built. Numerous times I have seen a great demo that has been built in a few weeks, or much less, sometimes as little as a weekend. The demo provides the high level overview, of course it is not fully functional, it is a demo. The demo provides the gist of the product idea. However to get from the demo to a delivered product can be mistakenly seen as a case of 'just polishing the demo, and fixing the things that do not work'. The word that starts that phrase is one of my 'red flag' words, 'just'. Whenever anyone says in a meeting 'you just need to...' I am triggered into high-alert as nothing worth doing is ever 'just do...', if it was then it would have already been done by many smarter people, which leads to the next reason. Before we get to that though, also consider the distance from demo to delivery to be serious effort and require more effort than you imagine. This is related to the High Level View challenge.
We are Smarter
It is a mistake to assume that for some reason the team building the product is smarter than everyone else on the planet, or that you have somehow assembled the smartest people in the world into your team. The chances against this are way higher than the chances you have somehow achieved this goal of hiring the worlds smartest people into a team. The mistake is assuming that no one else in the world has managed to 'just do' the work because they are not as smart as your team. It is better to assume you are not the first to have thought of an idea, or attempted to achieve that idea, and look for others that have tried, and attempt to learn from them. Apple did not invent the smartphone, or the portable MP3 player, Microsoft did not invent the operating system, or software to do word processing, Amazon did not invent e-commerce, or cloud computing. Each of these companies stood on the shoulders of the people that had delivered solutions in the past, and then worked really hard to improve and innovate the next generation solutions. Being smart will help solve problems, it will not reduce the total amount of work that needs to be done to deliver the product. As a team scales up, in order to deliver a product, the average intelligence in the team will get closer to the average intelligence of the human population. This is basic mathematics, even if you start with a team of one person, and that person happens to be the worlds most intelligent person (whatever that means), the next person hired into the team will bring the average intelligence closer to the human population average, even if (by some miracle) this second person, is the second most intelligent person on the planet. As your team grows, and you have 50, 100, 150 people in the team you will by definition be getting an average intelligence in the team, closer to the average in the larger population. So do not rely on being 'smarter than everyone else'.
Ability Divergence
Having ranted on about not being smarter than everyone else, it is important to recognize that within every team is going to exist a variation of abilities, some team members might be able to churn out code faster than anyone else, some might be better at solving deep problems, others might be more focussed on the customer experience. Be wary of making estimates of delivery time based on a multiplier of any individual or small subgroup effort. For example 'these two guys built this feature in a week' so if we have 150 (similar sized) more features to build then 'twenty people can finish them in fifteen weeks'. This assumes that every feature is the same amount of work, which is incorrect and discussed later, and also that the ability of the twenty people is exactly the same as the 'two guys'. Everyone is different and will go at a different pace, making different considerations. In software this is noticeable in the same way as in painting, drawing, or playing a musical instrument. When you encounter a sketch artist that is great, they make it look effortless to draw people, buildings, scenes, in fact almost everything they look at, they seem able to draw amazingly accurate sketches. This is a skill, learned and honed. Musicians have this ability too, I have known individuals who are so well attuned (pun intended) to how music is made that you can hand them a new instrument, they have never played before, and within thirty minutes being making it produce beautiful sounds. The musician has experience with a number of other instruments, and understands (implicitly) how to make beautiful music. This can be observed repeatedly in the creative world. However in the commercial world it seems to be often overlooked. Product creation, both hardware and software, is a talent that can be improved the more times you experience it. That is not to say that if you gave ten people the same experiences, they would have the same level of skills. There are other contributing factors, not least of which is passion. When I encounter people who have a passion (for any topic ) they will be researching, reading, and immersing themselves in that topic, if it is their 'job' or not. When an intersection of the passion and the work occurs then you get outstanding performers. This cannot be everyone, and as noted not everyone is going to have the same experiences. When estimating how long it will take to deliver a product, assume average ability and do not compare people in the team with each other, it is not going to be productive, or get you closer to delivery. Differences in the abilities of your team members can be vast, and if you play to each persons strength you are far more likely to succeed in the delivery of your new product.
Progress Is Lumpy
Be highly wary of non-linear progress when building a new product. At times it may feel like a big hurdle has been overcome, and the journey from this point will be straightforward. Then the next obstacle will appear, and it may feel like the product will never be finished. This is normal. As mentioned in the previous section, on the demo to delivery distance, building a demo that looks good can provide an illusion that there is not much more work to deliver the product. When predicting the delivery date for a product be conscious of this 'lumpiness', and take into account the speed of progress will not be constant. This is why techniques such as measure the 'velocity' of a team over time is important. A single data point, does not provide enough information to make any form of accurate prediction. Also be aware that the last 20% of most product development (feels like it) takes 80% of the time. To get a product delivery so that it is working correctly, and going to improve some aspect of a customer journey, requires a great deal of fine tuning. That fine tuning often cannot happen until closer to the delivery. Until you have the core of the product created, you do not have anything to tune. That final tuning will often be the difference between and 'ok' product and a 'great' product. Allow time for the tuning. Edge cases and niches are everywhere in a new product, or even in new versions of an existing product.
Solutions
In order to overcome this underestimation challenges here are some ideas for you, gained from my experiences building and delivering hardware and software products.
Start by laying strong, solid, foundations. As you build a new product, or spin out a new version of an existing product, ensure the foundations upon which you are going to build are solid. This requires creating a solid set of known hardware components to rely upon, or a an end-to-end slice through the software architecture. Validate that the core ideas will work, and that the product can be built. This foundation will also enable you to scale up the team more aggressively. As you hire more people, the foundations provide the guard rails, and guidance for building new components of the product.
Scaling up the teams for delivery is often considered hard, smart people working on a product often do not want to train up other people, they want to get the work done themselves, that is what they enjoy. However most products will never get delivered if the team cannot scale. As more people enter the team, train them to be able to support the next group of people entering the team. If you are reliant on an in-house recruitment team, you might find it hard to get enough people fast enough, so consider creating an elasticity in your team with contractors. Contractors may cost more to the business, however they also should bring more experience to the team, and enable faster scaling by training new employees.
Beware the word 'just'. Whenever someone says 'you just need to do...' then you should be ready for something hard coming up. It is rare anything is 'just do this', just see my points earlier on this topic.
Be alert for the 'this is how it is done' people, who would like to drag old techniques and technologies into new projects. These people are nearly always well intentioned, and they have experienced how things are done in previous projects. Sometimes they are correct. Sometimes they are wrong. The challenge is to work out what is baggage, that is holding back progress, and what is actually going to support the development of the new product going forward. When building a new product be prepared to experiment and try new ideas as well as throw out old ideas.
Keep quality high through the whole product development process. Watch out for the 'we will hack this in for now' approach. While it may provide the appearance of more progress in the short term, it is creating a debt that needs to be paid back before you can deliver the product. These 'hacks' will weaken the product foundations, and start to impact the ability to scale up the team size. This can occur through the entire lifecycle of all products, and can even reduce the ability for a large (eg 500 person) team to grow to a larger (eg 1000 person) team. Along with the 'hack' approach to quality, be wary of the 'lip service to quality' problem. This is where the 'process' (or ceremony) of the product development has the appearance of putting quality first, however in reality it is not producing a high quality product. The measure of quality output is the number of defects being found, along with the robustness of the foundations being built upon. If the outcome is not high quality then the process is 'lip service' and not real. As the product grows and the team scales then poor quality will lead to more time spent fixing issues, than developing the product. I have seen teams that spend their entire time fixing issues and have no capacity left to innovate or finish a product. This poor quality will impede the ability of a team to deliver in a reasonable time.
Created: September 18, 2022 01:34:26