After a Failure, Revert to the Last Known Working State

Both long and short-running projects will, at some point, face a failure. Sometimes it is possible to plan around them or include them as part of a risk management and mitigation program. Sometimes, however, failures appear unexpectedly, bringing an entire project to a halt and perhaps challenging its continuity.

To give some context, I can use my personal experience as an example. At Dispertech , we are building a device to characterize nanoparticles. It is based on technology developed in a lab, and we must transform a rough prototype into a usable product. Our biggest failure arose when I could not detect the same particles detected in the lab years before.

If I could not succeed at detecting these specific particles, the project was likely going to end. Therefore, rationalizing such a failure had a very high emotional toll. Moreover, reverting the many failed attempts required some detailed understanding of the factors that brought me to this particular conjuncture.

The Multiparemetric Optimization Problem

One of the biggest challenges in many projects is to define what we are trying to achieve. In our case, the general idea of "user-friendliness" has many different approaches. Software to control the experiment must be accessible. The external housing has to look nice and include informative LEDs to show the status of the device. The optics must be designed to minimize user intervention.

We rely on disposable cartridges to hold a hollow optical fiber, the core component of our device that would allow us unprecedented accuracy. These cartridges have many different aspects to consider, from the cost of production to environmental friendliness. But we must also include the alignment precision (we are talking in the range of 1/100th of a millimeter) and the type of glue used.

Overall, around 20 different parameters can be changed, and each one will impact the outcome. Optimizing for an abstract idea such as user-friendliness already generates an uncertain goal. Using a different laser simplifies the electronics, but the user will force have to spend ten more minutes waiting for the device to be ready. How can you weigh the consequences of such a decision?

Moreover, when changing any of the parameters generates an impact on the others, we end up with an actual multiparametric optimization problem. We need to explore a multi-dimensional decision space to achieve what it looks like, at least, a local maximum. But these processes are time and money-consuming. Therefore we probably will try, consciously or unconsciously, to find shortcuts in the process.

The real challenge with multiparametric optimization problems is that the result we obtain does depend on the path we follow. And this is not related to the problem itself but our psyche. If we introduce a system change to make it better, we can fail, months later, to acknowledge that it is holding us back. We are set to go always up from where we are standing, even though perhaps we could reach higher if we first take some steps down.

Early progress is not a persistent indicator

When we start with a new project, there'll likely be a lot of progress at the beginning. Perhaps it is fueled by motivation; maybe it is enabled by the number of things that must be done and have a relatively low difficulty. Early progress generates a fantastic feeling but making longer-term assumptions based on it can become risky.

Doing a linear extrapolation of what we expect to achieve in the coming six months based on how much we progressed in the first six months can give extremely biased outputs. The development stage is different, and we must adapt expectations also considering other factors. However, it is almost unavoidable to let our minds meander on those longer-term irrealities. Marathon runners know this better than anyone: the pace at which you run the first half-marathon can't be extrapolated to calculate the time it'll take to finish the race. Early progress is not a persistent indicator for future progress.

Once the output falls behind the projected progress, several feelings will emerge. Probably the most predominant is frustration generated by the apparent stagnation. These feelings can be self-motivated if we set expectations for ourselves based on what we have done up to a point. But it may also happen that others project their expectations on our company or us. When progress fuels motivation, instead of the other way around, lack of progress takes it away and gives space to the opposite feeling.

Stop and ask why

When building something no one else has made before, there is no reference to follow. We may relate to other products, but we are essentially on our own. Even after the initial phase, progress never stops completely. There are always some wins, either big or small, and plenty of learning opportunities. However, as deadlines pass by and the product is no closer to completion than before, it becomes a constant emotional struggle.

When we reach this point, we must realize there is a timeline that brought us here. We must be aware that there were plenty of smaller steps to arrive at where we are. It is the moment to stop, and ask why . It is a challenging exercise, but the more often we do it consciously, the better understanding we'll have about the process.

When someone new starts to work on a project, they will likely ask why we are doing things in the way we do. Even random questions such as why is the fridge so far away from the kitchen . Perhaps there was no kitchen when we bought the refrigerator, and when we built it, no one thought about moving it. If we are receptive to what they have to say, fresh pairs of eyes can be truly enlightening.

Asking why means reflecting on the end goal of each step we took in the past. For example, I was struggling with building disposable cartridges. They are one of the aspects of making the device user-friendly . The entire ethos of what we were building was designed around those cartridges. Because of the cartridges, many parts of the device became too convoluted, hard to maintain and use.

The obvious question to ask ourselves was: why are we still using the cartridges? They were the source of most of our problems. We built for so long on the idea that cartridges were a must-have that it took us a very long time to step back and decide that we needed to tackle the problem differently.

When we work in projects with many different dimensions, not all orthogonal to each other, the paths we follow may prevent us from reaching the desired outcome. Now that we know where each decision leads, we can go back and check if they were the correct ones. By asking ourselves "why" retrospectively, we can understand at which point a decision pushed us down a road that would eventually lead to failure.

Revert to the last known working state

Decisions have a compounding effect, making the moment at which our path started to depart from our original plan almost imperceptible. Borrowing from version control systems, we can try to revert to the last known working state . It is not an easy task, and it requires a lot of thorough introspection. But if we do the exercise, we may find a specific moment in time in which we followed a particular path that eventually leads to the state we wanted to avoid altogether.

Bisecting a timeline can be a helpful approach but not always practicable. We look at the last time we know we were on track, and the first time we realized we were off track. We check at an intermediate point and decide if it belongs to the first or the latter cases. Then, we split the sub-interval again and continue until we find the moment in which the course of action was wrong. This approach can work great for software development but is much tougher to follow in projects with many moving targets.

Therefore, another idea to identify the crucial moment is to go backward and ask why as many times as needed. When we find a moment where the reasoning is flawed or poorly supported, it is likely a moment of departure from the path we should have followed. However, this alone is not enough. Poorly supported decisions do not mean they were the wrong decisions. Since we are analyzing retrospectively, we can use all the information we have now to support or invalidate the choices we made.

Even if we find a moment when we took a failed approach, we must be sure it is the first time it happened. A decision could have been taken based on wrong assumptions, and we could try to correct it. However, there's no guarantee that decisions taken earlier were not problematic. There is no problem in going back to the project's genesis to understand what happened. It just takes time.

After a period of introspection, we would gain confidence about the moment in which things derailed. Ideally, we would go back in time to the moment we identified and make a different choice. However, it is impossible to revert ourselves, our history. The new decision will be based on what we already know, on what we learned. Our resources are not going to be the same now than in the past. The temporal and budgetary constraints are not going to be the same as the context in which we were.

If we are lucky, we can revert some of the choices without invalidating the rest of the project. In our case, for example, we decided to revert the decision of working with cartridges. It didn't mean we had to change the rest of the design, nor the electronics. It just required some geometric adjustments to the optical path. After some tweaking, things started working better than ever .

Dealing with failure is more complicated when alone

If there's something that I've learned, when you feel frustrated because of a failure, the best you can do is to lean on others. The worst you can do is to keep everything to yourself. From people who can tell you: you looked happier one year ago, to business partners who have a deeper understanding of the processes. Even people who do not have a complete picture of what you do but can challenge your decisions are of great help.

Every bit of input we can gather about the path we followed is of great value to reconstruct our history. What a lot of people miss to realize is that in many decisions, there are emotions involved. Acknowledging a failure, deciding to revert to a previous state will have a toll. If more people share these feelings and the project's responsibilities, it'll be easier for each individual. A failure can become a learning opportunity only if we are equipped to seize it.