In Greek Mythology, Procrustes, a bandit, killed his victims by force–fitting them into a standard iron bed. If the victims were shorter than the bed, he would forcefully stretch them; if taller, he would cut off their limbs as per the size of the bed.

This torture technique can be compared to how a predefined standard scale is used to forcefully “size up” work in an environment of high variability.

Sprint-based agile methodology of software development is suffering from the Procrustes phenomenon. This concept’s boundary conditions of applicability are highly limited in the real world. To understand why and how, one needs to go back to the history of software project management processes.

The Dubious History

The history of software project delivery performance is dubious. Probabilistically speaking, as per the annual Chaos Report, software projects are most likely to fail in terms of cost, quality and timelines. Interestingly, the trend has remained the same through the years.

There is now widespread consensus that the waterfall methodology of software development is a major reason for the poor track record.

Problem with Waterfall Methodology: Process Over Speed?

The waterfall methodology meets two main objectives in software development projects

Ensure clear accountability in the relationship between the supplier (one who is developing the software) and the customer (one who will use it)
Ensure the process is highly stable, and that the level of rework is minimal

For the first need, freezing requirements before starting the project is an absolute must. Once requirements are frozen, the accountability for any effort or time overrun can be clearly traced back to either of the stakeholders.

The second need requires one to approach software development like one would approach the development of a new physical product. Developing a new physical product requires the design to be complete before the start of the actual building. Allowing the design to remain fluid during implementation leads to uncontrolled rework and interruptions.

Borrowing from these principles, a waterfall methodology requires one to move the project into the following phases sequentially with clear “gates”, marking the completion of each phase

Requirements
Design
Development
Testing

Strict change management processes are to be put in place as a “deterrent” against any rethink late in the process. As a result, one of the overheads of this methodology is a lot of documentation and sign-offs.

However, when the customer-testing phase comes at the fag end of the entire project, “hindsight” clarity issues create severe conflicts between customer and developer on what was “told” initially in the requirement phase and what was “understood”. So, practically speaking, a strict change management process becomes difficult to implement, more so when the relationship between customer and supplier is that of unequals. The resultant effect is scope rework detected too late in the process, leading to uncontrolled delays. It is no surprise that contractual conflicts, delays, effort spiking and associated stress towards the end of the project are chronic to software development projects.

Agile Development: Speed over Process?

In sharp contrast, as the proponents claim, Agile Methodology takes a viewpoint that scope and design cannot be frozen upfront. Hence, it adopts a flexible working approach, which can embrace change much later in the development cycle. Instead of taking a “one perfect project delivery” viewpoint of software development, Agile believes in the principles of delivering usable features, good enough for customers to start using, and then keep improving based on frequent feedback. The idea is to deliver workable software as soon as possible and get feedback for further iterative development. This way of working also takes away the need to freeze requirements and design upfront.

So, in the Agile world, one can move away from sequential working between phases of software development. The team can work as a self-organizing group, which can design, develop, test and deliver workable features in an iterative manner. Time is managed by using the concept of sprints (a widely used version of Agile called Scrum) – a “time box” of 15/30 days, is used to deliver small batches of usable software features. While the ongoing sprint is frozen, subsequent ones can be changed based on customer feedback.

The “Not So” Silver Bullet

The silver bullet has been found. Or so, it seems. However, no conclusive, transparent and non-debatable study is available to establish the supremacy of the Scrum Methodology.

Beyond the data, what is of concern about Scrum specifically are the growing voices of practitioners against the methodology. Numerous blogs have been highlighting the increasing stress of developers, the ever-lengthening list of bugs’ backlog and customer dissatisfaction with the usability of software. Surprisingly these effects are exactly opposite of what Agile had promised!

As a counterargument, proponents of the methodology have pointed out that implementation gaps, not conceptual gaps, are the primary reason for the woes.

The argument of implementation gaps puts the onus of the problem of poor results on the abilities of the implementer rather than the inventor. (“They couldn’t implement – they were not disciplined!”). This narrative allows some management theories to stay as fads, longer than they really deserve to. So it is important to verbalize the hidden assumptions (or the boundary conditions) of a theory and check its validity in the practical world.

Flawed Assumptions

A widely cherished principle of agile is “self-organizing teams over strict processes”. The signoffs/handovers/checks or any kind of formal control for sequential movement as per a process is scoffed at, as self-organizing and iterative development becomes the guiding mantra.

The resultant effect of any agile intervention is the re-organization of a functional structure of design, development and testing into many small self-organizing teams, which are expected to be self-sufficient in terms of skills of design, development and testing.

This assumption of self-sufficiency of small, divided teams in terms of adequacy of high-end skills, particularly designing and requirement assessment is highly questionable in the vast majority of environments. High attrition rates, with many developers changing jobs every two to three years are hard realities of the industry, which is obsessed with outsourcing and reducing the cost of development. The high attrition rate does not impact the availability of generic skill resources like expertise in a programming language. But environment specific skills like design and requirement analysis which are only acquired with adequate experience, are never available in full strength as required by the multiple decentralized agile teams. These skills (also called SMEs or subject matter experts) are always part of a miniscule minority in most environments.

So it is highly likely, with skills decentralization after an agile intervention, that many agile teams will fall short of skills, particularly those contributed by experienced SMEs. Even though, each team may have the required numbers to be declared “self-sufficient” as per records.

This leads to a situation where some agile teams depend informally on others, outside of their team, for help. With decentralization, some teams also become ignorant that they need help outside of their structure! Without a formal process to control sequential movement, the first casualty is the adequacy of design and requirement assessment. Hence, testing becomes the primary source of feedback on design and development gaps.

Consequently, frequent re-work becomes inevitable. The lines of code for the same functionality are written many times over in an agile environment. At the same time, continuous “flow back” from testing causes frequent interruptions with developers. The problem of interruptions and rework further aggravates when different small agile groups are working independently on different sets of features in parallel; design decisions, at times, become “local” which causes conflicts between teams and creates further rework.

The few SMEs or designers in some self-organizing teams become overloaded. These scarce resources are forced to multi-task beyond their capacity; they have to not just work within their team but also get into issue management of other teams. This further deteriorates the productivity of the system as everyone waits for feedback from heavily loaded resources. The end result is an uncontrolled development process.

Agile has an antidote for this inherent chaotic working approach – sprint deadlines. Like Procrustes’s standard iron bed, the sprint cycle deadline force-fits open work fronts into closures. Effort spiking close to the sprint end pushes everyone to somehow drive closures and complete the sprint. But the efforts spiking compels compromises – some testing is skipped or some bugs are kept aside, and some scope is set aside for later sprints – the same way Procrustes force-fitted human bodies into beds. As almost all resources in the team become extremely busy towards a sprint closure, any planning activity for subsequent sprints is put off until the start of the next sprint.

Consequently, in every sprint, initial time is used up in planning work. Actual development and testing happen towards the second half of a sprint cycle, thus building pressure on time, followed by skips and misses. Testing resources stay less utilized during the beginning of a sprint and then get overloaded at the end, where all planned features land up together. Sprint ends rush and compromises become inevitable! The damaging effect of “forcing” a standard sprint cycle is not just seen in execution but also in planning. The arbitrary “cut” of deliverables based on a standard time, at times, leads sprint managers to size out work, which may not be usable at the end of the sprint.

When execution is chaotic, and features coming out from a sprint cycle are not “usable”, the predictability for the end customer is lost, particularly for those who might want a firm release schedule for a set of usable features to plan their own projects.

So, in many environments, it is not uncommon to find a hidden waterfall methodology-based project management imposed upon agile working. The resultant effect of this dual environment is a situation, which imbibes the worst of both solutions in terms of productivity losses.

The Hybrid Solution

Typically, an overall project plan is created for “final release” for customer commitment. The longer-term plan is broken into “sprints” with every sprint having its own predefined scope plan. The commitment to the end customer is the final release. Sprints turn into typical milestones of an overall project schedule.

When many series of consequent sprints are pre-defined with scope, flexibility to re-scope or repeat a sprint based on feedback from previous sprints becomes nearly impossible. So after every sprint, the pressure is to work on the scope of the next sprint rather than to deal with feedback/issues of the previous one.

This creates an ever-increasing bug list and leads to massive effort spiking towards the end of the “final release”. Some companies leave aside a time box of “final sprint” before the final release to clear all the old mess of previous sprints. So in the real sense, customers don’t get anything usable till the final release.

At times, even the “buffer” of one sprint is not enough to make up for all the past sins. Less available time, pressure to deliver and an overload of pending work is a perfect recipe for more compromises. Bad-quality products are likely to be delivered to customers who report the bugs back to the development team. Pressure to expand the team to tackle the huge list of bugs of previous releases while developing new releases increases the cost of development. Stress, poor productivity, delays and bad quality cannot be a rare phenomenon in such an environment.

At the same time, when an overall release project is super-imposed on agile cycles, the typical initial waste of time and development capacity spent in assessing the entire project efforts, scope and plan, negotiations becomes a reality. Software development companies lose out from all sides.

Agile Vs. Waterfall: Solution of Extremes?

One of the assumptions of the waterfall method, which was questioned by Agile, is the ability of customers to clearly specify requirements upfront. Many times, a requirement becomes clear only when customers actually see or start using something. At the same time, the need for perfection or the “best” can make the requirements phase go out of control. Hence, Agile did away with the overhead of trying to be perfect in design and requirements gathering.

But at the same time, Agile implementations also suffer from the bane of inadequate designs. What we have now is a more generic conflict underlying the software development process.

On one hand, we should freeze designs and requirements before implementation because this approach ensures rework is under control and hence, time is under control.
On the other hand, we should not freeze design and requirements because it is impossible to freeze them. Users get clarity on what they actually want after using or visually observing the product.

The only way out of the above conflict is to understand that there are clearly two types of rework which gets generated in any new product development process.

Type A (Foresight-led rework): They are misses and skips made under the pressure of time and lack of synchronization between team members. Some people in the team know about these inputs. Due to the pressure of time, these inputs are not incorporated in time; this leads to rework later in the process. These errors cause frustration among team members for missing the obvious. Using testing resources to get feedback on Type A errors is a criminal waste of time and capacity.

Type B (Hindsight-led rework): This is a rework created due to new knowledge generated after an event of testing or clear visualization. It is in a way value adding, because new knowledge is generated in the process. Prototypes generate Type 2 value-adding rework. Type B rework is inevitable and no amount of thought experiment can help one visualize the Type B errors upfront. The only way out is to devise a process to generate Type B errors early in the development cycle.

The Waterfall Method implicitly assumes that all errors are Type A and hence should be prevented. This school of thought has inspired the manufacturing slogan “First Time Right”. Process rigidity with checks and “no-go” gates becomes the key to avoiding Type A lead errors. Agile assumes all errors are Type B, hence it considers one-time handovers between phases as a waste of time. Hence agile propagates frequent back and forth between all phases.

Learning from the History of Automobile Manufacturing

Interestingly, this “Agile” way is exactly how automobiles used to be manufactured in the late 18th century. It was called the craft manufacturing method. A small group of mixed-skill resource groups were each given a car to be assembled from start till end. The cross-functional team had skills in design, fitting and even machine operations. The not-so-perfect parts could be filed and adjusted to fit with each other, and eventually, a vehicle would come out with arduous rework. This way of car manufacturing was indeed very expensive and only the rich could afford it in that era.

When Henry Ford presented the assembly line concept with a clear division of labour, car assembly was transformed. Clear sequential working with different resources specializing in different types of the assembly task, defined the assembly lines of Ford Motors. The cars moved between stations (instead of the workers moving) for a continuous flow of output. The specialization on one type of task, along with avoidance of switching losses (of craft manufacturing) between different types of skill work enabled a productivity jump of close to five times.

The revolutionary idea dramatically reduced the cost of making a car. The masses could afford a car because of the enhanced productivity. In a way, the car industry gave up the “agile methodology” of car making and achieved giant strides in productivity.

However, an assembly line working approach requires two critical conditions to be successful.

Flowbacks are negligible. Work moves forward in one direction. If the nature of work is such that it frequently comes back to previous workstations for rework, then it can create chaos with uncontrolled line stoppages and reduced output. (In such an environment, craft manufacturing, or the agile method, is a better way). In the case of Ford, assembly line working was enabled by standardization of parts, which could fit any car, without the need to file and adjust ill-fitting parts for a specific car, thus preventing the flowbacks.
Transfer batches should be small (ideally a single-piece flow). Transferring big batches not only expands lead time in every stage but also leads to late detection of quality issues.

Rapid Feature Flow Model

Let us examine how we can get above two critical conditions to resolve the inherent conflict of the software development process.

1. Minimizing Flow Backs

Organizing resource groups into requirements assessment team, design team, and development and testing group helps create the required division of labour for assembly line working. This allows the limited expert resources to be centralized in the requirements and the design groups. This grouping also helps in the proper utilization of their capacity in the type of work, that requires their expertise. However, without a clear definition of what constitutes the end of requirement and end of design, the assembly line type sequential working between resource groups will never materialize due to flow backs. It is also important to put in place a process of checking to mark completions – this is imperative to break the bad habit of proceeding without completing the phases.

Requirements Phase

This phase should focus not only on defining what should be done in a feature but also on defining what will not be given. It is also important to begin with the end in mind. The inputs of all test cases to be used in the final validation should be defined. At times, visual prototypes can also help customers visualize what they really want.

Design Phase

A high-level dependency understanding at an architectural level is usually done in many environments by using the first few sprints, (called the foundational sprints). Dependencies that are neglected are in the subsequent functional sprints where features are delivered.

The design stage should focus on unearthing detailed dependencies in terms of interaction with different functionalities. Identifying verification test cases upfront in this stage makes sure that the gap related to any kind of dependencies that can lead to regression is minimized. In the absence of proper design, a lot of capacity wastage can happen later in verification testing as fixing one issue might lead to the breaking of other things, creating a vicious loop leading to unpredictability in flow.

The gates of design and requirements along with the principle of “beginning with end in mind” should help in preventing type A errors and, unearthing type B errors early in the process.

(Interestingly, engineering of physical products focuses a lot on understanding dependencies as the cost of rework during the product building phase is high. For example, if the design of beams needs to accommodate the load-bearing capacity of a moving crane in a factory-building project, it is important such design dependencies are thrashed in the engineering phase. If the same dependencies are detected in execution for physical products, the huge physical waste of material becomes visible. However, in the case of software projects, wastages for bad design are never visible; a code going wrong can be re-written in the virtual world after it is detected in testing. No rejection of material or dismantling of build structure happens in the software development world. This inherent advantage of software development is also a bane. When testing is used for facilitating good design, the rework and loss of productivity is colossal, even if there is no loss of physical material).

2. Minimizing the Transfer Batch

Just having an assembly line with delivery coming out after a period of six months or a year is also undesirable. When different customers require different features at different points in time, having one delivery date for a large set of features is of no use to anyone. It is unreasonable to expect the customer to wait till all other features are complete. So, moving individual features through the assembly line in order of priority (sequence in which features are introduced) ensures better flow and a short lead time in delivering workable features that individual customers want.

The way to make a transfer batch of one feature is to institute a WIP (work in process) limit control in the different resource group types. The work in process control implies that a new feature will be introduced to the resource group only when one under progress is complete as per the criteria set for handovers. The rule of “one in when one is out” ensures a single-piece transfer batch flow at a feature level. (Due care should be taken to define features in a way that they are fairly independent entities).

The WIP control rule implies that tasks cannot be prescheduled or cut out using the “knife” of a time-boxed sprint. The rule of “one in when one is out” will ensure the output of features in a staggered manner and that the load on testing is levelled out. (Instead of a wave of lumpy work for testing in a sprint-based scheduling.) The staggered flow prevents temporary bottlenecks in the system.

The WIP control also aids transparency of queues, which in turn helps in better capacity planning between resource groups. Skill group sizes can be adjusted from time to time to manage any temporary elongation of the queue. (Change in complexity of features introduced in the system or loss of resources in some groups can create temporarily lengthen queues).

The above principles will help in the rapid flow of features from this system. The rapid delivery of individual deployable features also helps promote “a-version-at-a-time” thinking approach in the requirements phase to keep it under control. Customers (or marketers of the software) can keep improving the features by initiating work for the next versions.

The suggested model of working, however, has the following drawbacks:

The rule of WIP control makes the scheduling of tasks redundant. In the absence of a time schedule, predictability is lost.
It is in direct conflict with the typical sales planning process of a product company that works on annual or bi-annual software release plans.

Improving Predictability

The above system exposes queues in the system and helps differentiate touch time (actual work time) from waiting queues. If the features’ content can be standardized as multiples of a unit work package, then it is possible to model the expected arrival time to provide the desired predictability. When flow improves, with a drastic reduction in interruptions, priority changes and rework, the system predictability goes up many folds.

But it is important to use the scale of time to provide continuous predictability and not use it as a tool to drive “commitment” in execution or use it as a scheduling decision. It is a fallacy that schedules, and commitment around schedules, prevent time wastage. In reality, wastage of time is prevented by the daily process of issue resolution and close handholding by flow supervisors in respective groups.

The decoupling of prediction and commitment can be best understood with the analogy of driving using Google Maps. Google Maps offers expected time of arrival looking at queues on the road. But it is only a predictive tool, and the driver cannot commit to the time claimed by Google. What he can do effectively is only reduce wastage in every moment of driving. In the case of an overall manager of a system, the focus of predictability improvement comes from reducing queues in the system, and managing issues interrupting flow on a daily basis.

In this model, a feature is introduced in the system without trying to cut it out based on some arbitrary sprint cycle definition. The feature size introduced in the system is based on what is deployable.

Dynamic Software Release Planning

The new model mandates that one move away from managing software development as a large monolith project with a detailed plan aiming for a single release. Final releases can be done as frequently as one can manage based on the release overheads. Single features or a set of features can be bundled into releases as and when one wants. Instead of committing to a release plan upfront for a year, marketers of the software can decide on a release strategy based on the dynamics of the market. This provides the necessary flexibility to react to competition strategy, and hence change the queue of features in the waiting list to deal with market dynamics.

To summarize, software environments have had a chronic difficulty of not being able to clearly differentiate two independent measuring scales required to manage projects effectively

Scale of time (to provide predictability of when something will be completed)
Scale of work to understand the size

Both are different; they should not be mixed. Since defining a standard scale for work content has always been a challenge, using the time (the sprint cycle duration) as the proxy scale to “knife out” work in planning and execution is the source of all problems in the scrum methodology. Hence, the Procrustes phenomenon becomes unavoidable. The suggested Rapid Feature Flow model offers a way to differentiate the usage of the two scales to escape the wrath of Procrustes.

So what happened to Procrustes? His reign of terror ended with his death at the hands of Theseus, the king of Athens!

Case Study

Schneider Electric Engineering Workbench, a product from Schneider Electric’s Process Automation business helps in automating the engineering and configuration of the instrumentation and control systems of brown/greenfield projects. Faster engineering helps in reducing lead time to start of operations and thereby improves profitability and reduces the risks of penalties for the Automation vendor. The Agile methodology formerly in place for software development at Schneider Electric Engineering Workbench could not give the desired results. Therefore, the company innovated a flow model based on the core concepts of TOC for the automation software development process which not only created a more harmonious work environment for the development team but also increased productivity by a whopping 71%!

At the end of a year, Schneider reported the following additional benefits:

Let's Empower Growth