There has been a concerted recent effort within the software development community to develop and embrace new working models that aim to address weaknesses in old methodologies. The most prominent example of this, perhaps, is the transition towards Agile development practices in response to the inherent inflexibility and lack of parallelism of more traditional Waterfall development models.
However, while the merits of newer, more flexible methodologies are theoretically and intuitively evident, there remains a notable lack of metrics by which to prove these assertions. So, how does one arrive at a projected cost for a software project using objective, reliable metrics? Judging by the fact that 61% of software projects are cancelled or challenged (delivered late, over-budget or missing functionality), it can be reasonably suggested that such a metric does not, in reality, currently exist.1
This article discusses a reasonable approach to measuring development success, using requirements and metrics that can be defined in terms of those requirements, to be able to accurately project the cost of delivering quality software in a timely fashion. In addition, this article will make the case for one particular paradigm – the flowchart – illustrating how it can provide attractive solutions to the challenges of requirements design, cost projection and change management in Agile environments.
Requirements: The Genesis of Software
It is generally understood within the software industry that requirements are the backbone of the software development lifecycle (SDLC). They form the framework from which the rest of the process is predicated (one notable exception is testing, however, as explained later, some schools of thought are focused on correcting this). With this in mind, it stands to reason that, before any development starts, one can assume that a list of requirements will exist (in most cases, it will be the only information that exists at this stage). It is therefore reasonable to ask: since the cost of development needs to be projected as early as possible, can the requirements be of any use in estimating them?
Consider a software project – say an account management system for an online retailer – is at the point at which requirements are being drawn up. To begin with, a high-level design is put together, giving an overview of the major areas that need to be considered. For example:
| Highest Level Requirements |
| Account Management |
| Security |
| Payments |
| Integration With Orders System |
Once all the highest level requirements have been identified, each section in turn is then dissected to find the next level of requirements. Continuing our example:
| Highest Level Requirements | Next Level |
| Account Management | Add Account |
| Delete Account | |
| …. | |
| Security | Password security |
| Payment details security | |
| … |
So far, this is consistent with accepted best practices. However, since we are approaching this from a
cost perspective, how does this help us? Well, since the requirements have been subdivided according to granularity, it provides us with a methodology for calculating the total cost of development:
- For each lowest level requirement, calculate the cost (and/or complexity) of implementing it
- For each level above, its cost is simply the sum of the costs of the next level of requirements
- Repeat step 2 until all highest-level requirement have a calculated cost
- The total cost is simply the sum of the costs of the highest-level requirements
Using the previous example to illustrate, assume we have the following requirements:
| Highest Level Requirements | Next Level | Lowest Level |
| Account Management | Add Account | Customer First Name |
| Customer Last Name | ||
| Customer Address | ||
| … | ||
| Delete Account | Account Number | |
| Security | Password security | AES Encryption |
| SHA-256 password hashing | ||
| … | ||
| Payment details security | RSA Encryption | |
| … |
Calculate and fill out the lowest level costs first (highlighted in red):
| Highest Level Requirements | Next Level | Lowest Level |
| Account Management | Add Account | Customer First Name (2) |
| Customer Last Name (2) | ||
| Customer Address (6) | ||
| … (35) | ||
| Delete Account | Account Number (2) | |
| Security | Password security | AES Encryption (70) |
| SHA-256 password hashing (40) | ||
| … (125) | ||
| Payment details security | RSA Encryption (60) | |
| … (250) |
Once these have been identified and accounted for – for the next level, calculate the sum of the ones underneath it:
| Highest Level Requirements | Next Level | Lowest Level |
| Account Management (47) | Add Account (45) | Customer First Name (2) |
| Customer Last Name (2) | ||
| Customer Address (6) | ||
| … (35) | ||
| Delete Account (2) | Account Number (2) | |
| Security (545) | Password security (235) | AES Encryption (70) |
| SHA-256 password hashing (40) | ||
| … (125) | ||
| Payment details security (310) | RSA Encryption (60) | |
| … (250) |
So, for this example, the total cost is 592 (i.e. 47 + 545). In practice, costs would usually be denoted in units with each unit assigned a fixed cost (e.g. 1 unit = $1000). In this example, security = $545,000.
Bear in mind that, for the purposes of this example, the numbers are arbitrary and used for explanatory purposes only. In real application, it is necessary to calculate the cost of individual requirements (see the discussion on inner complexity which is set out further on in this article).
Above, we have provided a macro view on cost. The remainder of this article will deal with the frequently posed question: how does one calculate the cost of the lowest level requirements? This constitutes the micro view and a different approach.
The (Perceived) Problem With Requirements
Firstly, it is necessary to dispel the notion that requirements are subjective. Of all the engineering fields, this one appears to be totally isolated to software. For example, one would not expect engineers to proceed with the building of a bridge if there was any room for error in the underlying specifications. However, this is the expected norm of the software industry – the author’s experience certainly attests to that fact – and how this came to be the case is an interesting (but separate) issue.
However, the previous comparison with civil engineering does raise the prospect of ‘borrowing’ methodologies from the more mature engineering disciplines.
Requirements-Based Testing (RBT), which emerged in the early 1970s, is one such example. Based on the mathematics at the heart of hardware testing, it is a methodology that, because of the underlying logic, makes it possible to construct a (relatively small) suite of test cases to achieve 100% functional coverage. Those of you who are interested in its applicability to improving testing quality are invited to this reference.2 However, for this discussion, we are more interested in what RBT has to say on ambiguity in requirements as it forms the foundation of the notion that we are dispelling.
To properly build logic models for use in RBT, it is necessary to purge any and all ambiguities from the requirement before it can be tested. There same is true here: in order to be able to accurately calculate the cost of each requirement, it is necessary to ensure that there is one and only one interpretation of the requirement. This particular topic is beyond the scope of this article, but for an in-depth discussion of the ambiguity review process, the reader is invited to visit the following link.3
The reason for removing ambiguities is that minimizing and quantifying cost and complexity depend entirely on the interpretation of the requirement being consistent across every person who implements it. For instance, it is possible for a requirement to be interpreted in multiple ways (by the business analyst, by one or more developers, by testers etc.). Each misinterpretation involves an amount of rework to rectify the mistake, which reduces the accuracy of the cost projections. As interpretation of an ambiguous requirement is entirely subjective, it is not possible to predict the number of potential misinterpretations, only to give upper and lower bounds on this number.
If we consider ‘interpretations’ as a Venn diagram, it looks something like this:
Overlaps between the circles are the requirements that have been interpreted in a common way, while non-overlapping regions are those requirements which have not. The key here is to maximize the overlaps so that each group have interpreted requirements the same way. The only way to achieve this, however, is to ensure that the requirements are unambiguous to begin with.
By using a mathematical construct known as a finite Hausdorff metric, it is possible to measure the ‘distance’ between each set of interpretations. Since measuring ambiguity in a consistent and objective manner is not yet achievable (apart from the processes referenced earlier in this section, which leave short the existence of a single metric to quantify the results of ambiguity reviews), this is beyond the scope of this article. However, the notion of a ‘distance’ between interpretations is sufficient to outline a mathematical argument as to how it affects the margins of error for cost projections (see appendix A). For this section, it is sufficient to consider the rework required to bring the interpretations closer together, and how this must be factored into cost projections. In appendix A, we relate how requirement ambiguities can be factored into the mathematical model as the margin of error on cost projections.
Inner Complexity
Other approaches4 have proposed that the sizing of software is directly measurable from the number of ‘testable requirements’; this is a laudable approach and a marked improvement on the archaic lines of code (LOC) and functional point (FP) measures. However, without wishing to single this particular instance out, it is necessary to point out a few shortcomings to date.
For the most part, previous approaches have ignored the inner (or inherent) complexity of each requirements. In other words, they have treated every testable requirement as having equivalent weighting. This is a rather naïve assumption and can lead to massive errors when calculating the overall cost. In the initial example, each requirement was given an individual weighting to reflect its complexity, and therefore, cost – if each requirement had been given equal weighting this would have not reflected the differences between them. Intuitively, this can be thought of as requiring more coding to implement the ones with higher costs – this will be confirmed by almost any programmer!
Therefore, it is imperative that each requirement be assessed on its own merit. This includes avoiding assumptions such as ‘if requirement X cost £1000, then requirement Y should cost £2000’. This is because, along with the cost, it is also necessary to factor in a ‘margin of error’ for each individual requirement. As you might expect, these ‘margins for error’ are very rarely consistent.
Prediction Power as a Measure of Quality
Bearing in mind the above, a question naturally arises: what can we say about the margins of error in the predicted cost and can this be used as an interpretation of quality?
Of course, a prediction is merely that, and it will always be hindsight that determines whether or not it has borne fruit. Predictions also come with their own margin for error. However, a predicted figure will usually come within a high and low range, with the true value expected to lie somewhere between the two. Climate models are a good example of this, with an ‘expected value’ (i.e. the predicted true value) as well as a range of values that the model also factors in as its margin of error.
This concept can also be applied to our running example. Let’s suppose that we factor in a 10% error margin for each requirement. How does this affect the predicted cost as a whole? The table below shows complexity metrics (red) as well as a 10% margin of error (upper bound is blue, lower is green):
_____________
3 AmbiguityProcess. benderrbt.com. [Online] http://benderrbt.com/Ambiguityprocess.pdf
4 Wilson, Peter B. “Sizing Software With Testable Requirements
About The Author
Huw Price has been the lead technical architect for several US and European software companies over the last 30 years. Huw is Managing Director of Grid-Tools, a leading test data management vendor and Chief Technical Architect at Agile Designer. Huw has been guest speaker at many conferences including Oracle, HP, Star East and the IIBA’s UK Chapter. He was awarded “IT Director of the Year 2010″ by QA Guild and is an adviser to Kings College KCL. Huw also has experience offering strategic advice to large corporations like Deutsche Bank and Capgemini.
