Defect Containment: Find Once, Fix Everywhere - Krugle

White Paper

Defect Containment:
Find Once, Fix Everywhere

Defect Containment: Find Once, Fix Everywhere

Summary

The typical enterprise development organization today has millions of lines of software code. As
enterprises scramble to meet marketplace challenges, the code bases they maintain become
exponentially more complex every day, resulting in significant code cost and quality problems. Due to
poor maintenance practices, code defects often abound and propagate, and these result in critical
defects shipped to customers [1][2][3]. The irony of the situation is that some of these defects were
already fixed in other parts of the code base!

Just one of these critical defects may violate an SLA or bring down a critical customer system. One
telecom enterprise recently shipped a product to customers with a critical bug. The shipped defect had
an immediate bottom line impact: The organization incurred a total cost of $30M in emergency
customer code fixes and penalty fees, and violation of guaranteed SLAs with their customers. The
shipped defect also had an immediate internal impact: The head of the company’s development
organization was immediately removed for shipping the critical defect to customers.

In today’s increasingly complex development environment, the question that every enterprise
development executive must ask is: “Will the next bug that my organization ships be the one that
costs me my job?”

This paper examines the source of propagated defects and explains how modern automated defect
containment technology is being used to remedy these issues, and how an automated defect
containment solution can be used as a “safety net” for every development organization.

A look at the problem

Today’s software development methodologies and tools do a good job of making discrete and “project‐
specific” software releases more manageable; however, they do nothing to address cross‐project /
cross‐division code development and maintenance challenges. Nor do today’s software development
methodologies and tools prevent defect propagation throughout the code base.

Defects are introduced into a customer‐shipped product due to post‐release code changes. Because
developers have poor visibility into the body of code changes that are “in development” and
“successfully completed” outside of their own “project‐specific” software release, defect propagation
becomes endemic and entrenched in an enterprise’s code base.

The core problem stems from common development practices and situations:

• Developers frequently must work with code they didn’t write and don’t understand

• Complex component and distributed architectures make learning unfamiliar code extremely
difficult

Page 2

Defect Containment: Find Once, Fix Everywhere
• The inability to access and effectively share relevant insight (code, developer contact, pertinent
• documentation is ubiquitous
• Developers lack the time or tools to assess the impact of code change and determine change

requirements across the organization
• Mergers and acquisitions, multisite development teams, laptops, and management autonomy

result in little or no ability to detect or prevent the same problem from being “fixed”
independently and differently across projects
• The envelope of time pressure around the preceding issues results in incomplete fixes, “hacks”,
and code entropy.

How big of a problem is defect propagation?

In companies with 100+ developers and/or an active codebase of 500K+ lines of code, software
maintenance and bug fixing will account for more than half of the overall development budget. The
problem is even greater in larger organizations, where maintenance and bug fixing account for 60%‐80%
of the overall development budget. The result is that code maintenance and bug fixing increasingly
chew up a greater percentage of development spending, stealing budget from projects that are needed
to drive future financial returns.
Below are industry figures that illustrate the enormity of the defect propagation problem:

• Late stage defects cost 10x ‐ 100x what early stage defects cost
• A typical bug not sent to a customer is worth $20K ‐ $1M+
• 75% of testing and defect detection is manual and not coordinated across teams
• Defect injection rates in the code maintenance phase of development range from 10% to 35%
• Current methodologies focus solely on defect find & fix management within silo’ed projects,

rather than throughout a given enterprise’s code base
The defect propagation problem is growing, because of:

• Code interdependencies are increasing
• The rate of code change is increasing
• Poor code maintenance spawns additional defect propagation issues.

Page 3

Defect Containment: Find Once, Fix Everywhere
Defect propagation is a significant, growing problem that has not been contained by process and “best
practices”. Software development executives then are left “driving in the dark,” unable to predict when
and where defects might crop up, how to contain those defects, and how to prevent their development
organizations from shipping product to customers with those defects.

The Evolving Landscape of Software Development

Past Present

Release cycles Long Short
Requirements Detailed Basic
Interdependencies Many
Stack change rate Few Daily/Weekly
Planning Quarterly/Yearly Ad‐hoc
Points of coordination Many
% Developer time on someone Full
else’s code Few >75%
Maintenance as % of total effort
Ability to control lifecycle quality <25% More than 50%
Ability to control lifecycle costs ??
Ability to plan and manage dev Less than 50% ??
spending High
High ??

High

Table 1. Key Cost and Quality Drivers in Software Development

In the past, code was developed around a relatively stable and predictable framework, using trusted,
reliable and tractable code components. As a result, most development was applied to major releases
that could be carefully managed against a modest number of predictable dependencies. Tools and
methodologies (e.g., waterfall, extreme) ensured the efficient creation of these monolithic applications
through finely detailed requirements, development, and test planning.

The current development landscape is very different (see Table 1).

Page 4

Defect Containment: Find Once, Fix Everywhere

Code today is more complex, more dynamic and built from component modules for – at least in theory –
greater development leverage and improved quality. Many independent teams are involved in the
varied component modules that create a working stack of software. Development cycles for these
components are short. The overall change rate within an application code stack is very high. This, along
with the frequent need to provide quick, “backdoor” fixes to component source is a significant source of
problems late in the development process. As a result, the frequency of “post release” development
issues (defects, security vulnerabilities, performance issues, compatibility issues) has increased.

This accelerating cycle of change forces development behaviors out of the carefully managed and
predictable envelope of ALM best practices (detailed analysis, requirements definition, test planning,
etc.) and into the realm of “no time to do it right; we’ll do the best we can now and patch it later after
it’s live.”

Defect propagation is an even bigger issue within the realm of software maintenance. Less experienced
programmers and outsourced development teams are typically assigned to execute evolutionary
development, platform porting and bug fixing & QA. These less experienced staffers often lack deep
code knowledge and have limited visibility into relevant coding examples, dependences or
documentation. Much of their work is left to their discretion (uncontrolled or minimally controlled
processes) and they are forced to work under tight deadlines. The vast storehouse of reusable
components – and lessons learned ‐ goes mostly unused.

As a result, maintenance code changes are frequently the source of avoidable defects that result in
multiple “one‐off” code changes that:

(i) are costly
(ii) accelerate code entropy
(iii) are frequently the source of additional defects.

Ultimately, the development organization must face angry customers, violated SLAs, lost revenue, and
even penalty fees. Development executives are then forced to throw their development organizations
into “fire fighting” mode, dedicating all resources to fixing the critical bug for customers. However, even
then, development executives are rarely confident that the resulting fire fight has addressed all
propagated defects.

As one development executive put it: “I know that for every mission‐critical bug the customer brings to
our attention, and that we fix, there are probably 10 more hiding somewhere in the code base. The
problem is our code changes so often that I have no way of knowing where those bugs are at any given
time or even how to go about trying to find them.”

Page 5

Defect Containment: Find Once, Fix Everywhere

The 4 Critical Issues in Defect Propagation

Issue 1 Inability to discover existing fixes

• In many cases, the developer working on a defect isn’t the original code author
• Developers fixing bugs often may not have well developed problem solving approaches for the

defects they are fixing and they are frequently unaware of relevant fixes performed by others
• Once they’ve fixed a bug, developers are unlikely (too much work to spend time documenting

the work I’ve done) and unable (can’t find the people that really need to know about this) to
communicate their work to others who can benefit

As a result, maintenance code changes that could take advantage of existing code changes do not
leverage existing work. A separate fix is implemented. The overall code base diverges and the volume of
code that an organization must maintain increases.

Issue 2 Inability to learn from existing problem solving insight and solutions

• Even if a highly matched defect solution does not already exist in an organization, there is a
good chance that relevant problem solving insight exists within the development group or
organization

• Often, this knowledge is available in working code from “other projects” that rely on similar
development tooling, frameworks or integration components

• When access to the development work of other teams is limited, developers have little or no
visibility into the relevant development information – and a valuable knowledge transfer
opportunity is lost

• Commonly available search/discovery tools such as grep and grok are effective on “local” code,
but are expensive and difficult to synchronize when working with code outside of the immediate
SCM system

Propagated defects create redundant work for the organization. Poor information access prevents
teams from sharing information that unifies and coordinates similar work effort around propagated
defects. As a result, multiple fix streams are created. The overall code base diverges and the volume of
code that must be maintained increases.

Issue 3 Inability to find all occurrences of known defects

• Attempts to contain cross project duplication through process, development methodology or
tooling cannot eliminate the occurrence of duplicated defects; backdoor fixes, reliance on
external code frameworks and cut and paste coding increase the likelihood of cross‐project code
and defect duplication.

• Current discovery capabilities work only at the desktop or project (source code management
system) level; discovery capabilities of existing tools are limited and the amount of manual
effort to execute a comprehensive, cross project search is prohibitive.

Page 6

Defect Containment: Find Once, Fix Everywhere

• Defects are often created by patterns of code that contain subtle variation; token based search
or scanning technologies commonly used in IDEs and SCM systems are subject to inaccuracies
that create unacceptable levels of false positives and false negatives.

Current development tools and methodologies do not support cross project discovery of code and
defect patterns. This results in an unnecessary duplication of effort, high code maintenance costs and
higher than necessary latent defect densities in code.

Issue 4 Inability to measure and manage propagated defects

• Point‐in‐time defect duplication across SCM systems cannot be determined with current SCM
systems, even if the systems are from the same vendor.

• Semi‐automatic measurement of defect duplication requires that all active code projects be
periodically (usually daily or weekly) checked out of all SCM systems and analyzed for
duplication. Until now, this involves manual, error prone and time consuming steps using
externally managed metadata (project ownership, defect definitions, reporting configurations,
etc).

Current development tools and methodologies do not support automatic analysis or automatic
detection of cross project code defect patterns. With no way to measure the problem, the best that
development managers can do is to enact policies that if followed, hope to minimize propagated
defects. In reality, propagated defects persist and are a significant hidden problem for many
organizations.

Key Solution Requirements

Existing methodologies and tools rely on

1. Process and ad hoc project‐level orientation
2. Project‐level SCM tooling
3. Localized notification systems

In all 3 instances, the methodologies and tools provide for only project‐level and localized fixes. Within
any given discrete project, existing methodologies and tools can provide teams the ability to identify
defects in the local code, find local interdependencies, notify team members of those local defects, and
fix or merge local defects. However, due to the increasing complexity of growing code bases throughout
the enterprise, current solutions are not designed to address defects that have propagated beyond the
local project’s code base.

The common thread among these 4 Critical issues is a fundamental inability to access relevant
information quickly and accurately. A viable solution to these key defect propagation related problems
must:

Page 7

Defect Containment: Find Once, Fix Everywhere

• Make it fast and easy for developers to access, explore and deeply understand highly
interdependent code that is managed across many different systems and projects

• Rapidly pinpoint where similar code is defined and used
• Accurately determine defect injection points
• Precisely predict defect propagation across multiple versions and projects
• Quickly determine which projects will be impacted by code component, service (SOA) or third

party code modification
• Continuously discover violations of coding “best practices” before code changes are propagated
• Quickly assess organization‐wide exposure to known security and performance vulnerabilities
• Identify candidates for re‐factoring or SOA implementation
• Detect and monitor unmerged code changes in components or shared code
• Monitor code submits for known coding defects and compliance with key coding practices
• Help developers more quickly and deeply understand the code they are working on, but which

they did not author

These requirements address remediation of the top maintenance issues as summarized in Table 2:

Problem 1: Problem 2: Problem 3: Problem 4:
Fix Knowledge Defect Manage‐
Location
Discovery Transfer ment

Explore code 33
Locate similar code
Identify defect injection points 3 33
Find similar defects
Company‐wide Impact analysis 33
Global vulnerability assessment
Refactoring, SOA opportunities 333
Monitor code divergence
Code related info access 33

333

33

3 33

33

Table 2. Defect Containment Solution Requirements

Page 8

Defect Containment: Find Once, Fix Everywhere

Defect Containment with Krugle Enterprise

The root problem with defect containment is poor information access. Information access problems are
a classic area for solutions that are driven by search technology.

Several aspects of defect containment problems make code search uniquely useful in solving problems.
Development information is stored in proprietary Source Code Management (SCM) and database
systems that are related by a variety of proprietary metadata linking architectures. Being able to
continuously and reliably access the code and then generate a searchable library is essential.
Maintaining the project level code structure and preserving the information references to the code is
also very important.

Another issue is the ability to support fragment related searches. Code searches and traditional keyword
searches are fundamentally different. Being able to use syntactic and code structural characteristics of
the code is an important way to pinpoint code defects. It is important to detect both “exact” and “fuzzy”
matches which can both produce the offending error.

Lastly, the solution must generate results that are clearly and easily actionable. If a series of projects can
be found that contain a known defect, how will project owners become aware of and take action against
these errors – without ambiguity? The solution must document the problem and solution completely,
with as much automation as possible; project owners must be informed by appropriate means in a
timely fashion – and be given the authority to act on the propagated defect as appropriate.

All of this should be done in a way reliably automated and does not require existing development
tooling or methodology to be replaced or abandoned.

Figure 1. Krugle Enterprise Defect Containment Solution

Page 9

Defect Containment: Find Once, Fix Everywhere

One solution that addresses these capabilities is Krugle Enterprise. Three technologies are part of this
solution:

1. Comprehensive Code Inventory (CCI). Using its own patented technology, Krugle Enterprise
automatically catalogs and indexes all of the relevant code contained within an organization’s firewall.
This creates a single, comprehensive up‐to‐date inventory of source code and related information for an
organization. Information and links from resources related to the code are also indexed and fully
searchable.

Krugle Enterprise supports all the common SCM systems (including Clearcase, Team Foundation,
Perforce, CVS, SVN, StarTeam, etc.).

2. Krugle Enterprise delivers Code Optimized Search and Analysis – technology that helps pinpoint
code defect patterns. Krugle’s underlying index uses semantic information extracted from the structure
of each code file. It also offers a query capability that helps match similar multi‐line code patterns in
addition to simple keywords. This enables accurate identification and location of code fragments and
patterns.

3. Integration. Krugle uses a RESTFUL interface to integrate its query, result summary and detailed
results – with SCM, ALM and development portal applications. This ensures automated, predictable
measurement and monitoring of propagated defect.

Summary

Information access problems are responsible for major cost and quality problems in software
maintenance. These problems consume a significant portion of the budget in most software
development organizations, and can result in violated SLAs, penalty fees, lost revenues, and in some
cases, can result in the development executive’s job.

These problems can be addressed through defect containment technologies that are optimized for
source code and related development information. A viable technology must operate robustly across a
variety of SCM systems and development environments, without creating operational risk or requiring
changes to existing software methodologies. A practical solution must also employ code optimized
query capabilities that leverage non‐code information as well as the source code.

The defect containment technology then becomes a must have “safety net” for every development
organization. The defect containment system will make sure that shipped software never makes it out
the door with a bug you already fixed!

P a g e 10

Defect Containment: Find Once, Fix Everywhere

References

[1]Annual software maintenance cost in USA has been estimated to be more than $70 billion (Sutherland, 1995; Edelstein, 1993). SOURCE:
Software Maintenance Costs, Jussi Koskinen, Information Technology Research Institute, ELTIS-project. University of Jyväskylä, P.O. Box
35, 40014-Jyväskylä, Finland. Available at: <URL: http://www.cs.jyu.fi/~koskinen/smcosts.htm> , since 12th Sept. 2003. Email:
[email protected])

[2] An average Fortune 100 company maintains 35 million lines of code (Müller et al., 1994)…Older languages are not dead. E.g. 70% or
more of the still active business applications are written in COBOL (Giga Information Group)… There are at least 200 billion lines of COBOL-
code still existing in mainframe computers alone (Gartner Group)…Lientz, B.P. & Swanson, E. (1980). “Software Maintenance Management:
A Study of the Maintenance of Computer Application Software in 487 Data Processing Organizations”. Addison-Wesley: Reading, MA, 214 p.

[3] Studies of software maintainers have shown that approximately 50% of their time is spent in the process of understanding the code that
they are to maintain (Fjeldstad & Hamlen, 1983; Standish, 1984). SOURCE: Software Maintenance Costs, Jussi Koskinen,

P a g e 11