Algorithm_Design_Foundations_Michael_T.

¿4. A Lower Bound on Comparison-Based Sorting 239

4.4 A Lower Bound on. Comparison-Based Sorting

Recapping our discussiolis on sorting to this point, we have déscribedseveral meth-
ods with either a worst-case or expected running time of O(nlogn) on an input se-
quence of size n. These methods include merge-sort and quick-sort, described in
this chapter, as well as hapsort, described in Section2.4.4. A naturalquestion to
ask, then, is whethér it is possible to sort any faster than in O(nlogn) time

In this section, we show that if the computational primitive used by a sorting

algorithm is the comparison of two elements, then this is the best we can do-

comparison-based sorting has an Q(nlogn), worst-case lower bound on its running
time (Recall the notation Q() from Section 1.2.2.) To focus on the main cost of
comparison-based sorting, let us only count the comparisons that a sorting algo-

rithm performs. Since we want to derive a lower bound, this will be sufficient.

Suppose we are given a sequence S = (xo, xi,. . . ,x,_ 1) that we wish to sort, and
let us assume that all the elements of S are distinct (this is not a restriction since we
are deriving a lower bound). Each time a sorting algorithm compares two elements

x and xj (that is, it asks, "is x <xi?"), there are two outcomes: "yes" or "no?

Based on the result of.this comparison, the sorting algorithm may perform some
internal calculations (which we are not counting here) and will eventually perform
anòther comparison between two other elements of S, which again will have two
Outcomes. Therefore, we can represent a comparison-based sorting algorithm with
a decision tree T. Thatis, eäch internal node y in T corresponds to a comparison
and the edges from node v'to its children correspond to the computations resulting

from either a "yes" or "no" answer (see Figure 4.14).

It is important to note that the hypothetical sorting algorithm in question prob-
ably has no explicit knowledge of the tree T We simply use T to represent all the
possible sequences of comparisons that a sorting algorithm might make, starting
from the first comparison (associated with the root) and ending with the last com-
panson (associated with the parent of an external node) just before the algorithm

terminates its execution.

Each possible initial ordering, or permutatién, of the elements in S will cause
our hypothetical sorting algorithm to execute a serie of comparisons, traversing a
path in T from the root to some external node. Let us associate with each external
node y in T, then, the set of permutations of S that cause our sorting algorithm to
end p in y. The. most important observation in our lower-bound argument is that
each external node y in T can represent the sequence of comparisons for at most
one permutation of S The justification for this claim is simple if two different
permutations P1 and P2 of S are associated with the same external node, then there
are at least two objects x and x1, such that x is befOre x in P1 but x1 is after xj
in P2 At the same time, the output associated with y must be a specific reordering
of S, with either x, or x3 appearing before the other But if Pi and P2 both cause the
sòrting algorithm to oùtput the elemeñts of S in this order, then that implies there is

240 Chapter 4. Sorting, Sets, and Sel

a way to trick the algorithm into outputting x and xj in the wrong order. Si
cannot be allowed by a correct sorting algorithm, each external node of T'
associated with exactly one permutation of S. We use this property of the d
tree associated with a sorting algorithm to provç the following result

Theorem 4.12: The running time of any cornkarison-based algorithm for so
an n-element sequence is Q(nlogn) lathe worst case.

Proof: The running time of a comparison-based sorting algorithm must be
than or equal to the height of the decision tree T associated with this algont
described above. (See Figure 4.14.) By the above argument, each external ij'
T must be associated with one permutation of S; Moreover, each permutati,
S must result in a different external node of T. The number of permutations

objects is

n!=n(n-1)(n-2)..2.i.

Thus, T must have at least n! external nodes. By Theorem 2.8, the height of!

least log(n!) This immediately justifies the theorem, because there are at leâs

n/2terms that are greater than or equal to in the product n!; hence

log(n!)1ogç) =n logn,

which is Q(nlogn). x<x7
iJ
minimum height (time)

log (n!)

n!
Figure 4.14: Visualizing the lower bound for comparison-based sorting.

sucket0fl and Radix-Sort 241

Bucket-Sortafld Radix-Sort

In the previous section, we showed that Q(n log n) time is necessary, in the worst
case, to sort an n-element, sequence with a comparison-based sorting algorithm. A
natural question to ask, then, is whether there are other kinds of sorting algorithms

that can be designed to run asymptotically faster than O(nlogn) time Interest-
ingly, such algorithms exist, but they requiìe special assumptions about the input

sequence to be sorted. Even so, such scenarios often arise in practice, so discussing
them is worthwhile. In this section, we consider the problem of sorting a sequence
of items, ach a key-element pair.

4.5.1 Bucket-Sort

Consider a sequefice S of n items 'whose keys aré integers in the range [O, N -. 1],
for some integér N 2, and suppose that S should be sorted according to the keys
of the items. Iñ this case, it is possible to sort S in 0(n +N) time. It might seem
surprising, but this implies, for example, that.if .N is. 0(n), then we can sort S in
0(n) time Of cOurse, the crucial point is that, because of the restrictive assumption

about the format of the elements, we can avoid using comparisons.
The main idea is to use an algorithm called bucket-sort which is not based on

comparisons, but on using keys as indices iñto a bucket array B that has entries
from O to N - 1. An item with key k is placed in the "bucket" B [k], which itself is
a sequence (of items with key k). After inserting each item of the input sequence
S into its bucket, we can put the items back into S in sorted order by enumerating
the contents of the buckets B [O] , B [1],... , B [N - 1] in order. We give a pseudo-code

description of bucket-sort in Algorithm 4.15.

Algorithm bucketSort(S): t

Input: Sequence S of items with integer keys in the range [QN - 1]

Output: Sequence S sorted in nondecreasing order of the keys

let B be an array of N sequences, each of which is initially empty.

forleeacth.iktebmexitnhSdeokeyofx ..
remove x from S and insert it at the end of bucket (sequence) B[k]
foriiOtoNido
.

for each item x in sequence B[i] do

remo Ve x from B[i] and insert it at the end of S

Algorithm 4.15: Bucket-sOrt.

It is easy to see that bucket-sort runs in 0(n+IST) time and uses 0(n+N) space,

just by examining the two for loops.

242 Chaptet 4. Sorting, Sets, and Sel

Thus, bucket-sort is efficient when the range N of values for the keys is>

compared to the sequence size n, say N = 0(n) or N = 0(n log n). Still, its $

mance deteriorates as N grows compared to n.
In' addition, an important property. of the bucket-sOrt algorithm is that it

correctly eveh if there are many different elements with the same key. Inde
described it in a way that anticipates such occurrences.

Stable Sorting

When sorting kéy-element items, an important issue is how equal keys are han

Let S = ((ko,eo),... , (ko_1 ,e_i)) be a sequence. of items We Say that a

algorithm is stable if, for any two items (ki, e) arid. (ki, e) of S, such that kj
and (k,, e,) precedes (k3, e3) in S before sorting (that is, i <j), item (k,, e;)
precedes item (ks, e) after sorting. Stability is important for a sorting algóji
because applications may want to preserve the initial ordering of elements wi

same key.
Our informal description of bucket-sort in Algorithm 4.15 does not gu

stabthty This is not inherent in the bucket-sort method itself, however, for W
easily modify our descnption to make bucket-sort stable, while still preservin
O(n + N) running time Indeed, we can obtain a stable bucket-sort algon
always removing the first element from sequence S and from the sequencés

dunng the execution of the algonthm

a

4.5:2 Radix-Sort

One of the reasons that stable sorting is so important is that it allows the bue

sort approach to be applied to more general contexts than to sort integers. Sup

for example, that we want to sort items with keys that. are pairs (k,!), where k
i are integers in the range [O,N - 1], for some integer N 2. In a context s
as this, it is natural to define an ordering on these items using the lexkograph

(dictionary) convèntion, where (ki , ii) c (k2, 12) if

ki <k2 or
ki = k2 and l <12.

This is a pair-wise version of the lexicographic comparison function, usually

plied to equal-length character strings (and it easily generalizes to tuples of d n

bers for d> 2).

The radix-sort algorithm sorts a sequence of pairs, such as S; by applyin

stable bucket-sort on the sequence twice; first using one component of the p'

the ordering key and then using the second component. But which order is co

Should we first sort on the k's (the first componen.» and then .on the i's (the se3 F

component), or should it be the other way around? F

pucket Sort and Radix-Sort 243

45e

Before we answer this question, we consider the following example.
Example 4.13: Consider the following sequence S:

s= ((3,3),(i,5),(2,5);(1,2),(2,3),(1,7),(3,2),(2,2))

If we stably sort S on the first component, then we get the sequence

=- Sj ((l,g),(l,2),(l7),(25),(2,3),(2,2),(3,3),(3,2))

If we then stably sort this sequence Si using the seconçi component, then we get the
sequence

51,2

which is not exactly a sorted sequence. On the other hand, if we first stably sort S
using the second component, then we get the sequence

S2=«1,2)(3,2),(2,2),(3,3),(2,3),(1,5),(2,5),(1,7)).

if we then stably sort sequence 82 using the first component, then we get the se-
quence

=52,1 ((1,2),(1,5),(1,7),(2,2),(2,3),(2,5),(3,2),(3,3)),

which is indeed sequence S lexicographically ordered.

So, frorn this example, we are led to believe that we should first sort using
the second component and then again using the first component. This intuition is
exactly right. By first stably sorting by the second component and then again by
the first component, we guarantee that if two elements are equal in the second sort
(by the first component), then their ìelative order in the starting sequence (which
is sorted by the secoùd component) is preserved. Thus, the resulting sequenée is
guaranteed to be sorted lexicographically every time We leave the determination
of how this approach can be extended .to triples andothér d-tuples of numbers to a
simple exercise (R-4.15). We can summarize this section as follows:

Theorem 4.14: Let S be a sequence ofn key-element items, each of which has a

-key (k1 ,k2,... , kd), where k is an integer in the range [QN 1] for some integer

N 2. We can sort S lexicographically in time O(d(n + N)) using radix-sort.

As important as it is, sorting is not the only interesüng problèm dealing with
a total order relation on a set of elements. Theré re some applications, for ex-
ample, that do not require an ordered listing of an entire set, but nevertheless call
for some amount of ordering information about the set. Before we study such a
problem (called "selection"), et us step back and briefly compare Eli of the sorting
algorithms wehaye studied so far.

244 Chapter 4. Sorting, Sets, and SeFe

4.6 Comparison of Sorting Algorithms

At this point, it might be useful for us to take a breath and consider all th&
rithms we have studied in this book to sort an n-element sequence. As with.
things in life, there is no clear 'best" sorting algorithm, but, we can offerb
guidance and observations, based on the knpwn properties of "good" algorit

If implemented well, insertion-sort runs in O(n + k) time, where k is theiì
ber of inversions (that is, the number of pairs of elements out of order)
insertion-sort is an excellent algorithm for sorting small sequences (say, le$
50 elements) Also, insertion-sort is quite effective for sorting sequences th
already "almost" sorted By "almost," we mean that the number of inversi
small. But the O(n2)-time performanc& of insertion-sort makes it a poor

joutside of these special contexts
Merge-sort, on the other hand, mns in O(nlogn) time in the worst case,

is optimal for companson-based sorting methods Still, experimental studS!
shown that, since it is difficult to make merge-sort run in-place, the oven
needed to implement merge-sort make it less attractive than the in-place imple

tations of heap-sort and quick-sort for sequences that can fit entirely in a comp
main memory area Even so, merge-sort is an excellent algonthm for situ
where the input cannot all fit into main memory, but must be stored in blocks o
external memory device, such as a disk. In these contexts, the way that merg

processes runs of data in long merge streams makes the best use of all th4
brought into main memory in a block from disk (See Section 14 1 3)

Expenmental studies have shown that if an input sequence can fit entirej

main memory, then the in-place versions of quick-sort and heap-sort run
than merge-sort In fact, quick-sort tends, on average, to beat heap-sort ml

tests So, quick-sort is an excellent choice as a general-purpose, in-memory
algonthm Indeed, it is included in the qsort sorting utility provided in C laifg

libraries Still, its 0(n2) time worst-case perfonnance makes quick-sort 4
choice in real-time applications where we must make guarantees on the time ni
i
to complete a sorting operation

In real-time scenarios where we have a fixed amount of time to perform

ing operation and the input data can fit into main memory, the heap-sort alg
is probably the best choice It runs in 0(n log n) worst-case time and can east

made to execute in-place J

Finally, if our application involves sorting by integer keys or d-tuples of ni

keys, then bucket-sort or radix-sort is an excellent choice, for it runs in o(d@j
time, where [O,N - 1] is the range of integer keys (and d = i for bucket sort)4
if d(n + N) is "below" nlog n (formally, d(n + N) is o(nlog n)), then this S(

method should run faster than even quick-sort or heap-sort I

Thus, our study of all these different sorting algonthms pros ides us

versatile collection of sorting methods in our algonthm design "toolbox" I

2i Selection 245

Selection

There are a number of applications in which we are interested in identifying a sin-
gle element in terms of its rank relative to an ordering of the entire set. Examples
include, identifying the minimum and maximum elements, but we may also be in-
terested in, say, identifying the median element, that is, the element such that half
of the other elemènts are. smaller and the remaiñing half are larger. In general,
quedes that ask for an element with a given.rank are called order statistics.

In this section, we discuss the general order-statistic problem of selecting the
kth smallest element from an unsorted ccillection of n òomparable. elements. This is
known as the selection problem. Of course, we can solve this problem by sorting
the collection and then indexing into the sorted sequence ät rank index k - 1. Using
the best comparison-based sorting algorithms, this approach would take O(nlogn)
time. Thus; a natural question to ask is whether we can achieve 'an 0(n) running
time for all values of k, inclUding the interesting case of findiúg the median; where

k=In/2].

4.7.1 Prune-apd-Search

This may come as a small surprise, but we can indeed solve the selection problem in
0(n) time for any value of k. Moreover, the technique we use to achieve this result
involves an interesting algorithmic design pattern. This design pattern is known

as prune-and-search or decrease-and-conquer In applying this design pattern,
we solve a given problem that is defined on a collection of n objects by prumng
away a fraètion of the n objedts and recursively solving the smaller problem. When
we have finally reduced the problem to one defined on a constant-sized collection
of objects, then we solve the problem using some brute-force method. Returning
back from all the recursive calls completes the construction.. In some. cases, we
can avoid using recursion, in which case we simply iterate the prune-and-search
reduction step until we can apply a brute-force method and stop.

4.7.2 Randomized Quick-Select . ,,

In applying the prune-and-search pattern to the selection problem, we can design

a simple and practical method, called randomized quick-select, for finding the kth

smallest element in an unordered sequence of n elements on which a total order re-
lation is defined Randomized quick-select runs in 0(n) expected time, taken over
all possible random choices made by the algorithm, and this expectation does not
depend whatsoever on any randomness assumptions about the-input distribution
We note though that randomized quick-select runs in 0(n3tuíme in the worst-case
time, the justification of which is left as an exercise'CR-4 18) We also provide
an exercise (C-4 24) on modifying randomized qy4I-sèlect to get a deterministic

246 Chapter 4. Sorting, Sets, and Se1e

selection algorithm that runs in 0(n) worst-case time The existence of this
terministic algorithm is mostly of theoretical interest, however, since the con
factor hidden by the big-Oh notation is relatively large in this case.

Suppose we are given an unsorted sequénce S Of n comparable e1ement
gether with an integer k E [1, n]. At a high level, the quickselect algorithni
finding the kth smallest element in S is similar in structure to the randomized
sort algorithm described in Section 4 3 1 We pick an element x from S at rand
and use this as a "pivot" to subdivide S into three subsequences L, E, and G, sto
the elements of S less than x, equal to x, and greater than x, respectively This is
prune step Then, based on the value of k, we determine on which of these se
recur. Randomized quick-select is described iii Algorithm 4 16

Algorithm quickSelect(S,k):
Input: Sequenòe S of n comparable elements, and an integer k E [1, n]
Output: The kth smallest element of S

if n = ithen

return the (first) element of S
pick a random element x of S
remove all the elements from S and put them into three sequences:

L, storing the elements in S less than x
E, storing the elements in S equal to x
G, storing the eleménts in S greater than x

if k ILl then

quickSelect(L,k)

Sé if k ILI± EI then
return x {each element in E is equal to x}

else

quick5elect(G,k ILl - El) {note the new selection paramète

Algorithm 4.16: Randomized quick-select algorithm.

4.7.3 Analyzing Randomized Quick-Select

We mentioned above that the randomized quick-select algorithm runs in expect
0(n) time Fortunately, justifying this claim requires only the simplest of prob
nhtic arguments The main probabilistic fact that we use is the linearity of expec
talion -Recall that this fact states that if X and Y are random variables and c i

number, Ihen E(X +1') = È(X) +E(Y).and E(cX) = cE(X), where we Use E

to denote the exgected value, of theexpression Z.

247

Let t (n) denote the running time of randomized quick-select on a sequence of
size n. Since the randomized quick-select algorithm depends on the outcome of
random events, its running time, t(n), is a random variable. We are interested in
bounding E(t(n)), the expected value of «n). Say that a recursive invocation of
randomized quick-select is "good" if it partitions S, so that the size of L and G is at
most 3n/4. Clearly, a recursive call is good with probability 1/2. Let g(n) denote
the number of consecutive recursive invocations (mcludmg the present one) before

getting a good invocation. Then

t(n) bn.g(n)+t(3n/4),

where b > O is a constant (to account for the pverhead of each call). We are, of
course, focusing on the cäse where n is larger than 1, for we can easily characterize

in a closed form that t (1) = b. Applying the linearity of expectation property to the

general case, then, we get

E(t(n)) E(bn g(n)+t(3n/4)) =bn E(g(n))+E(t(3n/4))

Since a recursive öall is good with probability 1/2, and whether a recursive call is

good or noUs independent of its parent call being good, the expected value of g(n)

is the same as the expected number of times we must flip a fair coin before it comes

up "heads?' This implies that E(g(n)) = 2. Thus, if we let T(n) be a shorthand
notation for E(t(n)) (the expected running time of the randomized quick-select

algorithm), then we can write the case for n> I as

T(n)'< T(3n/4)+2bn. /

As with the merge-sort recurrence equation, we would like to convert this equation

'jnto a closed form. To do this, let us again iteratively apply this equation assuming

n is large. So, for example, after two iterative applications,, we get

T(n) <T((3/4)2n) +2b(3/4)n±2bn..

At this point, we see that the general case is

[Iog413n (3/4)i

T(n) <2bn

¡=0'

In other words, the expected running time of randomized quick-select is 2bn times
the sum of a gèometric progressioñ whose base 'is a positive number less than 1.
Thus, by Theorem 1.12 on geometric summations, we obtain the result that T (n)

is 0(n). To summarize, we have: ,.

Theorem 4.15: The expected running time of randomizqeudick-select on a se-

quenceofsizenisO(n) ''

.

As we mentioned earlier, there is a vanation of quick-select that does not use
randomization añd'runs in 0(n) worst'case time Execise C4.24 walks the inter-

ested.reader through 'the design and ànalysis cf this algôri.thnii F

248 Chapter 4. Sorting, Sets, an/d Sèiè$

4.8 Java Example: ln-Piàce Quick-Sort

Recall from Seçtion 2.4.4 that a sorting algorithm is in-place if it uses only a
stant amount of memory in addition to that needed for the objects being SO]
themselves The merge-sort algonthm, as we have descnbed it above, is no

place, and making it be in-place requires a more complicated merging method j
the one we discuss in Section 4 1 1 In-place sorting is not inherently diffi

however For, as with heap-sort, quick-sort can be adapted to be in-place

Performing the quick-sort algorithm in-place: requires a bit of ingenuity, h;
ever, for we must use the input sequence itself to store the subsequences foi
the recursivé calls. We show algorithm inPiaceQúickSort, which performs in-pi
quick-sort, in Algonthm 4 17 Algonthm inPiaceQuickSort assumes that the it
équence, S, has distinct elements The reason for this restriction is explore(
Exercise RL4 12 The extension to the general case is discussed in Exer,cise C-4
The algorithm accesses the elements of the input sequence, S, with rank-ba
methods. Hence, it runs efficiently provided Sis implemented.with an array.

Algorithm inPlaceQuickSort(S,a,b):

Input: Sequence S of distinct elements; integers a and b

Output: Sequence.S with elements originally from ranks from a to b, inclus

sorted in nondecreasing order from ranks a to b

if a b then return {empty subrange}
{pivot}
p - S.elèmAtRank(b)
i -a
r-b-i {will scan righ.tward}

{will scail leftward}

while l<rdo

{find an element larger than the pivot}

while lrandS.eIemAtRank(l)pdo

14-1+1

{find an element smaller than the pivot}

r<r-1while r i and S.elemAtRank(r) pdo

if l<r then

S.swapElements(S.atRank(1), S.atRank(r))

{put the pivot into its final place}

S.swapElements(S.atRank(l),S.atRank(b))

{recursive calls}

inPtaceQuickSort(S,a,l 1)

inPlaceQuickSort(S,l+1,b) i

Algonthm 4.17 In-place quick-sort for a sequence implemented with an arra'

a. Java Example: In-Place Qúkk-Sort 249

qsiaiunamnednudIdcnlotae-aepnrisiiseagnocihmoeutt-spqmelluyxioicpcfisrlkttiol-cyrmsiaotrlnery¡tkpfcmrorerr.eoswaetdhtniaefetriedseddusiabvbtnhsiyeddeaqeifur,nraeospnntmugecipeesrsoeib.sqfaIuppnceoedknÑrewcfeieotadiruro,mdsnia,sensdgsswupsbbeawyscpaeipsfqpicieunEadegnInbncepyimenaaigoerlnsfetthftoshtef-oemespieloeneqtspmaututereitanonsncntekess-
that are in reverse order, as shown in Figure 4.18. When these two indices "meet,"
subsequences L and G aie
completes by recursing on on opposite sides of the meeting point. The algorithm
these two subsequences.

In-place quick-sort reduces the running time, caused by the creation of new
sequences and the movement of eléments between them, by a constant factôr. We
show a Java version of in-place quiók-sort in Code Fragment 4.20.
Unfortunately, our implementation of quick-sort is, technically speaking, not
quite inplacc, as it still requires more than
Of course, we are using no additional space a constant amount of additional space.
for the subsequences, and we are using
only a constant amount of additional space for local variables (such as i and r).
So, where does dus additional space come from9 It comes from the recursion, for,
recalliñg Section 2.1.1, we note that we need space for à Stack proportional to the
depth of the recursion tree for quick-sort, which is at
In order to make quick-sort truly in-place, we müst leàst log n and at most n - 1.
implement
it nonrecursively
(and not use a stack). The key detail for sudi an implementation is that we need
an in-place way of determining the bounds for the left and right boundaries, of the
"current" subsequence. Such a scheme is not too difficult, however, and we lèave
iv the details of this implementation to an exercise (C-4.17).

(85 24 63 n n 96 50)
(31. 24 63
r

(85 24 63 45 17 31 96 so)

(a) Xb?

s 17 85 96 50)

(31 24 63 45 17 85 96 so)

(c)

(3! 24 n s oa 85 96 so)

(31 24 17 45 63 85 96 so)

(e) rf

U)

(31 17 45 50 85 96 63)

r (gf).. .H H

Figure 4.18: Divide step of in-place quick-sort. Index i scans the sequence from
left to right, and index r scans the sequence from right to left. A swap is performed
when I is at an element larger than the pivot and r is at an element smaller than the
y. pivot. A final swap with the pivot completes the divide step.

250 Chaptèr4.. Sorting, Sets, and Sei

* Sort theelements of sequence S in nondecreasing order according
* to comparator c, using the quick-sort ¿lgorithm. Most of the work
* is ddne by the auxiliary recursive method quickSortStep.

public static void quickSort (Sequence S, Comparator c) {
if (S.size() < 2)
return; // a sequence with O or i element is already sorted
quickSortStep(S,c, O, S.sizeQi); // recursive sort method

}

* Sort in. nondecreasing order the elements of sequence S between
* ranks leftBound and rightBound, using a recursive, in-place,

* implementation of thequick-sort algorithm.

private static void quickSortStep (Sequence S, Comparator c;
mt IeftBound, ¡nt rightBound ) {

if (lèftBbund >= rightBound)

return;

Object: pivot = S.atRank(rightBound) .elernentO;
mt lêftlndex = Ieftßound; J/ will scan rightward

intrightlndex = rightBoundi; // will scan leftward

while (leftlndex <= rightlndex) {
// scan rightward to find an element larger than thç pivot

while ( (leftlndex <= rightlndex) && ....

c. isLessThanOrEqualTo(S.atRaflk(leftltldex) .elementQ, pivot) )

leftlndex+±;
// scan leftward to findan element smaller than the pivot

while ( (rightlndex >= left!ndex) && .$

c.isGreaterThanOrEqualTo(S.atRank(rightlndex)emeh1tO, pivot

rightlndex--;
if (leftlndex < rightlndex) // both elements were found

S.swapElements(S.atRank(leftlfldeX), S.atRank(rightl ndex));

} // the loop continues until the indices cross
// place the pivot by swapping it with the element at leftlndex

S.swapElements(S.atRank(leftln.deX), S.atRank(rightBound));

// the pivot is now at leftlndex, so recur on both sides pf it

quickSortStep(S, c, leftBound, leftlndex-1);

quickSortStep(S, c, leftlndex+i, rightBound);

}

Code Fragment 4.20: Java implementation of in-piace quiçk-sort. It is assumS
the input sequence is implemented with an array and that it has distinct eleme

4.9. Exercises 251.

Reinforcement

R-4. i. Òive a complete justificatiòn of Theorem 4.1.

R-41 Give a pseudo-code description of the merge-sort algorithm.. You can call the
merge algorithm as a subroutine.

R-4.3 Give a pseudo-code description of a variation of the merge-sort algorithm that
operates on an array insteadof a general sequence.

Hint: Use an auxiliary array as a "buffer?'

R-4.4 Show that the running time of the merge-sort algorithm on an n-element sequence
j,s O(nlogn), even when n is not a power of 2.

Suppose we are given two n-element sorted sequences A and B that should not

be viewed as sets (that is, A and B may contain duplicate entries). Describe an

0(n)-time method for computing a sequence representing the set A U B (with no

duplicates). .

R-46 Showthat(XA)U(XB)=X.-.(AflB),foranythreesetsX,A, andB

R-4.7 Suppose we implement the tree-based partition (union-find) data structure us:
ing only the union-by-size heuristic. What is the amortized running time of a
sequènce of n union and find operations in this case?

R-4.8 Provide pseudo-code descriptions for performing methods insert and remove on
a set implemented with a sorted sequence.

R-4.9 Suppose we modify the deterministic version of.the quick-sort algorithm so that,

instead of selecting the last elementS in an n-element sequence as the pivot, we

choose the element at rank (index) [n/2j, that is, an element in the middle of the

sequence. What is the running time of this version of quick-sort on a sequence

that is already sorted? ..

R-4.10 Consider again the modification of the deterministic version of the quick-sort
algorithm so that, instead of selecting the last element in an n-element sequence
as the pivot, we choose the element at rank [n-/2j. Describe the kind of sequence
that would cause this version of quick-sort to run in 0(n2) time

R-4 11 Show that the best-case running time of quick-sort on a sequence of size n with

distinct elements is O(nlogn') -1

R-4.12 Suppose that algorithm inPiaceQuickSort (Algorithm 4.17) is executed ona se-
quence with duplicate elements. Show that, in this case, the algorithm correctly
sorts the input sequence, but the .result of the divide step may differ from the
high-level description given in Section 4 3 and may result in inefficiencies In
particular, what happens in the partition step when there are elements equal to
the pivot') Is the sequence E (storing the elements equal to the pivot) actually
computed') Does the algorithm recur on the subsequences L and R, or on some
other subsequences') What is the running time of the algorithm if all the elements

of the input sequence are equal?

252 Chapter 4. Sorting, Sets, and Selec ¿:

R-4.13 Give a pseudo-code description of the in-place version of quick-sort that is
cialized to take an array as input, rather than a general sequence, and retum iI
samearray as output.

R-4.14 Which, if any, of the algorithms bubble-sort, heap-sort, merge-sort, änd quic'
sort are stable?

R-4. 15 Describe a radix-sort method for lexicographically särting a sequence S of tripl

(k,1,,n), where k, 1, and m are integers in the range [0N - 1], for some N
2. How could this scheme be extended to sequencés of d-tuples (ki , k2,. . . , k,
where each Iq is an integer in the range [O,N - 1]?

R-4. 1.6 Is the bucket-sort algorithm in-place? Why or why not?

R-4.17 Give a pseudo-code description of an in-plâce quick-select algorithm.

R-4.18 Show that the worst-case running time of quick-select ¿n an n-element sequen.

isQ(n2). T

Creativity

C-4. Show how to implementmethod equals(B) on a setA, which tests whetherA

in O(IAI + BI) time by means of a variation of the generic merge algori

assuming A and B are implemented with sorted sequences.

C-4,2 Give a 'variation of the generic merge algorithm for computing A e B, which

the set of elements that are in A or B, but not in both.

C-4 3 Suppose that we implement the set ADT by representing each set using a b
anced search tree. Describe and analyze algorithms for each of the method&
the set ADT.

C-4 4 Let A be a collection of objects Describe an efficient method for converting
into a set, That is, remove all duplicates from A. What is the running time of tEl
method?

-C-4.5 Consider sets whose elements are (or can be mapped to) integers in the rañ
[O, N 1] A popular scheme for representing a set A of this type is by means
a Boolean vector, B, where we say that x is in A if and only if B{x] true Sin
each cell of B can be represented with a single bit, B is sometimes referred to
a bit vector Describe efficient algorithms for perfornung the Ufl ion, intersectio
and subtraction methods of the set ADT assuming this representation. What
the running times of these methods?

C-4 6 Suppose we implement the tree-based partition (union-find) data structure usin
the union-by-size heuristic and a partial path-compression heunstic The parti
path compression in this case means that, after perforimng a sequence of pointé
hops for a find operation, we update the parent pointer for each node u along tin
path to point to its grandparent Show that the ttal running time of performin
n i.inion and find operations is still 0(nlog* n).

C-4 7 Suppose we implement the tree-based partition (umon-find) data structure usiñ
the union-by-size and path-compression heuristics Show that the total runmn
timeof performiñg n union and find operations is 0(n) if all the unions coni
before all the finds.

253

-C-4.8 Show that randomized quick-sort runs in O(nlogn) time with probability i

1/n2.
Hint: Use the Chern off boun4 that states that if we flip a coin k times, then the
probability that we get fewer than k/16 heads is less than

C-4.9 Suppose we are :given a sequence S of n elements, each of which is colored red

or blue. Assuming S is represented as an affay, give an in-place method for

- ordering S so that all the blue elements are listed before all the red elements. Can
you extend your approach to three colors?

C-4.10 Suppose we are given an n-element sequence S such that each element inS repre-
sents a different vote in an election, where each vote is given as an integer repre-
seining the ID of the chosen candidate. Without making any assumptions about
who is running or eveñhow many candidates there are, design an O(nlogn)-tirne -
algorithm to see who wins the election S represents, assuming the candidate with
the most votes wins.

C-4.11 Cònsider the voting problem from the previous exercise, but now suppose that
we know the number k <n of candidates running. Describe an O(nlogk)-tirn
algorithm fordetermining who wins the election.

C-4.12 Show that any comparison-based sorting algorithm can be made to be stable,
without affecting the asymptotic running time of this algorithm.

flint: Change the way elements are compared with each other.

C-4.13 Suppose we are given two sequencés A and B of n elements, possibly containing
duplicates, on which a total order relation is defined. Describe an efficient algo-
rithm for determining if A and B contain the same set of elements (possibly in
different orders). What is the running time of this method?

-C-4.14 Suppose we are given a sequence S òf n elements, each of which is an integer in
the range [O »2 1]. Describe a simple method for sorting S in 0(n) time.

Hint: Think of alternate ways of viewing the elements.

C4. 15 Let S, S2, , S be k different sequences. whose elements have integer keys in

the range [O,N - 1], for some parameter N 2. 'Describe an algorithm running
in 0(n + N) time for sortiñg all the sequences (not äs ä union), where n denotes
the total size of all the sequences.

C-4.16 Suppose we are given a sequence S of nrçlements, on which a total order relation
is defined. Describe an efficient method for determining whether there are two
equal elements in S. What is the running time of your method?

C-4.17 Describe a noi7recursive, in-place version of the quiçk-sòrt algorithm. The algo-
rithm should still be based on the same divide-and-cÔnquer approach.
'Hint: Think about how io "mark" the left and right boundaries of the current
subsequence before making a recursive call from this one.

C-4 18 Modify Algonthm inPIaceQ.iickSort (Algonthm 4 17) to handle the general case
efficiently' when the input sequence, S, may have duplicate keys.

254 Chapter 4. Sorting, Sets and S

C-4. 1.9 Let S be a sequence of n elements on which a total order relation is defi
inversion in S is a pair of elements x and y such that x appears befor
but r> y. Describe an algorithm running in O(nlogn) time for dete

number of inversions in S. . .. -

Hint: Try to modify the merge-sort algorithm to solve this problem.

C-4.20 Let S be a sequence of n elements on which a total order relation is ¿

DeÑcribe a comparison-based method for sorting S in 0(n + k) time, wh
the number of inversions in S (recall the definition of inversion from the p
problem).

Hint: Think of an in-place version of the insertion-sort algorithm that,
linear-time preprocessing step, only swaps elements that are inverted

C-4.21. Give a sequence. of n integers with Q(n2) iflversions. (Recall the definf

inversion from Exercise C-4.19.) Y

C-4.22 Let A and B be two sequences of n integers each. Given an integer x, d
an 0(nlogn)-time algorithm for determining if there is an integer a in ¡Ç'
integer b in B such thatx= a + b;

C-4.23 Given an unordered sequence S of n comparable elements, describe an effi
method for finding the R/W} items whose rank in an ordered version o
closest to that of the median. What is the rünning time of your method?

C-4.24 This problem deals with the modification of the quick-select algorithm to
it deterministic, yet still run in 0(n) time on an n-element sequence. The i
to modify the way we choose the pivot so thatit is chosen deterministicall,
randomly, as follows:

Partition the set S into In/Si groups of size 5 each (except possibly.
for oñe group). Sort each little set and identify the median element in-
this set. From this set of In/Si 'baby" medians, apply the selectiotr
algorithm recursively to find the .median of the baby medians Use:
this element as the pivot and proceed as in the quick-select algorithm;

Show that this deterministic method runs in 0(n) time, by answering th
lowing questions (please ignore floor and ceIling functions if that simplifiq
mathematics, for the asymptotics are the same either way):

How many baby medians are less than or equal to the chOsen pivot?t
many are greater than or equal to the pivot?

For each baby median less than or equal tO the pivot, how many o
elements are less 'than or equal to the pivot? Is the same true for,
greater than or equal to the pivot?

Argue why the method for finding the deterministic pivoi and usingt
partition S takes 0(n) time.

Based.on these estimates, write a recurrence equation that bounds the w
case running time t (n) for this selection algorithm. (Note: In the worst
there are two recursive callsone to find the median of the baby medi
and one to then r cur on the larger of L and G.)

e. Using this recurrence equation, show by induction that t(n) is 0(n). i

255

£xeIX1

C-4.25 Bob has a set A of n nuts and a set B of n bolts, such that each nut in A has a
unique matching bolt in B. Unfortunately, the nuts in A all look the same, añd
the bolts in B all look the same as well. The only kind of a comparison that Bob
cWbsae. nehDaimeftsatichksreeitbhtihesertareounantdeansfkifeonicfgaiaentniaumtrtae-ebllgooaolfrtrgtipehthari,simrsalm(loagra,olbBlre)io,trhb,sm,uoctr,ohaimntphaetaetrtcfrahemcEust pAmofaaalntlncudohtf-bwbhoEiitslhtBntt,uehatsesntstdahnttrhedeasabttdoBistlottosobf.

must make?

C-4.26 Sahqouwickh-osworta-ldikeetesromrtiinnigstiaclg0o(rnit)h-tmimtheasterluencstiionn0a(lngolorigthnm) wcaonrsbt-ecausseedtitmoedoesnigann
n-element sequence.

C-4.27 Given an unsorted sequence S of n comparable elements, and an integer k, give
an 0(niogk) expected-time algorithm for finding the 0(k) elements that have

rank Fn/kl, 2 kiki, 3 [n/kl, and so on.

Projects

P-4. I 2Dstvie.maeqTlsuueihegeetnsnrciaoaaelflngsondoftroaiinmtnhtedepmlselÑtemsm,wheaeohnnñutedsttlhdawwesrirrtutihattnehbiilnaisentsemvh0geeo(ertnrrsht+ikoorNedenyp)dosotofritematthskdeeeei.nsnbPcdufrerceiorbkefmdieontr-rgtmsuhotnehraretiansacneltorgghideoeiesrs[iOattoihnmf,mdNbeet,fh-noef1cro]hr,rsmeofvsorauartrrilkNnitosignuogasf

these trials.

p-4 .2 Imrameraevprelkveriesmnregyesnto"etrrstamtensdedt?roog' mesWe-"seroliwortetohakâincisndhhgdoo,enratteserriemwspfeioanlrslittsaedtirsec.soYqcnuroeiiubscriktnth-egsasotttrhsatersaehncood"audpllemedraifonnosdcrtmltushodaeretsreseedersiq"ueulsotersonf"ocabeflestmnhthceoahsset-t

trials.

P-4.3 Ia"smhanplodmlueplomedsrtefi"noncrstmohdrdtaeeetds&se.ernqWiìeuisrneiniotsecfteiabcsesanthhncoadhrtmtraararenerpkdvoioierimtgryditzee"essrcdtasrnitvbdoeionsrmsegie"ot.lhnowesochkooiifcdnhetghoaeannsqedwuitsiehclfekla-arssetsoesorur.tnlYteasslogotuohfrratithtteheassmrtsee

trials.

P-4.4 Implement an ibne-pnlcahcme avrekrsiniognteosftisntsoerdtieotne-rsmoirnteanthdearnainng-eploafcevavleuresisonofonf quick-
sort. Perform where

quick-sort is, on average, faster than insertion-sort.

P-4.5 Ditthnoeissaeinxgchpnianlaaptiuntneidtri.tivhmYeispomluaerlamganoennnrieimttrha,anmtaiaontnndoimssshhoaoomtuiuoleldnodnfibloelerusuoatnnnraefnatooemfttaihttlheeieadkrsewowyriitptithnhrogtipeta.ex.lrgtWtoiaernrsidtihto/emfoatrshsdissoehusaoclnrgrtdiobrrseeiodpthoaimnrst
describing this animation.

P-4.6 Implement the partition (union-find) ADT using the tree-based approach with the
union-by-size and path-cOmpression heuristics.

256 Chaptèr 4. Sorting, Sets, and Sélec

Chapter Notes

Knuth's classic text on Sorting and Searching [119] contains an extensive history ¿
sorting problem and algorithms for solving it, starting 'with the census card sorting
chines of the late 19th century. Huang and Langston [103] describe how to merge
sorted lists in-place in linear time. Our set ADT is derived from the set ADT of<
Hoperoft, and Ullman [8]. The standard quick-sort algorithm is due to Hoare [96
tighter analysis of randomized quick-sort can be found im the book by Motwani and Ra
van [157]. Gonnet and Baeza-Yates [81] provide experimental comparisons and thee
ical analyses of a number of different sorting algorithms. The term "prune-and-se
originally comes from the computational geomSy literature (such as in the work of Cl
son [47] and Megiddo [145, 146]). The term "decrease-and-conquer" is from Levitin [

The analysis we give for the partition data structure cornes from Hoperoft and
man [99] (see also [7]). Tarjan [199] shows that a sequence of n union and find operan
implemented as described in this chapter, can be performed in 0(na(n)) time, where
is the very slow growing inverse of the Ackermann function, and this bound is fight i
worst case (see also [200]). Gabow and Tarjan [73] show that one can, in some
achieve a running time of 0(n), however.

Chapter Fundamental Techniques

5

Contents

5.1 The Greedy Method . . . . . . . . . . . . . . . . . . . 259
5.1.1 The Fractional Knapsack Problem . . . . . . . . . . 259
5.1.2 Task Scheduling . . . . . . . . . . . . . . . . . . . . 261

5.2 Divide-and-Conquer . . . . . . . . . . . . . . . . . . . 263
5.2.1 Divide-and-Conquer Recurrence Equations . . . . . . 263
5.2.2 Integer Multiplication . . . . . . . . . . . . . . . . . 270
5.2.3 Matrix Multiplication . . . . . . . . . . . . . . . . . 272

5.3 Dynamic Programming . . . . . . . . . . . . . . . . . 274
5.3.1 Matrix Chain-Product . . . . . . . . . . . . . . . . . 274
5.3.2 The General Technique . . . . . . . . . . . . . . . . 278
5.3.3 The 0-1 Knapsack Problem . . . . . . . . . . . . . . 278

5.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . 282

258 Chapter 5. Fundamental Techniques

A popular television network broadcasts two different shows about carpentry.
In one show, the host builds furniture using specialized power tools, and in the other
the host builds furniture using general-purpose hand tools. The specialized tools,
used in the first show, are good at the jobs they are intended for, but none of them
is very versatile. The tools in the second show are fundamental, however, because
they can be used effectively for a wide variety of different tasks.

These two television shows provide an interesting metaphor for data structure
and algorithm design. There are some algorithmic tools that are quite specialized.
They are good for the problems they are intended to solve, but they are not very
versatile. There are other algorithmic tools, however, that are fundamental in that
they can be applied to a wide variety of different data structure and algorithm design
problems. Learning to use these fundamental techniques is a craft, and this chapter
is dedicated to developing the knowledge for using these techniques effectively.

The fundamental techniques covered in this chapter are the greedy method,
divide-and-conquer, and dynamic programming. These techniques are versatile,
and examples are given both in this chapter and in other chapters of this book.

The greedy method is used in algorithms for weighted graphs discussed in
Chapter 7, as well as a data compression problem presented in Section 9.3. The
main idea of this technique, as the name implies, is to make a series of greedy
choices in order to construct an optimal solution (or close to optimal solution) for a
given problem. In this chapter, we give the general structure for the greedy method
and show how it can be applied to knapsack and scheduling problems.

Divide-and-conquer is used in the merge-sort and quick-sort algorithms of Chap-
ter 4. The general idea behind this technique is to solve a given problem by dividing
it into a small number of similar subproblems, recursively solve each of the sub-
problems until they are small enough to solve by brute force, and, after the recursive
calls return, merge all the subproblems together to derive a solution to the original
problem. In this chapter, we show how to design and analyze general divide-and-
conquer algorithms and we give additional applications of this technique to the
problems of multiplying big integers and large matrices. We also give a number of
techniques for solving divide-and-conquer recurrence equations, including a gen-
eral master theorem that can be applied to a variety of equations.

The dynamic programming technique might at first seem a bit mysterious, but
it is quite powerful. The main idea is to solve a given problem by characterizing its
subproblems using a small set of integer indices. The goal of this characterization
is to allow an optimal solution to a subproblem to be defined by the combination of
(possibly overlapping) solutions to even smaller subproblems. If we can construct
such a characterization, which is the hardest step in using the dynamic program-
ming technique, then we can build a rather straightforward algorithm that builds up
larger subproblem solutions from smaller ones. This technique underlies the Floyd-
Warshall transitive closure algorithm of Chapter 6. In this chapter, we describe the
general framework of dynamic programming and give several applications, includ-
ing to the 0-1 knapsack problem.

5.1. The Greedy Method 259

5.1 The Greedy Method

The first algorithmic technique we consider in this chapter is the greedy method.
We characterize this greedy method design pattern in terms of a general greedy-
choice property, and we give two applications of its use.

The greedy method is applied to optimization problems, that is, problems that
involve searching through a set of configurations to find one that minimizes or
maximizes an objective function defined on these configurations. The general for-
mula of the greedy method could not be simpler. In order to solve a given optimiza-
tion problem, we proceed by a sequence of choices. The sequence starts from some
well-understood starting configuration, and then iteratively makes the decision that
seems best from all of those that are currently possible.

This greedy approach does not always lead to an optimal solution. But there
are several problems that it does work optimally for, and such problems are said
to possess the greedy-choice property. This is the property that a global optimal
configuration can be reached by a series of locally optimal choices (that is, choices
that are the best from among the possibilities available at the time), starting from a
well-defined configuration.

5.1.1 The Fractional Knapsack Problem

Consider the fractional knapsack problem, where we are given a set S of n items,
such that each item i has a positive benefit bi and a positive weight wi, and we wish
to find the maximum-benefit subset that does not exceed a given weight W . If we
are restricted to entirely accepting or rejecting each item, then we would have the
0-1 version of this problem (for which we give a dynamic programming solution
in Section 5.3.3). Let us now allow ourselves to take arbitrary fractions of some
elements, however. The motivation for this fractional knapsack problem is that we
are going on a trip and we have a single knapsack that can carry items that together
have weight at most W . In addition, we are allowed to break items into fractions
arbitrarily. That is, we can take an amount xi of each item i such that

0 ≤ xi ≤ wi for each i ∈ S and ∑ xi ≤ W.
i∈S

The total benefit of the items taken is determined by the objective function

∑ bi(xi/wi).

i∈S

Consider, for example, a student who is going to an outdoor sporting event and
must fill a knapsack full of foodstuffs to take along. Each candidate foodstuff is
something that can be easily divided into fractions, such as soda pop, potato chips,
popcorn, and pizza.

260 Chapter 5. Fundamental Techniques

Algorithm FractionalKnapsack(S,W ):

Input: Set S of items, such that each item i ∈ S has a positive benefit bi and a
positive weight wi; positive maximum total weight W

Output: Amount xi of each item i ∈ S that maximizes the total benefit while not
exceeding the maximum total weight W

for each item i ∈ S do

xi ← 0 {value index of item i}
vi ← bi/wi

w←0 {total weight}

while w < W do

remove from S an item i with highest value index {greedy choice}

a ← min{wi,W − w} {more than W − w causes a weight overflow}

xi ← a
w ← w+a

Algorithm 5.1: A greedy algorithm for the fractional knapsack problem.

This is one place where greed is good, for we can solve the fractional knapsack
problem using the greedy approach shown in Algorithm 5.1.

The FractionalKnapsack algorithm can be implemented in O(n log n) time,
where n is the number of items in S. Specifically, we use a heap-based priority
queue (Section 2.4.3) to store the items of S, where the key of each item is its value
index. With this data structure, each greedy choice, which removes the item with
greatest value index, takes O(log n) time.

To see that the fractional knapsack problem satisfies the greedy-choice property,
suppose that there are two items i and j such that

xi < wi, x j > 0, and vi < v j.

Let

y = min{wi − xi, x j}.

We could then replace an amount y of item j with an equal amount of item i, thus
increasing the total benefit without changing the total weight. Therefore, we can
correctly compute optimal amounts for the items by greedily choosing items with
the largest value index. This leads to the following theorem.

Theorem 5.1: Given a collection S of n items, such that each item i has a benefit
bi and weight wi, we can construct a maximum-benefit subset of S, allowing for
fractional amounts, that has a total weight W in O(n log n) time.

This theorem shows how efficiently we can solve the fractional version of the
knapsack problem. The all-or-nothing, or “0-1” version of the knapsack problem
does not satisfy the greedy choice property, however, and solving this version of
the problem is much harder, as we explore in Sections 5.3.3 and 13.3.4.

5.1. The Greedy Method 261

5.1.2 Task Scheduling

Let us consider another optimization problem. Suppose we are given a set T of n
tasks, such that each task i has a start time, si, and a finish time, fi (where si < fi).
Task i must start at time si and it is guaranteed to be finished by time fi. Each task
has to be performed on a machine and each machine can execute only one task at
a time. Two tasks i and j are nonconflicting if fi ≤ s j or f j ≤ si. Two tasks can be
scheduled to be executed on the same machine only if they are nonconflicting.

The task scheduling problem we consider here is to schedule all the tasks in
T on the fewest machines possible in a nonconflicting way. Alternatively, we can
think of the tasks as meetings that we must schedule in as few conference rooms as
possible. (See Figure 5.2.)

Machine 3
Machine 2
Machine 1

123456789

Figure 5.2: An illustration of a solution to the task scheduling prob-
lem, for tasks whose collection of pairs of start times and finish times is
{(1, 3), (1, 4), (2, 5), (3, 7), (4, 7), (6, 9), (7, 8)}.

In Algorithm 5.3, we describe a simple greedy algorithm for this problem.

Algorithm TaskSchedule(T ):

Input: A set T of tasks, such that each task has a start time si and a finish time fi
Output: A nonconflicting schedule of the tasks in T using a minimum number

of machines

m←0 {optimal number of machines}

while T = ∅ do

remove from T the task i with smallest start time si

if there is a machine j with no task conflicting with task i then

schedule task i on machine j

else

m ← m+1 {add a new machine}

schedule task i on machine m

Algorithm 5.3: A greedy algorithm for the task scheduling problem.

262 Chapter 5. Fundamental Techniques

Correctness of Greedy Task Scheduling

In the algorithm TaskSchedule, we begin with no machines and we consider the
tasks in a greedy fashion, ordered by their start times. For each task i, if we have a
machine that can handle task i, then we schedule i on that machine. Otherwise, we
allocate a new machine, schedule i on it, and repeat this greedy selection process
until we have considered all the tasks in T .

The fact that the above TaskSchedule algorithm works correctly is established
by the following theorem.

Theorem 5.2: Given a set of n tasks specified by their start and finish times, Al-
gorithm TaskSchedule produces a schedule of the tasks with the minimum number
of machines in O(n log n) time.

Proof: We can show that the above simple greedy algorithm, TaskSchedule, finds
an optimal schedule on the minimum number of machines by a simple contradiction
argument.

So, suppose the algorithm does not work. That is, suppose the algorithm finds
a nonconflicting schedule using k machines but there is a nonconflicting schedule
that uses only k − 1 machines. Let k be the last machine allocated by our algorithm,
and let i be the first task scheduled on k. By the structure of the algorithm, when
we scheduled i, each of the machines 1 through k − 1 contained tasks that conflict
with i. Since they conflict with i and because we consider tasks ordered by their
start times, all the tasks currently conflicting with task i must have start times less
than or equal to si, the start time of i, and have finish times after si. In other words,
these tasks not only conflict with task i, they all conflict with each other. But this
means we have k tasks in our set T that conflict with each other, which implies
it is impossible for us to schedule all the tasks in T using only k − 1 machines.
Therefore, k is the minimum number of machines needed to schedule all the tasks
in T .

We leave as a simple exercise (R-5.2) the job of showing how to implement the
Algorithm TaskSchedule in O(n log n) time.

We consider several other applications of the greedy method in this book, in-
cluding two problems in string compression (Section 9.3), where the greedy ap-
proach gives rise to a construction known as Huffman coding, and graph algorithms
(Section 7.3), where the greedy approach is used to solve shortest path and mini-
mum spanning tree problems.

The next technique we discuss is the divide-and-conquer technique, which is a
general methodology for using recursion to design efficient algorithms.

5.2. Divide-and-Conquer 263

5.2 Divide-and-Conquer

The divide-and-conquer technique involves solving a particular computational prob-
lem by dividing it into one or more subproblems of smaller size, recursively solving
each subproblem, and then “merging” or “marrying” the solutions to the subprob-
lem(s) to produce a solution to the original problem.

We can model the divide-and-conquer approach by using a parameter n to de-
note the size of the original problem, and let S(n) denote this problem. We solve
the problem S(n) by solving a collection of k subproblems S(n1), S(n2), . . ., S(nk),
where ni < n for i = 1, . . . , k, and then merging the solutions to these subproblems.

For example, in the classic merge-sort algorithm (Section 4.1), S(n) denotes the
problem of sorting a sequence of n numbers. Merge-sort solves problem S(n) by
dividing it into two subproblems S( n/2 ) and S( n/2 ), recursively solving these
two subproblems, and then merging the resulting sorted sequences into a single
sorted sequence that yields a solution to S(n). The merging step takes O(n) time.
This, the total running time of the merge-sort algorithm is O(n log n).

As with the merge-sort algorithm, the general divide-and-conquer technique
can be used to build algorithms that have fast running times.

5.2.1 Divide-and-Conquer Recurrence Equations

To analyze the running time of a divide-and-conquer algorithm we utilize a recur-
rence equation (Section 1.1.4). That is, we let a function T (n) denote the running
time of the algorithm on an input of size n, and characterize T (n) using an equation
that relates T (n) to values of the function T for problem sizes smaller than n. In
the case of the merge-sort algorithm, we get the recurrence equation

T (n) = b if n < 2
2T (n/2) + bn if n ≥ 2,

for some constant b > 0, taking the simplifying assumption that n is a power of 2.
In fact, throughout this section, we take the simplifying assumption that n is an
appropriate power, so that we can avoid using floor and ceiling functions. Every
asymptotic statement we make about recurrence equations will still be true, even if
we relax this assumption, but justifying this fact formally involves long and boring
proofs. As we observed above, we can show that T (n) is O(n log n) in this case. In
general, however, we will possibly get a recurrence equation that is more challeng-
ing to solve than this one. Thus, it is useful to develop some general ways of solving
the kinds of recurrence equations that arise in the analysis of divide-and-conquer
algorithms.

264 Chapter 5. Fundamental Techniques

The Iterative Substitution Method

One way to solve a divide-and-conquer recurrence equation is to use the iterative
substitution method, which is more colloquially known as the “plug-and-chug”
method. In using this method, we assume that the problem size n is fairly large
and we then substitute the general form of the recurrence for each occurrence of
the function T on the right-hand side. For example, performing such a substitution
with the merge-sort recurrence equation yields the equation

T (n) = 2(2T (n/22) + b(n/2)) + bn
= 22T (n/22) + 2bn.

Plugging the general equation for T in again yields the equation

T (n) = 22(2T (n/23) + b(n/22)) + 2bn
= 23T (n/23) + 3bn.

The hope in applying the iterative substitution method is that, at some point, we
will see a pattern that can be converted into a general closed-form equation (with
T only appearing on the left-hand side). In the case of the merge-sort recurrence
equation, the general form is

T (n) = 2iT (n/2i) + ibn.

Note that the general form of this equation shifts to the base case, T (n) = b, when
n = 2i, that is, when i = log n, which implies

T (n) = bn + bn log n.

In other words, T (n) is O(n log n). In a general application of the iterative substitu-
tion technique, we hope that we can determine a general pattern for T (n) and that
we can also figure out when the general form of T (n) shifts to the base case.

From a mathematical point of view, there is one point in the use of the iterative
substitution technique that involves a bit of a logical “jump.” This jump occurs at
the point where we try to characterize the general pattern emerging from a sequence
of substitutions. Often, as was the case with the merge-sort recurrence equation,
this jump is quite reasonable. Other times, however, it may not be so obvious what
a general form for the equation should look like. In these cases, the jump may
be more dangerous. To be completely safe in making such a jump, we must fully
justify the general form of the equation, possibly using induction. Combined with
such a justification, the iterative substitution method is completely correct and an
often useful way of characterizing recurrence equations. By the way, the colloqui-
alism “plug-and-chug,” used to describe the iterative substitution method, comes
from the way this method involves “plugging” in the recursive part of an equation
for T (n) and then often “chugging” through a considerable amount of algebra in
order to get this equation into a form where we can infer a general pattern.

5.2. Divide-and-Conquer 265

The Recursion Tree

Another way of characterizing recurrence equations is to use the recursion tree
method. Like the iterative substitution method, this technique uses repeated sub-
stitution to solve a recurrence equation, but it differs from the iterative substitution
method in that, rather than being an algebraic approach, it is a visual approach. In
using the recursion tree method, we draw a tree R where each node represents a
different substitution of the recurrence equation. Thus, each node in R has a value
of the argument n of the function T (n) associated with it. In addition, we associate
an overhead with each node v in R, defined as the value of the nonrecursive part
of the recurrence equation for v. For divide-and-conquer recurrences, the overhead
corresponds to the running time needed to merge the subproblem solutions coming
from the children of v. The recurrence equation is then solved by summing the
overheads associated with all the nodes of R. This is commonly done by first sum-
ming values across the levels of R and then summing up these partial sums for all
the levels of R.

Example 5.3: Consider the following recurrence equation:

T (n) = b if n < 3
3T (n/3) + bn if n ≥ 3.

This is the recurrence equation that we get, for example, by modifying the merge-
sort algorithm so that we divide an unsorted sequence into three equal-sized se-
quences, recursively sort each one, and then do a three-way merge of three sorted
sequences to produce a sorted version of the original sequence. In the recursion tree
R for this recurrence, each internal node v has three children and has a size and an
overhead associated with it, which corresponds to the time needed to merge the sub-
problem solutions produced by v’s children. We illustrate the tree R in Figure 5.4.
Note that the overheads of the nodes of each level sum to bn. Thus, observing that
the depth of R is log3 n, we have that T (n) is O(n log n).

Overhead

bn

bn

bn

Figure 5.4: The recursion tree R used in Example 5.3, where we show the cumula-
tive overhead of each level.

266 Chapter 5. Fundamental Techniques

The Guess-and-Test Method

Another method for solving recurrence equations is the guess-and-test technique.
This technique involves first making an educated guess as to what a closed-form
solution of the recurrence equation might look like and then justifying that guess,
usually by induction. For example, we can use the guess-and-test method as a kind
of “binary search” for finding good upper bounds on a given recurrence equation.
If the justification of our current guess fails, then it is possible that we need to use
a faster-growing function, and if our current guess is justified “too easily,” then it
is possible that we need to use a slower-growing function. However, using this
technique requires our being careful, in each mathematical step we take, in trying
to justify that a certain hypothesis holds with respect to our current “guess.” We
explore an application of the guess-and-test method in the examples that follow.

Example 5.4: Consider the following recurrence equation (assuming the base case
T (n) = b for n < 2):

T (n) = 2T (n/2) + bn log n.

This looks very similar to the recurrence equation for the merge-sort routine, so we
might make the following as our first guess:

First guess: T (n) ≤ cn log n,

for some constant c > 0. We can certainly choose c large enough to make this true
for the base case, so consider the case when n ≥ 2. If we assume our first guess is
an inductive hypothesis that is true for input sizes smaller than n, then we have

T (n) = 2T (n/2) + bn log n

≤ 2(c(n/2) log(n/2)) + bn log n

= cn(log n − log 2) + bn log n

= cn log n − cn + bn log n.

But there is no way that we can make this last line less than or equal to cn log n for
n ≥ 2. Thus, this first guess was not sufficient. Let us therefore try

Better guess: T (n) ≤ cn log2 n,

for some constant c > 0. We can again choose c large enough to make this true
for the base case, so consider the case when n ≥ 2. If we assume this guess as an
inductive hypothesis that is true for input sizes smaller then n, then we have

T (n) = 2T (n/2) + bn log n
≤ 2(c(n/2) log2(n/2)) + bn log n
= cn(log2 n − 2 log n + 1) + bn log n
= cn log2 n − 2cn log n + cn + bn log n
≤ cn log2 n,

provided c ≥ b. Thus, we have shown that T (n) is indeed O(n log2 n) in this case.

5.2. Divide-and-Conquer 267

We must take care in using this method. Just because one inductive hypothesis
for T (n) does not work, that does not necessarily imply that another one propor-
tional to this one will not work.

Example 5.5: Consider the following recurrence equation (assuming the base case
T (n) = b for n < 2):

T (n) = 2T (n/2) + log n.

This recurrence is the running time for the bottom-up heap construction discussed
in Section 2.4.4, which we have shown is O(n). Nevertheless, if we try to prove
this fact with the most straightforward inductive hypothesis, we will run into some
difficulties. In particular, consider the following:

First guess: T (n) ≤ cn,

for some constant c > 0. We can choose c large enough to make this true for the
base case, certainly, so consider the case when n ≥ 2. If we assume this guess as an
inductive hypothesis that is true for input sizes smaller than n, then we have

T (n) = 2T (n/2) + log n

≤ 2(c(n/2)) + log n

= cn + log n.

But there is no way that we can make this last line less than or equal to cn for n ≥ 2.
Thus, this first guess was not sufficient, even though T (n) is indeed O(n). Still, we
can show this fact is true by using

Better guess: T (n) ≤ c(n − log n),

for some constant c > 0. We can again choose c large enough to make this true for
the base case; in fact, we can show that it is true any time n < 8. So consider the
case when n ≥ 8. If we assume this guess as an inductive hypothesis that is true for
input sizes smaller than n, then we have

T (n) = 2T (n/2) + log n
≤ 2c((n/2) − log(n/2)) + log n
= cn − 2c log n + 2c + log n
= c(n − log n) − c log n + 2c + log n
≤ c(n − log n),

provided c ≥ 3 and n ≥ 8. Thus, we have shown that T (n) is indeed O(n) in this
case.

The guess-and-test method can be used to establish either an upper or lower
bound for the asymptotic complexity of a recurrence equation. Even so, as the
above example demonstrates, it requires that we have developed some skill with
mathematical induction.

268 Chapter 5. Fundamental Techniques

The Master Method

Each of the methods described above for solving recurrence equations is ad hoc
and requires mathematical sophistication in order to be used effectively. There is,
nevertheless, one method for solving divide-and-conquer recurrence equations that
is quite general and does not require explicit use of induction to apply correctly. It
is the master method. The master method is a “cook-book” method for determining
the asymptotic characterization of a wide variety of recurrence equations. Namely,
it is used for recurrence equations of the form

T (n) = c if n < d
aT (n/b) + f (n) if n ≥ d,

where d ≥ 1 is an integer constant, a > 0, c > 0, and b > 1 are real constants, and
f (n) is a function that is positive for n ≥ d. Such a recurrence equation would arise
in the analysis of a divide-and-conquer algorithm that divides a given problem into
a subproblems of size at most n/b each, solves each subproblem recursively, and
then “merges” the subproblem solutions into a solution to the entire problem. The
function f (n), in this equation, denotes the total additional time needed to divide
the problem into subproblems and merge the subproblem solutions into a solution to
the entire problem. Each of the recurrence equations given above uses this form, as
do each of the recurrence equations used to analyze divide-and-conquer algorithms
given earlier in this book. Thus, it is indeed a general form for divide-and-conquer
recurrence equations.

The master method for solving such recurrence equations involves simply writ-
ing down the answer based on whether one of the three cases applies. Each case is
distinguished by comparing f (n) to the special function nlogb a (we will show later
why this special function is so important).

Theorem 5.6 [The Master Theorem]: Let f (n) and T (n) be defined as above.

1. If there is a small constant ε > 0, such that f (n) is O(nlogb a−ε), then T (n) is
Θ(nlogb a).

2. If there is a constant k ≥ 0, such that f (n) is Θ(nlogb a logk n), then T (n) is
Θ(nlogb a logk+1 n).

3. If there are small constants ε > 0 and δ < 1, such that f (n) is Ω(nlogb a+ε)
and a f (n/b) ≤ δ f (n), for n ≥ d, then T (n) is Θ( f (n)).

Case 1 characterizes the situation where f (n) is polynomially smaller than the
special function, nlogb a. Case 2 characterizes the situation when f (n) is asymptoti-
cally close to the special function, and Case 3 characterizes the situation when f (n)

is polynomially larger than the special function.

5.2. Divide-and-Conquer 269

We illustrate the usage of the master method with a few examples (with each
taking the assumption that T (n) = c for n < d, for constants c ≥ 1 and d ≥ 1).

Example 5.7: Consider the recurrence

T (n) = 4T (n/2) + n.
In this case, nlogb a = nlog2 4 = n2. Thus, we are in Case 1, for f (n) is O(n2−ε) for
ε = 1. This means that T (n) is Θ(n2) by the master method.

Example 5.8: Consider the recurrence

T (n) = 2T (n/2) + n log n,
which is one of the recurrences given above. In this case, nlogb a = nlog2 2 = n.
Thus, we are in Case 2, with k = 1, for f (n) is Θ(n log n). This means that T (n) is
Θ(n log2 n) by the master method.

Example 5.9: Consider the recurrence

T (n) = T (n/3) + n,

which is the recurrence for a geometrically decreasing summation that starts with n.
In this case, nlogb a = nlog3 1 = n0 = 1. Thus, we are in Case 3, for f (n) is Ω(n0+ε),
for ε = 1, and a f (n/b) = n/3 = (1/3) f (n). This means that T (n) is Θ(n) by the
master method.

Example 5.10: Consider the recurrence
T (n) = 9T (n/3) + n2.5.

In this case, nlogb a = nlog3 9 = n2. Thus, we are in Case 3, since f (n) is Ω(n2+ε)
(for ε = 1/2) and a f (n/b) = 9(n/3)2.5 = (1/3)1/2 f (n). This means that T (n) is
Θ(n2.5) by the master method.

Example 5.11: Finally, consider the recurrence
T (n) = 2T (n1/2) + log n.

Unfortunately, this equation is not in a form that allows us to use the master method.
We can put it into such a form, however, by introducing the variable k = log n,
which lets us write

T (n) = T (2k) = 2T (2k/2) + k.
Substituting into this the equation S(k) = T (2k), we get that

S(k) = 2S(k/2) + k.

Now, this recurrence equation allows us to use master method, which specifies that
S(k) is O(k log k). Substituting back for T (n) implies T (n) is O(log n log log n).

Rather than rigorously prove Theorem 5.6, we instead discuss the justification
behind the master method at a high level.

270 Chapter 5. Fundamental Techniques

If we apply the iterative substitution method to the general divide-and-conquer
recurrence equation, we get

T (n) = aT (n/b) + f (n)

= a(aT (n/b2) + f (n/b)) + f (n) = a2T (n/b2) + a f (n/b) + f (n)

= a3T (n/b3) + a2 f (n/b2) + a f (n/b) + f (n)

...

logb n−1
∑= alogb nT (1) + ai f (n/bi)

i=0

logb n−1 ai f (n/bi),
∑= nlogb aT (1) +
i=0

where the last substitution is based on the identity alogb n = nlogb a. Indeed, this

equation is where the special function comes from. Given this closed-form char-

acterization of T (n), we can intuitively see how each of the three cases is derived.

Case 1 comes from the situation when f (n) is small and the first term above domi-

nates. Case 2 denotes the situation when each of the terms in the above summation

is proportional to the others, so the characterization of T (n) is f (n) times a loga-

rithmic factor. Finally, Case 3 denotes the situation when the first term is smaller

than the second and the summation above is a sum of geometrically decreasing

terms that start with f (n); hence, T (n) is itself proportional to f (n).

The proof of Theorem 5.6 formalizes this intuition, but instead of giving the

details of this proof, we present two applications of the master method below.

5.2.2 Integer Multiplication

We consider, in this subsection, the problem of multiplying big integers, that is,
integers represented by a large number of bits that cannot be handled directly by
the arithmetic unit of a single processor. Multiplying big integers has applications
to data security, where big integers are used in encryption schemes.

Given two big integers I and J represented with n bits each, we can easily
compute I + J and I − J in O(n) time. Efficiently computing the product I · J using
the common grade-school algorithm requires, however, O(n2) time. In the rest
of this section, we show that by using the divide-and-conquer technique, we can
design a subquadratic-time algorithm for multiplying two n-bit integers.

Let us assume that n is a power of two (if this is not the case, we can pad I and J
with 0’s). We can therefore divide the bit representations of I and J in half, with one
half representing the higher-order bits and the other representing the lower-order
bits. In particular, if we split I into Ih and Il and J into Jh and Jl, then

I = Ih2n/2 + Il,
J = Jh2n/2 + Jl.

5.2. Divide-and-Conquer 271

Also, observe that multiplying a binary number I by a power of two, 2k, is
trivial—it simply involves shifting left (that is, in the higher-order direction) the
number I by k bit positions. Thus, provided a left-shift operation takes constant
time, multiplying an integer by 2k takes O(k) time.

Let us focus on the problem of computing the product I ·J. Given the expansion
of I and J above, we can rewrite I · J as

I · J = (Ih2n/2 + Il) · (Jh2n/2 + Jl) = IhJh2n + IlJh2n/2 + IhJl2n/2 + IlJl.

Thus, we can compute I · J by applying a divide-and-conquer algorithm that divides
the bit representations of I and J in half, recursively computes the product four
products of n/2 bits each (as described above), and then merges the solutions to
these subproducts in O(n) time using addition and multiplication by powers of two.
We can terminate the recursion when we get down to the multiplication of two 1-bit
numbers, which is trivial. This divide-and-conquer algorithm has a running time
that can be characterized by the following recurrence (for n ≥ 2):

T (n) = 4T (n/2) + cn,

for some constant c > 0. We can then apply the master theorem to note that the
special function nlogb a = nlog2 4 = n2 in this case; hence, we are in Case 1 and T (n)
is Θ(n2). Unfortunately, this is no better than the grade-school algorithm.

The master method gives us some insight into how we might improve this al-
gorithm. If we can reduce the number of recursive calls, then we will reduce the
complexity of the special function used in the master theorem, which is currently
the dominating factor in our running time. Fortunately, if we are a little more clever
in how we define subproblems to solve recursively, we can in fact reduce the num-
ber of recursive calls by one. In particular, consider the product

(Ih − Il) · (Jl − Jh) = IhJl − IlJl − IhJh + IlJh.

This is admittedly a strange product to consider, but it has an interesting property.
When expanded out, it contains two of the products we want to compute (namely,
IhJl and IlJh) and two products that can be computed recursively (namely, IhJh and
IlJl). Thus, we can compute I · J as follows:

I · J = IhJh2n + [(Ih − Il) · (Jl − Jh) + IhJh + IlJl]2n/2 + IlJl.

This computation requires the recursive computation of three products of n/2 bits
each, plus O(n) additional work. Thus, it results in a divide-and-conquer algorithm
with a running time characterized by the following recurrence equation (for n ≥ 2):

T (n) = 3T (n/2) + cn,

for some constant c > 0.

Theorem 5.12: We can multiply two n-bit integers in O(n1.585) time.

Proof: We apply the master theorem with the special function nlogb a = nlog2 3;
hence, we are in Case 1 and T (n) is Θ(nlog2 3), which is itself O(n1.585).

272 Chapter 5. Fundamental Techniques

Using divide-and-conquer, we have designed an algorithm for integer multipli-
cation that is asymptotically faster than the straightforward quadratic-time method.
We can actually do even better than this, achieving a running time that is “almost”
O(n log n), by using a more complex divide-and-conquer algorithm called the fast
Fourier transform, which we discuss in Section 10.4.

5.2.3 Matrix Multiplication

Suppose we are given two n × n matrices X and Y , and we wish to compute their
product Z = XY , which is defined so that

e−1

Z[i, j] = ∑ X [i, k] ·Y [k, j],
k=0

which is an equation that immediately gives rise to a simple O(n3) time algorithm.

Another way of viewing this product is in terms of submatrices. That is, let
us assume that n is a power of two and let us partition X , Y , and Z each into four
(n/2) × (n/2) matrices, so that we can rewrite Z = XY as

IJ = AB EF .
KL CD GH

Thus,

I = AE + BG
J = AF + BH
K = CE + DG
L = CF + DH.

We can use this set of equations in a divide-and-conquer algorithm that com-
putes Z = XY by computing I, J, K, and L from the subarrays A through G. By the
above equations, we can compute I, J, K, and L from the eight recursively com-
puted matrix products on (n/2) × (n/2) subarrays, plus four additions that can be
done in O(n2) time. Thus, the above set of equations give rise to a divide-and-
conquer algorithm whose running time T (n) is characterized by the recurrence

T (n) = 8T (n/2) + bn2,

for some constant b > 0. Unfortunately, this equation implies that T (n) is O(n3) by
the master theorem; hence, it is no better than the straightforward matrix multipli-
cation algorithm.

Interestingly, there is an algorithm known as Strassen’s Algorithm, that orga-
nizes arithmetic involving the subarrays A through G so that we can compute I, J,
K, and L using just seven recursive matrix multiplications. It is somewhat myste-
rious how Strassen discovered these equations, but we can easily verify that they
work correctly.

5.2. Divide-and-Conquer 273

We begin Strassen’s Algorithm by defining seven submatrix products:

S1 = A(F − H)
S2 = (A + B)H
S3 = (C + D)E
S4 = D(G − E)
S5 = (A + D)(E + H)
S6 = (B − D)(G + H)
S7 = (A −C)(E + F).

Given these seven submatrix products, we can compute I as

I = S5 + S6 + S4 − S2
= (A + D)(E + H) + (B − D)(G + H) + D(G − E) − (A + B)H
= AE + DE + AH + DH + BG − DG + BH − DH + DG − DE − AH − BH
= AE + BG.

We can compute J as

J = S1 + S2
= A(F − H) + (A + B)H
= AF − AH + AH + BH
= AF + BH.

We can compute K as

K = S3 + S4
= (C + D)E + D(G − E)
= CE + DE + DG − DE
= CE + DG.

Finally, we can compute L as

L = S1 − S7 − S3 + S5
= A(F − H) − (A −C)(E + F) − (C + D)E + (A + D)(E + H)
= AF − AH − AE +CE − AF +CF −CE − DE + AE + DE + AH + DH
= CF + DH.

Thus, we can compute Z = XY using seven recursive multiplications of matrices of
size (n/2) × (n/2). Thus, we can characterize the running time T (n) as

T (n) = 7T (n/2) + bn2,

for some constant b > 0. Thus, by the master theorem, we have the following:

Theorem 5.13: We can multiply two n × n matrices in O(nlog7) time.

Thus, with a fair bit of additional complication, we can perform the multiplica-
tion for n × n matrices in time O(n2.808), which is o(n3) time. As admittedly com-
plicated as Strassen’s matrix multiplication is, there are actually much more com-
plicated matrix multiplication algorithms, with running times as low as O(n2.376).

274 Chapter 5. Fundamental Techniques

5.3 Dynamic Programming

In this section, we discuss the dynamic programming algorithm-design technique.
This technique is similar to the divide-and-conquer technique, in that it can be
applied to a wide variety of different problems. Conceptually, the dynamic pro-
gramming technique is different from divide-and-conquer, however, because the
divide-and-conquer technique can be easily explained in a sentence or two, and can
be well illustrated with a single example. Dynamic programming takes a bit more
explaining and multiple examples before it can be fully appreciated.

The extra effort needed to fully appreciate dynamic programming is well worth
it, though. There are few algorithmic techniques that can take problems that seem
to require exponential time and produce polynomial-time algorithms to solve them.
Dynamic programming is one such technique. In addition, the algorithms that re-
sult from applications of the dynamic programming technique are usually quite
simple—often needing little more than a few lines of code to describe some nested
loops for filling in a table.

5.3.1 Matrix Chain-Product

Rather than starting out with an explanation of the general components of the dy-
namic programming technique, we start out instead by giving a classic, concrete
example. Suppose we are given a collection of n two-dimensional matrices for
which we wish to compute the product

A = A0 · A1 · A2 · · · An−1,
where Ai is a di × di+1 matrix, for i = 0, 1, 2, . . . , n − 1. In the standard matrix
multiplication algorithm (which is the one we will use), to multiply a d × e-matrix B
times an e × f -matrix C, we compute the (i, j)th entry of the product as

e−1

∑ B[i, k] ·C[k, j].

k=0

This definition implies that matrix multiplication is associative, that is, it implies
that B · (C · D) = (B · C) · D. Thus, we can parenthesize the expression for A any
way we wish and we will end up with the same answer. We will not necessar-
ily perform the same number of primitive (that is, scalar) multiplications in each
parenthesization, however, as is illustrated in the following example.

Example 5.14: Let B be a 2 × 10-matrix, let C be a 10 × 50-matrix, and let D be
a 50 × 20-matrix. Computing B · (C · D) requires 2 · 10 · 20 + 10 · 50 · 20 = 10400
multiplications, whereas computing (B ·C) · D requires 2 · 10 · 50 + 2 · 50 · 20 = 3000
multiplications.

5.3. Dynamic Programming 275

The matrix chain-product problem is to determine the parenthesization of the
expression defining the product A that minimizes the total number of scalar multi-
plications performed. Of course, one way to solve this problem is to simply enu-
merate all the possible ways of parenthesizing the expression for A and determine
the number of multiplications performed by each one. Unfortunately, the set of all
different parenthesizations of the expression for A is equal in number to the set of
all different binary trees that have n external nodes. This number is exponential in
n. Thus, this straightforward (“brute force”) algorithm runs in exponential time, for
there are an exponential number of ways to parenthesize an associative arithmetic
expression (the number is equal to the nth Catalan number, which is Ω(4n/n3/2)).

Defining Subproblems

We can improve the performance achieved by the brute force algorithm signifi-
cantly, however, by making a few observations about the nature of the matrix chain-
product problem. The first observation is that the problem can be split into subprob-
lems. In this case, we can define a number of different subproblems, each of which
is to compute the best parenthesization for some subexpression Ai · Ai+1 · · · A j. As
a concise notation, we use Ni, j to denote the minimum number of multiplications
needed to compute this subexpression. Thus, the original matrix chain-product
problem can be characterized as that of computing the value of N0,n−1. This obser-
vation is important, but we need one more in order to apply the dynamic program-
ming technique.

Characterizing Optimal Solutions

The other important observation we can make about the matrix chain-product prob-
lem is that it is possible to characterize an optimal solution to a particular subprob-
lem in terms of optimal solutions to its subproblems. We call this property the
subproblem optimality condition.

In the case of the matrix chain-product problem, we observe that, no matter how
we parenthesize a subexpression, there has to be some final matrix multiplication
that we perform. That is, a full parenthesization of a subexpression Ai · Ai+1 · · · A j
has to be of the form (Ai · · · Ak) · (Ak+1 · · · A j), for some k ∈ {i, i + 1, . . . , j − 1}.
Moreover, for whichever k is the right one, the products (Ai · · · Ak) and (Ak+1 · · · A j)
must also be solved optimally. If this were not so, then there would be a global
optimal that had one of these subproblems solved suboptimally. But this is impos-
sible, since we could then reduce the total number of multiplications by replacing
the current subproblem solution by an optimal solution for the subproblem. This
observation implies a way of explicitly defining the optimization problem for Ni, j
in terms of other optimal subproblem solutions. Namely, we can compute Ni, j by
considering each place k where we could put the final multiplication and taking the
minimum over all such choices.

276 Chapter 5. Fundamental Techniques

Designing a Dynamic Programming Algorithm

The above discussion implies that we can characterize the optimal subproblem so-
lution Ni, j as

Ni, j = i≤mki<nj{Ni,k + Nk+1, j + didk+1d j+1},

where we note that

Ni,i = 0,

since no work is needed for a subexpression comprising a single matrix. That is, Ni, j
is the minimum, taken over all possible places to perform the final multiplication,
of the number of multiplications needed to compute each subexpression plus the
number of multiplications needed to perform the final matrix multiplication.

The equation for Ni, j looks similar to the recurrence equations we derive for
divide-and-conquer algorithms, but this is only a superficial resemblance, for there
is an aspect of the equation Ni, j that makes it difficult to use divide-and-conquer
to compute Ni, j. In particular, there is a sharing of subproblems going on that
prevents us from dividing the problem into completely independent subproblems
(as we would need to do to apply the divide-and-conquer technique). We can,
nevertheless, use the equation for Ni, j to derive an efficient algorithm by computing
Ni, j values in a bottom-up fashion, and storing intermediate solutions in a table of
Ni, j values. We can begin simply enough by assigning Ni,i = 0 for i = 0, 1, . . . , n− 1.
We can then apply the general equation for Ni, j to compute Ni,i+1 values, since
they depend only on Ni,i and Ni+1,i+1 values, which are available. Given the Ni,i+1
values, we can then compute the Ni,i+2 values, and so on. Therefore, we can build
Ni, j values up from previously computed values until we can finally compute the
value of N0,n−1, which is the number that we are searching for. The details of this
dynamic programming solution are given in Algorithm 5.5.

Algorithm MatrixChain(d0, . . . , dn):
Input: Sequence d0, . . . , dn of integers
Output: For i, j = 0, . . . , n − 1, the minimum number of multiplications Ni, j
needed to compute the product Ai · Ai+1 · · · A j, where Ak is a dk × dk+1 matrix
for i ← 0 to n − 1 do
Ni,i ← 0
for b ← 1 to n − 1 do
for i ← 0 to n − b − 1 do
j ← i+b
Ni, j ← +∞
for k ← i to j − 1 do
Ni, j ← min{Ni, j, Ni,k + Nk+1, j + didk+1d j+1}.

Algorithm 5.5: Dynamic programming algorithm for the matrix chain-product
problem.

5.3. Dynamic Programming 277
Analyzing the Matrix Chain-Product Algorithm

Thus, we can compute N0,n−1 with an algorithm that consists primarily of three
nested for-loops. The outside loop is executed n times. The loop inside is exe-

cuted at most n times. And the inner-most loop is also executed at most n times.
Therefore, the total running time of this algorithm is O(n3).

Theorem 5.15: Given a chain-product of n two-dimensional matrices, we can
compute a parenthesization of this chain that achieves the minimum number of
scalar multiplications in O(n3) time.

Proof: We have shown above how we can compute the optimal number of scalar
multiplications. But how do we recover the actual parenthesization?

The method for computing the parenthesization itself is is actually quite straight-
forward. We modify the algorithm for computing Ni, j values so that any time we
find a new minimum value for Ni, j, we store, with Ni, j, the index k that allowed us
to achieve this minimum.

In Figure 5.6, we illustrate the way the dynamic programming solution to the
matrix chain-product problem fills in the array N.

Nj

i i,j
i,k

+ didk+1dj+1 k+1,j

Figure 5.6: Illustration of the way the matrix chain-product dynamic-programming
algorithm fills in the array N.

Now that we have worked through a complete example of the use of the dy-
namic programming method, let us discuss the general aspects of the dynamic pro-
gramming technique as it can be applied to other problems.

278 Chapter 5. Fundamental Techniques

5.3.2 The General Technique

The dynamic programming technique is used primarily for optimization problems,
where we wish to find the “best” way of doing something. Often the number of
different ways of doing that “something” is exponential, so a brute-force search
for the best is computationally infeasible for all but the smallest problem sizes.
We can apply the dynamic programming technique in such situations, however, if
the problem has a certain amount of structure that we can exploit. This structure
involves the following three components:

Simple Subproblems: There has to be some way of breaking the global optimiza-
tion problem into subproblems, each having a similar structure to the original
problem. Moreover, there should be a simple way of defining subproblems
with just a few indices, like i, j, k, and so on.

Subproblem Optimality: An optimal solution to the global problem must be a
composition of optimal subproblem solutions, using a relatively simple com-
bining operation. We should not be able to find a globally optimal solution
that contains suboptimal subproblems.

Subproblem Overlap: Optimal solutions to unrelated subproblems can contain
subproblems in common. Indeed, such overlap improves the efficiency of a
dynamic programming algorithm that stores solutions to subproblems.

Now that we have given the general components of a dynamic programming
algorithm, we next give another example of its use.

5.3.3 The 0-1 Knapsack Problem

Suppose a hiker is about to go on a trek through a rain forest carrying a single
knapsack. Suppose further that she knows the maximum total weight W that she
can carry, and she has a set S of n different useful items that she can potentially take
with her, such as a folding chair, a tent, and a copy of this book. Let us assume that
each item i has an integer weight wi and a benefit value bi, which is the utility value
that our hiker assigns to item i. Her problem, of course, is to optimize the total
value of the set T of items that she takes with her, without going over the weight
limit W . That is, she has the following objective:

maximize ∑ bi subject to ∑ wi ≤ W.
i∈T i∈T

Her problem is an instance of the 0-1 knapsack problem. This problem is called
a “0-1” problem, because each item must be entirely accepted or rejected. We
consider the fractional version of this problem in Section 5.1.1, and we study how
knapsack problems arise in the context of Internet auctions in Exercise R-5.12.

5.3. Dynamic Programming 279

A First Attempt at Characterizing Subproblems

We can easily solve the 0-1 knapsack problem in Θ(2n) time, of course, by enu-
merating all subsets of S and selecting the one that has highest total benefit from
among all those with total weight not exceeding W . This would be an inefficient
algorithm, however. Fortunately, we can derive a dynamic programming algorithm
for the 0-1 knapsack problem that runs much faster than this in most cases.

As with many dynamic programming problems, one of the hardest parts of
designing such an algorithm for the 0-1 knapsack problem is to find a nice char-
acterization for subproblems (so that we satisfy the three properties of a dynamic
programming algorithm). To simplify the discussion, number the items in S as
1, 2, . . . , n and define, for each k ∈ {1, 2, . . . , n}, the subset

Sk = {items in S labeled 1, 2, . . . , k}.

One possibility is for us to define subproblems by using a parameter k so that sub-
problem k is the best way to fill the knapsack using only items from the set Sk. This
is a valid subproblem definition, but it is not at all clear how to define an optimal
solution for index k in terms of optimal subproblem solutions. Our hope would be
that we would be able to derive an equation that takes the best solution using items
from Sk−1 and considers how to add the item k to that. Unfortunately, if we stick
with this definition for subproblems, then this approach is fatally flawed. For, as we
show in Figure 5.7, if we use this characterization for subproblems, then an optimal
solution to the global problem may actually contain a suboptimal subproblem.

(a) (3,2) (5,4) (8,5) (4,3)

(b) (3,2) (5,4) (8,5) (10,9)

20

Figure 5.7: An example showing that our first approach to defining a knapsack
subproblem does not work. The set S consists of five items denoted by the the
(weight, benefit) pairs (3, 2), (5, 4), (8, 5), (4, 3), and (10, 9). The maximum total
weight is W = 20: (a) best solution with the first four items; (b) best solution with
the first five items. We shade each item in proportion to its benefit.

280 Chapter 5. Fundamental Techniques
A Better Subproblem Characterization

One of the reasons that defining subproblems only in terms of an index k is fa-

tally flawed is that there is not enough information represented in a subproblem
to provide much help for solving the global optimization problem. We can correct
this difficulty, however, by adding a second parameter w. Let us therefore formulate
each subproblem as that of computing B[k, w], which is defined as the maximum to-
tal value of a subset of Sk from among all those subsets having total weight exactly
w. We have B[0, w] = 0 for each w ≤ W , and we derive the following relationship
for the general case

B[k, w] = B[k − 1, w] if wk > w

max{B[k − 1, w], B[k − 1, w − wk] + bk} else.

That is, the best subset of Sk that has total weight w is either the best subset of Sk−1
that has total weight w or the best subset of Sk−1 that has total weight w − wk plus
the item k. Since the best subset of Sk that has total weight w must either contain
item k or not, one of these two choices must be the right choice. Thus, we have

a subproblem definition that is simple (it involves just two parameters, k and w)

and satisfies the subproblem optimization condition. Moreover, it has subproblem

overlap, for the optimal way of summing exactly w to weight may be used by many

future subproblems.

In deriving an algorithm from this definition, we can make one additional ob-
servation, namely, that the definition of B[k, w] is built from B[k − 1, w] and possibly
B[k −1, w−wk]. Thus, we can implement this algorithm using only a single array B,
which we update in each of a series of iterations indexed by a parameter k so that at
the end of each iteration B[w] = B[k, w]. This gives us Algorithm 5.8 (01Knapsack).

Algorithm 01Knapsack(S,W ):
Input: Set S of n items, such that item i has positive benefit bi and positive
integer weight wi; positive integer maximum total weight W
Output: For w = 0, . . . ,W , maximum benefit B[w] of a subset of S with total
weight w

for w ← 0 to W do
B[w] ← 0

for k ← 1 to n do
for w ← W downto wk do
if B[w − wk] + bk > B[w] then
B[w] ← B[w − wk] + bk

Algorithm 5.8: Dynamic programming algorithm for solving the 0-1 knapsack
problem.

5.3. Dynamic Programming 281

Analyzing the 0-1 Knapsack Dynamic Programming Algorithm

The running time of the 01Knapsack algorithm is dominated by the two nested
for-loops, where the outer one iterates n times and the inner one iterates at most W
times. After it completes we can find the optimal value by locating the value B[w]
that is greatest among all w ≤ W . Thus, we have the following:

Theorem 5.16: Given an integer W and a set S of n items, each of which has a
positive benefit and a positive integer weight, we can find the highest benefit subset
of S with total weight at most W in O(nW ) time.

Proof: We have given Algorithm 5.8 (01Knapsack) for constructing the value of
the maximum-benefit subset of S that has total weight at most W using an array B of
benefit values. We can easily convert our algorithm into one that outputs the items
in a best subset, however. We leave the details of this conversion as an exercise.

Pseudo-Polynomial-Time Algorithms

In addition to being another useful application of the dynamic programming tech-
nique, Theorem 5.16 states something very interesting. Namely, it states that the
running time of our algorithm depends on a parameter W that, strictly speaking, is
not proportional to the size of the input (the n items, together with their weights and
benefits, plus the number W ). Assuming that W is encoded in some standard way
(such as a binary number), then it takes only O(logW ) bits to encode W . Moreover,
if W is very large (say W = 2n), then this dynamic programming algorithm would
actually be asymptotically slower than the brute force method. Thus, technically
speaking, this algorithm is not a polynomial-time algorithm, for its running time is
not actually a function of the size of the input.

It is common to refer to an algorithm such as our knapsack dynamic program-
ming algorithm as being a pseudo-polynomial time algorithm, for its running time
depends on the magnitude of a number given in the input, not its encoding size. In
practice, such algorithms should run much faster than any brute-force algorithm,
but it is not correct to say they are true polynomial-time algorithms. In fact, there is
a theory known as NP-completeness, which is discussed in Chapter 13, that states
that it is very unlikely that anyone will ever find a true polynomial-time algorithm
for the 0-1 knapsack problem.

Elsewhere in this book, we give additional applications of the dynamic pro-
gramming technique for computing reachability in a directed graph (Section 6.4.2)
and for testing the similarity of two strings (Section 9.4).

282 Chapter 5. Fundamental Techniques

5.4 Exercises

Reinforcement

R-5.1 Let S = {a, b, c, d, e, f , g} be a collection of objects with benefit-weight values
as follows: a: (12, 4), b: (10, 6), c: (8, 5), d: (11, 7), e: (14, 3), f : (7, 1), g: (9, 6).
What is an optimal solution to the fractional knapsack problem for S assuming

we have a sack that can hold objects with total weight 18? Show your work.

R-5.2 Describe how to implement the TaskSchedule method to run in O(n log n) time.

R-5.3 Suppose we are given a set of tasks specified by pairs of the start times and finish
times as T = {(1, 2), (1, 3), (1, 4), (2, 5), (3, 7), (4, 9), (5, 6), (6, 8), (7, 9)}. Solve
the task scheduling problem for this set of tasks.

R-5.4 Characterize each of the following recurrence equations using the master method

(assuming that T (n) = c for n < d, for constants c > 0 and d ≥ 1).

a. T (n) = 2T (n/2) + logn
b. T (n) = 8T (n/2) + n2
c. T (n) = 16T (n/2) + (n logn)4
d. T (n) = 7T (n/3) + n
e. T (n) = 9T (n/3) + n3 log n

R-5.5 Use the divide-and-conquer algorithm, from Section 5.2.2, to compute 10110011·
10111010 in binary. Show your work.

R-5.6 Use Strassen’s matrix multiplication algorithm to multiply the matrices

X= 32 and Y= 15 .
48 96

√
R-5.7 A complex number a + bi, where i = −1, can be represented by the pair (a, b).

Describe a method performing only three real-number multiplications to compute

the pair (e, f ) representing the product of a + bi and c + di.

R-5.8 Boolean matrices are matrices such that each entry is 0 or 1, and matrix multipli-

cation is performed by using AND for · and OR for +. Suppose we are given two
n × n random Boolean matrices A and B, so that the probability that any entry

in either is 1, is 1/k. Show that if k is a constant, then there is an algorithm for
multiplying A and B whose expected running time is O(n2). What if k is n?

R-5.9 What is the best way to multiply a chain of matrices with dimensions that are
10 × 5, 5 × 2, 2 × 20, 20 × 12, 12 × 4, and 4 × 60? Show your work.

R-5.10 Design an efficient algorithm for the matrix chain multiplication problem that
outputs a fully parenthesized expression for how to multiply the matrices in the
chain using the minimum number of operations.

R-5.11 Solve Exercise R-5.1 for the 0-1 knapsack problem.

R-5.12 Sally is hosting an Internet auction to sell n widgets. She receives m bids, each
of the form “I want ki widgets for di dollars,” for i = 1, 2, . . . , m. Characterize
her optimization problem as a knapsack problem. Under what conditions is this

a 0-1 versus fractional problem?

5.4. Exercises 283

Creativity

C-5.1 A native Australian named Anatjari wishes to cross a desert carrying only a sin-
gle water bottle. He has a map that marks all the watering holes along the way.
Assuming he can walk k miles on one bottle of water, design an efficient algo-
rithm for determining where Anatjari should refill his bottle in order to make as
few stops as possible. Argue why your algorithm is correct.

C-5.2 Consider the single machine scheduling problem where we are given a set T
of tasks specified by their start times and finish times, as in the task scheduling
problem, except now we have only one machine and we wish to maximize the
number of tasks that this single machine performs. Design a greedy algorithm
for this single machine scheduling problem and show that it is correct. What is
the running time of this algorithm?

C-5.3 Describe an efficient greedy algorithm for making change for a specified value
using a minimum number of coins, assuming there are four denominations of
coins (called quarters, dimes, nickels, and pennies), with values 25, 10, 5, and 1,
respectively. Argue why your algorithm is correct.

C-5.4 Give an example set of denominations of coins so that a greedy change making
algorithm will not use the minimum number of coins.

C-5.5 In the art gallery guarding problem we are given a line L that represents a long
hallway in an art gallery. We are also given a set X = {x0, x1, . . . , xn−1} of real
numbers that specify the positions of paintings in this hallway. Suppose that a
single guard can protect all the paintings within distance at most 1 of his or her
position (on both sides). Design an algorithm for finding a placement of guards
that uses the minimum number of guards to guard all the paintings with positions
in X.

C-5.6 Design a divide-and-conquer algorithm for finding the minimum and the maxi-
mum element of n numbers using no more than 3n/2 comparisons.

C-5.7 Given a set P of n teams in some sport, a round-robin tournament is a collection
of games in which each team plays each other team exactly once. Design an
efficient algorithm for constructing a round-robin tournament assuming n is a
power of 2.

C-5.8 Let a set of intervals S = {[a0, b0], [a1, b1], . . . , [an−1, bn−1]} of the interval [0, 1]
be given, with 0 ≤ ai < bi ≤ 1, for i = 0, 1, . . . , n − 1. Suppose further that we
assign a height hi to each interval [ai, bi] in S. The upper envelope of S is defined
to be a list of pairs [(x0, c0), (x1, c1), (x2, c2), . . . , (xm, cm), (xm+1, 0)], with x0 =
0 and xm+1 = 1, and ordered by xi values, such that, for each subinterval s =
[xi, xi+1] the height of the highest interval in S containing s is ci, for i = 0, 1, . . . , m.
Design an O(n log n)-time algorithm for computing the upper envelope of S.

C-5.9 How can we modify the dynamic programming algorithm from simply comput-
ing the best benefit value for the 0-1 knapsack problem to computing the assign-
ment that gives this benefit?

C-5.10 Suppose we are given a collection A = {a1, a2, . . . , an} of n positive integers that
add up to N. Design an O(nN)-time algorithm for determining whether there is
a subset B ⊂ A, such that ∑ai∈B ai = ∑ai∈A−B ai.

284 Chapter 5. Fundamental Techniques

C-5.11 Let P be a convex polygon (Section 12.5.1). A triangulation of P is an addition
of diagonals connecting the vertices of P so that each interior face is a triangle.
The weight of a triangulation is the sum of the lengths of the diagonals. Assuming
that we can compute lengths and add and compare them in constant time, give an
efficient algorithm for computing a minimum-weight triangulation of P.

C-5.12 A grammar G is a way of generating strings of “terminal” characters from a
nonterminal symbol S, by applying simple substitution rules, called productions.
If B → β is a production, then we can convert a string of the form αBγ into the
string αβγ. A grammar is in Chomsky normal form if every production is of the
form “A → BC” or “A → a,” where A, B, and C are nonterminal characters and a
is a terminal character. Design an O(n3)-time dynamic programming algorithm
for determining if string x = x0x1 · · · xn−1 can be generated from start symbol S.

C-5.13 Suppose we are given an n-node rooted tree T , such that each node v in T is given
a weight w(v). An independent set of T is a subset S of the nodes of T such that
no node in S is a child or parent of any other node in S. Design an efficient
dynamic programming algorithm to find the maximum-weight independent set
of the nodes in T , where the weight of a set of nodes is simply the sum of the
weights of the nodes in that set. What is the running time of your algorithm?

Projects

P-5.1 Design and implement a big integer package supporting the four basic arithmetic
operations.

P-5.2 Implement a system for efficiently solving knapsack problems. Your system
should work for either fractional or 0-1 knapsack problems. Perform an experi-
mental analysis to test the efficiency of your system.

Chapter Notes

The term “greedy algorithm” was coined by Edmonds [64] in 1971, although the concept
existed before then. For more information about the greedy method and the theory that
supports it, which is known as matroid theory, please see the book by Papadimitriou and
Steiglitz [164].

The divide-and-conquer technique is a part of the folklore of data structure and al-
gorithm design. The master method for solving divide-and-conquer recurrences traces its
origins to a paper by Bentley, Haken, and Saxe [30]. The divide-and-conquer algorithm
for multiplying two large integers in O(n1.585) time is generally attributed to the Russians
Karatsuba and Ofman [111]. The asymptotically fastest known algorithm for multiplying
two n-digit numbers is an FFT-based algorithm by Scho¨ nhage and Strassen [181] that runs
in O(n log n log log n) time.

Dynamic programming was developed in the operations research community and for-
malized by Bellman [26]. The matrix chain-product solution we described is due to God-
bole [78]. The asymptotically fastest method is due to Hu and Shing [101, 102]. The dy-
namic programming algorithm for the knapsack problem is found in the book by Hu [100].
Hirchsberg [95] shows how to solve the longest common substring problem in the same
time given above, but with linear space (see also [56]).



Part

Ii Graph Algorithms

Chapter

Contents

6.1 The Graph Abstract Data Type 289
293
6.1.1 Graph Methods
296
6.2 Data Structures for Graphs
296
6.2.1 The Edge List Structure
299
6.2.2 The Adjacency List Structure
301
6.2.3 The Adjacency Matrix Structure
303
6.3 Graph Traversal. .

6.3:1 Depth-First Search 303

-1 307
........ .6.3.2 Biconnected Components 313

6.3.3 Breadth-Ffrst Search 316
318
6.4 Directed Graphs . . . ......T

6.4.1 Traversing a Diflh 320

6.4.2 Transitive Closure 323

6.4.3 DFS and Garbage Collection : 325

6.4.4 Directed Acyclic Graphs ....... 329

6.5. Java Example: Depth-First Search 329

The Decorator Pattern
.........6.5.1

6.5.2
A:DFS Engine .............. . . . . . . . 330
331Y
6.5.3 The Template Method Design Pattern

6.6 Exercises . . 335

.

288 Chapter 6. Gj

Greek mythology tells of an elaborate labyrinth that was built to hou
monstrous part bull, part man Minotaur This labynnth was so complex that n
beast nor human could escape it. No human, that is, until the Greek hero, TIi
with the help of the king's daughter, Ariadne, decided to implement one
algorithms discussed in this chapter. Theseus fastened a ball of thread to 11w
of the labyrinth and unwound it as he traversed the twisting passages in sé
of the monster. Theseus obviously knew about good algorithm design, for,
finding añd defeating the beast, Theseus eaÑily'follòwed the string back out
labyrinth to the loving arms of Ariadne.

Being able to determine which objects, such as labyrinth passages, are
nected to which other objects may not always be as vitally important as it w
this story, but it is nevertheless fundamental Connectivity information is prese ii
for example, in city maps, where the objects are roads, and also in the routinjft

bIes for the Internet, where the objécts are computers. Connectivity informa.

is also present in the parent-child relationships defined by a binary tree, where
objects are tree nodes Indeed, connectivity information can be defined by all n
of relationships that exist between pairs of objects The topic we study in ,

chaptergraphsis therefore focused on representatioñs and algorithms for4

ing efficiently with such relationships That is, a graph is a set of objects, c
vertices, together with a collection of pairwise connections between them By
way, this notion of a "graph" should not be confused with bar charts and func
plots, as these kinds of "graphs" are unrelated to the topic of this chapter.

Graphs have applications in a host of different domains, including mapp
(in geographic information systems), transportation (in road and flight networ
electncal engineenng (in circuits), and computer networking (in the connectiö
of the Internet) Because applications for graphs are so widespread and divers
people have developed a great deal of terminology to describe different componé
and properties of graphs Fortunately, since most graph applications are relativé
recent developments, this terminology is fairly intuitive.

We therefore begin this chapter by reviewing muòh of this terminology andp
senting the graph ADT, including some elementary properties of graphs Havi
given the graph ADT, we then present, in Section 6 2, three main data structuf
for representing graphs As with trees, traversals are important computations fo
graphs, and we discuss such computations in Section 6 3 We discuss direc
graphs in Section 6.4; where relationships have a given direction, and connecthì
problems become much more interesting. Finally, in Section 6.5, we give a c
study of depth-first search in Java This case study involves the use of two so
ware engineenng design patternsthe decorator pattern and the template metht
patternas well as a discussion of how depth-first search can be used for garb
collection.