Java Caching with Guava

Java Ca

Charles Fry (fry@

aching with Guava

@google.com)

Google Confidential and Proprietary

Introduction

● The Guava project is an op
Google's core Java librarie
○ Stuff like collections, pri
libraries, string process
○ These are the libraries
on

● The package com.google
our caching libraries
○ Simple, in-memory cach
○ Thread-safe implement
ConcurrentHashMap)
○ No explicit support for d

pen-source release of
es
imitives support, concurrency
sing, & cetera
that other projects are built

e.common.cache contains

hing
tation (internally similar to
)
distributed caching

Google Confidential and Proprietary

Types of Caches

● We provide two types of ca
○ LoadingCache: knows
cache miss occurs
■ LoadingCache.ge
associated with key
○ Cache: does not autom

● We're going to focus on the
usually what you want

aches
s how to load entries when a
et(key) returns the value
y, loading it first if necessary
matically load entries
e loading case here; it's

Google Confidential and Proprietary

Simple Loading Ca

CacheLoader<String, St
new CacheLoader<Stri
public String load
return key.toUpp
}
};

LoadingCache<String, S
CacheBuilder.newBuil
.build(loader);

ache

tring> loader =
ing, String>() {
d(String key) {
perCase();

String> cache =
lder()

Google Confidential and Proprietary

Simple Loading Ca

cache.size(); // retur

cache.getUnchecked("si
// cache miss, invokes
// returns "SIMPLE TES

cache.size(); // retur

cache.getUnchecked("si
// cache hit
// returns "SIMPLE TES

ache

rns 0
imple test");
s CacheLoader
ST"
rns 1
imple test");
ST"

Google Confidential and Proprietary

Concurrency

● Cache instances are intern
to ConcurrentHashMap
○ And are thus thread-saf

● But what happens if multip
request the same key?

● CacheLoader.load will b
each key, regardless of the
threads
○ The result will be return
and inserted into the ca
putIfAbsent

nally implemented very similar
fe
ple threads simultaneously
be invoked a single time for
e number of requesting
ned to all requesting threads
ache using the equivalent of

Google Confidential and Proprietary

Checked Exception

● What if loading causes a ch

CacheLoader<String, St
new CacheLoader<Stri
public String load
throws IOExcep
return loadFromD
}
};

ns

hecked exception?
tring> checkedLoader =
ing, String>() {
d(String key)
ption {
Disk(key);

Google Confidential and Proprietary

Checked Exception

LoadingCache<String, S
CacheBuilder.newBuil
.build(checkedLoad

try {
cache.get(key);

} catch (ExecutionExce
// ensure stack trac
throw new IOExceptio

}

ns

String> cache =
lder()
der);

eption e) {
ce is for this thread
on(e.getCause());

Google Confidential and Proprietary

Weak Keys

● What if the cache keys are
requests), which don't belo
no other references elsewh

LoadingCache<Request,
CacheBuilder.newBuil
.weakKeys()
.build(loader);

● Allow the garbage collector
keys when other reference

● Causes key equality to be
● Cost: 3 new references, ad

e transient objects (e.g.
ong in the cache if there are
here?

Metadata> cache =
lder()

r to immediately collect cache
es are gone
determined using ==
dding 16 bytes per entry

Google Confidential and Proprietary

Eviction

● So far the caches we've sh
bound

● CacheBuilder can autom
on various criteria

hown you will grow without
matically evict elements based

Google Confidential and Proprietary

Eviction: Maximum

LoadingCache<String, S
CacheBuilder.newBuil
.maximumSize(200)
.build(loader);

● Elements will be evicted in
● Costs:

○ Every access now beco
record access order)

○ Evictions occur on write
○ 2 new references, in a d

adding 16 bytes per ent

m Size

String> cache =
lder()

n approximate LRU order
omes a lightweight write (to
e operations
doubly-linked access queue,
try

Google Confidential and Proprietary

Eviction: Maximum

Weigher<String, String
new Weigher<String,
public int weigh(
String key, St
return value.len
}
};

LoadingCache<String, S
CacheBuilder.newBuil
.maximumWeight(200
.weigher(weighByLe
.build(loader);

m Weight

g> weighByLength =
String>() {

tring value) {
ngth();

String> cache =
lder()
00)
ength)

Google Confidential and Proprietary

Eviction: Maximum

● Eviction order is the same
○ In fact they share the sa
○ However more than one
time (making room for a

● Weight is only measured o
to the cache

● Weight is only used to dete
over capacity; not for selec

m Weight

as maximumSize
ame data structure (and cost)
e entry may be evicted at a
a single large entry)
once, when an entry is added
ermine whether the cache is
cting what to evict

Google Confidential and Proprietary

Cache Stats

● With an automatic eviction
wonder about cache perfor
○ What ratio of requests a
cache?
○ How much time is spen

● These and other questions

LoadingCache<String, S
CacheBuilder.newBuil
.recordStats()
.build(loader);

// cumulative stats si
CacheStats stats = cac

policy in play, one starts to
rmance
are served directly from

nt loading entries?
s can be answered with:

String> cache =
lder()

ince cache creation
che.stats();

Google Confidential and Proprietary

Cache Stats

CacheStats stats = cac
stats.hitRate();
stats.missRate();
stats.loadExceptionRat
stats.averageLoadPenal

CacheStats delta = cac
.minus(stats);

delta.hitCount();
delta.missCount();
delta.loadSuccessCount
delta.loadExceptionCou
delta.totalLoadTime();

che.stats();

te();
lty();
che.stats()

t(); Google Confidential and Proprietary
unt();
;

Eviction: Time to Id

LoadingCache<String, S
CacheBuilder.newBuil
.expireAfterAccess
.build(loader);

● Elements will expire after th
since the most recent acce

● Eviction order is the same
○ They share the same d
○ However cache size wil
○ Evictions performed on

● Cost: 2 new references, in
adding 16 bytes per entry

● Tests can advance time wi

dle

String> cache =
lder()
s(2, TimeUnit.MINUTES)

he specified time has elapsed
ess
as maximumSize
data structure (and cost)
ll be dynamic instead of static
n read or write operations

a doubly-linked write queue,

ith CacheBuilder.tickerGoogle Confidential and Proprietary

Eviction: Time to L

LoadingCache<String, S
CacheBuilder.newBuil
.expireAfterWrite(
.build(loader);

● Elements will expire after th
since the entry's creation o

● Useful for dropping stale da
○ Unlike other expiration
data correctness than r

● Cost: 2 new references, in
adding 16 bytes per entry

Live

String> cache =
lder()
(2, TimeUnit.MINUTES)

he specified time has elapsed
or update
ata from the cache
strategies this is more about
resource conservation

a doubly-linked write queue,

Google Confidential and Proprietary

Eviction: Soft Valu

LoadingCache<String, S
CacheBuilder.newBuil
.softValues()
.build(loader);

● Allow the garbage collector
○ VMs "bias against clear
recently-used soft refer
○ But in practice "SoftRef
for at least one GC afte

● Cost: 4 new references, ad
● Performance: O(?), large p

very adversely affected by
○ Consider maximumSiz

ues

String> cache =
lder()

r to collect cached values
ring recently-created or
rences"
ferences will always be kept
er their last access"
dding 16 bytes per entry
production systems can be
many soft references
ze instead (or also) Google Confidential and Proprietary

Cache Configuratio

● CacheStats give insight i
open the door for optimizin

● Cache configuration param
without recompiling code w

// from command-line f
String spec =

"maximumSize=200,exp
LoadingCache<String, S

CacheBuilder.from(sp
.build(loader);

on

into cache performance, and
ng the cache configuration
meters can be changed
with CacheBuilderSpec

flag or config file

pireAfterWrite=2m";
String> cache =
pec)

Google Confidential and Proprietary

Removal Notificatio

● Sometimes cached entries
resources which need to be

● Removal notifications can b
is removed from the cache
and value (if available) and

LoadingCache<String, S
CacheBuilder.newBuil
.maximumSize(200)
.removalListener(l
.build(loader);

ons

s are associated with
e closed or cleaned up
be sent for each entry which
e, containing the removed key
d the cause of removal

String> cache =
lder()

listener)

Google Confidential and Proprietary

Removal Notificatio

RemovalListener<String, St
new RemovalListener<St
public void onRemova
RemovalNotificat
if (n.wasEvicted()
cleanupEntry(n.g
}
}
};

ons

tring> listener =
tring, String>() {
al(
tion<String, String> n) {
)) {
getKey(), n.getValue());

Google Confidential and Proprietary

Removal Notificatio

● Removal notifications inclu
it is generally sufficient to c

● Removal listeners are calle
operations
○ Consider implementing
asynchronously (or wra
RemovalListeners.a

● Removal listeners shouldn
elements back into the cac

ons

ude a RemovalCause, though
check wasEvicted()
ed synchronously during user

g RemovalListener
apping with
asynchronous)
n't blindly re-insert removed
che

Google Confidential and Proprietary

Refreshing Stale E

● We've already seen how e
remove stale entries

● In cases where stale data s
data is being loaded, the m
refresh(K) can be used
○ Reload will be performe
reload(K, V), which
asynchronously
○ Reload can take the old
consideration for higher
○ The stale value will con
reload completes

Entries

expireAfterWrite can

should be served while fresh
method LoadingCache.
d to request a reload
ed by calling CacheLoader.
h can be implemented

d cached value into
r efficiency
ntinue to be returned until

Google Confidential and Proprietary

Automatic Refresh

● Alternatively, stale entries c
refreshed

LoadingCache<String, S
CacheBuilder.newBuil
.refreshAfterWrite
.build(loader);

● We call LoadingCache.r
get is called after the timeo

● Inactive entries will not be
○ Couple with expireAf

h

can be automatically

String> cache =
lder()
e(2, TimeUnit.MINUTES)

refresh for you the first time
out
proactively refreshed
fterWrite to purge these

Google Confidential and Proprietary

Automatic Refresh

● Benefits of automatic refres
with stale data:
○ Reload can be optimize
cached value
○ The stale value will con
reload (rather than bloc
○ Reload can be impleme
decreasing cache laten

h

sh over expiration for dealing
ed based on the previous
ntinue to be served during
cking other threads)
ented asynchronously,
ncy

Google Confidential and Proprietary