-
Notifications
You must be signed in to change notification settings - Fork 1
/
1_baseball.html
665 lines (596 loc) · 27.2 KB
/
1_baseball.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
<!--The design and styling of this site was taken from http://redbook.io-->
<!DOCTYPE html>
<html>
<head>
<title>Consistency in Distributed Systems</title>
<link href='css/style.css' rel='stylesheet'>
<link href='css/svg.css' rel='stylesheet'>
<link href='css/1_baseball.css' rel='stylesheet'>
<meta name=viewport content="width=device-width, initial-scale=1">
<link rel="shortcut icon" href="favicon.ico">
</head>
<body>
<div id="header">
<div id="grouptitle">
<a href="index.html">Consistency in Distributed Systems</a>
</div>
</div>
<div id="container">
<article>
<h1 id="baseball" class="lecturetitle">
<a href="https://scholar.google.com/scholar?cluster=3008756295145383805">
Replicated Data Consistency Explained Through Baseball
</a>
</h1>
<p>
In this lecture, we'll see why storage systems replicate their data and
how unfettered replication can lead to anomalies. We'll then discuss how
strong and weak consistency can help tame anomalous behavior before
concluding with a review of the 2013 CACM article <em>Replicated Data
Consistency Explained Through Baseball</em>.
</p>
<h2 id="replication">Replication</h2>
<p>
Often, distributed storage systems—like file systems, relational
databases, or key-value stores—store a copy of the same data on
multiple computers. This is known as <strong>replication</strong>. To
motivate why storage systems replicate their data, we'll look at an
example.
</p>
<p>
Consider a non-distributed key-value store running on a single
computer. The key-value store is nothing more than a map (or dictionary)
from string-valued keys to string-valued values. The key-value store
supports a dirt simple interface. Clients can issue
<ul>
<li>
a <code>get(k)</code> request to retrieve the value associated with
key <code>k</code> or
</li>
<li>
a <code>set(k, v)</code> request to associate the value
<code>v</code> with key <code>k</code>.
</li>
</ul>
</p>
<p>
An example interaction between a client (<code>a</code>) and the
key-value store (<code>s</code>) is animated below. The client first
sends a <code>set(x,42)</code> request and then a <code>get(x)</code>
request to the server. The top half of the animation shows how messages
flow between the client and the server while the bottom half traces a
timeline of every request and response.
</p>
<div class="svgbox">
<svg viewBox="0 0 400 105" id="one_client_one_kvs">
</svg>
</div>
<p>
Even though there's only one key-value store, there can be multiple
clients. Below, we animate an interaction where client <code>a</code> and
client <code>b</code> concurrently set and then get a value from the
key-value store <code>s</code>.
</p>
<div class="svgbox">
<svg viewBox="0 0 400 155" id="two_clients_one_kvs">
</svg>
</div>
<p>
Unfortunately, computers don't live forever. Eventually, they crash. In
our example, the entire key-value store is stored on <code>s</code>. This
means that if <code>s</code> were to fail, we would irrevocably lose all
of our data. That's no good! In reality, storage systems—like our
key-value store—replicate data across multiple computers so that
their data survives even when any single computer fails.
</p>
<h2 id="strong_and_weak_consistency">Strong and Weak Consistency</h2>
<p>
Replication allows a distributed storage system to tolerate computer
failures. Unfortunately, a naively replicated storage system can behave
very weirdly. For example, consider a key-value store replicated across
two servers (<code>s1</code> and <code>s2</code>). If a client
(<code>a</code>) issues a <code>set(x,42)</code> request to
<code>s1</code> and then a <code>get(x)</code> request to
<code>s2</code>, the <code>get</code> could return something that's not
42! This is animated below.
</p>
<div class="svgbox">
<svg viewBox="0 0 400 182" id="anomaly1">
</svg>
</div>
<p>
This behavior is bananas! If a client issues a <code>set(x, 42)</code>
request followed by a <code>get(x)</code> request, we expect the
key-value store to return 42. When a storage system exposes an unbridled
number of anomalies like this and acts completely bananas, we colloquially
say it is <strong>inconsistent</strong>.
</p>
<p>
Ideally, a storage system can hide the fact that it's replicated from
clients and act indistinguishably from a storage system running on a
single computer. The replication would increase the system's
fault-tolerance but would not cause it to expose any anomalous behavior
to clients. For example, a <code>get(x)</code> request following a
<code>set(x,42)</code> request would always return 42. When a replicated
storage system behaves indistinguishably from a storage system running on
a single computer, we say it is <strong>strongly consistent</strong>.
</p>
<p>
Reconsider the execution above where client <code>a</code> sends a
<code>set(x,42)</code> command to server <code>s1</code> and then a
<code>get(x)</code> command to server <code>s2</code>. If <code>s1</code>
propagates the effects of the <code>set(x,42)</code> command to
<code>s2</code> before responding to <code>a</code>, then <code>s2</code>
will correctly return 42 when it receives a <code>get(x)</code> request.
By doing this, the system implements strongly consistency. This is
animated below.
</p>
<div class="svgbox">
<svg viewBox="0 0 400 182" id="example_strong_consistency">
</svg>
</div>
<p>
Note that the term "strongly consistent" is not well-defined. It means
different things in different contexts. In some contexts, it might be
used colloquially to express a general notion that a storage system
doesn't act bananas. In other contexts, it might be used as a synonym for
a very formally defined form of consistency like
<strong>linearizability</strong> (which we'll define in the next
lecture).
</p>
<p>
Unfortunately, implementing strong consistency—that is,
implementing a system that is strongly consistent—is both
challenging and costly. The algorithms used to implement strong
consistency are often complex and nuanced. Worse yet, strong consistency
is fundamentally at odds with low-latency and availability. We'll defer
an in-depth discussion of this fact to the next lecture, but we already
saw hints of it in the previous animation. When we made our key-value
store strongly consistent, the <code>set(x, 42)</code> request took
longer than it did when the key-value store was inconsistent.
</p>
<p>
As a consequence, systems often eschew strong consistency in favor of
<strong>weak consistency</strong>. Weakly consistent storage systems do
not behave indistinguishably from storage systems running on a single
computer. While they do expose various anomalous behaviors, weakly
consistent systems do not completely throw consistency out the window.
Instead, they try to sit somewhere between acting completely bananas and
acting with strong consistency. They provide some number of basic
guarantees that are hopefully sufficient to satiate clients.
</p>
<p>
For example, consider one of the weakest forms of weak consistency:
<strong>eventual consistency</strong>. An eventually consistent system
guarantees that if all clients stop issuing requests for a while, then
all the system's replicas will converge to the same state.
</p>
<p>
Let's again reconsider the example execution from above, except this time
we'll look at an eventually consistent key-value store. Each server in
the key-value store buffers write (i.e. <code>set</code>) requests and
propagates them to the other servers every so often. As with the
inconsistent key-value store, a <code>get(x)</code> requests following a
<code>set(x,42)</code> request can return something other than 42. But,
if a client waits long enough, <em>eventually</em> a <code>get(x)</code>
request will return 42. This is animated below.
</p>
<div class="svgbox">
<svg viewBox="0 0 400 182" id="example_weak_consistency">
</svg>
</div>
<h2 id="baseball_intro">Replicated Data Consistency Explained Through Baseball</h2>
<p>
In addition to strong consistency, there are a buffet of flavors (or
models) of weak consistency:
<a href="https://scholar.google.com/scholar?cluster=4308857796184904369">eventual consistency</a>,
<a href="https://scholar.google.com/scholar?cluster=4496511741683930603">strong eventual consistency</a>,
<a href="https://scholar.google.com/scholar?cluster=11945551128380183299">causal consistency</a>,
<a href="https://scholar.google.com/scholar?cluster=16870210484225303236">causal+ consistency</a>,
<a href="https://scholar.google.com/scholar?cluster=4316742817395056095">RedBlue consistency</a>,
etc. Each consistency model exposes various degrees of inconsistency with
various performance characteristics. Some simple ones can be implemented
efficiently but are borderline bananas. Others are more complex but
provide stronger consistency.
</p>
<p>
Is the baffling number of consistency models actually useful? Are the
weaker consistency models too weak? Are the stronger consistency models
too strong?
</p>
<p>
Doug Terry's 2013 CACM article, <em>Replicated Data Consistency Explained
Through Baseball</em>, answers these questions using the game of
baseball. Terry defines six consistency models and then shows how
different clients of a baseball application can benefit from each.
</p>
<h2 id="consistency_models">Consistency Models</h2>
<p>
In this section, we define six consistency models for a replicated
key-value store. As with our examples above, there can be multiple
clients issuing read (i.e. <code>get</code>) and write (i.e.
<code>set</code>) requests to the replicated key-value store
concurrently. We'll also augment <code>get</code> requests by allowing
them to read multiple keys at once. For example, the request <code>get(x,
y)</code> returns values for <code>x</code> and <code>y</code>
simultaneously. This is animated below.
</p>
<div class="svgbox">
<svg viewBox="0 0 400 105" id="multi_get">
</svg>
</div>
<p>
To make life easier for ourselves, we make a couple of simplifying
assumptions. First, we assume that when a client issues a write request,
the effects of the write are eventually propagated to all replicas.
Second, we assume that all write requests are executed in the same order
at all replicas (we'll see how to implement something like this in the
third lecture). With these assumptions in place, the six consistency
models will differ only in how read requests behave.
</p>
<p>
We briefly define each of the consistency models here. In the next
section, we'll give an example illustrating the differences between the
six.
</p>
<ol>
<li>
<strong>Strong Consistency.</strong> With a strongly consistent
key-value store, a <code>get(x<sub>1</sub>,...,x<sub>n</sub>)</code>
request is guaranteed to return the most recently written values of
every key from <code>x<sub>1</sub></code> to
<code>x<sub>n</sub></code>.
</li>
<li>
<strong>Eventual Consistency.</strong> With an eventually consistent
key-value store, a <code>get(x<sub>1</sub>,...,x<sub>n</sub>)</code>
request is guaranteed to return values
<code>v<sub>1</sub>,...,v<sub>n</sub></code> where
<code>v<sub>i</sub></code> is any previously written value of key
<code>x<sub>i</sub></code>. With our assumption that writes are
eventually propagated to all replicas, if clients stop issuing write
requests for a while, reads will (typically) return the most recently
written values.
</li>
<li>
<strong>Consistent Prefix.</strong> Recall our assumption that writes
are executed in the same order on all replicas. With a key-value store
guaranteeing consistent prefixes, a <code>get</code> request is
guaranteed to return values that are consistent with some prefix of
this sequence of writes. Note that for <code>get</code> requests
reading a single value, consistent prefix is equivalent to eventual
consistency.
</li>
<li>
<strong>Bounded Staleness.</strong> With a key-value store guaranteeing
bounded staleness, a <code>get(x<sub>1</sub>,...,x<sub>n</sub>)</code>
request is guaranteed to return values
<code>v<sub>1</sub>,...,v<sub>n</sub></code> where
<code>v<sub>i</sub></code> is some value that key
<code>x<sub>i</sub></code> took on during the last <em>t</em> minutes
for some fixed <em>t</em>.
</li>
<li>
<strong>Monotonic Reads.</strong> With a key-value store guaranteeing
monotonic reads, a client's initial read of value <code>x</code> is
only guaranteed to return some previously written value of
<code>x</code> (this is equivalent to eventual consistency). However,
each subsequent read of <code>x</code> by the same client is guaranteed
to return the same value of <code>x</code> or a more up-to-date value
of <code>x</code> compared with the previous read of <code>x</code>.
</li>
<li>
<strong>Read My Writes.</strong> With a key-value store guaranteeing
read my writes, if a client writes a value <code>v</code> to key
<code>x</code>, then any subsequent reads of <code>x</code> by the same
client will return <code>v</code> or a more recently written value of
<code>x</code>.
</li>
</ol>
<h2 id="game_of_baseball">Baseball as a Sample Application</h2>
<p>
The name of the game is baseball. A baseball game consists of nine
rounds, called innings. During the first half of each inning (called the
top of the inning), the visiting team bats until they make three outs.
Then, during the second half of each inning (called the bottom of the
inning), the home team bats until they make three outs. Whichever team
scores more runs by the end of the game wins.
</p>
<p>
In the rest of this lecture, we'll model a baseball game using the
pseudocode below. The state of a baseball game is stored in a
replicated key-value store. The visiting score is stored under the key
<code>"visitors"</code>, and the home score is stored under the key
<code>"home"</code>.
</p>
<pre><code><strong>set</strong>("visitors", 0);
<strong>set</strong>("home", 0);
for inning = 1 .. 9
outs = 0;
while outs < 3
visiting player bats;
for each run scored
score = <strong>get</strong>("visitors");
<strong>set</strong>("visitors", score + 1);
outs = 0;
while outs < 3
home player bats;
for each run scored
score = <strong>get</strong>("home");
<strong>set</strong>("home", score + 1);
end game;</code></pre>
<p>
For simplicity, we'll assume that a single client (the official
scorekeeper) updates the score; no other clients write to the
<code>"visitors"</code> or <code>"home"</code> key in the key-value
store. To illustrate the differences between the six consistency models
defined above, let's jump into an example baseball game during the middle
of the seventh inning. At this point in the game, the following sequence
of writes has been produced by the official scorekeeper.
</p>
<ul>
<li><code>set("visitors", 0)</code></li>
<li><code>set("home", 0)</code></li>
<li><code>set("home", 1)</code></li>
<li><code>set("visitors", 1)</code></li>
<li><code>set("home", 2)</code></li>
<li><code>set("home", 3)</code></li>
<li><code>set("visitors", 2)</code></li>
<li><code>set("home", 4)</code></li>
<li><code>set("home", 5)</code></li>
</ul>
<p>
This sequence of writes corresponds to the following inning-by-inning
line score. We see that the visiting team scored once in the third and
fifth inning. The home team scored once in the first, third, and fourth
inning and scored twice in the sixth inning. Thus, the visiting team is
behind two runs to five. We abbreviate this score as 2-5, visiting team
first.
</p>
<table id="baseball_scoresheet">
<tr id="baseball_scoresheet_1">
<th> </th> <th>1</th> <th>2</th> <th>3</th> <th>4</th>
<th>5</th> <th>6</th> <th>7</th> <th>8</th> <th>9</th>
<th>RUNS</th>
</tr>
<tr id="baseball_scoresheet_2">
<td id="visitors"><strong>Visitors</strong></td>
<td>0</td> <td>0</td> <td>1</td> <td>0</td> <td>1</td>
<td>0</td> <td>0</td> <td> </td> <td> </td> <td>2</td>
</tr>
<tr id="baseball_scoresheet_3">
<td id="home"><strong>Home</strong></td>
<td>1</td> <td>0</td> <td>1</td> <td>1</td> <td>0</td>
<td>2</td> <td> </td> <td> </td> <td> </td> <td>5</td>
</tr>
</table>
<p>
Now, for each of the six consistency models defined above, we list all
the possible scores that could be returned from issuing a
<code>get("visitors", "home")</code> request.
</p>
<table id="scoresheet_examples">
<tr>
<td><strong>Strong Consistency</strong></td>
<td>2‑5</td>
</tr>
<tr>
<td><strong>Eventual Consistency</strong></td>
<td>
0‑0, 0‑1, 0‑2, 0‑3, 0‑4, 0‑5,
1‑0, 1‑1, 1‑2, 1‑3, 1‑4, 1‑5,
2‑0, 2‑1, 2‑2, 2‑3, 2‑4, 2‑5
</td>
</tr>
<tr>
<td><strong>Consistent Prefix</strong></td>
<td>
0‑0, 0‑1, 1‑1, 1‑2, 1‑3, 2‑3,
2‑4, 2‑5
</td>
</tr>
<tr>
<td>
<strong>Bounded Staleness</strong>
<em>(scores that are at most one inning out-of-date)</em>
</td>
<td>2‑3, 2‑4, 2‑5</td>
</tr>
<tr>
<td>
<strong>Monotonic Reads</strong>
<em>(after reading 1‑3)</em>
</td>
<td>
1‑3, 1‑4, 1‑5, 2‑3, 2‑4, 2‑5
</td>
</tr>
<tr>
<td>
<strong>Read My Writes</strong>
<em>(for the writer)</em>
</td>
<td>2‑5</td>
</tr>
<tr>
<td>
<strong>Read My Writes</strong>
<em>(for anyone other than the writer)</em>
</td>
<td>
0‑0, 0‑1, 0‑2, 0‑3, 0‑4, 0‑5,
1‑0, 1‑1, 1‑2, 1‑3, 1‑4, 1‑5,
2‑0, 2‑1, 2‑2, 2‑3, 2‑4, 2‑5
</td>
</tr>
</table>
<p>
A few things to note:
</p>
<ul>
<li>
With a <em>strongly consistent</em> key-value store, the only possible
result is 2-5: the current score of the game.
</li>
<li>
With an <em>eventually consistent</em> key-value store, the visiting
score can be read as 0, 1, or 2, and the home score can be read as 0,
1, 2, 3, 4, or 5. Note that some of the scores read—like 2-0 or
0-5—were never actually the score of the game at any point in
time.
</li>
<li>
With a key-value store guaranteeing <em>consistent prefixes</em>, the
score that's read may not be the most up-to-date score, but it is
guaranteed to have been the score of the game at some point in the
past.
</li>
<li>
With a key-value store guaranteeing <em>bounded staleness</em> within
the last inning, the visiting score must be read as 2 because the
visiting team did not score in the sixth inning. The home score can be
read as 3, 4, or 5: the three home scores that occurred in the sixth
inning.
</li>
<li>
After reading 1-3 from a key-value store guaranteeing <em>monotonic
reads</em>, the visiting score can be read as 1 or 2, and the home
score can be read as 3, 4, or 5.
</li>
<li>
Since there is only a single client who updates the score, <em>read my
writes</em> is equivalent to strong consistency for the writer and
equivalent to eventual consistency for everyone else.
</li>
</ul>
<h2 id="participants">Read Requirements for Participants</h2>
<p>
In this section, we'll look at six different people—a scorekeeper,
an umpire, a radio reporter, a sportswriter, a statistician, and a stat
watcher—and discuss the weakest consistency model each person
requires. In doing so, we'll see why all six consistency models are
useful.
</p>
<h3 id="scorekeeper">Official Scorekeeper</h3>
<p>
As discussed above, the official scorekeeper is responsible for updating
the score whenever the visiting team or home team scores a run. To ensure
that the score is always correct, the scorekeeper must read the most
up-to-date score just before updating it. This could be accomplished with
strong consistency, but since the scorekeeper is the only one who updates
the score, read my writes (which in this case is equivalent to strong
consistency) suffices.
</p>
<h3 id="umpire">Umpire</h3>
<p>
The umpire officiates the game. After the top of the ninth inning, the
home team finishes the game at bat. If the home team has more runs than
the visiting team at this point, then they are guaranteed to win, no
matter how poorly they perform in the bottom of the ninth inning. In this
event, it is the responsibility of the umpire to end the game early. This
responsibility is codified below.
</p>
<pre><code>if first half of 9th inning complete then
vScore = get("visitors");
hScore = get("home");
if vScore < hScore
end game;</code></pre>
<p>
In order to decide whether or not to end the game, the umpire needs the
current score of the game. Thus, the umpire requires <strong>strong
consistency</strong>.
</p>
<h3 id="radio_reporter">Radio Reporter</h3>
<p>
Every thirty minutes, the radio reporter announces the score of the game
over the radio.
</p>
<pre><code>do {
vScore = get("visitors");
hScore = get("home");
report vScore and hScore;
sleep(30 minutes);
}</code></pre>
<p>
The reported score does not have to be up-to-date; listeners are
accustomed to hearing out-of-date scores reported on the radio. However,
as not to mislead listeners, the radio reporter should not report a score
that never actually occurred. Thus, the radio reporter requires
<strong>consistent prefix</strong>.
</p>
<p>
Moreover, once the radio reporter reports a score, the reporter would
look foolish to announce a previous score. For example, it would be a
faux pas to announce a score of 2-3 (middle of the fifth inning) and then
announce a score of 0-1 (middle of the second inning) 30 minutes later.
Thus, the reporter also requires either <strong>monotonic reads</strong>
or <strong>30-minute bounded staleness</strong>.
</p>
<h3 id="sportswriter">Sportswriter</h3>
<p>
The sportswriter is tasked with writing an article about the game after
it finishes. A typical sportswriter executes the following pseudocode.
</p>
<pre><code>while not end of game {
drink beer;
smoke cigar;
}
go out to dinner;
vScore = get("visitors");
hScore = get("home");
write article;</code></pre>
<p>
The sportswriter's article must be written with the correct final score.
While strong consistency would suffice, the sportswriter can take
advantage of a lengthy dinner. If the sportswriter eats for an hour, then
<strong>1-hour bounded staleness</strong> is equivalent to strong
consistency and meets the sportswriter's needs.
</p>
<h3 id="statistician">Statistician</h3>
<p>
The statistician, say for the home team, is responsible for recording
season-long statistics such as the total number of runs scored. The
statistician stores this statistic in the key-value store under the
<code>"season-runs"</code> key. A couple hours after each game, the
statistician adds the number of runs from the game to the season-long
aggregate.
</p>
<pre><code>Wait for end of game;
score = get("home");
state = get("season-runs");
set("season-runs", stat + score);</code></pre>
<p>
As with the sportswriter, the statistician can use <strong>bounded
staleness</strong> to read the home score. Since the statistician is
the only person who updates the season-long number of runs, the
statistician can use <strong>read my writes</strong> to read the latest
value of <code>"season-runs"</code>, similar to how to official
scorekeeper used read my writes to read the score during the game.
</p>
<h3 id="stat_watcher">Stat Watcher</h3>
<p>
A stat watcher checks on their team's statistics once every day and
discusses them with friends. Because the stat watcher checks the stats
infrequently and because reading up-to-date stats is not essential,
<strong>eventual consistency</strong> suffices for a stat watcher.
</p>
<pre><code>do {
stat = get("season-runs");
discuss stats with friends;
sleep(1 day);
}</code></pre>
</article>
</div>
<script src="js/snap.svg.js"></script>
<script src="js/distributed_systems.js"></script>
<script src="js/1_baseball.js"></script>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-90310997-1', 'auto');
ga('send', 'pageview');
</script>
</body>
</html>