Quelea

This benchmark implements a simple replicated BankAccount data type supporting three operations - Deposit, Withdraw and GetBalance.

Building the Benchmark

Navigate to ~/git/quelea/tests/BankAccount, and run make. On the stdout, along with GHC’s build information, you should find something similar to the following getting printed:

The above lines indicate the consistency class for each Bank Account operation (i.e., ), along with the time taken to classify the operation, and also the total time consumed by the contract classification procedure. Similar information is also displayed for Bank Account transactions (namely, saveTxn, and totalBalanceTxn). Observe that compile-time overhead of classifying each operation is very low, and the total overhead of contract classification is roughly the sum of overheads incurred in each case.

Succesfully building the benchmark results in four binaries - BankAccount_EC, BankAccount_CC, BankAccount_SC, and BankAccount_Q, that correspond to curves named EC, CC, SC and Q in the graph shown in Fig. 9(a) of our paper.

Running the Benchmark

First, we run a basic experiment to measure latency and throughput, when a single client sends 1000 successive requests (with an inter-request delay of 1ms (1000 μs)). We take the measurements for each of the EC, CC, SC, and Q cases. Client requests are roughly 25% getBalances, 25% deposits, and 50% withdraws. EC, CC and SC binaries run all operations under eventual, causal and strong consistency levels, respectively. On the other hand, the Q testcase runs operations at appropriate consistency levels as determined by the contract classification procedure. The commands to execute, and output generated in our sample runs are shown below. Please note that experiments can be terminated either manually (via CTRL+C), or by setting a hard time limit via --terminateAfter option, which accepts number of seconds as argument.

EC

./BankAccount_EC --kind Daemon --measureLatency --delayReq 1000 --numThreads 1 --numRounds 1000

Once you have aggregate latency and throughput data, terminate the experiment by pressing CTRL-C. If you have encountered an error, or if you suspect that execution is not making progress, please refer to the troubleshooting guide for quick fix.

CC

./BankAccount_CC --kind Daemon --measureLatency --delayReq 1000 --numThreads 1 --numRounds 1000

SC

./BankAccount_SC --kind Daemon --measureLatency --delayReq 1000 --numThreads 1 --numRounds 1000

Q

./BankAccount_Q --kind Daemon --measureLatency --delayReq 1000 --numThreads 1 --numRounds 1000

The experiment can now be repeated varying the number of client threads (--numThreads) from 2 to 8. Screenshots capturing measurements for sample runs when --numThreads is 4 are shown below:

EC

CC

SC

Q

Observations

As shown in the sample runs, you should be able to observe that:

• Throughput is highest when all operations are executed under eventual consistency (EC). Causal consistency (CC) throughput is comparable to that of EC. However, in both cases, withdraw operation can exhibit incorrect behaviour.
• Throughput is lowest when all operations are executed under Strong Consistency (SC).
• Testcase BankAccount_Q, which executes operations at appropriate consistency levels, delivers performance which is between the above two extremes.