However, the graphs do not match exactly with the ones in the blog post. The 99% data are quite random as already suggested by the authors. We also observe a big discrepancy in the CDF of the average completion time for the web workload. In our results it seems like TCP and pFabric have similar performance instead of pFabric clearly outperforming TCP for large load as the authors claim.

To conclude, it was very easy to reproduce the results following the authors’ instructions. However, the fact that tail latency is a very volatile metric and the selection of Mininet as emulation platform lead to unstable results. The problem could be potentially alleviated by running more iterations.

The instructions were very clear and the results are easy to reproduce due to well-setup VM that is provided. Unfortunately, we were not able to get fast enough connection through to the virtual machine to successfully run the live experiment, but the offline experiment ran perfectly, and the results matched identically to the results in the blog post. Running the experiment takes no longer than 5 minutes from booting the VM to displaying the plots.

For reference, here are the plots we reproduced:
fig1: https://drive.google.com/open?id=0BwuLLdq7RbLQY0xjOWVhQ1BGX1k

fig2: https://drive.google.com/open?id=0BwuLLdq7RbLQNzBETmpzaWxRTmc

Everything went very smoothly. I cloned their repo into a VM, and performed experiments 1 and 2, which both produced output that was nearly identical to the data presented in the original blog post. Script 1 (analyzeStatic) completed in roughly 30 minutes, and Script 2 (runRand) completed overnight. The processing and analysis scripts that turned the output into tables and graphs were also straightforward, and the HTTP presentation interface was a nice touch. Since we did a random sample of the blog posts’s findings, our absolute numbers were different, but it was reassuring to see that our percentages were nearly identical, differing by at most a percentage point or two. (eg their main result was 23.44% of sites had weak DH parameters, we found 23.28% in our random sample).

The blog post demonstrated understanding of the original paper, and chose an interesting subset of results to reproduce. The results of the blog post, and of our reproduction, show that the number of misconfigured DHE parameter has decreased dramatically from roughly 82% to 23%. This is encouraging to see!

Most figures we got closely matched what was published in the blog post.

There was however a (small) difference in one of the plots. In the figure of average response latency for web search, we see no improvement for 256 Kbps bandwidth when repeating the experiment.

Our experiment reproduced figure: https://github.com/jervisfm/tcp-init-cwnd/blob/70d5990ec0d1f3f1fa43c135c87755b9d5a2b35d/results/Figure5_BW.png

Blog post figure: https://reproducingnetworkresearch.files.wordpress.com/2017/06/figure5_bw2.png

We asked the author what might be contributing factors here and they mentioned potential issues with cloud cpu scheduling, effects from a noisy neighbor running next to us or insufficient experiment duration for the low bandwidth case.

Plots we reproduced: