Gaming with the Throughput and the Latency Benchmarking Measurement Procedures of RFC 2544

Gábor Lencse, Ákos Kovács, Keiichi Shima


In this paper, we investigate three potential issues of the benchmarking measurement procedures defined in RFC 2544 and also used in RFC 5180 and RFC 8219. One of them is the lack of proper timeout usage. We use a Linux box, which can selectively delay a specified ratio of the packets. Using carefully selected parameters based on our preliminary measurements, we demonstrate that the experienced speed of the HTTP download is much less, than what could have been expected on the basis of the throughput results of the RFC 2544 tests. The other critical issue is the strict, absolutely zero loss criterion. We use a Linux box, which drops a specified small ratio of the packets. Whereas the RFC 2544 throughput results tend to be zero, the experienced speed of the HTTP download is quite good. The third problem is the lack of requirement for statistically relevant number of tests in the RFC 2544 benchmarking procedures. We demonstrate its severity with the latency benchmarking procedure defined in RFC 2544 and kept unchanged in RFC 5180 but redefined in RFC 8219.

Full Text:



G. Lencse, “Benchmarking methodology for IPv6 transition technologies”, IIJ Lab seminar, Tokyo, Oct. 10, 2017, [Online]. Available:

S. Bradner and J. McQuaid, “Benchmarking methodology for network interconnect devices”, IETF RFC 2544, 1999. DOI: 10.17487/RFC2544

C. Popoviciu, A. Hamza, G. Van de Velde, and D. Dugatkin, “IPv6 benchmarking methodology for network interconnect devices”, IETF RFC 5180, 2008. DOI: 10.17487/RFC5180

M. Georgescu, L. Pislaru L, and G. Lencse, “Benchmarking methodology for IPv6 transition technologies, IETF RFC 8219, 2017. DOI: 10.17487/RFC8219

G. Lencse, K. Shima, “Performance analysis of SIIT implementations: Testing and improving the methodology”, Computer Communications, vol. 156, no. 1, pp. 54-67, April 15, 2020, DOI: 10.1016/j.comcom.2020.03.034

C. Bao, X. Li, F. Baker, T. Anderson, and F. Gont, “IP/ICMP translation algorithm”, IETF RFC 7915, 2016. DOI: 10.17487/RFC7915

B. Constantine, G. Forget, R. Geib, R. Schrage, “Framework for TCP Throughput Testing”, IETF RFC 6349, 2011. DOI: 10.17487/RFC6349

P. Lettieri and M. B. Srivastava, “Adaptive frame length control for improving wireless link throughput, range, and energy efficiency” Proc. IEEE INFOCOM '98, San Francisco, CA, USA, 1998. DOI: 10.1109/INFCOM.1998.665076

Y. Bai, A.T. Ogielski, G. Wu, “Interactions of TCP and radio link ARQ protocol” Proc. IEEE VTS 50th Vehicular Technology Conference, Amsterdam, The Netherlands, 1999. DOI: 10.1109/VETECF.1999.801596

V. Tsaoussidis, H. Badr, X. Ge, K. Pentikousis, “Energy/throughput tradeoffs of TCP error control strategies”, Proc. ISCC 2000, Antibes-Juan Les Pins, France, 2000. DOI: 10.1109/ISCC.2000.860618

N. Celandroni, F. Potorti, “Maximizing single connection TCP goodput by trading bandwidth for BER”, International Journal of Communication Systems, vol. 16, no. 1, pp. 63-79, February 18, 2003. DOI: 10.1002/dac.580

Peng Zhang ; Hongbo Wang ; Shiduan Cheng, “Shrinking MTU to Mitigate TCP Incast Throughput Collapse in Data Center Networks”, Proc. 2011 Third International Conference on Communications and Mobile Computing, Qingdao, China, 2011. DOI: 10.1109/CMC.2011.68

G. Lencse and Y. Kadobayashi, “Benchmarking DNS64 implementations: Theory and practice”, Computer Communications, vol. 127, no. 1, pp. 61-74, September 1, 2018. DOI: 10.1016/j.comcom.2018.05.005

G. Lencse, “Siitperf: an RFC 8219 compliant SIIT (stateless NAT64) tester written in C++ using DPDK”, source code, [Online]. Available:

G. Lencse, “Design and implementation of a software tester for benchmarking stateless NAT64 gateways”, accepted for publication in IEICE Transactions on Communications, revised version is available:

G. Lencse, K. Shima, “An upgrade to benchmarking methodology for network interconnect devices”, individual Internet Draft, May 20, 2020. available:



  • There are currently no refbacks.