ec2 benchmarks

This commit is contained in:
Josh Baker 2017-11-03 17:14:09 -07:00
parent 8ea2a4a98f
commit 6a6c22352b
12 changed files with 92 additions and 129 deletions

View File

@ -1,3 +1,2 @@
bin/
socket
*.png
socket

View File

@ -6,16 +6,21 @@ Required tools:
- [tcpkali](https://github.com/machinezone/tcpkali) for Echo
- [Redis](http://redis.io) for Redis
Required Go packages:
```
go get gonum.org/v1/plot/...
go get -u github.com/valyala/fasthttp
go get -u github.com/kataras/iris
go get -u github.com/tidwall/redcon
```
And of course [Go](https://golang.org) is required.
Run `bench.sh` for all benchmarks.
## Notes
- The current results were run on an Ec2 c4.xlarge instance.
- The servers started in single-threaded mode (GOMAXPROC=1).
- Network clients connected over Ipv4 localhost.

BIN
benchmarks/out/echo.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

View File

@ -2,35 +2,33 @@
--- BENCH ECHO START ---
--- GO STDLIB ---
2017/11/03 16:37:10 echo server started on port 5001
2017/11/04 00:06:31 echo server started on port 5001
*** 50 connections, 10 seconds, 6 byte packets
Destination: [127.0.0.1]:5001
Interface lo0 address [127.0.0.1]:0
Interface lo0 address [127.94.0.1]:0
Using interface lo0 to connect to [127.0.0.1]:5001
Interface lo address [127.0.0.1]:0
Using interface lo to connect to [127.0.0.1]:5001
Ramped up to 50 connections.
Total data sent: 6131.1 MiB (6428962876 bytes)
Total data received: 6132.4 MiB (6430294582 bytes)
Bandwidth per channel: 205.668⇅ Mbps (25708.5 kBps)
Aggregate bandwidth: 5142.242↓, 5141.177↑ Mbps
Packet rate estimate: 472952.1↓, 446187.4↑ (6↓, 12↑ TCP MSS/op)
Test duration: 10.0039 s.
Total data sent: 9409.8 MiB (9866903552 bytes)
Total data received: 9165.8 MiB (9611005954 bytes)
Bandwidth per channel: 311.589⇅ Mbps (38948.6 kBps)
Aggregate bandwidth: 7687.378↓, 7892.059↑ Mbps
Packet rate estimate: 721925.3↓, 677380.8↑ (6↓, 45↑ TCP MSS/op)
Test duration: 10.0019 s.
--- DONE ---
--- EVIO ---
2017/11/03 16:37:21 echo server started on port 5002
2017/11/04 00:06:43 echo server started on port 5002
*** 50 connections, 10 seconds, 6 byte packets
Destination: [127.0.0.1]:5002
Interface lo0 address [127.0.0.1]:0
Interface lo0 address [127.94.0.1]:0
Using interface lo0 to connect to [127.0.0.1]:5002
Interface lo address [127.0.0.1]:0
Using interface lo to connect to [127.0.0.1]:5002
Ramped up to 50 connections.
Total data sent: 17129.1 MiB (17961204524 bytes)
Total data received: 17117.6 MiB (17949094958 bytes)
Bandwidth per channel: 574.050⇅ Mbps (71756.2 kBps)
Aggregate bandwidth: 14346.408↓, 14356.087↑ Mbps
Packet rate estimate: 1333916.3↓, 1238381.2↑ (12↓, 24↑ TCP MSS/op)
Test duration: 10.009 s.
Total data sent: 15867.3 MiB (16638083072 bytes)
Total data received: 15855.5 MiB (16625705220 bytes)
Bandwidth per channel: 532.133⇅ Mbps (66516.6 kBps)
Aggregate bandwidth: 13298.362↓, 13308.263↑ Mbps
Packet rate estimate: 1217575.2↓, 1142257.4↑ (12↓, 45↑ TCP MSS/op)
Test duration: 10.0017 s.
--- DONE ---
--- BENCH ECHO DONE ---

BIN
benchmarks/out/http.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 18 KiB

View File

@ -2,57 +2,57 @@
--- BENCH HTTP START ---
--- GO STDLIB ---
2017/11/03 16:36:23 http server started on port 8081
2017/11/04 00:05:45 http server started on port 8081
*** 50 connections, 10 seconds
Running 10s test @ http://127.0.0.1:8081
1 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 702.61us 289.11us 4.36ms 65.96%
Req/Sec 71.71k 4.47k 75.96k 91.09%
719868 requests in 10.10s, 89.93MB read
Requests/sec: 71263.20
Transfer/sec: 8.90MB
Latency 1.20ms 676.37us 4.51ms 59.55%
Req/Sec 42.00k 2.95k 43.23k 96.04%
422031 requests in 10.10s, 52.72MB read
Requests/sec: 41786.07
Transfer/sec: 5.22MB
--- DONE ---
--- IRIS ---
Now listening on: http://localhost:8082
Application started. Press CMD+C to shut down.
2017/11/03 16:36:35 http server started on port 8082
Application started. Press CTRL+C to shut down.
2017/11/04 00:05:57 http server started on port 8082
*** 50 connections, 10 seconds
Running 10s test @ http://127.0.0.1:8082
1 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.86ms 556.33us 5.49ms 89.37%
Req/Sec 62.49k 2.32k 67.04k 67.33%
627701 requests in 10.10s, 78.42MB read
Requests/sec: 62138.33
Transfer/sec: 7.76MB
Latency 1.22ms 751.36us 4.69ms 65.91%
Req/Sec 42.23k 553.92 43.22k 68.00%
420122 requests in 10.00s, 52.49MB read
Requests/sec: 42009.70
Transfer/sec: 5.25MB
--- DONE ---
--- FASTHTTP ---
2017/11/03 16:36:47 http server started on port 8083
2017/11/04 00:06:09 http server started on port 8083
*** 50 connections, 10 seconds
Running 10s test @ http://127.0.0.1:8083
1 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 442.36us 128.03us 4.22ms 81.00%
Req/Sec 110.18k 12.49k 135.57k 63.00%
1095146 requests in 10.00s, 155.62MB read
Requests/sec: 109505.28
Transfer/sec: 15.56MB
Latency 303.56us 174.00us 2.04ms 64.86%
Req/Sec 167.97k 29.11k 186.31k 77.00%
1670611 requests in 10.00s, 237.39MB read
Requests/sec: 167050.74
Transfer/sec: 23.74MB
--- DONE ---
--- EVIO ---
2017/11/03 16:36:58 http server started on port 8084
2017/11/04 00:06:20 http server started on port 8084
*** 50 connections, 10 seconds
Running 10s test @ http://127.0.0.1:8084
1 threads and 50 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 429.45us 239.79us 9.92ms 95.43%
Req/Sec 113.16k 16.86k 128.84k 85.15%
1136765 requests in 10.10s, 112.75MB read
Requests/sec: 112542.41
Transfer/sec: 11.16MB
Latency 119.48us 36.20us 613.00us 87.72%
Req/Sec 227.28k 1.21k 229.55k 51.00%
2260225 requests in 10.00s, 224.17MB read
Requests/sec: 226013.08
Transfer/sec: 22.42MB
--- DONE ---
--- BENCH HTTP DONE ---

View File

@ -2,39 +2,26 @@
--- BENCH REDIS PIPELINE 1 START ---
--- REAL REDIS ---
6747:M 03 Nov 16:37:33.512 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
6747:M 03 Nov 16:37:33.513 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
6747:M 03 Nov 16:37:33.513 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.2.8 (db8a945c/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6392
| `-._ `._ / _.-' | PID: 6747
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
6747:M 03 Nov 16:37:33.517 # Server started, Redis version 3.2.8
6747:M 03 Nov 16:37:33.517 * The server is now ready to accept connections on port 6392
15719:C 04 Nov 00:09:50.971 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
15719:C 04 Nov 00:09:50.971 # Redis version=4.0.2, bits=64, commit=00000000, modified=0, pid=15719, just started
15719:C 04 Nov 00:09:50.971 # Configuration loaded
15719:M 04 Nov 00:09:50.972 * Increased maximum number of open files to 10032 (it was originally set to 1024).
15719:M 04 Nov 00:09:50.973 * Running mode=standalone, port=6392.
15719:M 04 Nov 00:09:50.973 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
15719:M 04 Nov 00:09:50.973 # Server initialized
15719:M 04 Nov 00:09:50.973 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
15719:M 04 Nov 00:09:50.973 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
15719:M 04 Nov 00:09:50.973 * Ready to accept connections
*** 50 connections, 1000000 commands, 1 commands pipeline
PING_INLINE: nan PING_INLINE: 116748.00 PING_INLINE: 115610.00 PING_INLINE: 117642.66 PING_INLINE: 116757.00 PING_INLINE: 116677.60 PING_INLINE: 113024.00 PING_INLINE: 108711.43 PING_INLINE: 108675.50 nan nan nan nan PING_INLINE: 116748.00 nan PING_INLINE: 115610.00 nan PING_INLINE: 117642.66 nan PING_INLINE: 116757.00 nan PING_INLINE: 116677.60 nan PING_INLINE: 113024.00 nan PING_INLINE: 108711.43 nan PING_INLINE: 108675.50 PING_INLINE: 116748.00 PING_INLINE: 116748.00 nan PING_INLINE: 116748.00 PING_INLINE: 116748.00 PING_INLINE: 116748.00 PING_INLINE: 115610.00 PING_INLINE: 116748.00 PING_INLINE: 117642.66 PING_INLINE: 116748.00 PING_INLINE: 116757.00 PING_INLINE: 116748.00 PING_INLINE: 116677.60 PING_INLINE: 116748.00 PING_INLINE: 113024.00 PING_INLINE: 116748.00 PING_INLINE: 108711.43 PING_INLINE: 116748.00 PING_INLINE: 108675.50 PING_INLINE: 115610.00 PING_INLINE: 115610.00 nan PING_INLINE: 115610.00 PING_INLINE: 116748.00 PING_INLINE: 108829.38 PING_INLINE: 108837.09 PING_INLINE: 108793.41 PING_INLINE: 115610.00 PING_INLINE: 116677.60 PING_INLINE: 115610.00 PING_INLINE: 113024.00 PING_INLINE: 108896.88 requests per second
PING_INLINE: 0.00 PING_INLINE: 127231.09 PING_INLINE: 128309.38 PING_INLINE: 128535.29 PING_INLINE: 138613.38 PING_INLINE: 146055.95 PING_INLINE: 151113.92 PING_INLINE: 154639.62 PING_INLINE: 157245.88 PING_INLINE: 159290.55 PING_INLINE: 160986.81 PING_INLINE: 162353.69 PING_INLINE: 163625.12 PING_INLINE: 164611.81 PING_INLINE: 165530.70 PING_INLINE: 166273.53 PING_INLINE: 166944.02 PING_INLINE: 167518.00 PING_INLINE: 168095.98 PING_INLINE: 168532.52 PING_INLINE: 169000.80 PING_INLINE: 169338.98 PING_INLINE: 169724.23 PING_INLINE: 170033.73 PING_INLINE: 170241.73 requests per second
--- DONE ---
--- EVIO REDIS CLONE ---
2017/11/03 16:37:44 redis server started on port 6393
2017/11/03 16:37:44 redis server started at socket
2017/11/04 00:09:58 redis server started on port 6393
2017/11/04 00:09:58 redis server started at socket
*** 50 connections, 1000000 commands, 1 commands pipeline
PING_INLINE: nan PING_INLINE: 117884.00 PING_INLINE: 124196.00 PING_INLINE: 119600.00 PING_INLINE: 118939.00 PING_INLINE: 119103.20 PING_INLINE: 119974.00 PING_INLINE: 120057.14 PING_INLINE: 118809.00 nan nan nan nan PING_INLINE: 117884.00 nan PING_INLINE: 124196.00 nan PING_INLINE: 119600.00 nan PING_INLINE: 118939.00 nan PING_INLINE: 119103.20 nan PING_INLINE: 119974.00 nan PING_INLINE: 120057.14 nan PING_INLINE: 118809.00 PING_INLINE: 117884.00 PING_INLINE: 117884.00 nan PING_INLINE: 117884.00 PING_INLINE: 117884.00 PING_INLINE: 117884.00 PING_INLINE: 124196.00 PING_INLINE: 117884.00 PING_INLINE: 119600.00 PING_INLINE: 117884.00 PING_INLINE: 118939.00 PING_INLINE: 117884.00 PING_INLINE: 119103.20 PING_INLINE: 117884.00 PING_INLINE: 119974.00 PING_INLINE: 117884.00 PING_INLINE: 120057.14 PING_INLINE: 117884.00 PING_INLINE: 118809.00 PING_INLINE: 124196.00 PING_INLINE: 124196.00 nan PING_INLINE: 116605.42 PING_INLINE: 116377.88 PING_INLINE: 115540.48 PING_INLINE: 114599.41 PING_INLINE: 114117.14 PING_INLINE: 114090.12 requests per second
PING_INLINE: 0.00 PING_INLINE: 166804.80 PING_INLINE: 171942.12 PING_INLINE: 173320.91 PING_INLINE: 174023.97 PING_INLINE: 174624.30 PING_INLINE: 175023.98 PING_INLINE: 175253.56 PING_INLINE: 175374.31 PING_INLINE: 175518.89 PING_INLINE: 175424.64 PING_INLINE: 175532.53 PING_INLINE: 175496.17 PING_INLINE: 175522.61 PING_INLINE: 175594.97 PING_INLINE: 175592.38 PING_INLINE: 175558.36 PING_INLINE: 175556.11 PING_INLINE: 175517.89 PING_INLINE: 175528.52 PING_INLINE: 175507.70 PING_INLINE: 175480.30 PING_INLINE: 175471.00 PING_INLINE: 175438.61 requests per second
--- DONE ---

View File

@ -2,39 +2,26 @@
--- BENCH REDIS PIPELINE 16 START ---
--- REAL REDIS ---
6857:M 03 Nov 16:37:59.270 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
6857:M 03 Nov 16:37:59.270 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
6857:M 03 Nov 16:37:59.270 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.2.8 (db8a945c/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6392
| `-._ `._ / _.-' | PID: 6857
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
6857:M 03 Nov 16:37:59.271 # Server started, Redis version 3.2.8
6857:M 03 Nov 16:37:59.271 * The server is now ready to accept connections on port 6392
15833:C 04 Nov 00:10:09.722 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
15833:C 04 Nov 00:10:09.722 # Redis version=4.0.2, bits=64, commit=00000000, modified=0, pid=15833, just started
15833:C 04 Nov 00:10:09.722 # Configuration loaded
15833:M 04 Nov 00:10:09.723 * Increased maximum number of open files to 10032 (it was originally set to 1024).
15833:M 04 Nov 00:10:09.723 * Running mode=standalone, port=6392.
15833:M 04 Nov 00:10:09.723 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
15833:M 04 Nov 00:10:09.723 # Server initialized
15833:M 04 Nov 00:10:09.723 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
15833:M 04 Nov 00:10:09.723 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
15833:M 04 Nov 00:10:09.723 * Ready to accept connections
*** 50 connections, 1000000 commands, 16 commands pipeline
PING_INLINE: nan PING_INLINE: 773376.00 PING_INLINE: 788832.00 PING_INLINE: 687808.00 PING_INLINE: 688784.00 PING_INLINE: 696985.62 PING_INLINE: 689655.12 requests per second
PING_INLINE: 0.00 PING_INLINE: 1023617.56 PING_INLINE: 1029014.00 PING_INLINE: 1030647.12 PING_INLINE: 1031991.75 requests per second
--- DONE ---
--- EVIO REDIS CLONE ---
2017/11/03 16:38:02 redis server started on port 6393
2017/11/03 16:38:02 redis server started at socket
2017/11/04 00:10:12 redis server started on port 6393
2017/11/04 00:10:12 redis server started at socket
*** 50 connections, 1000000 commands, 16 commands pipeline
PING_INLINE: 0.00 PING_INLINE: 1550024.00 PING_INLINE: 1538682.62 PING_INLINE: 1536098.25 requests per second
PING_INLINE: -nan PING_INLINE: 2138688.00 PING_INLINE: 2150537.50 requests per second
--- DONE ---

View File

@ -2,39 +2,26 @@
--- BENCH REDIS PIPELINE 8 START ---
--- REAL REDIS ---
6815:M 03 Nov 16:37:54.009 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
6815:M 03 Nov 16:37:54.009 # Server can't set maximum open files to 10032 because of OS error: Operation not permitted.
6815:M 03 Nov 16:37:54.009 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
_._
_.-``__ ''-._
_.-`` `. `_. ''-._ Redis 3.2.8 (db8a945c/0) 64 bit
.-`` .-```. ```\/ _.,_ ''-._
( ' , .-` | `, ) Running in standalone mode
|`-._`-...-` __...-.``-._|'` _.-'| Port: 6392
| `-._ `._ / _.-' | PID: 6815
`-._ `-._ `-./ _.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' | http://redis.io
`-._ `-._`-.__.-'_.-' _.-'
|`-._`-._ `-.__.-' _.-'_.-'|
| `-._`-._ _.-'_.-' |
`-._ `-._`-.__.-'_.-' _.-'
`-._ `-.__.-' _.-'
`-._ _.-'
`-.__.-'
6815:M 03 Nov 16:37:54.010 # Server started, Redis version 3.2.8
6815:M 03 Nov 16:37:54.010 * The server is now ready to accept connections on port 6392
15775:C 04 Nov 00:10:04.957 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
15775:C 04 Nov 00:10:04.957 # Redis version=4.0.2, bits=64, commit=00000000, modified=0, pid=15775, just started
15775:C 04 Nov 00:10:04.957 # Configuration loaded
15775:M 04 Nov 00:10:04.958 * Increased maximum number of open files to 10032 (it was originally set to 1024).
15775:M 04 Nov 00:10:04.959 * Running mode=standalone, port=6392.
15775:M 04 Nov 00:10:04.959 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
15775:M 04 Nov 00:10:04.959 # Server initialized
15775:M 04 Nov 00:10:04.959 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
15775:M 04 Nov 00:10:04.959 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
15775:M 04 Nov 00:10:04.959 * Ready to accept connections
*** 50 connections, 1000000 commands, 8 commands pipeline
PING_INLINE: nan PING_INLINE: 611616.00 PING_INLINE: 595296.00 PING_INLINE: 589664.00 PING_INLINE: 594752.00 PING_INLINE: 581139.19 PING_INLINE: 584784.00 PING_INLINE: 583090.38 requests per second
PING_INLINE: -nan PING_INLINE: 646432.00 PING_INLINE: 648144.00 PING_INLINE: 648789.31 PING_INLINE: 648896.00 PING_INLINE: 649043.19 PING_INLINE: 649210.69 PING_INLINE: 649350.69 requests per second
--- DONE ---
--- EVIO REDIS CLONE ---
2017/11/03 16:37:57 redis server started on port 6393
2017/11/03 16:37:57 redis server started at socket
2017/11/04 00:10:07 redis server started on port 6393
2017/11/04 00:10:07 redis server started at socket
*** 50 connections, 1000000 commands, 8 commands pipeline
PING_INLINE: nan PING_INLINE: 886112.00 PING_INLINE: 893152.00 PING_INLINE: 874496.00 PING_INLINE: 865888.00 PING_INLINE: 864304.25 requests per second
PING_INLINE: 0.00 PING_INLINE: 1234804.88 PING_INLINE: 1235928.12 PING_INLINE: 1250045.25 PING_INLINE: 1253132.88 requests per second
--- DONE ---

Binary file not shown.

After

Width:  |  Height:  |  Size: 15 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 16 KiB