Skip to: Site menu | Main content

Handling large connection/second rates on Amazon EC2

Some people say Amazon EC2 aka “the Cloud” is the answer to every technical problem one can have.
Just add more servers and things will go faster, unfortunately this is not always true on Amazon EC2.

Part of our EC2 infrastructure is responsible for handling about 2500/requests/s on average, this requests are very small and do not put a lot of stress on the web server, despite this we used to have a lot of instances handling that traffic (which were mostly idling around 5% CPU usage)
After migrating from us-east to eu-west and reducing the number of servers everything died, EBS were reporting:

"HTTP/1.1 503 Service Unavailable: Back-end server is at capacity"

After checking up the load was about 5-10% and everything looked good.
The Problem:
netstat  showed a lot of TIME_WAIT connections

# netstat -tan | grep ':80 ' | awk '{print $6}' | sort | uniq -c
 17 ESTABLISHED
 1 LISTEN
 2 SYN_RECV
 23816 TIME_WAIT

why only over 11k connections even when there are over 60k ports?
It has to do with a way Linux chooses source port for outgoing connection

$ cat /proc/sys/net/ipv4/ip_local_port_range
 32768 61000

61000-32768=28232

So with the default linux kernel configuration there are only 28k port numbers that can be used for outgoing connection, not more

Solution:

There are a few possible solutions to the problem, the fastest and safest is to set:

net.ipv4.tcp_tw_reuse = 1

in sysctl

From the kernel documentation:

Enable fast recycling TIME-WAIT sockets. Default value is 0.
It should not be changed without advice/request of technical
experts.

This will force the kernel to reuse sockets that are in TIME_WAIT state if this is safe from the protocol point of view

There is also another tuning parameter:

tcp_tw_recycle - BOOLEAN
Enable fast recycling TIME-WAIT sockets. Default value is 0.
It should not be changed without advice/request of technical
experts.

This option will forcefully recycle TIME_WAIT sockets, the side effect is that it will break NATed connections.

 

Running Half Life dedicated server on 64bit linux

If while installing Counter Strike Source server you get en error like:

unable to execute ./hldsupdatetool.bin: No such file or directory

and strace also shows something strange:

mag@entropy:~/cs$ strace ./hldsupdatetool.bin
execve(“./hldsupdatetool.bin”, ["./hldsupdatetool.bin"], [/* 22 vars */]) = -1 ENOENT (No such file or directory)
dup(2) = 3
fcntl(3, F_GETFL) = 0×8002 (flags O_RDWR|O_LARGEFILE)
fstat(3, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), …}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f25d9a22000
lseek(3, 0, SEEK_CUR) = -1 ESPIPE (Illegal seek)
write(3, “strace: exec: No such file or di”…, 40strace: exec: No such file or directory
) = 40
close(3) = 0
munmap(0x7f25d9a22000, 4096) = 0
exit_group(1) = ?

Just install 32bit glibc:
apt-get install libc6-i386

Execve has a strange behaviour and returns ENOENT when it cannot find a system library, this leeds to misleading error.

KVM performance “qcow2″ vs “raw”, “ide” vs “virtio”

I just got a new server for private use and needed to put some virtual machines, I decided to go with KVM as it seems it will be supported longer than XEN

Of course regardless of virtualization platform the decision has to be made – how to store Virtual Disks?

KVM wiki suggests that qcow2 is fast enough, is it really?

Some benchmarks of clean install install of Ubuntu 9.10 with KVM, Virtual Disk stored in qcow2 format:

qcow2+ide

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
entropy.be       4G 38684  43 53081   9 64187  13 55412  89 360629  43  5278  12

qcow2+virtio

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
entropy.be       4G 58859  73 76112  13 74583  14 52058  91 546065  44  5725  32

raw(file)+ide

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
entropy.be       4G 74111  83 86947  15 66292  14 50945  89 347763  41  5892   7

raw(file)+virtio:

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
entropy.be       4G 74607  90 86478  16 20930   7 51069  88 658254  48  8676  64

as a comparison, native I/O on host filesystem:

Version 1.03c       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
serwer3.itcoms. 16G 75336  82 89196  14 34498   5 46891  79 99439   5 421.1   1

It seems qcow2 perfomance in block input and output is still slightly worse than raw image. Also virtio is faster than ide for qcow2.

All benchmarks were run using `bonnie++ -u root -d /home` using kernel 2.6.31-20-server #58-Ubuntu SMP Fri Mar 12 05:40:05 UTC 2010 x86_64 GNU/Linux

How badly debian/ubunutu openssl is fscked up?

Executive Summary: BADLY!

At first I want to admit that this test may be silly but is fast and simple
All test were done on my Lenovo Z61p running recent Fedora9 – debian openssl was compiled from source

So, how to test what commenting out one line in openssl code changes?

Let’s generate some random numbers using openssl. Quick, dirty and lame solution:

$ for i in `seq 100000`; do ./openssl rand -base64 40 >> test; done 

and now quick check for randomness of this data:

$ wc -l test; cat test | sort| uniq -c | sort | tail   

Results:

Fedora openssl:

mag@viper:~$ wc -l test

100000 test   



mag@viper:~$ wc -l test; cat test | sort| uniq -c | sort | tail

100000 test

      1 ZzvfJF8Sh8s59kU+/hNoK8F4FfbyIoG7qY9NObjd24PwvMzm6CHuzw==

      1 ZZVN89z8LONs8nbk49enpEYbm47iZF8bFc0kvFwyEr0MzLuzOiQqQA==

      1 zZwbc+wuPBMCdR6gEXjkkaxrFF+Capaw8INv0DPhi8paqcTbvxX86w==

      1 ZZx1dP0xMsceZl9ozVeUvQgEP9AmE/56a+CgdIO8kCCDI6DdDTrxeA==

      1 zzxnavFvysTxniMWv+a0Rxyu1XtCZ2UZN5pVAB3b5FAtIZx+5EttqA==

      1 ZzyKbqW87iekJtK9niDu0FwKnR5ENXnKdKJFcCbNAp2qqEw9yrTn5A==

      1 zZZ9K7L4n7qKH5E5GYDaIZLchQvSn7e/g1FPL8SmZF+NW+1i3H3XRw==

      1 Zz/z9kwIDywXrxtrTNmWy9gFGGcass9k3g8fe8DTae3lCiAukILUtg==

      1 +zZ/ZKKqv5YQagg3HSpp2hLrFSuPIRRg0nDOEwZ2hf935Gk6zS7TFA==

      1 zZZwoogf3Ib5Xg9WMnMU/4IP2mbbvXhuecSnofxqscLg1QpA4R6N3g==   

Everything looks fine all seeds are unique, now debian openssl:

Debian openssl:

mag@viper:~/tmp/openssl-0.9.8g/apps$ wc -l test

100000 test   



mag@viper:~/tmp/openssl-0.9.8g/apps$ wc -l test; cat test | sort| uniq -c | sort | tail

100000 test

      4 zYgla7UVURkIGH9Bol2otXKSTYfr/NBIVZa/68FKeidHgbtSAIVNog==

      4 zYJLIAIxG8AMtrf6smNMsdZFBADCQPyuZHQTmGFWnJg6PwESLL2e8g==

      4 ZyKePqCM93yGcYkNxlHJ0/y7ZOhAEtXJyyi4H6JVzUcRpe+2zhJeZg==

      4 zyuCidG1FPJm8Ut4CshJvI5A+g3SDvmFZ4lIN+ESUCA/m8DU1aJ5ww==

      4 ZZbFf3M7R0Pl+VPkJD9TJzsqSVbgCxydAJVzS8f752pYfXCyzVNP1g==

      4 zzrbuvwbw8XN61Bygi8KEKMUI7qOVZmQmglJYua/Zp7dDFH/Z45mJQ==

      4 zztQ5wdvkWiwCIEJpkSvyqkrVgb32UccogSmZwNRlLH7ieIZvXU+/A==

      4 zZUGNwhNUFHCdkPnHFYW0ME1+MlFwSo3eb1rLOwjDfL5FyHHqvFQ4g==

      4 zZVYlHlCIH4XHveWvvWgES2AwHuhy3aiIQk2JvyE7T3VKdNCWZU9QA==

      4 zzZPx/Z5iE29WkCpTVKunmo102m4p3GYHzXZ6B/TRgKSUFkJScGVqA==  

Quick interpretation of results – first column is count of repeating seed. So in 100000 tries debian openssl random() generated 4 times same 40 byte string.
This was used for openssh/ssl/etc key generation…

Conclusion: debian openssl was UNABLE to generate random numbers AT ALL!

and nobody noticed :-/

I leave the rest to you, happy regenerating all CA and ssl certificates. I only wonder what if Verisign or other “trusted” company used this for main CA generation.

If you think something is terribly wrong with this test please comment

Ubuntu Feisty – slightly better setup

After my opensuse Factory install crashed last week I deceided to try something new and installed Ubuntu feisty on my private laptop.

XFS

Strange thing but grub which is shipped with ubuntu can not boot system from XFS partition.

So far I had not investigated a problem and I am booting system with LILO.

KDE

First thing was to deinstall all gnome apps ( leaving only openofice )  then using metapackage I installed kde. I do not know why all big distributions are sticking to gnome and most users still use KDE.

BOOTING

Booting process in ubuntu is rather fast, but if you add some more daemon it is usefull to turn on concurency

edit /etc/init.d/rc

and change:

CONCURRENCY=none

to:

CONCURRENCY=shell

FIREFOX

If you mean to use firefox it is allways good to disable pango, just add

MOZ_DISABLE_PANGO=”1″

to /etc/environment

PRELOAD

If you have plenty of RAM installing preload may help

# sudo apt-get install preload

XOrg

Ubuntu default config adds entries in /etc/X11/xorg.conf for wacom tablets. Unless you own one this is useless and only generates huge ~/.xsession-errors log file. Commenting out all devices besides used ones will help.

.NET and MONO

Yesterday i have listened to very interesting presentation that featured  .NET and the compatible open source version – MONO

I was impressed with the tools used - Microsoft Visual Studio 2005 and Visual Web Developer are pretty fast and easy to use ( unlike Vista ;-). Comparing to eclipse they were really fast.

Mono aims to be 100% compatible with .NET compiles C#, Java, boo and other languages to bytecode that is run with mono. Same compiled code can be run on windows/linux/MacOS/etc.  Also windows forms can be used on Linux without modyfying the codebase.

You can find more information on the official site http://mono-project.com and also on some community sites like http://gotmono.com