Bidirectionally Testing Network Connections
Beginning in Spring 2009, I started seeing network problems at some customer sites: data transmissions would hang and time out; connections via OpenVPN links would work at first but time out as soon as more data was transmitted; VPN connections using UDP would not work at all (but would work magically when changing OpenVPN's port to 80/TCP). All these symptoms are usually tied to traffic shaping, problems with the maximum packet size (maximum transfer unit, MTU) or other issues usually found between the end-points of the transmission. But how do you test and record the performance of "your" Internet?
Using Network Tools for Debugging
First of all, you can use network tools to inspect the path between the two communication partners.
- hping3 is a tool to construct arbitrary TCP/IP packets and send them to hosts. This is very useful for sending TCP SYN packets and seeing if TCP SYN+ACK packets return. You can also send packets with a custom payload, and check for MTU issues.
- ping is a well-known tool for sending ICMP Echo Request packets.
- tracepath is used for discovering the MTU between source and destination. It uses UDP packets, starting with a size of 65536 byte and reducing the size after receiving timeouts or errors.
- traceroute is the classic tool for displaying the intermediate hops between source and destination.
Most tools display error messages, round-trip/response times and other useful information. Although many people rely on the results of tools, always keep in mind that sending test packets is not the same as sending real data over different protocols. There are routers and other network devices out there that do not treat every packet equally.
One site reported very strange effects. Immediately after you used a Bittorrent client or a protocol over "bad ports", the bandwidth for "regular" network traffic dropped notably. After half an hour or more, the performance would return to normal. This behaviour may indicate a violation of net neutrality, the presence of filters, or other defects. The big problem is to find the cause, since it could be the network path, flaky hardware, broken software, prioritisation of network traffic, policy enforcement, or some entirely different reason. If there are intermediate network devices manipulating your packet flow, then there's a chance you will never identify or "see" these devices with the tools available.
Using Data Transmissions for Debugging
Using "real" data and "real" protocols for testing is better, but how do you do that? Clicking on download links in your Web browser and using a stop watch isn't well suited to finding hard facts. What about the other direction? Downloads are treated differently from uploads, if you're on an asymmetric Internet connection. You need to test transfers in both directions, and you need to use different protocols and ports. Why not let a script do all of this, and collect a nice summary?
First, we thought to use a shell script containing a few tools, and send data around. The problem is that you might want to do some statistics. Filtering the output and calculating transmission rates can be quite difficult. That's why a Perl script was created. It uses HTTP, HTTPS, SCP, FTP, SMTP, and IMAP. All protocols except SMTP are automatically used bidirectionally. The script uses a configuration file where login, password, paths, and other parameters can be configured. You can use different servers for every protocol, if you want. The data to be transferred can be given explicitly by file. If no file is given, then the script will create a given amount of random data using OpenSSL's pseudo-random generator code. The full configuration file with all entries looks like this:
# Size of data - you can either provide a file or a number of bytes. # If the file is set to "none", the number of random bytes is used. data_bytes=65536 data_chunk=8192 data_file="none" logfile="result.log" tests=3 # Send report by email email="recipient@example.net" cc="none" smtp_relay="localhost" # HTTP URLs http_download="http://www.example.net/tmp/crypto_schrift.png" http_upload="http://www.example.net/internettester.php" # HTTPS URLs https_download="https://www.example.net/tmp/crypto_schrift.png" https_upload="https://www.example.net/internettester.php" # FTP ftp_server="ftp.example.net" ftp_username="ftpuser" ftp_password="password" ftp_path="none" # IMAP connection (we use the INBOX) imap_server="mail.example.net" imap_port="143" imap_messages=5 imap_username="mailuser" imap_password="password" imap_ssl="no" # SCP connection scp_server="www.example.net" scp_port=22 scp_path="/tmp" scp_username="user" # SMTP smtp_server="mail.example.net" smtp_port=25 smtp_subject="This is a automatically generated email from internettester.pl (%u)" smtp_from="tester@example.net" smtp_to="devnull@example.net" # SSL parameters ssl_ca="" ssl_cert="" ssl_key="" # Temporary directory tmpdir="/tmp"
It is important that the HTTP and HTTPS upload URLs handle POST requests. One way of doing this is to place the PHP script internettester.php on a Web server. The script receives the POST requests, checks the parameters, and discards the upload data. If you need to use SSL with certificates, then you can add the path to a certificate, key, and certificate authority to the configuration, too. (See the section "SSL parameters".)
The results are printed on standard output. If you want the results by e-mail, then you can set the parameters email, cc, and smtp_relay. The output is then put into an e-mail message and sent via SMTP.
In order to do the actual measurements, you will need at least one endpoint with an Internet connection with known parameters. You can use a place on a co-located server, but bear in mind that the connectivity will most probably be shared. 100 Mbit/s or more sounds a lot - if you have well-behaving neighbours. The bandwidth should at least be higher than the one at the endpoint you are interested in testing.
The script itself needs only Perl and a couple of modules for all the protocols used. You can check if you have all modules by using perl -cw internettester.pl on the command line. On Debian or Ubuntu systems, the packages of the Perl modules start with lib and end with -perl. So the module Crypt::OpenSSL::Random translates to the package libcrypt-openssl-random-perl.
Running the Script
Running the script is fairly straightforward. It takes only two optional parameters. By using --config, you can direct the script to use a configuration file. By default, it uses internettester.cfg. The other parameter sets the debug mode. It lets you see what the script does.
lynx@nightfall:~$ ./internettester.pl --config lg.cfg +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ + Test run - using configuration file lg.cfg - + Sun Dec 13 03:35:01 CET 2009 + + Test file has a size of 3284992 bytes. + +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ --- HTTP download --------------------------------------------------------- Number of tests : 5 Mean : 227.16 KiB/s Minimum : 221.96 KiB/s Maximum : 231.75 KiB/s Standard deviation : 4.66 KiB/s --- HTTP upload ----------------------------------------------------------- Number of tests : 5 Mean : 1195.61 KiB/s Minimum : 1171.40 KiB/s Maximum : 1208.61 KiB/s Standard deviation : 14.47 KiB/s --- HTTPS download -------------------------------------------------------- Number of tests : 5 Mean : 227.64 KiB/s Minimum : 222.47 KiB/s Maximum : 233.43 KiB/s Standard deviation : 4.13 KiB/s --- HTTPS upload ---------------------------------------------------------- Number of tests : 5 Mean : 1054.77 KiB/s Minimum : 978.15 KiB/s Maximum : 1138.29 KiB/s Standard deviation : 64.00 KiB/s [...] lynx@nightfall:~$
I cut the output after the first two protocols. The script measures only bandwidth. Packet loss and round-trip time are not recorded. You can see the mean, maximum, and minimum of the measured rates. They are computed with the help of the Statistics::Descriptive module. The fourth value is the standard deviation. It indicates how far away the measurements are away from the mean. Low values of standard deviation indicate low variations of the individual transfer rate. Make sure the number of tests is not too low for a statistical analysis.
A word of warning: The tests will probably saturate your Internet link. All tests use TCP, which will tend to grab all your bandwidth (on a "standard" link). Measurements during office hours should be coordinated.
What's next?
The script is no magic bullet, and won't detect all of your network problems, but can perform automated tests and record the bandwidth used. Feel free to adapt the code to your requirements.
Useful resources
Talkback: Discuss this article with The Answer Gang
René was born in the year of Atari's founding and the release of the game Pong. Since his early youth he started taking things apart to see how they work. He couldn't even pass construction sites without looking for electrical wires that might seem interesting. The interest in computing began when his grandfather bought him a 4-bit microcontroller with 256 byte RAM and a 4096 byte operating system, forcing him to learn assembler before any other language.
After finishing school he went to university in order to study physics. He then collected experiences with a C64, a C128, two Amigas, DEC's Ultrix, OpenVMS and finally GNU/Linux on a PC in 1997. He is using Linux since this day and still likes to take things apart und put them together again. Freedom of tinkering brought him close to the Free Software movement, where he puts some effort into the right to understand how things work. He is also involved with civil liberty groups focusing on digital rights.
Since 1999 he is offering his skills as a freelancer. His main activities include system/network administration, scripting and consulting. In 2001 he started to give lectures on computer security at the Technikum Wien. Apart from staring into computer monitors, inspecting hardware and talking to network equipment he is fond of scuba diving, writing, or photographing with his digital camera. He would like to have a go at storytelling and roleplaying again as soon as he finds some more spare time on his backup devices.
