Network latency on Amazon EC2 t1.micro to Dukascopy

While the processing power of a EC2 t1.micro server sucks (benchmarks have shown it is slower than a Nokia N900 phone), network performance is well-known to be exceptional for EC2 servers throughout the spectrum of their offerings. Here's a ping test from my home computer with a ADSL connection in Ottawa, Canada to 194.8.15.1, one of the Dukascopy web servers.

PING 194.8.15.1 (194.8.15.1) 56(84) bytes of data.
64 bytes from 194.8.15.1: icmp_req=1 ttl=242 time=149 ms
64 bytes from 194.8.15.1: icmp_req=2 ttl=242 time=134 ms
64 bytes from 194.8.15.1: icmp_req=3 ttl=242 time=135 ms
64 bytes from 194.8.15.1: icmp_req=4 ttl=242 time=148 ms
64 bytes from 194.8.15.1: icmp_req=5 ttl=242 time=135 ms
--- 194.8.15.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss,
time 4003ms rtt min/avg/max/mdev = 134.364/140.579/149.726/7.053 ms

Then the same test from a EC2 t1.micro instance located in Dublin, Ireland (closest Amazon EC2 server location to Geneva).

PING 194.8.15.1 (194.8.15.1) 56(84) bytes of data.
64 bytes from 194.8.15.1: icmp_req=1 ttl=243 time=40.9 ms
64 bytes from 194.8.15.1: icmp_req=2 ttl=243 time=41.2 ms
64 bytes from 194.8.15.1: icmp_req=3 ttl=243 time=49.0 ms
64 bytes from 194.8.15.1: icmp_req=4 ttl=243 time=41.2 ms
64 bytes from 194.8.15.1: icmp_req=5 ttl=243 time=41.2 ms
--- 194.8.15.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss,
time 4005ms rtt min/avg/max/mdev = 40.961/42.728/49.013/3.154 ms

Exceptional, indeed. With the monthly pricing of a t1.micro going for around \$12/month, there are few reasons not to get a remote server to run your trading system rather than pray for your home internet connection to remain reliable 24/7.

Update with ping test from Berlin courtesy of commenter Holger:

Ping wird ausger 194.8.15.1 mit 32 Bytes Daten:
Antwort von 194.8.15.1: Bytes=32 Zeit=43ms TTL=244
Antwort von 194.8.15.1: Bytes=32 Zeit=43ms TTL=244
Antwort von 194.8.15.1: Bytes=32 Zeit=43ms TTL=244
Antwort von 194.8.15.1: Bytes=32 Zeit=43ms TTL=244
Ping-Statistik  194.8.15.1:
Pakete: Gesendet = 4, Empfangen = 4, Verloren = 0 (0% Verlust),
Ca. Zeitangaben in Millisek.: Minimum = 43ms, Maximum = 43ms, Mittelwert = 43ms

Update April 12, 2011: I've been using Rackspace Cloud (#2 cloud server provider, after Amazon). Here's the ping result from their London, UK server.

PING 194.8.15.1 (194.8.15.1) 56(84) bytes of data.
64 bytes from 194.8.15.1: icmp_seq=1 ttl=246 time=69.2 ms
64 bytes from 194.8.15.1: icmp_seq=2 ttl=246 time=69.6 ms
64 bytes from 194.8.15.1: icmp_seq=3 ttl=246 time=68.8 ms
64 bytes from 194.8.15.1: icmp_seq=4 ttl=246 time=66.6 ms
64 bytes from 194.8.15.1: icmp_seq=5 ttl=246 time=68.5 ms
64 bytes from 194.8.15.1: icmp_seq=6 ttl=246 time=64.6 ms
64 bytes from 194.8.15.1: icmp_seq=7 ttl=246 time=68.5 ms
64 bytes from 194.8.15.1: icmp_seq=8 ttl=246 time=69.1 ms
64 bytes from 194.8.15.1: icmp_seq=9 ttl=246 time=66.6 ms
--- 194.8.15.1 ping statistics ---
9 packets transmitted, 9 received, 0% packet loss,
time 8012ms rtt min/avg/max/mdev = 64.677/67.986/69.617/1.573 ms

Update April 26, 2011: I'm testing Quickweb Germany's VPS.

PING 194.8.15.1 (194.8.15.1) 56(84) bytes of data.
64 bytes from 194.8.15.1: icmp_seq=1 ttl=247 time=15.4 ms
64 bytes from 194.8.15.1: icmp_seq=2 ttl=247 time=15.3 ms
64 bytes from 194.8.15.1: icmp_seq=3 ttl=247 time=15.2 ms
64 bytes from 194.8.15.1: icmp_seq=4 ttl=247 time=15.3 ms
64 bytes from 194.8.15.1: icmp_seq=5 ttl=247 time=15.1 ms
64 bytes from 194.8.15.1: icmp_seq=6 ttl=247 time=15.2 ms
64 bytes from 194.8.15.1: icmp_seq=7 ttl=247 time=15.4 ms
--- 194.8.15.1 ping statistics ---
7 packets transmitted, 7 received, 0% packet loss,
time 6000ms rtt min/avg/max/mdev = 15.152/15.306/15.463/0.116 ms

Update September 2013: DigitalOcean offers $5/month 512MB server plan with 20GB SSD. I've used them exclusively for other work. Here's a ping test from one of their Amsterdam servers.

PING 194.8.15.1 (194.8.15.1) 56(84) bytes of data.
64 bytes from 194.8.15.1: icmp_req=2 ttl=245 time=21.5 ms
64 bytes from 194.8.15.1: icmp_req=3 ttl=245 time=22.4 ms
64 bytes from 194.8.15.1: icmp_req=4 ttl=245 time=22.6 ms
64 bytes from 194.8.15.1: icmp_req=5 ttl=245 time=21.5 ms
64 bytes from 194.8.15.1: icmp_req=6 ttl=245 time=22.4 ms

--- 194.8.15.1 ping statistics ---
6 packets transmitted, 5 received, 16% packet loss, time 5013ms
rtt min/avg/max/mdev = 21.540/22.142/22.686/0.484 ms

Rev your trading system on the cloud with a free Amazon cloud server

I always try to keep my trading and any related costs as cheap as possible. And there's nothing cheaper than free, which that is what Amazon Web Services (AWS) is offering starting November 1, 2010. They are offering a full year of free t1.micro EC2 instance along with other basic level of AWS services (see link below). That seems to be a great incentive to move my trading system on to a remote cloud server. One of the biggest benefits of running your trading system on a virtual private server or a cloud instance is reliability. I don't know about you, but my internet connection sometimes acts up or even goes down for hours. There's the occasional electrical power disruption too, but it's been good these days. On the other hand, there are many disadvantages for running your trading system remotely. Costs, technical know-hows, and limited processing power, just to name a few. The added costs is not an issue for now as Amazon is offering a year of free trial. With regard to technical know-hows, I don't know how to setup a cloud server yet. Although it shouldn't be a terribly difficult task for me as I am familiar with Linux. What's at issue though, is the limited processing power. The processing power of a t1.micro instance is an unspecified fraction of one ECU, which is equivalent to a 1 GHz 2007 Intel Xeon processor. So you get a fraction of that, which doesn't sound much. CloudHarmony's benchmark shows that a Micro instance is about 1/10th of that of the Small instance. However, a Micro instance can burst to up to 2 ECUs (double that of the Small) of processing power occasionally. This is good if you can keep your continuous processes minimal and only make calculations infrequently (such as on arrival of a new price bar). The t1.micro gets 613 MB of memory along with 10 GB of EBS (for low-latency files) and 5 GB of S3 (for persistent storage). These numbers are generous, so the bottleneck obviously remains with the processor. Your best bet for minimal background processing use is to run a command-line-only trading system. That's something that can be easily done on JForex. However, there remains the question of is the minimum power of a Micro sufficient to sustain a trading system? Any hiccup in a trading system could cost you more than the annual charge for even a Small instance. This freebie from Amazon may not be worth the risk. The only way to know is to test it out. I won't be able to get to that soon as I'm been busy programming. So if anyone is running a trading system on a t1.micro instance, I'd really appreciate if you can provide some feedback. Update Nov 6: See my first failed test and then my first time running remote desktop on t1.micro. Otherwise, enjoy the freebie, AWS Free Usage Tier.

Productivity on the road

I am waiting for a flight to Geneva. For a guy that works in an unexciting cubicle, I seem to travel a lot. As Europe is the fourth continent which I am visiting this year. (More about this very exciting trip once I am back) The thing with travelling is that I will be away from my development desktop, amongst other things of course. As I work a day job and can only do my trading analysis and quant development work between the hours that I can squeeze in every day, every free minute is precious. One productivity habit that I adapted from frequent travelling is that I have a portable workflow system implemented. For the sake of completeness by stating the obvious, there are three keys to being productive while on the road.

  1. Exploiting downtime.
  2. Being prepared.
  3. Planning ahead.

Being productive doesn't mean that I coup myself up in a hotel room to work. Enjoying a foreign city and absorbing a different culture help enhance my personal experience in the long run. Being productive means exploiting down-times. The times waiting at an airport (like now), stuck in a plane, the wee hours of the morning when you couldn't sleep because of jet-lag, etc. Being prepared means have your work ready and available. There are three free software/services that I can't live without these days. They are Dropbox, Subversion, and SSH.

  • Dropbox is a web-based file hosting and synchronizing service. Basically, after installing the Dropbox software, anything you put in your "My Dropbox" directory in your computer will be mirrored on their online server. Then you can install the Dropbox software in a second computer and choose to link your private dataset with this second computer. Thus the "My Dropbox" directory will be synchronized in real-time between these two (or more) computers and an online version. All your most important files will be up-to-date and available to you anywhere. No more need for USB drives.
  • A software version control system to keep track and backup my source code revisions. I use Apache Subversion in particular. You can use Subversion to archive any file other than source code. For example, you can use it to keep track of revisions and branches to your presentation slides. [Assembla offers a free 2GB online private repository][] so that you can access it from anywhere and save the hassle of rolling your own repository server.
  • Secure Shell (SSH) is not a service like the other two but a network protocol. It is not as easy to setup so I'll leave it to Google. I use it to securely and remotely access my main computer's file system or the desktop when necessary. A simpler alternative to SSH is to use LogMeIn, a remote desktop service.

Note that some of these links are affiliate programs. But I just use their free services, so I am only recommending you do that too. These tools not only help me stay productive while away, they also ease the trouble of using more than one computers between my office, my home desktop, and my wife's laptop. Together they create a true virtual development environment. While mobile and cloud computing technologies are certainly useful, nothing beats old fashioned planning and organizing. Plan ahead, prepare your work, and have them available with you wherever you are. A lot can be done with plain pen and paper.

How to backup your data on the cloud with ease and for free

Dr. Steenbarger's recent harddrive failure reminded me about the importance of backing up my data. Everything that I do with trading is on the computer. Consequently, I have many files that are vital to my trading. From web bookmarks to chart templates to source codes, I would have to make a checklist to ensure that I backed up everything frequently. But only if I were to do my backup manually. Being the lazy and cheap person that I am, I don't like the idea of forcing myself to backup frequently. That is why my folders are synchrnized automatically to an online file server using Mozy Remote Backup. I don't have to do a thing to know that all my important stuff are safe. There are two reasons why I choose Mozy. It is one of the two biggest names in the business (the other is Carbonite). It is the only one of the two to offer a free for lifetime 2GB online backup space (affiliate). The benefit of using Mozy is that it actively synchronize my files to their cloud storage server. These are files that I change often. It makes sense to use Mozy to monitor and back them up automatically so that I don't have to worry about them. Still, I have another 50GB+ of personal photos, videos, and documents that won't fit in this free 2GB space. Which is why I only use Mozy for my frequently changing or critical files (mostly trading related). For the rest of my 50GB+ data, they are typically static content (e.g. photos and media files). So there is not much point for me to pay a monthly fee. I rather spend a few minutes a month to make backup DVDs or upload them to my Amazon S3 account. That way, I have all my data backed up and get to keep my \$5 a month (price of monthly subscription fee for an unlimited storage space backup) too. I am not being paranoid. Another one of my computers died on me a few years back with no prior warning. Its power supply suddenly blew up in smoke and sparks one day. The scene was quite spectacular. More than a few components were damaged then. I doubt that I will see that kind of fireworks again. But who knows what else could go wrong with this aging computer (4 years and counting)? A data failure with my computer would indeed be very costly for me. I invested a lot of time and energy on my trading work. If my harddrive were to fail, I may lose all of that work if I don't have a backup. That is why I have a free automated data backup plan which doesn't require me to lift a finger.

continue   →