Graphs from network monitors on my remote trading server

Here are some server monitoring graphs from my remote trading server. These graphs are produced by the server monitoring application, Munin. I ran both JForex and Metatrader 4 simultaneously on my server last week. The 512 MB of memory was noticeably insufficient. Which caused the system to use swap files, thus leading to I/O latency, and which held up the CPU. All of which are evident from the graphs below. This week I am only running JForex. The server load is within tolerable range. The thing with picking your own VPS is that no VPS provider is the same. You can't really compare just the numbers as there are just too many factors to be considered behind the scene on a VPS service. So it is important to run applications that you intend to run and stress your system in real-world conditions. If your system seems slow, figure out where exactly is the bottleneck before throwing away money for upgrades. This is where a server monitoring application comes handy. [caption id="attachment_5437" align="aligncenter" width="497" caption="CPU data"][/caption] [caption id="attachment_5438" align="aligncenter" width="497" caption="Memory data"][/caption] [caption id="attachment_5439" align="aligncenter" width="497" caption="Interrupts data"][/caption] [caption id="attachment_5440" align="aligncenter" width="497" caption="Disk I/O latency"][/caption] [caption id="attachment_5441" align="aligncenter" width="497" caption="Network transfer"][/caption]

Porting JForex QTD to Metatrader 4

I just finished porting Quantised Trading Desk to Metatrader 4. So that I can make use of my dormant trading account at Oanda. Seeing that Oanda has enabled use of Metatrader 4, I thought I might give them a second try. I've always preferred to have more than one active trading accounts to serve as a backup and comparison. I settled on Questrade and TD Waterhouse for stocks and options, after hopping around brokerages for years. And I've been using Dukascopy to trade forex after much contention. This is a good chance to diversify as I don't want to pigeonhole myself into one broker and one platform. The move to Metatrader 4 up until this point only took a couple of weeks as opposed to JForex version's year-long development time. This is because the underlying QTD system dynamics and algorithms were ironed out in JForex. I don't think I can finish QTD if I only had MQL4 at my disposal. Debugging and extending MQL4 scripts are a nightmare to say the least. I am so glad that I moved away from these "easy" languages. For the MQL4 version of QTD, I am merely doing a direct source code port from JForex to MQL4. MQL4, if anything else, is a very easy scripting language. I just taped out an alpha version of MT4 QTD this weekend as it is up to a stage where it is functional. However, porting QTD to MQL4 isn't without its difficulties. For one, MQL4 is tailored for simple trading with technical analysis indicators. If price move above moving average, go long, for example. Whereas QTD rely on statistics and numerical methods in its trading algorithm. As such, I had to go back to basics and define all of my numerical methods from the fundamentals. I am now moving on to do forward testing on the MT4 QTD to ensure that both the JForex and the MT4 versions work similarly. I can't wait til I can run QTD on Dukascopy and Oanda simultaneously.

Encrypting your communications to minimize eavedropping

The case of Galleon Group founder Raj Rajaratnam shows a corporate security crack that shouldn't happen in this day and age. In the Rajaratnam case, government wiretaps served as vital evidence against him. The question though, is that, if the government is able to do that on a multi-billion dollar company without them knowing, what about their competitors or enemies with ill-intent? With all the resources available to them, how can something so old-school like this happen to a hedge fund? Secrecy and conspiracy shouldn't be unfamiliar in the hedge fund industry. So I had assumed that they operated inside an iron wall. Government wiretapping or corporate espionage are certainly not my worry. However, I still try to employ prudent security procedures in the financial quantitative research and development that I do at Quantisan Systems just because I value my intellectual property. For example, in my email correspondence with business associates, I make use of GPG with a 2048-bit RSA asynchronous key pair. (my public key is below) GPG is an open-source equivalent of Pretty Good Privacy. It is most likely that your email client supports it too. I use Thunderbird and it is a built-in feature. You just need to enable it and set it up. On voice communications, I prefer to use Skype rather than regular voice calls because it's free and Skype inherently uses 256-bit AES encryption. So no setup is even required for this added security. For my important files, I place them in a TrueCrypt volume encrypted with 256-bit AES encryption. The encrypted container is synchronized to my Dropbox account so that I can use it anywhere while ensuring security. TrueCrypt is available on Windows, Mac, and Linux. It's only a matter of following the installation instructions to safeguard your files from prying eyes. All of these encryption tools are open-source and free. I configured them to work in the background and they don't hinder my work flow at all. It is certainly not a sophisticated setup, yet it isn't too dingy either. At the very least, if someone is to steal my notebook computer, an ordinary crook wouldn't be able to gain access to my sensitive information. In view of the fact that Quantisan Systems is just an one-man operation with practically no resource available, I can't believe security at some financial companies is worse than this homebrew setup. -----BEGIN PGP PUBLIC KEY BLOCK----- Version: GnuPG v1.4.11 (GNU/Linux) mQENBE1Ul0gBCACZGqPoYrzRF9YcuUvMeES+etPUAeNQ6uvw42k7Sf1ydBp3lFGQ ggYnOsmQ8Nf/62sB9CoYkAV+h+CPTuTOT5N4gwTojtPX3bIrjA6JgTDpt7bpwyMM j+mnZboXdBgZwz1xqiB38Oj51OBYPPLbTc/YphYfjXEuO4tQwsQufd5dvJqF+Yse CGeLRroJt9EmGVYW/NaCEBKHaMnD9gkDyWLxm7GFQYhl7ZWU+VQXjBxzZyYtpup2 s1N5/igjiRj8t5CGnMwrG2yZkPW+n8m5sTLjcHWFY9OxiqFRaYgm9Vk3mVhxE5ML DPtSPySF+KkyIFxeL78MscU2W1DMH5pNM/+fABEBAAG0HVBhdWwgTGFtIDxwYXVs QHF1YW50aXNhbi5jb20+iQE+BBMBAgAoBQJNVJdIAhsjBQkFo5qABgsJCAcDAgYV CAIJCgsEFgIDAQIeAQIXgAAKCRD8QV1yPcrlhIphB/9cxkbWJ+19axfZxFYRE1u4 Gq1p5/hkm3990yT8kc+4Qg7xa2cm+SgBmEnAy24xdQpR/Up/EF13D15ZBiE1SG+D 0+EUZp4Ir5+TTuTrmJxKRb56IgFZVXP9KatNuK3aPvNFl4gZUZUZHmvUvvBzqtSM HrfzjmuirXVKI6M5WPZchLNUGE1sDtR2SR7l+rjvcPTehaJf+9srK1gsl3Vb3Pzo D5/JBoNK1ueDpfRxNhdryCUQr2uJ1pWLgauDz0iSRPOczV8x1n+051DJQXHx16tY I0VOErTVxscE/KeuuiiFwOwSLteIMDIY2vg6KuUw3cQxxMcfeNTu+jTJXuroib+O uQENBE1Ul0gBCAD05TZ2/oF1Lb8ggg9nKlgB+ZXWBVCc9CNlguZpZAmC7ezc72qT gSho/5+3ldEqvA1AxnWubaWV26QLL2fNybKd1DG9uYOsnOwx6L5H8toQswcurC9y 3jKVQSpuIJrsrQiTT3aIZ3ZONGsg0k0D+ZiMC9qSLmFXh3f2fio5QsyRsEWlC9Qj CDXrgpNZqpDegMvvRXXJfRGulh4sMTGXrxE2pXI/+9Hv64+BuBMkixA7UwiOBQbd V5GkVm7V2hvgQpLc2QaXofLdBVIda6hvUuU/BK5m7HShNPKc21VgcSbYFOggYDal efTeFdCfHEg08ND8JSa2Bv+fdEmOFjPF5Z9hABEBAAGJASUEGAECAA8FAk1Ul0gC GwwFCQWjmoAACgkQ/EFdcj3K5YTRxAf8DspugdiTZUxIT/1EAa/0GWpsT8YZmITm 0iUebID3jZRLXpyqfDr9KUZvk6IMYru1h6c6pZdzY4IJ/p/sZ+p2gtLr1pmOqEjd W94Q3adtjH/42zauFOdCzamtbc7Q5cJTIeAFcvFsVqDLWkK1vWiUhqUoacgN6TWS MActe4B7hYdcMWsbj4j3xVoGkkiJOL81/MJW9f/xDqcNbLsiEK7DxOaWUb2CkuUl 7c/VVaSuWFBKGsj2fgEoTErfsmhjMm+WUcoE1Zjn2UwJzxxIcfzj3A0IEvMIua6u UJWnKv8t55G9f+NQ3yrKtPC/+O7N1sCSilmB/IekaSrBX9KvZgHM8Q== =DjK6 -----END PGP PUBLIC KEY BLOCK-----

Choosing a budget VPS provider for hosting automated trading programs

I've been looking for a cheap but reliable virtual private server (VPS) to run my trading program for the last few months. I ran my QTD program on 3 remote servers in UK and Germany in April for an entire month as performance testing. One in UK on a Xen cloud. One in Germany on OpenVZ. And a last one also in UK on KVM technology. This post is a summary of my initial findings on what to watch for when choosing a server provider for running a remote trading server. Keep in mind that this is for low-end, low-frequency, retail trading. Where latencies in the range of tens of milliseconds are considered good. General target price range is US\$10 - \$20 per month for an unmanaged server with 512MB of ram and 600MHz-equivalent of circa 2006 Intel CPU. Low price is the primary consideration here. If you don't care for the technical details, the top VPS names are Linode, Hetzner, and 6sync for Linux-based VPSs. Linode offers Xen VPSs in US and UK. 6sync offers KVM VPSs in US only. Both companies offer Linux-based 512MB instances for \$20 as a starter server. Hetzner offers KVM VPSs in Germany for €7.90. Cloud servers like Amazon EC2 and Rackspace Cloud are too expensive for running 24/7.

Location, Location, Location

Use a datacenter that's at least in the same sub-continent as your broker's datacenter. You wouldn't want your data to travel half-way around the world just to save a few bucks. It adds unnecessary latency and prices are comparable in either the states or Europe nowadays. Although Asia is another story. If you use more than one broker, you can use multiple servers or pick a server location that sits between them on the internet backbone. TeleGeography provides global internet maps to identify best locations. I trade with Dukascopy and Oanda. One in Switzerland and another in the states. As such, Internet hubs in cities like New York, London, and Frankfurt are prime targets for low latency between both brokers. As I trade more at Dukascopy, I'm biased to European servers. Furthermore, London and Frankfurt are the #1 and #2 internet hub in the world. London has 7,723 Gbps and Frankfurt has 7,218 Gbps capacity in 2010. Whereas New York, 5th in the world, has 3,850 Gbps.

Virtualization Technologies

Virtual private servers are merely reserved resource chunks of a computer in a datacenter. This is achieved through the use of virtualization technologies. And there are quite many of those as I have discovered. Here are three that you'll likely hear about in your search.

  1. OpenVZ. This is the most popular and least preferred platform for trading servers. It's often used by budget hosts because resources between virtualized instances are not well isolated. As such, a host can oversell a server's resource as most webapps have sporadic resource utilization trend. In other worlds, it is unlikely all the virtual private servers would demand their maximum allocated resources at the same time. However, trading servers require a consistent and guaranteed level of computing resource. So stay away from VPS that runs on OpenVZ unless you want to see server hiccups.
  2. Xen. This is what Amazon EC2 and Rackspace Cloud runs on. Xen offers true resource isolation so that you're less likely affected by your virtual server's neighbours. What you see is also what you get. So if you're promised 512MB of memory, you'll get 512MB of memory. However, processing power varies tremendously across different VPS providers. A micro instance on Amazon EC2 offers 613MB of memory, for example. But it is no match for even a 256MB instance on Rackspace Cloud. I was able to run a LXDE desktop GUI plus a java program on Rackspace but couldn't do it on Amazon.
  3. KVM. KVM uses the Linux kernel to virtualize. It's said to offer lower overhead to the host server so that it can provide better value than Xen. Like Xen, it offers true resource isolation.

In summary, either Xen or KVM are good but stay away from OpenVZ.

Operating System

Key question here is: Linux or Windows? If you're unfamiliar with Linux and is not interesting in learning about it, use a Windows Server 2003 or 2008 provider. Windows Server 2003 is preferred as it uses fewer resources just to run so it's cheaper. Do note that Windows VPSs are \$10-\$20 more expensive anyway because you have to pay for a monthly license lease. With the use of a Windows-based VPS, you can connect to your server through the remote desktop protocol (RDP). It's just a matter of running the Remote Desktop application on your local machine to your remote server. Then you'll see the remote machine's desktop on your own machine. And then you can control the remote server just like any other Windows computer. Very simple. However, I prefer to run Linux because of its renowned stability and lower cost. The downside is that you need to know what you're doing to tweak the server for trading. Linux uses very little computing resource to run. I can squeeze JForex and Metatrader together on a Linux VPS with only 256MB memory. By comparison, you need at least 512MB on a Windows box just to get the OS running. And there's no licensing fee for Linux too. A double savings.

Summary

When choosing a VPS provider, consider factors such as location, virtualization technology, and operating system(s) offered. Once you've narrowed your search to a shortlist of providers, the next step is to compare their specific VPS offerings by looking at price, services, and technical specifications.

←   newer continue   →