First time setting up JForex on Amazon EC2 t1.micro

I bumped into a myriads of obstacles setting up the JForex trading platform on an Amazon AWS’s free t1.micro intance. I will go through the steps I went through to setup a cloud server for JForex in this post. In the end, I find out that the t1.micro instance chokes up from running the GNOME desktop environment on Ubuntu Maverick. I enabled Amazon’s CloudMonitor utility and the CPU measurement is locked at 100% from running the JForex platform and the desktop. This is expected as I suspected that the t1.micro wouldn’t be able to handle all that graphics display.

I chose to run Ubuntu on EC2 because that’s what I’m familiar with at home. I also considered running CentOS because it is legendary as an enterprise server. Yet I read reviews from individuals running their own private VPS saying that CentOS is very secure but it is too “tight-assed”. As JForex needs a relatively recent commercial Sun Java version to run, I chose a easier Linux distro for my EC2 instance.

A minor gripe I have with Ubuntu is that their EBS Amazon Machine Image (AMI) comes in 15 GB. Whereas the free offer from Amazon only provides 10 GB of free EBS use. So there’s a 5 GB extra that will be charged on a monthly basis. This has been discussed on the developer forum and it looks as though subsequent versions of Ubuntu release AMI will be in 10GB. However, this 5 GB amounts to only $0.55 a month. Still, I want free!

I started out my trial on EC2 using the server variant of Ubuntu. It has less clutter and potentially more secure than the regular desktop variant. However, getting remote desktop running on the server took me two evenings to figure out! My problem is in getting a NX server to work. I tried the commercial, but free, nxserver from NoMachine. I tried the GPL implementation, FreeNX. And I tried Google’s open source adaptation, Neatx. It just wouldn’t work! As soon as I solved one problem something else breaks. At first it was an authentication problem because the SSH keys were mixed between the NX server and SSH server. Then once that’s resolved, the desktop just wouldn’t start and without any error message to tell me what’s wrong.

That’s when I gave up on NX and switched to using X2go. It only took me a few minutes to install X2go. It ran fine fresh from installation. So many hours wasted on NX.

Once I had my remote desktop running, I tried to install Sun Java for JForex. After a few failed attempts, I found out about this problem. Apparently there’s a kernel bug on Ubuntu in which installing Sun Java on a t1.micro would crash the installer. Just my luck.

By then my curiosity waned and it’s just a matter of getting the job done. So I restarted the whole setup process yet another time with a Ubuntu 10.10 desktop edition (has been using server edition) 64-bit (to circumvent the Sun Java installation bug), installed Sun Java, Google Chrome, and X2go. Logged in to the remote desktop through X2go. Launched Chrome to access the Dukascopy website. Started JForex. It takes only a few minutes once I know what I’m doing.

Then I watched the t1.micro instance come to a crawl.

There’s my first attempt at running JForex on a free t1.micro. My recommendation? Don’t do it.

Rev your trading system on the cloud with a free Amazon cloud server

I always try to keep my trading and any related costs as cheap as possible. And there’s nothing cheaper than free, which that is what Amazon Web Services (AWS) is offering starting November 1, 2010. They are offering a full year of free t1.micro EC2 instance along with other basic level of AWS services (see link below). That seems to be a great incentive to move my trading system on to a remote cloud server.

One of the biggest benefits of running your trading system on a virtual private server or a cloud instance is reliability. I don’t know about you, but my internet connection sometimes acts up or even goes down for hours. There’s the occasional electrical power disruption too, but it’s been good these days.

On the other hand, there are many disadvantages for running your trading system remotely. Costs, technical know-hows, and limited processing power, just to name a few. The added costs is not an issue for now as Amazon is offering a year of free trial. With regard to technical know-hows, I don’t know how to setup a cloud server yet. Although it shouldn’t be a terribly difficult task for me as I am familiar with Linux. What’s at issue though, is the limited processing power.

The processing power of a t1.micro instance is an unspecified fraction of one ECU, which is equivalent to a 1 GHz 2007 Intel Xeon processor. So you get a fraction of that, which doesn’t sound much. CloudHarmony’s benchmark shows that a Micro instance is about 1/10th of that of the Small instance. However, a Micro instance can burst to up to 2 ECUs (double that of the Small) of processing power occasionally. This is good if you can keep your continuous processes minimal and only make calculations infrequently (such as on arrival of a new price bar).

The t1.micro gets 613 MB of memory along with 10 GB of EBS (for low-latency files) and 5 GB of S3 (for persistent storage). These numbers are generous, so the bottleneck obviously remains with the processor.

Your best bet for minimal background processing use is to run a command-line-only trading system. That’s something that can be easily done on JForex. However, there remains the question of is the minimum power of a Micro sufficient to sustain a trading system? Any hiccup in a trading system could cost you more than the annual charge for even a Small instance. This freebie from Amazon may not be worth the risk.

The only way to know is to test it out. I won’t be able to get to that soon as I’m been busy programming. So if anyone is running a trading system on a t1.micro instance, I’d really appreciate if you can provide some feedback.

Update Nov 6: See my first failed test and then my first time running remote desktop on t1.micro.

Otherwise, enjoy the freebie, AWS Free Usage Tier.

First look at Google App Engine for automated trading and quant analysis on the cloud

I just spent the last few hours looking into Google App Engine to use it for trading. Google App Engine (GAE) is a cloud computing development and hosting platform for web applications. GAE is similar to the well-known Amazon EC2 service but it is also very different. The main difference between GAE and EC2 is that GAE is not as flexible as EC2 but it is a lot easier to develop on. Think of it like using C++ (EC2) to write your own trading platform to using EasyLanguage on TradeStation or MQL on Metatrader (GAE).

Another advantage for using Google App Engine is that it is free! Well, to a certain extent. It is free for a limited monthly usage. However, Google’s free quota is known to be very generous (think Gmail) and this is no exception. Based on the numbers, it looks comparable to a typical $20/month virtual private server (VPS) hosting package. My guess is that you can run a decent personal quant program on it if you keep the resources for yourself (private use). For more information on the free quota, you can read the current GAE quota web page or the Wikipedia page for GAE.

You might be wondering how did I spend a few hours just reading on GAE? The answer is that I’ve done a brief research, planning, and assessment to see how GAE can be put to use in terms of quantitative analysis for automated and discretionary trading. Without going through all of my notes and logic, here’s the gist of my conclusion.

For Automated Trading

Since GAE is also a SDK, it has a restricted programming API. You can’t just plug in any custom library of your own (i.e. your broker’s API) into your program. That is its advantage and its disadvantage. Furthermore, GAE only support HTTP and HTTPS. So you can’t open a socket for your connection either. These limitations won’t be changing anytime soon as GAE is designed to be like that for security reasons. Basically, there’s no way for your web application on GAE to connect with your broker for trading. So much for that idea.

For Discretionary Trading

Things look brighter for discretionary trading though. A big bonus with using Google App Engine is that it supports the Google Finance API. Google Finance offers free real-time stock quotes. Combine it with the programming capability of the Google App Engine and you have yourself a free number cruncher in the cloud. However, GAE is definitely no match for your personal computer, so why would you go through all that trouble?

One possible scenario in which this could be useful is for traders that is often on the road or traders such as myself in which we have a day job. In that case, you can set you quant analytics on GAE as a private web application and then you can access it as a web page anywhere on your mobile phone.

One of the obvious limitation with using GAE is that it has limited features for scientific computing. For example, it doesn’t support Numpy in the Python SDK. As such, I can’t imagine using GAE for anything more than simple technical analysis for now. I certainly hope that GAE can include more powerful scientific computing capabilities in the near future.

Until then, I really can’t see Google App Engine offering much help aside from performing algorithmic tasks to help with your trading analysis. Their open issues log does announce that Numpy is being integrated into GAE though. So that’s a great sign for things to come! I am very hopeful of GAE’s potential as a free cloud server to host my quant analytics on. We’ll just have to wait for now.