Building a distributed back-tester with Hadoop on Amazon AWS

Testing is arguably the single most important aspect of trading system development. You can’t tell how well an idea works unless you test it out. Testing can also help you identify weaknesses or strengths in your model. The downside to testing is that it takes time to churn through those gigabytes of data.

Backtesting is inherently a linear process. You feed in your tick data into your algorithm and expect some actions. You can’t really make use of fork/join to let other threads steal from the process queue as the later process depends on results from the earlier calculations. However, often times than not, you’re interested in testing many variations of a strategy. This is where MapReduce comes into play.

MapReduce is a Google software framework. It is inspired by the map and reduce functions ubiquitous in functional programming. They are as common as for-loops in the Java world.

The map function partitions an input into smaller problems and run them concurrently, e.g. each of the strategy’s variant is executed on a node.

The reduce function takes the results from all the nodes and aggregate them to get an output, e.g. back-test results from each strategy.

Having used functional programming for some time now, using map/reduce is very natural for me. Where my knowledge falls short is in implementing a distributed infrastructure for running these map and reduce with massive scaling beyond my own multi-core computer.

It just so happens that Amazon AWS has a hosted Hadoop PaaS. Where Hadoop is the Apache’s framework for MapReduce. Hardware, check. Framework, check. This will be the first second system that I’ll be working on in my goal to build a complete trading R&D platform.

Expect some technical discussions in the coming months as I work my way through. Now, where should I start…

Towards a broker-agnostic trading system

The first generation of my trading system uses a broker’s proprietary script on that same broker’s platform. The second generation of my trading system uses a generic programming language on another broker’s platform. I am now working on the third generation of my trading system. It is language-agnostic, platform-agnostic, and broker-agnostic.

Why? Because designing trading systems is as much about creativity as the science. An artist would not limit themselves to a defined toolset. I want to be free to use whatever tools are necessary to get the job done efficiently and being able to try new things as technologies evolve.

Having a broker-agnostic system frees me from being locked into any particular broker. Conversely, I could use one system to gather data and trade from multiple brokers for the best execution prices.

Yes, this is over-engineering for a small entity like myself. But heck, this is something that I enjoy building. So why not?

Graphs from network monitors on my remote trading server

Here are some server monitoring graphs from my remote trading server. These graphs are produced by the server monitoring application, Munin.

I ran both JForex and Metatrader 4 simultaneously on my server last week. The 512 MB of memory was noticeably insufficient. Which caused the system to use swap files, thus leading to I/O latency, and which held up the CPU. All of which are evident from the graphs below.

This week I am only running JForex. The server load is within tolerable range.

The thing with picking your own VPS is that no VPS provider is the same. You can’t really compare just the numbers as there are just too many factors to be considered behind the scene on a VPS service.

So it is important to run applications that you intend to run and stress your system in real-world conditions. If your system seems slow, figure out where exactly is the bottleneck before throwing away money for upgrades. This is where a server monitoring application comes handy.

CPU data

Memory data

Interrupts data

Disk I/O latency

Network transfer

Porting JForex QTD to Metatrader 4

I just finished porting Quantised Trading Desk to Metatrader 4. So that I can make use of my dormant trading account at Oanda. Seeing that Oanda has enabled use of Metatrader 4, I thought I might give them a second try.

I’ve always preferred to have more than one active trading accounts to serve as a backup and comparison. I settled on Questrade and TD Waterhouse for stocks and options, after hopping around brokerages for years. And I’ve been using Dukascopy to trade forex after much contention. This is a good chance to diversify as I don’t want to pigeonhole myself into one broker and one platform.

The move to Metatrader 4 up until this point only took a couple of weeks as opposed to JForex version’s year-long development time. This is because the underlying QTD system dynamics and algorithms were ironed out in JForex. I don’t think I can finish QTD if I only had MQL4 at my disposal. Debugging and extending MQL4 scripts are a nightmare to say the least. I am so glad that I moved away from these “easy” languages.

For the MQL4 version of QTD, I am merely doing a direct source code port from JForex to MQL4. MQL4, if anything else, is a very easy scripting language. I just taped out an alpha version of MT4 QTD this weekend as it is up to a stage where it is functional.

However, porting QTD to MQL4 isn’t without its difficulties. For one, MQL4 is tailored for simple trading with technical analysis indicators. If price move above moving average, go long, for example. Whereas QTD rely on statistics and numerical methods in its trading algorithm. As such, I had to go back to basics and define all of my numerical methods from the fundamentals.

I am now moving on to do forward testing on the MT4 QTD to ensure that both the JForex and the MT4 versions work similarly. I can’t wait til I can run QTD on Dukascopy and Oanda simultaneously.

What is a trading system?

For some traders, building a trading system means that they focus on finding the best indicators they could get their hands on or the one trading setup that is backtested to work 99% of the time. All I can say is, to each his own.

For me, the keyword of a trading system lies not with trading, but with system. Remember, it is a system. First of all, it is a system for trading. Not a system to trade. What do I mean by that?

Consider this saying, “traders don’t go broke by missing opportunities, they go broke by taking too many.” A trading system needs to be selective. It needs prudent risk management.

Secondly, you need to generate your own market view to trade. Rather it be technical, fundamental, sentimental, or just following some joe, you trade based on your view. Trading is a test of your market hypothesis using money. If you think the market will rise, you buy. If you think the market will fall, you short.

Lastly, a trading system needs to execute trades. Otherwise you’d be an anlayst. It needs to time entries and exits. Again, this could be as simple as on a whim or as complicated as some n-th order algorithm.

Thus we have the three pillars of a trading system.

1. Risk Management
2. Market Analysis
3. Trade Execution

What’s more, if we combine some of these pillars we get some well known concepts. For example, market analysis + trade execution = trading signals, wherein you derive signals based on market conditions to execute trades. Or risk management + trade execution = position management, wherein you manage your holdings by manipulating your trades based on your risk profile.

All these relationships and composition of a trading system can be summarized with a Venn diagram. So, this is my view of a trading system:

Trading system Venn diagram