It's an open buffet in a small business

One of the few benefits of working for myself is that I don't need to worry about compatibility with legacy systems. I am free to use whatever open source tools to get the job done well. The downside to this is that there are so many technologies out there that it's hard to choose the right ones for the job. To give you a sense of what I meant, here are some of the topics that I have either tried or seriously considered in the past year.

  • Programming languages: Java, Python, R, C#, F#, Scala, Clojure, Haskell.
  • Data storage: HDF5, CSV, Binary, MySQL, PostgreSQL, MongoDB, MonetDB, Redis, HBase.
  • Cloud server: Amazon EC2, Microsoft Azure, [Google App Engine][], Rackspace Cloud, plain old VPSs.

Programming language choices are important because they sit at the bottom of a technology stack. Most of the work that I do are built on using them. For a long while, I settled on using a combination of Java, Python, and R. I prototype small ideas in Python. Implement production code in Java. And perform analysis in R. I discussed why use the right tool for the right task a year ago. By the end of my previous project, I am finding that the popular development triplet of Java, Python, and R, is not ideal for a solo-operation. Seeing that I have more time on my hands because I am using QTD to trade for me now, I am taking a break this summer to expand my knowledge and learn new technologies. Some of the technologies that I am experimenting with includes:

  • an in-memory data store for instantaneous and persistent tick data
  • parallel programming for concurrent processing with no locks
  • mathematically intuitive algorithm implementations using high order functions

Don't mind me as I help myself in this open buffet of technologies.

A brief comparison of double arrays for high-performance numerical computing in Java

Numerical calculation performance matters if you're calculating gigantic datasets or have limited processing capability. I belong in the latter case as I intend to run my trading system on a free cloud server in the future. There are plenty of research and studies to push the boundary of high-performance numerical computing in Java (see ref. 1 and 2). Most of the latest and greatest is beyond the scope of my quantitative work. What I seek is performance with simplicity and efficiency. That is how I became aware of the Colt Project by CERN, and its multi-threaded branch. Colt is a set of open source Java libraries for scientific and technical computing problems characterized by large data sizes while demanding high performance and low memory footprint. Just what I needed. Out of curiosity, I tested its implementation of double arrays and compared its performance to standard Java libraries.

Method

A random sample of a million decimal values (e.g. prices) are generated and stored in four Java data structures. All data structures have the same one million values in their own respective memory spaces.

  1. Primitive double array
  2. ArrayList of Double objects
  3. Colt
  4. Parallel Colt

Three tests were ran.

  1. A simple diff function. A traversal of the million data points and calculating array[i] - array[i - shift] for all i = shift to 1,000,000. Where shift is a random integer from 0 to 50. This type of calculation is common place in technical analysis indicators.
  2. A sort function sorting the million data points in the array in ascending order. This come in handy for some statistical analysis. In the case of the primitive array, I am using java.util.Arrays.sort(). Otherwise I'm using the sort() method implicit in the other interfaces.
  3. A binary search function on the sorted arrays. For the primitive array, I am using java.util.Arrays.binarySearch().

Each test is ran 3 times with different random seeds and the results are averaged. The system runs Java SE build 1.6.0_22-b04 and Linux kernel 2.6.35-22 on an Intel Core 2 Duo 6300 @ 1.86 GHz.

Result

For the diff, primitive array averaged 9.3 millisecond; ArrayList 27 ms.; Colt 14 ms.; Parallel Colt 15 ms. For sort, primitive 354.7 ms.; ArrayList 1,316 ms.; Colt 270.3 ms.; Parallel Colt 273 ms. For search, primitive 559.7 ms.; ArrayList 1,561.7 ms.; Colt 575.7 ms.; Parallel Colt 551.3 ms.

Discussion

As this isn't a vigorous test, I wouldn't mind too much on the actual numbers. The take home message here is that the performance of Colt is at least in the same magnitude of a primitive array. Whereas a standard ArrayList object is a magnitude worse. However, where Colt really shines is in its methods for handling matrices. It offers plenty of convenient functions for matrix calculations while guaranteeing good performance. To work with matrices in Java, you either use third-party libraries such as Colt or manage a bunch of for loops. Which isn't pretty.

References

[1] Moreira, et al., "Java programming for high-performance numerical computing," IBM Systems Journal, Vol. 39, No. 1, pp. 21-56, 2000. [2] Wendykier and Nagy, "Parallel Colt: A High-Performance Java Library for Scientific Computing and Image Processing," ACM Transactions on Mathematical Software, Vol. 37, No. 3, September 2010.