High-Performance Computing in Finance: A Customer’s Perspective [1/7]
11 Jun 2007
Excerpted from a paper I delivered on January 16, 2007 at the Microsoft High-Performance Computing in Financial Services event in New York.
Software engineers like myself, who work in finance and have a keen interest in high-performance computing, are kids in a candy store. Everywhere we look — from front-office to back, from portfolio construction to trade execution, from data capture to quarterly reports — we see algorithms and business processes that are inherently parallel and deeply amenable to high-performance computing. We delight in the fact that our business comes pre-sliced into portfolios, asset classes, markets, securities, trading sessions, and ticks. And even when we get all fancy and use things like Monte Carlo analysis, genetic algorithms, and simulated annealing, we’re still dealing with a rain of independent calculations that rarely rendezvous. They call these types of problems embarrassingly parallel in the business. I can tell you, though, there’s nothing embarrassing about them. Frankly, if you want to do high-performance computing, they’re the types of problems you’d like to have.
But it isn’t all rosy. At my firm, we’ve only parallelized a fraction of what is possible. Despite having an optimal problem set, we are still hampered in our efforts to put all this parallelism potential into production. As it turns out, high-performance computing in finance has some peculiarities that are not entirely addressed by the stable of tools currently available. We are making progress, but only by writing services and infrastructure totally orthogonal to our primary business of investing money.
Historically, high-performance computing has been the stock in trade for research and academic institutions doing “grand challenge”-type problems. Simulations of nuclear weapon explosions. Computational fluid dynamics. Brute-force attacks on cryptographic keys. Analysis of extraterrestrial radio signals. These efforts carried the flag of high-performance computing into the public consciousness and created MPI and grid computing and other foundational technologies and patterns that many of us use today. But just as knowing Latin doesn’t get you cross-town in Mexico City, replicating these classic efforts doesn’t quite deliver the functionality we need in finance today.
In some ways, our problems are simpler. Broadly speaking, there aren’t as many mesh-like algorithms in finance that require constant coherence of shared state. We either operate at less granular levels of parallelism or have truly independent tasks that have no shared state at all. On the flip-side, we deal with heterogeneous data sources, security requirements, and configuration and deployment concerns that, while largely absent in the classic applications, are painfully prominent in ours.
Regardless of the obstacles, though, all of our roadmaps point to high-performance computing. It’s inescapable. More literally than any other industry, finance is all about turning data into information, information into action, and action into profit. And that transformation gorges on computing power. The volume, variety, and relative unreliability of the data that we have to work with is always proliferating. We are constantly evolving more sophisticated financial models and weaving ever more complex and conditional business rules. We are expanding into new markets and dealing in new instruments. We are constantly trying all we can to get a leg up in age of extreme market and information efficiency and find new ways to transmute data into gold.