• Buy-side firms engaging in high-frequency trading (HFT), either onsite or at a co-location facility.
• Sell-side firms offering smart order routing or internal order crossing to increase revenue or save on costs of order processing.
• Exchanges accelerating and enhancing their services to compete with ever more sources of liquidity.
• All kinds of capital markets participants improving how they handle real-time risk across trading systems, and/or automate middle-office settlement and record keeping.

Reg NMS and the Explosion in HFT

The path to these systems began with decimilization of equities in 2001 which dramatically increased electronic trading. Then in 2007, Reg NMS opened the door for many more competitors to supply liquidity, which created a n-way set of very-short lived arbitrage opportunities across trading venues. Reg NMS included rules that regulated order fills to assure they were at the best price (across all available liquidity) within a time window (National Best Bid and Offer — NBBO). That led all sell-side participants to require very low latency technology, because they had to fall within the NBBO window to play at all. Mix in a bunch of clever quants to develop models and programmers to automate them and you have the HFT explosion in equities.

Dodd-Frank Sets the Stage for HFT Across More Assets

Over the past 12-18 months we’ve had more and more conversations with people looking to improve capacity of systems that support electronic trading of other asset classes such as foreign exchange, fixed-income and derivatives. I didn’t think too much about why until I read something Patrick Whalen, head of trading for AllianceBerstein, said in this article about the growth of HFT and electronic trading. In a nutshell, Patrick points out that the Volcker Rule will prevent banks from holding assets like bonds or currency positions on their books — they’ll need to move ‘em out quickly and efficiently, which means more electronic trading. As part of the new Dodd-Frank rules about the clearing of swaps, regulators are also pushing for an end to the over-the-phone transactions in favor of more transparent electronic trading. That will change the dynamics of those markets in terms of the need for efficiency, and the mechanics of acquiring those assets.

Thinking about his observations makes the future of electronic trading far more clear. Dodd-Frank will drive more asset classes to electronic trading for transparency and increased monitoring of adherence to rules. Most of these asset classes trade across many venues, which means there will be HFT-style arbitrage opportunities between them. Most of the trading firms have already made investments in low-latency technology, so the implementation times can be very short as the markets increase in liquidity. Therefore the barriers to HFT in asset classes like derivatives, FX or fixed income should be far lower than when equities underwent the shift.

Settle in and Watch the Trading Landscape Shift

Every significant new financial regulation has its own batch of winners and losers — the winners are those who best understand the secondary implications of the new rules, individually and collectively, and position themselves to capitalize on change. “Skating to where the puck is going to be,” as the Wayne Gretzky line goes. As Reg NMS gave birth to accelerated HFT for equities, the changes driven by Dodd-Frank will leave at least as big a change in the markets across many more asset classes, and will accelerate the spread of electronic trading across capital markets. Grab a big bowl of popcorn and settle in to see which firms ride the wave to new levels of success. Like the best movies, there will be lots of plot twists as well as plenty of heroes and villains.

• Big data – for anyone in IT this is a “no kidding Sherlock” inclusion, but it is remarkable to step back and appreciate how quickly big data is changing the landscape and its long term implications.
• Smart manufacturing – The next wave in manufacturing – engineering from the molecular level to manufacture products using methods more like the ones we use to print documents. The emphasis shifts to design and IP, and away from cheap labor and physical factories.
• The wireless revolution – The extension of internet access to more than a billion of the world’s people, paired with a new generation of applications and services, is fundamentally changing the way the world connects, socializes and engages in commerce.

The author does a little flag waving around America being an epicenter for these three trends, but the more important point is that all three are underway now and are good bets to literally change the world.

These trends line up very nicely with the big picture principles of Solace — to unshackle information and make it available wherever it is needed in ways not currently possible. Whether loading up a big data repository, collaborating on a design for smart manufacturing, or giving the world’s mobile user base real-time access to…well, anything…we’re in step with the vision outlined in this article.

If you haven’t read it, I recommend you check it out: The Coming Tech-led Boom (Wall Street Journal)

This is part of a project called GOES-R (Geostationary Operational Environmental Satellite R-Series). Here is its stated mission as described on the GOES-R website:

The advanced spacecraft and instrument technology used on the GOES-R series will result in more timely and accurate weather forecasts. It will improve support for the detection and observations of meteorological phenomena and directly affect public safety, protection of property, and ultimately, economic health and development.

GOES-R provides essential information related to air quality, coastal and marine monitoring, fire monitoring, hurricane forecasts, precipitation and floods, land cover observations, volcanoes, lightning detection, severe thunderstorms, tornado warnings and more. If you’re interested in learning more about this project and the advances it will deliver, I highly recommend checking out the GOES-R site. Here are some fun facts I learned in just a few minutes of browsing:

• The instruments on the GOES-R series will produce more than 50 times the information provided by the current GOES system and will offer unique observations of the environment, with emphasis on hazardous weather in the western hemisphere and space weather impacts.
• Data provided by the GOES-R instruments will be used to create 65 different products that meteorologists can use to monitor the weather.
• Information from the GOES-R satellite will improve short-term weather forecasts such as severe thunderstorms and tornadoes, as well as long-term climatological trends.
• GOES-R will be the first of a new fleet of GOES satellites (R/S), marking the first significant upgrade in GOES weather monitoring capabilities since the start of the GOES I/M series in 1994.

Certainly, LAN and WAN network technology introduces many limits and the largest commercial databases (e.g. Netezza, Teradata) or open source big data stores (e.g. Hadoop, Splunk) can only store data so fast. Even in memory data grids are limited by how many in-memory writes can be performed per second. Managing the distributed information is usually some kind of middleware, once again, usually a commercial product (e.g. JMS or MQ) or open source code (e.g. Kafka or Qpid).

Even at full speed, a single instance of the middleware layer runs at far less capacity than the network, in-memory grid, or data store can process, making it the weakest link. This means to keep up, the software middleware traffic has to be scaled horizontally across many middleware brokers or servers. Each application becomes a fragile layered mess of servers and any disruption can lead to significant cascading problems of volume and backlog.

An increasing number of our customers with big data projects (e.g. in capital markets, internet infrastructure and  transportation) have thrown in the towel on attempting to use traditional JMS, MQ, or open source for this scale of data capture. Instead, they’re opting for Solace’s hardware messaging to feed their big data stores. Where software messaging peaks at a few thousand messages per second, Solace’s failsafe queuing solution exceeds 150,000 messages per appliance. That means you would need to horizontally scale a typical JMS, MQ or open source alternative to 30 or more servers (assuming it could sustain 5,000 msgs per JMS server) to match the throughput of one Solace appliance. It just makes everything easier if the layers and moving parts in your scaling architecture stay light and lean. Fewer servers, less datacenter space, fewer outages = cheaper and less headaches.

Many customers initially think a commercial solution like Solace’s has to be more expensive than open source, after all open source is free and Solace costs money. But it is easy to show that when you factor in server costs, rack space, power, and management it’s far cheaper to pay for an appliance that replaces 30 or more servers.

Big data is right in the sweet spot of (one of the many) use cases that this company was built to address. If you are struggling with these problems, we’d like the opportunity to talk to you about solving them.

Today we announced a new partnership integrating our message routers with the B2B and Managed File Transfer (MFT) components of SEEBURGER’s Business Integration Server platform. The big “why” behind this partnership is captured in the “growth of global data” factoid above. SEEBURGER is a powerhouse in the B2B and MFT spaces, and does a lot more application integration (A2A/EAI) than people think.

Today, the growth in data generated by these systems puts stress on the enterprise services bus underneath their software, but with Solace integrated into BIS and MFT, SEEBURGER customers get worry-free capacity for their distributed information requirements in an easy-to-use appliance form factor.

As the volume of information grows, so does the need for smarter distribution technology. Remember, most popular middleware technologies were invented more than a decade ago, when information volumes were 1/32^nd what they are today. At Solace, we’ve changed the middleware game by boosting throughput by orders of magnitude to help companies deal with the flood of data without suffering from the cost and complexity of datacenter sprawl.

That’s just what we recently announced with our partner, BCC Group of Germany. BCC Group introduced a solution called CALCNODE that can run thousands of very complex Excel spreadsheets with better performance and accountability than ever before.

CALCNODE offloads complex calculations to a high-performance computing environment, and uses our appliance to supply the spreadsheets with real-time market data and keep real-time data flowing between spreadsheets and the powerful calculation engine which lives in a shared back end.

This lets companies boost the productivity of their employees by making their existing Excel spreadsheets run faster, while simultaneously giving them unprecedented oversight and accountability so they can better comply with increasingly stringent government regulations.

It’s an ingenious use of high-speed messaging and high performance computing to solve a long term challenge – reducing the risk of using of Excel in trading.

We’re especially proud of a leadership award we were given, described by Deloitte’s Mark Noonan in an Ottawa Business Journal article:

“Solace won a special leadership award in the category of hardware for demonstrating their ability to create a distinct competitive advantage in a high-growth market, which allows them to dominate their sector.”

We’re proud of our #25 spot on the list, and as Noonan further explained we would have cracked the top 10 if our fiscal year mapped to the calendar year. “If we look at their current year revenue model, they have achieved accelerated revenue in 2011. If we were to annualize that, Solace’s growth rate would almost be 2,600% percent, which would have been top 10,” he said.

And since real growth and results are what matter, we’re OK with that.

If efforts regarding regulations proceed, I hope HFT gets a fair trial. In the current economic climate and election cycle, there’s a real risk that media punditry, political grandstanding, and whatever voice the Occupy Wall Street movement ends up contributing might skew an outcome before the data and facts are understood.

It seems to me that the genie is out of the bottle where electronic trading is concerned. We need to make sure that we understand and tweak the model for maximum benefit, not initiate change based on fear of the unknown.

Their site includes:

• In-Play sports betting
• A wide range of popular casino, poker, lottery and bingo games.
• The ability to bet on financial markets, such as the closing prices of currency pairs and market indices.

It’s a remarkably rich and diverse offering, and if you take a look at their site, www.bet365.com, you’ll quickly realize how hard the company works to offer the best experience they can for their customers.

That’s all great, but why am I writing about gambling innovation in a middleware blog? Let’s take a closer look at what’s going on behind the scenes. Each day bet365 can serve over one hundred thousand concurrent users on their main In-Play system and over a million on their partner banner system. Odds information is updated every few seconds and around a million transactions can be processed in a day. That all adds up to the need for a seriously secure, robust and efficient data distribution infrastructure that gets the odds data out and can efficiently process inbound activity.

For bet365, Solace gives the confidence that as the business continues to evolve, they can reliably meet the demand for their services far into the future. We’re proud to be of service to such an innovative and respected market leader.

This article ties together three of my favorite topics – big data analytics, cloud computing infrastructure and baseball. Take a look at the iPad dashboard at right and think about a starting pitcher and catcher sitting together on a flight using this kind of highly-visual tool to decide how to pitch each hitter. Now think about the rudimentary paper-based systems from five or ten years ago. Which pitcher has the edge? I’m sure that comparable data exists for batters on pitcher’s tendencies and release points, but it does appear that the overwhelming advantage of this technology favors the pitcher/catcher game plan.

But do the results back up the theory? Take a look at the aggregate statistics across major league baseball the last 5 years. Since 2006 both league batting average and runs per game have fallen every single year. Batting averages are down from .269 to .255 and runs per game are down from 4.86 to 4.28 (per team). Only one offensive stat is up in each of those years – strikeouts, from 6.52 per team per game to 7.06. MLB is marketing this as a pitching renaissance. Maybe it’s a data/knowledge renaissance? Pitchers simply have better tools than hitters and it’s showing up as a reduction in offense.

This phenomenon is not at all unique to baseball. What Moneyball and sabermetrics did for baseball happened a decade before in financial services. Human traders that specialized in arbitrage and technical chart reading have been overrun by automated systems that perform split-second algorithmic trading resulting in billions in profits. Online giants like Google and Amazon have applied similar big data analytics techniques to know what you want to buy before you do. I would be willing to bet that a similar explanation is behind recent record big company profits, despite the luke warm global economy. The past decade has seen an explosion in the number and accuracy of tracked company metrics, along the lines of the baseball dashboard above. It’s the same story – interactive dashboards and process automation are buoying decision making accuracy, resulting in fewer mistakes and greater corporate efficiency. This is a classic example of what Greenspan would have called “worker productivity improvements enabled by technology”.

We are marching inexorably towards a interconnected world of huge volumes of ever changing data and anywhere, anytime access which will make today’s incredible improvements look quaint when we look back ten years from now. Whether you work in a baseball front office, at a leading internet company, in corporate IT or supply companies like these with the world’s most scalable messaging middleware it’s an exciting time to be building applications that connect the web and mobile worlds with the processing power of the cloud in real time.