• Big data – for anyone in IT this is a “no kidding Sherlock” inclusion, but it is remarkable to step back and appreciate how quickly big data is changing the landscape and its long term implications.
• Smart manufacturing – The next wave in manufacturing – engineering from the molecular level to manufacture products using methods more like the ones we use to print documents. The emphasis shifts to design and IP, and away from cheap labor and physical factories.
• The wireless revolution – The extension of internet access to more than a billion of the world’s people, paired with a new generation of applications and services, is fundamentally changing the way the world connects, socializes and engages in commerce.

The author does a little flag waving around America being an epicenter for these three trends, but the more important point is that all three are underway now and are good bets to literally change the world.

These trends line up very nicely with the big picture principles of Solace — to unshackle information and make it available wherever it is needed in ways not currently possible. Whether loading up a big data repository, collaborating on a design for smart manufacturing, or giving the world’s mobile user base real-time access to…well, anything…we’re in step with the vision outlined in this article.

If you haven’t read it, I recommend you check it out: The Coming Tech-led Boom (Wall Street Journal)

This is part of a project called GOES-R (Geostationary Operational Environmental Satellite R-Series). Here is its stated mission as described on the GOES-R website:

The advanced spacecraft and instrument technology used on the GOES-R series will result in more timely and accurate weather forecasts. It will improve support for the detection and observations of meteorological phenomena and directly affect public safety, protection of property, and ultimately, economic health and development.

GOES-R provides essential information related to air quality, coastal and marine monitoring, fire monitoring, hurricane forecasts, precipitation and floods, land cover observations, volcanoes, lightning detection, severe thunderstorms, tornado warnings and more. If you’re interested in learning more about this project and the advances it will deliver, I highly recommend checking out the GOES-R site. Here are some fun facts I learned in just a few minutes of browsing:

• The instruments on the GOES-R series will produce more than 50 times the information provided by the current GOES system and will offer unique observations of the environment, with emphasis on hazardous weather in the western hemisphere and space weather impacts.
• Data provided by the GOES-R instruments will be used to create 65 different products that meteorologists can use to monitor the weather.
• Information from the GOES-R satellite will improve short-term weather forecasts such as severe thunderstorms and tornadoes, as well as long-term climatological trends.
• GOES-R will be the first of a new fleet of GOES satellites (R/S), marking the first significant upgrade in GOES weather monitoring capabilities since the start of the GOES I/M series in 1994.

Certainly, LAN and WAN network technology introduces many limits and the largest commercial databases (e.g. Netezza, Teradata) or open source big data stores (e.g. Hadoop, Splunk) can only store data so fast. Even in memory data grids are limited by how many in-memory writes can be performed per second. Managing the distributed information is usually some kind of middleware, once again, usually a commercial product (e.g. JMS or MQ) or open source code (e.g. Kafka or Qpid).

Even at full speed, a single instance of the middleware layer runs at far less capacity than the network, in-memory grid, or data store can process, making it the weakest link. This means to keep up, the software middleware traffic has to be scaled horizontally across many middleware brokers or servers. Each application becomes a fragile layered mess of servers and any disruption can lead to significant cascading problems of volume and backlog.

An increasing number of our customers with big data projects (e.g. in capital markets, internet infrastructure and  transportation) have thrown in the towel on attempting to use traditional JMS, MQ, or open source for this scale of data capture. Instead, they’re opting for Solace’s hardware messaging to feed their big data stores. Where software messaging peaks at a few thousand messages per second, Solace’s failsafe queuing solution exceeds 150,000 messages per appliance. That means you would need to horizontally scale a typical JMS, MQ or open source alternative to 30 or more servers (assuming it could sustain 5,000 msgs per JMS server) to match the throughput of one Solace appliance. It just makes everything easier if the layers and moving parts in your scaling architecture stay light and lean. Fewer servers, less datacenter space, fewer outages = cheaper and less headaches.

Many customers initially think a commercial solution like Solace’s has to be more expensive than open source, after all open source is free and Solace costs money. But it is easy to show that when you factor in server costs, rack space, power, and management it’s far cheaper to pay for an appliance that replaces 30 or more servers.

Big data is right in the sweet spot of (one of the many) use cases that this company was built to address. If you are struggling with these problems, we’d like the opportunity to talk to you about solving them.

If efforts regarding regulations proceed, I hope HFT gets a fair trial. In the current economic climate and election cycle, there’s a real risk that media punditry, political grandstanding, and whatever voice the Occupy Wall Street movement ends up contributing might skew an outcome before the data and facts are understood.

It seems to me that the genie is out of the bottle where electronic trading is concerned. We need to make sure that we understand and tweak the model for maximum benefit, not initiate change based on fear of the unknown.

This article ties together three of my favorite topics – big data analytics, cloud computing infrastructure and baseball. Take a look at the iPad dashboard at right and think about a starting pitcher and catcher sitting together on a flight using this kind of highly-visual tool to decide how to pitch each hitter. Now think about the rudimentary paper-based systems from five or ten years ago. Which pitcher has the edge? I’m sure that comparable data exists for batters on pitcher’s tendencies and release points, but it does appear that the overwhelming advantage of this technology favors the pitcher/catcher game plan.

But do the results back up the theory? Take a look at the aggregate statistics across major league baseball the last 5 years. Since 2006 both league batting average and runs per game have fallen every single year. Batting averages are down from .269 to .255 and runs per game are down from 4.86 to 4.28 (per team). Only one offensive stat is up in each of those years – strikeouts, from 6.52 per team per game to 7.06. MLB is marketing this as a pitching renaissance. Maybe it’s a data/knowledge renaissance? Pitchers simply have better tools than hitters and it’s showing up as a reduction in offense.

This phenomenon is not at all unique to baseball. What Moneyball and sabermetrics did for baseball happened a decade before in financial services. Human traders that specialized in arbitrage and technical chart reading have been overrun by automated systems that perform split-second algorithmic trading resulting in billions in profits. Online giants like Google and Amazon have applied similar big data analytics techniques to know what you want to buy before you do. I would be willing to bet that a similar explanation is behind recent record big company profits, despite the luke warm global economy. The past decade has seen an explosion in the number and accuracy of tracked company metrics, along the lines of the baseball dashboard above. It’s the same story – interactive dashboards and process automation are buoying decision making accuracy, resulting in fewer mistakes and greater corporate efficiency. This is a classic example of what Greenspan would have called “worker productivity improvements enabled by technology”.

We are marching inexorably towards a interconnected world of huge volumes of ever changing data and anywhere, anytime access which will make today’s incredible improvements look quaint when we look back ten years from now. Whether you work in a baseball front office, at a leading internet company, in corporate IT or supply companies like these with the world’s most scalable messaging middleware it’s an exciting time to be building applications that connect the web and mobile worlds with the processing power of the cloud in real time.

This week, Nick Harris (of the Information Sharing Environment) posted an update on the ISE website detailing the substantial progress in tracking and accurately communicating information about radiological events. The multi-city, multi-agency demonstration took place earlier in July, and Solace was very proud to have participated with DNDO to demonstrate and communicate the role and value of the Mission Critical Messaging (MCM) backbone.

It’s pretty cool stuff, check it out here.

Since Thursday, US market data volume peaks have been:

• 5.3 million mesgs/second on Thursday August 4^th
• 5.2 million msgs/sec on Friday August 5^th
• 4.7 million msgs/sec today

That’s some pretty thin air for systems that consume market data. We have heard from many on the buy side that feeds from their market data vendors (consolidators) are struggling mightily to keep up, and investment banks are experiencing hiccups in their multicast messaging networks. This kind of infrastructure stress and hand wringing is exactly why so many companies are turning to hardware to both handle feeds and distribute the market data. Hardware is especially good at keeping systems stable and predictable when unexpected volume spikes occur.

Those rates will no doubt back off again when market volatility settles, but this spike is also happening in August, normally one of the slowest times of the year. So it’s a good bet that either natural volume progression or another black swan event will come along and we’ll be talking about climbing market data peaks of 6 million msgs/sec before the end of the year. With more and more electronic trading, market data volumes will only go in one direction — up.

FT.com published another good article on the state of high frequency trading that aligns pretty well with the shifts we have seen in the market. The theme of the article is that opportunities in HFT are becoming less abundant and, as a result, many HFT firms are turning back to quant strategies that differentiate based on unique market pairings or predictions of where the markets will move.

As one of our customers that chose to sidestep HFT from the beginning told us:

“Differentiating on speed is not appealing to us. There can only be one fastest, and even if it is you today, it probably won’t be tomorrow.”

There’s no doubt that electronic trading will continue to dominate volumes, but expect the focus to shift back towards unique arbitrage and quantitative strategies for many current HFT participants.

I would add to Gordon’s observations that the rapid adoption of appliances within enterprises is being driven by a desire for key advantages of cloud computing: simplified operations and cookie cutter horizontal scaling.

Further evidence exists in how many of our customers are using Solace virtualization capabilities, which let administrators easily partition each appliance into many discrete messaging environments, to give different applications and departments their own message bus without the cost or hassle of deploying new messaging platforms with their own software licenses and hardware. Solace virtualization functionality was originally conceived for multi-tenant service providers that wanted to virtually segregate traffic between different companies’ applications to ensure security and operational autonomy on shared equipment. Enterprises quickly adopted this technology to consolidate traffic from many internal applications or groups onto a shared backbone. It even given them the option to easily track internal usage for charge backs across cost centers, making data centers look more and more like service providers.

That’s cloud technology at work folks, and it’s not happening at Amazon or Google, it’s happening inside the banks, airlines and government agencies as each gets smarter about how they deploy their current generations of applications.

Real time reporting of short transactions and/or positions would effectively create another feed that provides new information about a given security’s short position, and inevitably becomes a new source of input for algorithms which would be able to monitor net short positions in a way never before available. Either way, this would mean still more real time infrastructure in the middle-office (for capturing and reporting short sales) and the front office (where the new consolidated “short feed” would be consumed). Just another example of the non-stop migration to real time data. Expect more of this. It only makes sense.

As an aside: I will confess that I was surprised by one of the claims of the article, that the SEC says short selling accounted for almost half of US equities volume in the month of June 2010. That was a net down month for the market, but still, it seems hard to believe on the surface. But when you consider that high frequency trading represents on the order of 70% of all trading volume, and hedge funds are responsible for the lion’s share of high frequency trading, it probably shouldn’t be that surprising that shorting would be that prevalent during a market descent. But regardless, if shorting is that prevalent, and the SEC and congress are aiming to put some kind of controls on market manipulation, you can expect that this proposal will eventually pass — sooner than later.