Solace Systems

Why peer to peer is a flawed architecture for most messaging needs

2012-05-18T22:39:28Z

If you’ve spent any time architecting or implementing solutions that rely on messaging, you’re surely familiar with the concepts behind broker-based messaging. In a nutshell, message brokers centralize common networked application requirements like:

figuring out what information is needed where
making the application fault tolerant
scaling as requirements grow
securing data access
finding and fixing faults

By performing these functions in a shared broker, you offload a lot of work from each application, and get centralized management, security, scaling and so on.

In the area of high-frequency trading, some algorithmic trading strategies live and die by their ability to get market data a few microseconds before the competition. For these scenarios every single microsecond architects can cut out of their system drives bottom line value, to the point that they’re willing to sacrifice the functions and advantages described above to do so. In response to this need, along came “peer-to-peer” messaging, which features no broker and pushes responsibility for messaging logic to publishing and subscribing applications. For latency sensitive market data customers that want to trade manageability for microseconds, and are willing to code messaging logic into their apps, peer-to-peer can be a good solution.

Outside of this arena of ultra low latency market data delivery, brokered messaging is the preferred architecture 99% of the time. This hasn’t deterred some of the peer-to-peer vendors from marketing their market-data messaging solutions for use in the middle or back office where a handful of microseconds makes no meaningful difference to the application. Especially for guaranteed messaging, peer-to-peer is fundamentally flawed as an architecture, and the trade off in features, manageability and function is not worth the claimed benefits.

With input from multiple customers who have taken a hard look at peer-to-peer vs. brokered messaging, our CTO Shawn McAllister recently wrote a white paper outlining the two architectures to highlight when each is best. If you’re evaluating peer-to-peer, or using it and struggling with reliability, I encourage you to check it out.

Solace technical blog — unleashing the techies

2012-05-05T15:29:16Z

If you’ve ever been to a customer meeting between a vendor of cutting edge technology and the techies who put it to work, there is an unmistakable cadence to how they unfold. The sales rep usually kicks off the meeting with a few words, maybe declares the objectives or agenda for the meeting, then sits down and shuts up.

The rest of the meeting is a Q&A mindmeld between the vendor techie specialists, who understand their products better than anyone else in the room, and the enterprise’s techies who understand their problems and projects better than anyone in the room.

You can tell it’s a good meeting when the white board looks nothing like the clean stock art above, and a lot like this:

As a non-techie, the discussion, design and deployment of technology like ours can be a fascinating process to watch, but other times it’s more productive to just give up and go get a coffee. Either way, there’s no denying that it’s tete-a-tete exchanges between like-minded tech types that make the technology world turn.

With that in mind, today we’re launching a new Solace technical blog to directly connect our best and brightest technical minds with the people that get their hands dirty with technology every day. If you’ve read this blog, you know it resides more in the market and business domains of technology. This new technical blog is an ongoing virtual incarnation of the techie-to-techie meeting described above.

Check it out — if you’re a technical reader, you’re sure to appreciate the insights shared. If you’re a non-technical reader, you’re also more than welcome to wade in to the extent you feel comfortable. Perhaps we’ll set up a page where we can step out for a virtual coffee when the dialog gets over our heads.

Is the real-time web streaming data in the wrong direction?

2012-04-12T15:08:01Z

Examples of web streaming have become rather predictable and yawn-worthy. It’s always some variation of streaming real-time stock market data, news and status updates from the cloud to your browser, tablet or phone – classic filtered fan-out data distribution. Sure, there are a few upstream bits like the character inputs used for real-time keyword search completion, or chat applications, but the upstream data is a trickle compared to the fire hose coming downstream. However this model is beginning to flip directions and applications are more frequently streaming large volumes of upstream data with a downstream trickle.

Consider how most Big Data is being collected at the server side today. Click streams, log data, activity streams, search queries – they are all pouring into Kafka, Scribe, or Flume and ending up in a variety of big data repositories. As users increasingly run thicker smartphone, tablet or desktop apps the view from the web server becomes less and less complete. Take for example a shopping site that monitors user behavior to optimize user experience or increase click-to-sales ratios. Before rich apps, it was the clickstream that contained all the needed information to know who looked at an item, who put the item in their shopping cart, and who actually purchased or abandoned the cart. With a rich app on your phone or tablet, commerce sites can now track new events like:

who swiped which part of the screen?
who likes to contemplate the purchase button, but never clicks?
who is scrolling up/down versus right/left?
which color, position or shape of button attracts more activity?

It’s now possible to put these apps in verbose mode and stream huge amounts on interaction data upstream for real-time and historical analysis. Imagine real-time heat-maps of every user interface across all client devices at your fingertips. All this data improves the small but important trickle downstream – perhaps recommended specials or dynamically generated menus that perfectly target each user.

Of course, it isn’t all about shopping. There are endless other telemetry-style data collection use cases. The client app can just as easily be a vending machine, railroad switch sensor, electricity smart meter, or your car. Shouldn’t the HD back-up camera in your new car take snapshots every 30 seconds and publish them to a private real-time stream that you can monitor and archive? It’s sitting in your driveway, why not put it to work as a home security webcam?

Hit and run drivers and burglars better beware the upstream real-time web.

Recapping TEDx New Wall Street – re-imagining Financial Services

2012-03-13T12:43:57Z

Yesterday, I participated in TEDx New Wall Street in Mountain View, California. In the spirit of all TED events, this was a day of sharing ideas — specifically about re-imagining financial services. TEDx events operate under license of TED in support of locally organized events that hold to their TED-like vision. This event was curated by Bruce Cahan, an impressive guy with a background in academia who is (surprise) starting a new kind of online bank called GoodBank.

There was an odd undercurrent of Silicon Valley chest thumping with respect to idea generation and lordship over all things technology that seemed out of place, especially since so many of the speakers were from other places. Regardless, here is a summary of the topics presented grouped by major themes.

The underserved:

The banking world finds a huge percentage of the world (and a surprising number of Americans) so unattractive as customers that they offer virtually no relevant products and leave them to the world of predatory “payday loan” companies.
In poor countries with a lot of violence we should replace paper currency with digital money so Grandma doesn’t keep getting robbed.
The hundreds of attributes that make up your web identity should replace your FICO score (a 40 year old formula) for defining your creditworthiness.
When it comes to the housing crisis, game theory suggests that consumers and banks working together would lead to a much better outcome than the alternatives of walking away or foreclosing.

Small business:

Combining micro-lending and small business venture investing could help grow the smallest of businesses, especially in developing nations.
Small businesses would prosper if big companies paid their bills in 10 days instead of 3 months. (more on this below)

The philosophical:

What should money be? Answer: Virtual, secure, personal.
Innovation, the digital identity spectrum and how Björk changed everything. Best talk of the day, I will expand below.
Is it our world or the algorithms’? More on this below.
Sustainable investing (i.e. investing in the companies that support the big picture of our planet) is a good thing.

The TED-esque bizarre:

How scientists are using biomimicry to learn the design lessons of nature. (Pre-recorded)
The history of double entry accounting in Italy and France from 1200 on. (Interesting, but it’s anyone’s guess what it has to do with renewing Wall Street.)
Economists vs scientists and how economists are nitwits, but not really. (Don’t ask.)
If the federal government and states ran all the banks, we’d be borrowing from ourselves and paying interest on our debt to ourselves, which would fix everything. Kind of reminded me of the old song I’m my own Grandpa.

So there you have it — the future Wall Street! Ok, so maybe the conference promised a little more than it could deliver in one day, but there were a few thought provoking presentations none-the-less. My 3 favorites:

Ami Kassar on 10 Day Pay

Ami Kassar is a small business advocate with an idea. Big businesses are hoarding cash, and can borrow at 4%. Small businesses are struggling to get loans and have to borrow (often against receivables) at rates more like 15-40% (no source quoted). He claimed that current average payment against receivables by big companies to small business is 94 days. He points out that big companies take advantage of how important they are to small business by abusing/delaying payment schedules.

Ami would like to see a social campaign that motivates big business to pay sooner (he suggests 10 days). This would reduce the burden on small businesses by reducing high-rate interest payments currently needed to float their cash flow. With current accounting rules, it wouldn’t change the big company’s profit picture at all, only cashflow, and small business would flourish. It would work like this:

some new web service would track how big corporations pay their bills and give them a score (think BBB for small-business friendly practices).
consumers that care would choose who they buy from based on how that company supports small businesses (using that score).
small businesses would compete to serve the companies that embrace this concept, delivering better service.
companies that embrace this could advertise that they support small business growth (ie, market to consumers that care).

And with that…voila! Healthier small business, more jobs, etc.

Shawn Gourley (live) and Kevin Slavin (recorded) on Algorithmic Trading

Shawn Gourley made the point that algorithms play offense and defense. As they are finding market inefficiencies and exploiting them, they also try to disguise their actions to minimize impact on the market (and maximize their own profits). Meanwhile, other algorithms are watching for these fake outs because they create the opportunity to take advantage of what another algorithm is up to. What we end up with is algorithms fighting with each other all over the place in mini trading wars that have unknown outcomes. Everyone knows about the flash crash of 2:45 where the entire market melted down under some still unclear combination of algorithmic behavior. But about 10 times a day (yes, someone is keeping track), some individual stock goes haywire and charges up or down 5% or more (and usually comes right back inline) for no reason other than two algorithms flared up against each other.

So the big philosophical question: If algorithms we barely understand are warring in ways we can’t predict, at speeds we literally can’t imagine — is it still our market, or do the algorithms own it? His take is that the genie is out of the bottle and we have to learn to work with the algos.

This would have been a great talk if the conference organizers had not played a video of a thematically similar talk by Kevin Slavin from July 2011 earlier in the day. Not sure why we needed to hear it twice.

Peter Vander Auwera on Innovation

The best TED talks are a combination of mind expanding ideas, lateral thinking and good old fashioned entertainment. When all of these videos make their way online, the presentation that I predict will end up getting the most views is Peter Vander Auwera’s pitch on babies, innovation and Bjork. I can’t say with certainty if Peter’s message was deeper than the very simple idea that innovation and taking chances is what makes the world a better place. I did learn about some Flemish painters though. And that if you were born before Bjork (in 1965) you value fitting in, while if you were born after Bjork you value being weird/unique. And that Bucky Fuller was a cool guy with big ideas (which I already knew).

Peter snuck in a little commercial for SWIFT, where he is the leader of their innovation tribe. He highlighted a couple projects related to 1) a digital identity spectrum (think of a secure way to manage your credit details, medical details, community details, military service, etc — who sees what, how these attributes securely represent you in various societal roles) and 2) the digital asset map (what he called the Google Maps for digital assets). He was short on details, just attempting to move the image of SWIFT from stodgy old inter-banking company to innovator.

All in all, not a bad way to spend a Sunday. I didn’t get a clear picture of how Wall Street will change in the future, but I did get a chance to think about financial services in some new and interesting ways.

What the big banks can learn about cutting costs from Apple, Google and Facebook

2012-03-02T02:09:52Z

Apple does it, Facebook does it, Google does it. Why not Goldman, JP Morgan and UBS?

No, I’m not talking about deploying open source software or embracing cloud computing whole hog, I’m talking about choosing datacenter locations based on cost instead of convenience.

Yahoo!, Google, Microsoft and Dell have all built massive datacenters in Quincy, Washington along the Columbia River to take advantage of cheap, clean power (hydro from the river) and local tax incentives. Apple, Google and Facebook followed the same motivations to rural North Carolina to bring down the costs of their online services. Meanwhile, if you ask the CIO of any of the big banks, the cost of building and operating datacenters in New York, New Jersey or London are among their top concerns. What’s stopping them from employing the same strategy?

Much has been written about how high-speed trading is all about proximity to exchanges and that logic has led to an explosion in datacenter growth in New Jersey. But how many applications within a large investment bank really need to be across the street from the exchange? Less than 5%? I read this article about a company pitching the economics of running datacenters in cost advantageous places like Iceland or Wales and got to thinking about how private cloud and distributed computing advances make this kind of architecture entirely viable.

Imagine for a minute if front office operations remained collocated with exchanges in New York or London in a dramatically scaled down datacenter, and less latency-sensitive middle and back-office applications were moved somewhere more affordable. It doesn’t matter if it is Iceland, North Carolina or Timbuktu – the key is to follow the logical economic blueprint laid out by the internet companies.

Sure, a hedge fund or boutique trading firm would not have sufficient scale of operations to benefit from such an approach, but the global investment banks have huge datacenter operations measuring in the tens of thousands of servers, very much like the internet companies. The people cost of operations would be much lower as well, as the cost of living in a rural outpost like Wales or North Carolina is a fraction of the NY or London areas.

It’s not even a huge shift architecturally – front, middle and back-office applications are already decoupled by message passing architectures. It’s really about parceling out the pieces of the private cloud differently and modeling to make sure there are no data flow hotspots that exceed the physics of the distance. The supporting technologies like high-throughput wide-area links, in-memory data grids, data synchronization, high-rate messaging, and WAN optimization are available today to make this a reality.

Exchanges getting proactive about improving the impact of HFT on markets

2012-02-29T21:58:45Z

Last week, I posted on how regulation has unintentionally led to increased high frequency trading across various asset classes. Today, we saw a different angle on the HFT debate, with the IntercontinentalExchange (ICE) highlighting the results of changes they implemented to encourage HFT as a source of liquidity, but discourage counterproductive high-volume quote generation.

Many other exchanges, like LSE and Deutsche Borse, have chosen to penalize market participants with increased fees if they have a high order-to-trade ratio, that is, if their algos generate lots of traffic but result in few actual trades. The claim is the high level of short-lived orders affects exchange performance for all participants.

About a year ago, ICE instituted a more nuanced approach for their high frequency messaging policy. They welcome any order volume, provided it is within a narrow boundary close to the best bid or ask. They claim this encourages HFT order volume that is useful as liquidity and penalizes algorithmic traders that are flooding the market with out of the money orders that rarely trade.

Naturally, ICE’s statement today highlights the positive impact these changes have had on HFT traffic – orders far from current market values were down 63% over the year, and most market participants modified their algos to avoid the penalties. To me, this is the more interesting aspect of the story. Whether it is the simplistic approach many of the exchanges are taking or this differentiated approach at ICE, the market works best when it corrects itself without waiting for slow-moving and heavy handed regulations to come down from law makers.

The reasons HFT is shifting asset classes may surprise you

2012-02-19T15:24:10Z

At Solace, the shift to electronic trading has been at the core of our business in capital markets for many years. We’ve seen a variety of recurring use cases that are now mainstream, specifically for equities. For example:

Buy-side firms engaging in high-frequency trading (HFT), either onsite or at a co-location facility.
Sell-side firms offering smart order routing or internal order crossing to increase revenue or save on costs of order processing.
Exchanges accelerating and enhancing their services to compete with ever more sources of liquidity.
All kinds of capital markets participants improving how they handle real-time risk across trading systems, and/or automate middle-office settlement and record keeping.

Reg NMS and the Explosion in HFT

The path to these systems began with decimilization of equities in 2001 which dramatically increased electronic trading. Then in 2007, Reg NMS opened the door for many more competitors to supply liquidity, which created a n-way set of very-short lived arbitrage opportunities across trading venues. Reg NMS included rules that regulated order fills to assure they were at the best price (across all available liquidity) within a time window (National Best Bid and Offer — NBBO). That led all sell-side participants to require very low latency technology, because they had to fall within the NBBO window to play at all. Mix in a bunch of clever quants to develop models and programmers to automate them and you have the HFT explosion in equities.

Dodd-Frank Sets the Stage for HFT Across More Assets

Over the past 12-18 months we’ve had more and more conversations with people looking to improve capacity of systems that support electronic trading of other asset classes such as foreign exchange, fixed-income and derivatives. I didn’t think too much about why until I read something Patrick Whalen, head of trading for AllianceBerstein, said in this article about the growth of HFT and electronic trading. In a nutshell, Patrick points out that the Volcker Rule will prevent banks from holding assets like bonds or currency positions on their books — they’ll need to move ‘em out quickly and efficiently, which means more electronic trading. As part of the new Dodd-Frank rules about the clearing of swaps, regulators are also pushing for an end to the over-the-phone transactions in favor of more transparent electronic trading. That will change the dynamics of those markets in terms of the need for efficiency, and the mechanics of acquiring those assets.

Thinking about his observations makes the future of electronic trading far more clear. Dodd-Frank will drive more asset classes to electronic trading for transparency and increased monitoring of adherence to rules. Most of these asset classes trade across many venues, which means there will be HFT-style arbitrage opportunities between them. Most of the trading firms have already made investments in low-latency technology, so the implementation times can be very short as the markets increase in liquidity. Therefore the barriers to HFT in asset classes like derivatives, FX or fixed income should be far lower than when equities underwent the shift.

Settle in and Watch the Trading Landscape Shift

Every significant new financial regulation has its own batch of winners and losers — the winners are those who best understand the secondary implications of the new rules, individually and collectively, and position themselves to capitalize on change. “Skating to where the puck is going to be,” as the Wayne Gretzky line goes. As Reg NMS gave birth to accelerated HFT for equities, the changes driven by Dodd-Frank will leave at least as big a change in the markets across many more asset classes, and will accelerate the spread of electronic trading across capital markets. Grab a big bowl of popcorn and settle in to see which firms ride the wave to new levels of success. Like the best movies, there will be lots of plot twists as well as plenty of heroes and villains.

The Journal on three world-changing tech trends

2012-03-09T00:19:59Z

There is an excellent article in today’s Wall Street Journal that details the technology-driven, macro shifts that are happening right now in the world of business. The three mega-trends they cite:

Big data – for anyone in IT this is a “no kidding Sherlock” inclusion, but it is remarkable to step back and appreciate how quickly big data is changing the landscape and its long term implications.
Smart manufacturing – The next wave in manufacturing – engineering from the molecular level to manufacture products using methods more like the ones we use to print documents. The emphasis shifts to design and IP, and away from cheap labor and physical factories.
The wireless revolution – The extension of internet access to more than a billion of the world’s people, paired with a new generation of applications and services, is fundamentally changing the way the world connects, socializes and engages in commerce.

The author does a little flag waving around America being an epicenter for these three trends, but the more important point is that all three are underway now and are good bets to literally change the world.

These trends line up very nicely with the big picture principles of Solace — to unshackle information and make it available wherever it is needed in ways not currently possible. Whether loading up a big data repository, collaborating on a design for smart manufacturing, or giving the world’s mobile user base real-time access to…well, anything…we’re in step with the vision outlined in this article.

If you haven’t read it, I recommend you check it out: The Coming Tech-led Boom (Wall Street Journal)

Ground control to major storm…

2012-01-10T15:04:41Z

Today, we announced a new customer relationship with communications and information technology giant Harris Corporation detailing that they have selected Solace to power the on-the-ground portion of a satellite weather system jointly developed by NASA (the space guys) and NOAA (the weather guys).

This is part of a project called GOES-R (Geostationary Operational Environmental Satellite R-Series). Here is its stated mission as described on the GOES-R website:

The advanced spacecraft and instrument technology used on the GOES-R series will result in more timely and accurate weather forecasts. It will improve support for the detection and observations of meteorological phenomena and directly affect public safety, protection of property, and ultimately, economic health and development.

GOES-R provides essential information related to air quality, coastal and marine monitoring, fire monitoring, hurricane forecasts, precipitation and floods, land cover observations, volcanoes, lightning detection, severe thunderstorms, tornado warnings and more. If you’re interested in learning more about this project and the advances it will deliver, I highly recommend checking out the GOES-R site. Here are some fun facts I learned in just a few minutes of browsing:

The instruments on the GOES-R series will produce more than 50 times the information provided by the current GOES system and will offer unique observations of the environment, with emphasis on hazardous weather in the western hemisphere and space weather impacts.
Data provided by the GOES-R instruments will be used to create 65 different products that meteorologists can use to monitor the weather.
Information from the GOES-R satellite will improve short-term weather forecasts such as severe thunderstorms and tornadoes, as well as long-term climatological trends.
GOES-R will be the first of a new fleet of GOES satellites (R/S), marking the first significant upgrade in GOES weather monitoring capabilities since the start of the GOES I/M series in 1994.

The instruments on the GOES-R satellite are particularly cool:

The GOES-R Advanced Baseline Imager (ABI), a sixteen channel imager with two visible channels, four near-infrared channels, and ten infrared channels, will provide three times more spectral information, four times the spatial resolution, and more than five times faster temporal coverage, than the current system.
The Geostationary Lightning Mapper (GLM) is an optical transient detector and imager that provides early indication of storm intensification and severe weather events, tornado warning lead time of 20 minutes or more and data for long-term climate variability studies.
Solar UV Imager (SUVI) locates coronal holes, flares and coronal mass ejection source regions. SUVI data characterizes active region complexity, enabling improved forecasting of space weather and early warnings of possible impacts to the Earth environment.
Space Environmental In-Situ Suite (SEISS) is an ensemble of electron, proton and heavy ion detecting sensors. SEISS data drives the Solar Radiation Storm portion of NOAA’s Space Weather Scales and other NOAA operational Alerts and Warnings.
Extreme UV/X-ray Irradiance Sensor (EXIS) detects solar soft X-ray irradiance (XRS) and solar Extreme UltraViolet (EUVS) spectral irradiance in the 5-127 nm range. XRS monitors solar flares (and helps predict proton events) that can disrupt communications and degrade navigational accuracy. EUVS monitors solar variations that directly affect satellite drag/tracking and ionospheric changes, which impact communication and navigation operations.
The magnetometer measures the time-varying field in the magnetosphere. It provides the only operational measure of the impact of geomagnetic storms at geosynchronous orbit, and it is key for interpreting solar radiation storm measurements by SEISS.

The GOES-R satellite is scheduled to launch in 2015. We are very proud to to be working with Harris to make the GOES-R vision a reality.

Capturing data streams at “big data” scale

2012-03-09T00:20:41Z

Before you can break into a cold sweat about tackling the design of a system that analyzes big data volumes, you first need to be able to capture the data. More often than not, the design parameters feel like a traffic engineering problem — there are simply too many cars and not enough road.

Certainly, LAN and WAN network technology introduces many limits and the largest commercial databases (e.g. Netezza, Teradata) or open source big data stores (e.g. Hadoop, Splunk) can only store data so fast. Even in memory data grids are limited by how many in-memory writes can be performed per second. Managing the distributed information is usually some kind of middleware, once again, usually a commercial product (e.g. JMS or MQ) or open source code (e.g. Kafka or Qpid).

Even at full speed, a single instance of the middleware layer runs at far less capacity than the network, in-memory grid, or data store can process, making it the weakest link. This means to keep up, the software middleware traffic has to be scaled horizontally across many middleware brokers or servers. Each application becomes a fragile layered mess of servers and any disruption can lead to significant cascading problems of volume and backlog.

An increasing number of our customers with big data projects (e.g. in capital markets, internet infrastructure and transportation) have thrown in the towel on attempting to use traditional JMS, MQ, or open source for this scale of data capture. Instead, they’re opting for Solace’s hardware messaging to feed their big data stores. Where software messaging peaks at a few thousand messages per second, Solace’s failsafe queuing solution exceeds 150,000 messages per appliance. That means you would need to horizontally scale a typical JMS, MQ or open source alternative to 30 or more servers (assuming it could sustain 5,000 msgs per JMS server) to match the throughput of one Solace appliance. It just makes everything easier if the layers and moving parts in your scaling architecture stay light and lean. Fewer servers, less datacenter space, fewer outages = cheaper and less headaches.

Many customers initially think a commercial solution like Solace’s has to be more expensive than open source, after all open source is free and Solace costs money. But it is easy to show that when you factor in server costs, rack space, power, and management it’s far cheaper to pay for an appliance that replaces 30 or more servers.

Big data is right in the sweet spot of (one of the many) use cases that this company was built to address. If you are struggling with these problems, we’d like the opportunity to talk to you about solving them.