Issue 1: Heavy CPU Loading/CPU Offload

CPU intensive operations are difficult for general purpose software to execute on general purpose CPUs. Examples might be monte carlo simulations, complex algorithms or complex transformation of large data records. Think of it this way: let’s say that the CPU cost of running simulations in software is as follows:

You can see that relative to the task of preparing and presenting data, the simulation is a CPU hog. If that task is handed off to an FPGA, two things happen:

• Special purpose hardware might complete the simulation 20-100 times faster than software (in terms of elapsed time). So you can allow more iterations to complete in the same time.
• Equally important, you have unburdened the general purpose processor of 1000 CPU work units, or in this example, by over 99% per of its work per iteration. Obviously there is additional operating system CPU work in the formula, but the key is that the hardware offload very significantly lightens the load for the general purpose CPU.

So ignoring the challenges of how the FPGA simulation code get written, this CPU offload use case is the most likely scenario for FPGA co-processing as suggested by Intel and AMD in their architectures. I suppose you could execute the prepare data and present data steps in hardware as well, but 99% of the performance savings come from focusing on the high CPU cost of the simulation.

Issue 2: Eliminate Context Switching

The second problem that FPGAs and similar technology can address comes about when you swap out ALL of the software for hardware. Take a look software pseudochart below for a simple view of how pub/sub messaging works:

Each one of the steps is very light on CPU cost and it all works great if the number of messages is low, but when you try to execute lots of messages a second, a rate limiting issue comes up. The cost of ‘context switching’ – the time it takes for the operating system to pass control from the network stack to an operating system function to the application space and back again – becomes very high. Each individual context switch is fast, but a single publish to multiple subscribers can cause hundreds or thousands of them which leads to the mathematical principle:

fast * many iterations = slow

This is fundamental to software. No co-processing architecture can address it. If you were to try to apply CPU offloading from the prior example to this scenario, you might pick route lookups as the task you implement in hardware. But in the context of all the work performed, making that one step 100 times faster might only reduce performance by 10% or less. All the other steps are the same, and context switching becomes the performance killer at high volume.

This is why an all-hardware messaging approach is so fundamentally different from software messaging.

With FPGAs doing the application processing and network processors performing networking, security, compression etc you have eliminated the operating system and coupled the application and network stack all in hardware. By doing so, you eliminate context switching. FPGAs and network processors can also more naturally parallelize operations that need to be iterative in software, providing further major performance gains. Of course, removing context switches and parallelizing also substantially lowers end-to-end latency.

These two concepts are most of the reason software often runs out of steam or gets very erratic in terms of behavior and latency at just a few hundred thousand messages/second and hardware can scale all the way up to wire saturation, above 10 million messages per second per hardware blade. An all hardware solution rewrites the rulebook for what is possible.

So in the context of the prior blog post, this hopefully clarifies why FPGA co-processing can help with CPU offloading, but it does little to resolve throughput issues relating to context switching.

Of course the idea of hosting FPGA co-processing is not new, AMD has been offering a version of this approach for over two years. Intel is clearly playing catch-up here. It’s also not surprising that Intel would be concerned about any key market moving processing to specialized hardware that can outperform software on Intel processors by 10, 20 or even 50 times. Especially if one box of non-Intel special-purpose hardware can replace the work of 10 to 30 Intel boxes running software.

This is a classic case of if you can’t beat ‘em join ‘em, which has been a successful strategy for Intel in the past. The question is, who exactly will they be joining?

For FPGA manufacturers, this is a mixed blessing. The positive is that it can allow more FPGA processors to be sold. The negative case is that it levels the playing field and pushes the technology towards low margin commodity faster than through custom platform. It will be harder for FPGA manufacturers to differentiate their products in a plug-and-play world where they are subservient to Intel’s processors.

Companies that rely on FPGA technology to deliver differentiated products (like Solace) could benefit in some cases. While custom designed boards are likely to always be faster, the lower costs from using commodity parts could allow hardware companies to line extend to more tiers of offerings at more attractive hardware price points, while releveraging the same FPGA code. Perhaps software and servers on general processors result in performance X, FPGA co-processing on an Intel or AMD motherboard might offer performance improvements of 3-5X and a custom hardware solution may offer performance improvements of 10-50X. That could be intriguing to firms looking to take high-end products downmarket at lower price points. But the co-processor architecture will not displace the very high-end requirements. Co-processing will favor a blended software/hardware solution, which introduces some amount of context switching, which in non-CPU intensive applications (feed handlers, messaging, etc) is the primary source of bottlenecks today.

Who will NOT be coming along for the ride is most software ISVs or enterprise customers. Designing hardware solutions is not for the faint of heart, it is a significant long term commitment that is expensive and time consuming. There may be one FGPA code designer available for every 1,000 or maybe even 10,000 software developers. This scarcity makes this risky to staff within IT and even within most software vendors. That leaves only the most performance-sensitive, highly motivated suppliers to commit to the FPGA path. By definition, those are the suppliers that need all the performance juice they can get, not a half-way, lower-cost solution.

There are niche applications that are hardcore CPU crunchers that may jump on board. Simulation and quant engines are candidates. Specialized algorithmic trading applications, where the algorithms are sufficiently long lived to be worth coding in FPGAs, may be another. But the target in financial services is not broad. It’s a couple niche cases within the already narrow low-latency financial services market.

This awkward mismatch of motivations between Intel, FPGA providers and low-latency specialist solution vendors leaves Intel with ambitions that will be challenging to convert to meaningful market share. It’s telling that AMD’s been at this for a while and has not made noteworthy progress thus far.

Please comment if you have opinions on how this may play out for Intel (or AMD).