By Fergal Toomey - Chief Scientist and Co-Founder at Corvil

A market data ‘microburst’ is a short burst of high activity in a feed, typically lasting much less than a second but posing a risk of temporarily saturating the resources available to a receiver (and thereby adding latency). While the term ‘microburst’ might suggest a duration measured in microseconds, in practice it’s often used to describe any burst short enough to pass under the radar of traditional load-monitoring tools (that measure average loads over seconds or minutes). Detecting microbursts requires a monitoring system that can accurately sample activity patterns at very short timescales – a ‘microscope’, if you will.

Microbursts are short, but they are important to market data practitioners who nowadays cannot tolerate the extra microseconds or milliseconds that a slow system might take to clear them. To some practitioners, a burst only qualifies as a microburst if it does actually cause saturation at some point in the market data path. To others, any unusually high spike of activity qualifies. The latter definition is a reasonable one to apply if you do not know the capacities of the downstream systems that will have to handle the spike.

From the volume metrics presented on LatencyStats.com we can see that both the ArcaBook and OpenBook feeds contain microbursts, in the sense of short-term activity spikes that greatly exceed the long-term average data rate. A closer look at the activity pattern reveals that busy periods tend to consist of many high-rate bursts in succession. For example, here is a view of 1-millisecond bit-rates in the ArcaBook feed during a fairly active 1-minute period from a few weeks back. The average bit-rate during this period is less than 10 Mbps, but there are multiple microbursts extending to over 200 Mbps:

Strikingly, the activity pattern in the OpenBook feed over the same timeframe reproduces the timing of the ArcaBook spikes with uncanny precision (albeit at lower peak bit-rates):

The OpenBook and ArcaBook feeds are published by different servers in separate data centers and are delivered to the client connection point (where the LatencyStats.com measurements are made) via separate network paths. The spikes are clearly not an artifact of machine- or network-processing along either delivery path. Each burst consists of a very large number of order book updates covering a broad range of different securities. They represent events in which significant changes occur in both order-books almost simultaneously. Below is a close-up view of one of the microbursts in the chart for ArcaBook. Most of the activity (over 25,000 updates) occurs within the first 100 milliseconds after the onset of the event:

It’s not surprising to find correlation between the ArcaBook and OpenBook feeds, since the corresponding markets trade many of the same securities and orders can be forwarded between them. Nevertheless, the degree of coincidence between the feed microbursts is remarkably close. The onset of the ArcaBook event shown above is separated from that of the equivalent OpenBook event by less than 1 millisecond.

This coincidence of microbursts has implications when it comes to sizing systems to keep up with the highest peak data rates. If the spikes in each feed rarely coincided, then it would be possible to consume both feeds together using relatively modest additional resources. It appears however that for these two feeds at least, the individual peak data rates must be added to get an idea of the combined impact.