Reactive Application Design For High-Volume Multidimensional Temporal Data Series @pidster Stuart Williams == ‘Pid’ • Lead Engineer, RTI • SpringSource / VMware / Pivotal • A little OSS: Apache, Eclipse @pidster Mumbling Isn’t a Sign of Laziness — It’s a Clever Data-Compression Trick Source: http://nautil.us/blog/mumbling-isnt-a-sign-of-lazinessits-a-cleverdata_compression-trick Practical • We built a real-time* application (RTI) – Using Spring (yes! I know!) – Share some lessons – Do some demos – Reveal a secret go-faster switch! * for some definition of real-time Built with… • Spring IO Platform – Boot, Data, Integration, Reactor, AMQP, SpEL, Shell (and a little Groovy) • GemFire, RabbitMQ • C24 Questions… • Do you know your system load & input rates? – No. – Yes! • • • • • • Up to 1k/s? Up to 5k/s? Up to 10k/s? Up to 100k/s? Up to 1M/s? Above 1M/s? Questions… • Heard of Spring Integration? – Tried it? – Used it in production? • Heard of Spring Reactor? – Tried it? – Used it in production? Some discussion about… DESIGN Design Goals & Challenges • High throughput – versus • Low latency • Enable low-impact analysis on live streams – We don’t know in advance what this will be… – User accessible API for analytics Many whiteboards later… Show me the… BIG PICTURE! The Big Picture Reactive expressions Query, Adapt, React feedback loop Analytics Stream Analytics Queries Metrics Firehose AMQP Ingester Ingest Grid Distribution End User Applications Databases Queries - Unstructured - Structured HTTP Ingestion & Filtering HTTP Analytics & Distribution End-user / Consumers Input Data Rates RTI Twitter*  100k/s baseline  6k/s average  ~120k/s daily peak  9k/s daily peak  >1M/s annual peak  30k/s large events OK, so Twitter’s internal fan-out & timeline access rates & storage problems are vastly different! (see also Redis…) **Source @catehstn twitter.com/catehstn/status/494918021358813184 Load Characteristics  Low numbers of inbound connections  High rates, micro-bursts  Occasional peaks of nearly 2x, rare peaks of 10x  Variable payload size (200B – 300KB)  Internal fan-outs multiply event rates More statistics…  100k/s order of magnitude – 8,640,000,000 (per day) – An Integer based counter will ‘roll over’ in ~3 days  400Mbps of raw data – 10Gbps NICs required to support traffic peaks – Logging! Verbose errors can fill a disk quickly – Queues backing up == #fail  Upcoming 10x existing rates! What’s all this fuss about… REACTIVE APPLICATIONS? Reactive Applications www.reactivemanifesto.org Responsive Depends on Scalable Resilient Depends on Event (or message) driven Reactive Streams • Collaboration between key industry players 18 Reactive Streams: Specification • Semantics – Single document listing all rules – Open enough to allow for various patterns • 4 API interfaces • TCK to verify implementation behaviour 19 Reactive Streams github.com/reactive-streams org.reactivestreams.Processor org.reactivestreams.Publisher org.reactivestreams.Subscriber org.reactivestreams.Subscription A quick look at the… REACTIVE STREAMS API Spring Reactor LMAX Disruptor – a RingBuffer reactor.bus.EventBus reactor.core.Dispatcher reactor.rx.Stream A quick look at the… REACTOR API Spring Integration Enterprise Integration Patterns See http://www.enterpriseintegrationpatterns.com/ by Hohpe & Woolf Messages Channels Endpoints (pipes & filters architecture) SI Pipeline Example A quick look at the… SPRING INTEGRATION JAVA DSL Spring Integration Performance • 3.x – Take out all the SpEL • 4.x – 4.0 • Improved – 4.1 (Q4 2014) • Put back all the SpEL – 4.2 (late 2015) • Rather good Back to the… BIG PICTURE AGAIN The Big Picture Key Reactor usages Reactive expressions Query, Adapt, React feedback loop Analytics Stream Analytics Queries Metrics Firehose AMQP Ingester Ingest Grid Distribution End User Applications Databases Queries - Unstructured - Structured HTTP Ingestion & Filtering HTTP Analytics & Distribution End-user / Consumers Reactor Usage • UDP/TCP Servers (or clients) • Outputs – batching • Dispatchers & Streams – Expression evaluation engine Spring Integration + Reactor • Batching – Adaptive sizing • Routing – With … batching An example or two… SPRING INTEGRATION + REACTOR But what about the… TEMPORAL DATA SERIES? Temporal Data Expressions RingBuffer Old Data New Data And there was something about a… SECRET ‘GO-FASTER’ SWITCH? Spring Expression Language (SpEL) • Powerful expression language • Supports querying and manipulating an object graph at runtime • Similar to Unified EL – Additional features, include method invocation and string templating. SpEL is slow!  Enter SpEL Compilation • 3 modes – Immediate – Mixed – Off • -Dspring.expression.compiler.mode=mixed And now for a quick SPEL DEMO SpEL is fast!  Fin And relax And now for some… QUESTIONS? @pidster @smaldini