Originally published over at Medium
On the heels of the release of Reactive Streams 1.0.0, I decided to grab a couple of engineers — Ben Christensen of RxJava and Netflix fame, Stephane Maldini of Pivotal’s Project Reactor and the inimitable Dr. Roland Kuhn of the Akka Project at Typesafe — that have been engaged in the project, to ask them a couple of quick questions about Reactive Streams, their involvement, and what the future has in store for Reactive Streams.
Massive thanks to everyone who has contributed to the effort, both via code, via discussions, via ideas and trying things out, and giving most excellent feedback throughout the process — none of it would be possible without you.
Viktor: How did you get involved and what led to the Reactive Streams initiative?
Ben: In 2013 a handful of us on the RxJava team were exploring the addition of backpressure to RxJava. We were prototyping various solutions and learning how challenging it was. In October of that year Viktor Klang reached out and introduced himself. Viktor and Roland Kuhn had an initial Skype call with George Campbell and myself where we all met for the first time and started talking about reactive programming and async, non-blocking flow control and backpressure. Viktor drove collaboration over the following months with other companies including a memorable meeting at Twitter with Marius Eriksen. Out of those early collaborations we evolved our thinking, broadened our targeted use cases and built prototypes that helped increase our confidence in what we were striving for. In early 2014 Viktor formalized the collaboration under the Reactive Streams name. Prior to this point each team had been sharing ideas but exploring the solution space independently. With the creation of the reactive-streams Github namespace it pushed us into collaborating concretely on specific code and specifications.
Stephane: I don’t remember, I know Viktor approached us at Pivotal but the details are blurry, I was drunk and naturally we said yes as we all do in this kind of situation. More seriously, end 2013, early 2014, Viktor and Roland reached out to Jon and I as Pivotal Reactor representatives. Reactor is on a mission to support Spring apps and beyond transformation to non-blocking programming model, and being part of an industrial group specifically focusing on asynchronous matters was essential to us. We ignored competition between different vendors and the open source idea of this collaboration took precedence, because this group was composed with real engineers interested in solving problems once for all. We solved object-allocation and throughput issues in asynchronous eventing using RingBuffer structures for Reactor message-passing but still our producers were time to time blocked by … the bounded RingBuffers. So what do you do at this stage ? You can’t afford unbounded queue and you don’t have any time to move this overflow over IO, file or network ? Well that’s where Reactive Streams has been a revelation: Why do you have this overflow at first ? Can you prevent that ? Can you just tell your data source you only want to read this much data ?
Roland: The Akka story is rather similar, we found that people love the Actor model, but it is easy to forget about flow control — which occasionally manifests in questions on the mailing list about OutOfMemoryErrors, excessive GC pauses, or degraded performance under load. Another aspect is that Actor communication has so far been dynamically typed, meaning that the compiler cannot track possible mismatches in sent messages vs expected messages (something we’re working on under the name of Akka Typed). The use-case of letting messages of known types flow through a pipeline of Actor-based processing stages is a common one, so we set out to define interfaces for modeling a restricted language for this domain — an internal DSL for non-blocking, bounded, asynchronous stream processing — but were there others facing the same issue that we could collaborate with? What led me to propose an initial collaboration with RxJava was that while recording the video lectures for the Coursera course «Principles of Reactive Programming» I met Erik Meijer and we started talking about adding back-pressure to Rx; he is the inventor of Rx.NET and ported it to the JVM as RxJava as well. Discussing this with Viktor we started reaching out and what we found was a lot of demand for a common solution as well as a lot of expertise and brain power, which inspired faith that such a large venture could actually be successful.
Viktor: What is the purpose of Reactive Streams?
Stephane: Of course the common name for the behavior of “please don’t send more than I want” is “back-pressure”. The main purpose of Reactive Streams is to define a protocol for informing publishers (upstream) of their subscribers’ (downstream) processing capacity. I think the software world could do with more informed decision when it comes to asynchronous processing. It’s a total waste of resources to try requesting a service that is going to be blocked at some point. Think about network efficiency as well, all producing clients could just adapt to the current server processing rate and avoid bottlenecks. Better they could determine over-time the optimal number of messages in-flight based on demand latency. In the end data should be where you expect it should be in a processing chain, bringing a bit of determinism in this data location is one of the main goals of the specification. Resources saved can then be properly allocated to things that actually do work. Note that this pattern applies very well to any kind of distributed systems, including microservice architecture since it’s all about requesting more input data when it has been actually processed or forwarded to another remote component.
Roland: Exactly, I’d like to reinforce this. We set out to define a local protocol that avoids to overwhelm data recipients without having to block any Threads, but in the end the scope in which we apply it is much larger: when streaming an HTTP payload to or from a server the natural back-pressure propagation of TCP should seamlessly integrate with the stream transformations performed on the data before or after sending it over the network. Through much discussion and by incorporating experience from working with rather different programming models we have arrived at a minimal specification that at the same time can express very well how to pass data across all sorts of asynchronous boundaries, be that between Actors, Threads, or machines.
Ben: Adding to the above, I think we have all been interested in two key points. First to ensure that stream implementations could exist without any unbounded buffering and switch between reactive and interactive models automatically based on consumption rate. Second, we wanted a solution that could allow interop between libraries, systems, networks and processes.
Viktor: What have the biggest challenges been?
Ben: The end result of Reactive Streams seems quite simple, 4 Java interfaces and a fairly short specification document. However, it is only after many hours of debate, discussion and prototyping on dozens of issues that led to this state. There were indeed times when it seemed we wouldn’t find alignment and times I’m sure when people were cursing at their computers! The quality of people involved, experience in applying against real systems, and willingness to endure contention allowed us to achieve an outcome better than what I think any one of us could have done on our own. This type of collaboration is easy to have fail due to bike-shedding and personal opinion. Legitimate challenges existed because of differing technology stacks and languages (despite targeting the JVM). Additionally, despite there being lots of related prior art, there wasn’t anything out there exactly (that I’m aware of) like what we were pursuing and thus we couldn’t just model our solution after something already done or defined in a scientific paper. This meant we were exploring the problem domain and teaching each other as we went.
Stephane: There is an inherent inertia in any collaboration, take it and multiply it by 13,2 for inter-companies collaboration. But that is one of the challenges only and I’ve rarely seen such focused group. The special mention comes to the TCK work achieved by Konrad Malawski: that alone made this specification real because it is all about behavior and trust between implementations. The specific challenge for us came then to re-engineer Reactor to work with the spec in a maximum of components: network, dispatchers and composition API. Such refactoring took a long time and the TCK is not forgiving even when you start beating the last issues. Random CI issues raised by the TCK were common until we got a stamped 2.0.1.RELEASE. I’m now happy to help any newcomer in the specification (e.g. Spring Integration has started some work here too) because we can now negate the steep learning curve we had in implementing it.
Roland: The biggest challenge certainly was to converge on a common solution while coming from extremely different directions. Everyone on the project had previously implemented a successful framework or toolkit for creating asynchronous and concurrent applications, meaning that a lot of prior knowledge existed that needed to be understood — as a group — and forged into a cohesive set of requirements. Unlike in politics where this is usually done by including ever more complicated special rules for accommodating everyone’s opinion we had to build something that actually works in practice and that can be applied consistently to each of our own projects. Like Ben said, our success hinged upon the willingness of everyone to work towards this one common goal, setting aside politics or ego. I am deeply impressed by the quality of the debates we had — we beat the ideas hard to reveal their core and harden them. This is what turned our greatest challenge into our greatest strength: we admitted a diverse set of viewpoints into the process and distilled them to their common essence, finding such an effective and simple mechanism that we believe will be a solid foundation for many things to come.
Viktor: What aspect about Reactive Streams do you find the most interesting?
Stephane: We expanded our knowledge about non-blocking behaviors along with the spec progression and we internally used real world feedbacks as a handful of customers at Pivotal have started using this work and the implementation we made from it. Like the specification flow-control, this feedback loop has been crucial to validate some of the theorycrafting we were fighting with. Yes it’s awesome to only pass the data you have requested for but how does it work at scale ? How these first users would start using this ? I’ve also greatly appreciated the engineering-based collaboration, working with some of the greatest individuals I had the chance to interact with and just focusing on the s.. sorry result. It looks like the Hacker Way Erik Meijer is talking about in fact, where only code is worth a million meetings and discussions (https://vimeo.com/110554082). That’s the way we worked with the specification on the Reactor side, implementing it since 0.1 (or near ;)) to make sure the early adopters we had got the same chance to run the feedback loop and help with spotting some issues. In the end this task force challenged everyone involved to get ready for the next wave of asynchronous libraries and make all sorts of applications *really non-blocking.
Roland: In addition to the learning effects that Stephane mentions I found it very interesting to observe the magnitude of interest that built up around this project rather quickly. It seems that we are solving a genuinely ubiquitous problem that was not yet tackled otherwise and consequently there was a lot of hunger for good solutions in this field.
Viktor: What’s next for Reactive Streams?
Ben: It’s exciting to see the proposal to adopt the interfaces in Java 9 as j.u.c.Flow. Hopefully the full specification will follow with it since that’s a critical piece. An unsolved problem right now is standardization on how to achieve composition with Reactive Stream. Today each concrete implementation must effectively wrap the Publisher type. Function composition necessitates methods on the Publisher and unfortunately the JVM does not offer extension methods to make this easy. I would like to see the JVM add extension methods, or perhaps the Reactive Streams project can explore the addition of extensibility or key operators such as flatMap. I’m also hopeful that we can succeed in defining a network protocol with the ReactiveStreams.IO initiative. This would allow async message passing with application level flow control semantics in a standardized way.
Stephane: Now it is all about making sure this protocol expands beyond the JVM because modern distributed architectures require it. With the current specification we can make sure the JVM will push back the demand until its very source, like Async IO server components or Non-Blocking Data drivers. But that will stop the data at the system level in some cases (TCP buffers etc). We should work on expanding to the network first and then to other languages possibly interacting with JVM services, for instance Javascript for HTTP servers. Another point I think we should elaborate more is on the “Hot” sequences, how do you apply this flow-control on user-generated demand such as mouse clicks ? There are various strategies for it: composition libraries such as Rx-based ones address this with operators like sampling or aggregating ones. But there is probably a path to expand some rules and maybe core interfaces to this use-case, or maybe it’s just a matter of explaining clearly the possible solutions as it might be hard to find a common ground. Finally, we have a responsibility to make sure the specification is clear enough, understood and appealing to the JVM ecosystem too. Be sure to see ourselves and more roaming in a few conferences around to explain why it matters and where everyone can start making use of it. With the coming Java 9 support started by Doug Lea, we’re not far before that notion starts being core to any data-related (IO or not) JVM solution. Still it is first a use-case driven approach and developers will adopt the Reactive Streams standard if they perceive the potential benefit for the cool next-gen apps they are working on!
Roland: In addition to transcending the JVM boundaries as Ben and Stephane already outlined I am excited by the movement we are seeing the whole ecosystem towards offering Publishers and Subscribers for more and more natively stream-oriented interfaces, most notably storage solutions like databases, document stores, persistent queues. For these it is not so important to govern their own network protocols, but it will be highly beneficial to consume data from or provide data to these services in a controlled yet non-blocking fashion. On a larger scale I am sure that the Reactive Streams project will be recognized also for another achievement. The software industry has been changing fundamentally in the past 1–2 decades and we start seeing examples of commercially successful open-source companies, a sign that this model is here to stay. The Reactive Streams project has demonstrated that with enough dedication and discipline a new standard can be developed by competing technologies in a completely open and collaborative fashion, for the greater benefit of all. Personally I find this inspiring, a very positive sign for the development of our industry as a whole.