How we chose our tech stack
At Canvas we're building the frontend for the modern data stack. Fittingly, we’ve ended up implementing this on what we call the “modern web stack” - though some of these technologies are so bleeding-edge this appellation could be premature. The choice of stack might be the most consequential decision an early technical founder makes. Using an emerging technology is fraught with risk; there's relatively sparse documentation, unknown limitations, and little prior art. Yet one year in I remain confident we picked the right tech.
For our inaugural engineering post I’ll explain how we ended up on our frontend stack at the end of the world.
Requirements
Our mission is to bring the fruits of the data revolution to the business users that have until now been locked out. From the beginning we saw this as a design problem first and foremost. Any analysis is possible with SQL. Distilling this expressiveness to a familiar, usable UI would be the challenge.
We decided spreadsheets were the key. This interface is not only universally familiar, but also maps elegantly to SQL operations. Filters become where clauses; pivots become group by statements; a sort becomes an order by. Data teams and business teams had been speaking the same language all along without even knowing it.
Looking over the landscape of existing spreadsheet tools and their shortcomings for data analysis we arrived at other principles for our tool.
Our spreadsheets should work with billions of rows - scrolling, filtering, sorting, and writing formulas. And this should be fast. In our book there's no difference between slow and broken.
We also decided the traditional one-sheet-per-view model was incorrect. The models in an analysis form a dependency graph. A pivot table aggregates over another table, which itself might be the result of joining two other tables. Data analysis almost always involves multiple related tables like this. Our UI needed to be expressive enough to reflect that.
Relatedly, our users wanted to be able to create pixel-perfect dashboards - something that's mostly impossible in existing tools. The state of the art is copying and pasting charts into a PowerPoint presentation.
Finally, we wanted users to be able to collaborate inside the app in real-time. Users should be able to work on the same table without worrying about conflicts. This called for an architecture where the client and server would run identical code, with the server being the final arbiter of truth - a familiar paradigm from multiplayer video games.
Weighing the options
Our performance requirements pushed us away from Javascript and towards WebAssembly. Building a web application that's highly performant under heavy workloads, for all intents and purposes, requires using WebAssembly (Wasm). Wasm is a machine code language understood by modern browsers. Wasm has a number of performance advantages over JavaScript; it's compiled, has memory management so there's no garbage collection, and has a strict type system that allows for superior optimizations, to name a few.
Wasm isn't written directly, but rather you compile your code into Wasm from a higher-level language. Here we had to choose between C++ and Rust as both languages can compile into WebAssembly. Our multiplayer requirement came into play here. Since we wanted our web server running the same core code as the client we chose Rust for its superior web framework ecosystem. Rust’s excellent type safety was an added benefit.
Our performance and design goals led us to use WebGL for our core UI instead of the DOM. We wanted to load and display data fast and then let users scroll through tables and drag elements at 60fps. Updates through the DOM would not meet these responsiveness goals. The further requirements that users be able to create pixel-perfect reports, and then be able to view the graph of tables backing those charts, meant that the DOM would be too restrictive.
Finally, we decided writing the entire application in Wasm and WebGL would be wasteful. For menu pages and sidebar components that are not performance constrained there’s little value in re-implementing existing JavaScript components in WebGL. Since our stack already had plenty of cutting-edge technologies, we decided on tried-and-true React for our boiler-plate application code. As my mentor once told me, only use one new technology at a time.
Putting it together
Our current frontend stack is:
- WebGL for rendering the core canvas UI
- Rust compiled into WebAssembly for multi-player state management and performance-bound application code
- React to handle the boiler-plate application code
On the backend we have a Rust web server running the same core code as the browser clients. The clients communicate with the server - and by proxy with each other - via WebSockets, with the server acting as the arbiter of truth in deciding what messages to publish.
One year in we’re happy with our choices. Rust is simply a pleasure to work with. WebAssembly and WebGL are blazing fast and we’ve had surprisingly few browser compatibility issues.
There are times when we feel plainly that we’re on the bleeding edge of the web stack. Writing components in WebAssembly and WebGL is significantly slower than using existing React components, so I would caution against using these frameworks unless you absolutely need the performance benefits. There’s also a dearth of practical documentation or articles about deploying this stack. And you'll find yourself re-implementing much of the functionality you took for granted in the DOM, such as copy and pasting and even scrolling.
For the most part these are fun challenges. It's thrilling to be at the cutting edge of a technology that we hope and believe will become massively important. At Canvas we’re looking forward to building out this ecosystem by building component libraries, discovering new ways to interoperate with React, and publishing articles about our stack. If this sounds exciting to you, we’re always looking for new adventurers to join our expedition.
Citations
https://hacks.mozilla.org/2017/02/what-makes-webassembly-fast/
