Debugging MySQL replication lag by diving into the internals
Recently one of our PMs flagged to us that our product analytics data was nearly 9 days behind the production database! We quickly found that this was due to MySQL replication lag. But to fix this lag, we had to dive deep into the internals of MySQL replication! In this...
Learnings from JAM London 2019
JAM is an annual conference that brings together the best and brightest minds in the product and design industry, for a jam-packed day (pun intended) of talks and panel discussions from industry experts.
Hot SRE trends in 2019 (brought to you from SREcon EMEA)
A couple of weeks ago I attended SREcon EMEA 2019 in Dublin. I felt like I learned a lot over the (exhausting!) 3 days, 19 talks and 2 workshops I attended, so I wanted to write up my experience.
Splash the cache: how caching improved our reliability
In the wake of an incident, we implemented a caching solution to help us handle spiky load better. Here’s how we did it.
Production-ready Database Connection Pooling in Go
Here at Pusher, our new products, Beams (our push notification API) and Chatkit (our chat API), have recently undergone extensive benchmarking of their systems as part of going from Beta to GA. And both make use of PostgreSQL.
How Pusher Channels has delivered 10,000,000,000,000 messages
At the Pusher office, we have a little counter, constantly incrementing. It shows the total number of messages which have ever been delivered via Pusher Channels. On Friday night, at 22:20 UTC, that counter gained another digit, reaching 10,000,000,000,000 messages. There are 13 zeroes in that number, or ten trillion....
Ephemeral port exhaustion and how to avoid it
Recently, a production incident caused a part of our system to reject all new connections. The culprit: “ephemeral port exhaustion.” By default, your OS will not give any warning when this is about to happen – scary, given the severe consequences! In this post we try to remedy this. We’ll...
Six Tips for Running Scalable Workloads on Kubernetes
Kubernetes is a powerful platform for running scalable and highly available applications. In this post, I discuss the things Kubernetes expects when deploying new applications, as well as the things Kubernetes provides to ensure your applications are always available.
Setting up an SSH CA with Vault
Even with modern tools, managing SSH access to hundreds of machines is daunting. At Pusher we use Hashicorp’s Vault as an SSH certificate authority, allowing us to manage ssh access from one central place. Vault makes signing SSH keys so easy that we were able to implement a self-serve mechanism...
Single Sign On For Kubernetes: An Introduction
User management is hard. At Pusher, with an expanding engineering team, we wanted to build a simple identity management experience within our Kubernetes infrastructure. Kubernetes supports a protocol known as Open ID Connect. A system based on OAuth2 that allows you to move your user login experience away from Kubernetes...
Alternatives to sum types in Go
Many statically typed programming languages, such as Haskell or Swift, have a feature called “sum types”. Sum types (also known as tagged unions or variant types) allows a new type to be defined as the “union” of a set of other types and values, and allows users to “pattern match”...
Per-IP rate limiting with iptables
Every second, Pusher’s main pub-sub system handles 9,000 new WebSocket connections. No sweat. But earlier this year, when the system started receiving spikes of 20,000 new connections every second per server, the sweat began to bead on our foreheads. What, or who, were these new connections? Were they malicious or...
go tool trace
Do you want to know what your Go program is really doing? go tool trace can show you: it visualizes all the runtime events over a run of your Go program, in exquisite detail. This under-documented tool is one of the most useful tools in the Go ecosystem for diagnosing...
Redis Pub/Sub under the hood
Do you want to code a chat app? Here you’ll see how to do it the hard way. I show how Redis Pub/Sub works in detail, all the way down to the bits and bytes! This is the first part of a series of deep dives into Redis.
Golang’s Real-time GC in Theory and Practice
Each day, Pusher sends billions of messages in real-time: source to destination in less than 100ms. How do we achieve this? A key factor is Go’s low-latency garbage collector.
What I Learned Building the PusherSwift Framework
Creating a framework in the Apple developer ecosystem is a bit of a mixed bag. Some of the tools you come into contact with are intuitive, reliable, and a joy to use. Others are incredibly frustrating and you can lose hours (and days) to battling with them, before eventually emerging...
Announcing Go Interface Fuzzer
We’re excited to announce the open source release of Go Interface Fuzzer, a tool for automating the boilerplate of writing a fuzz tester to compare implementations of an interface.
Low latency, large working set, and GHC’s garbage collector: pick two of three
Pusher makes it easy for developers to reliably deliver data at scale. A few features we are working on required a rewrite of our internal message bus, but we knew anything we build has to perform as reliably as the existing platform. Our chosen language for this job was Haskell,...
3 lessons learned after 6 months in a startup
There are thousands of articles and blogs out there about what it’s like working in a startup. Most of them extoll the virtues of Friday beers and remote working, but they struggle to give you much detail about what your work will actually entail.
Fuzz testing distributed systems with QuickCheck
Here at Pusher we’re always trying new approaches to web development and engineering to uncover ways of making our software more reliable. That’s why I’ve recently been using QuickCheck to fuzz test our Raft implementation with some great results.
3 Approaches to Monadic API Design in Haskell
Designing a good API for a library is a challenging problem. In Haskell getting the mix right between specialised and generic can be tricky. For example, just look at the controversy around FTP, where the functions that used to defined for Lists, were lifted to the Foldable and Traversable typeclasses....
My 5 favourite features of Go and how to use them
Ever since I joined Pusher a little over a year ago, I’ve had the opportunity to work on some amazing projects and learn new programming languages. One of the languages I came across was Go.
Approaching Realtime as a Blank Slate
When Pusher started as a company the main environment in which WebSockets were used was the web browser. Since the birth of WebSockets many new JavaScript runtimes have emerged. Now JavaScript runs everywhere, including servers and mobile applications which could benefit from a WebSocket connection. Unfortunately, those environments differ in...
Making Efficient use of memory in Haskell
In part two of my series on Haskell we spoke about identifying memory leaks and tuning the GC to improve performance. In part three we’ll be looking at a number of other techniques and libraries we have found that maximise efficient use of memory in Haskell when writing performance critical...
Memory profiling in Haskell
At Pusher we are currently writing a high performance system in Haskell and blogging about it along the way. In our first blog post we explained some tools and tips that we have used for improving CPU time of our program.
4 questions you need to ask before deploying Docker
Docker was released as open source in March 2013, so in software terms it’s relatively new. As always with shiny things, nerds like me see their potential and start thinking of ways of using them everywhere.
HackDays: Connect Pusher, Slack, Spotify and Sonos for office audio
One of the things that makes working at Pusher great is our appetite for building. We’ve created various internal tools, members of the team work on various open source projects — and of course, Pusher itself is designed to help others in this way.
Top tips and tools for optimising Haskell
(This blog post discusses CPU time profiling, but not space (memory) profiling. We’ll come back to that in Part 2.)
Here’s what I learned from 3 months of startup interviews
At Pusher, we take great pride in making developers’ lives easier. Making messaging APIs simple means they can focus time and resources elsewhere. So it’s slightly ironic that our interview process is designed to make prospective engineers’ lives a little more difficult, if only briefly.
Unit testing IO in Haskell
The generally side-effect free nature of Haskell code makes it convenient to test. Haskell programs can interact with the outside world – otherwise they would be useless – but these side-effects are only possible in the IO monad. It is still important to test this code that performs IO, but...
What the Hack?
Since joining Pusher last year I have participated in a few office hack days, each one unique, and a good opportunity to get our creative juices flowing. I love seeing the imaginative ways people use Pusher in their apps.
Useful online resources to learn Haskell
Such is the volume of books and online resources available that it can be difficult to decide where to start when learning Haskell. If in doubt, take a quick look at the monad tutorials timeline.
10 things I learnt diving in the functional programming deep end - with Haskell
Having recently joined Pusher as a Junior Platform Engineer, I had my hands full from day one; cleaning up infrastructure code, writing integration tests, implementing client certificates and a fully blown Haskell project. Yes, Haskell! My colleague, Will, and I were tasked with replacing our existing integration test framework written...
Porting the Pusher integration tests to Haskell
Those of you who use Pusher will know that the APIs we expose are small and straightforward to use. But this hides the complexity of what is going on behind the scenes. The sheer volume of messages (around 5 billion messages a day and approaching 1.25 trillion since we started...