Debugging MySQL replication lag by diving into the internals

· 13 March 2020

Recently one of our PMs flagged to us that our product analytics data was nearly 9 days behind the production database! We quickly found that this was due to MySQL replication lag. But to fix this lag, we had to dive deep into the internals of MySQL replication! In this...

Learnings from JAM London 2019

· 20 November 2019

JAM is an annual conference that brings together the best and brightest minds in the product and design industry, for a jam-packed day (pun intended) of talks and panel discussions from industry experts.

Hot SRE trends in 2019 (brought to you from SREcon EMEA)

· 31 October 2019

A couple of weeks ago I attended SREcon EMEA 2019 in Dublin. I felt like I learned a lot over the (exhausting!) 3 days, 19 talks and 2 workshops I attended, so I wanted to write up my experience.

Splash the cache: how caching improved our reliability

· 17 September 2019

In the wake of an incident, we implemented a caching solution to help us handle spiky load better. Here’s how we did it.

Production-ready Database Connection Pooling in Go

and · 30 August 2018

Here at Pusher, our new products, Beams (our push notification API) and Chatkit (our chat API), have recently undergone extensive benchmarking of their systems as part of going from Beta to GA. And both make use of PostgreSQL.

How Pusher Channels has delivered 10,000,000,000,000 messages

· 18 June 2018

At the Pusher office, we have a little counter, constantly incrementing. It shows the total number of messages which have ever been delivered via Pusher Channels. On Friday night, at 22:20 UTC, that counter gained another digit, reaching 10,000,000,000,000 messages. There are 13 zeroes in that number, or ten trillion....

Ephemeral port exhaustion and how to avoid it

· 22 May 2018

Recently, a production incident caused a part of our system to reject all new connections. The culprit: “ephemeral port exhaustion.” By default, your OS will not give any warning when this is about to happen – scary, given the severe consequences! In this post we try to remedy this. We’ll...

Six Tips for Running Scalable Workloads on Kubernetes

· 16 April 2018

Kubernetes is a powerful platform for running scalable and highly available applications. In this post, I discuss the things Kubernetes expects when deploying new applications, as well as the things Kubernetes provides to ensure your applications are always available.

Setting up an SSH CA with Vault

Phil Taprogge · 23 March 2018

Even with modern tools, managing SSH access to hundreds of machines is daunting. At Pusher we use Hashicorp’s Vault as an SSH certificate authority, allowing us to manage ssh access from one central place. Vault makes signing SSH keys so easy that we were able to implement a self-serve mechanism...

Single Sign On For Kubernetes: An Introduction

· 12 March 2018

User management is hard. At Pusher, with an expanding engineering team, we wanted to build a simple identity management experience within our Kubernetes infrastructure. Kubernetes supports a protocol known as Open ID Connect. A system based on OAuth2 that allows you to move your user login experience away from Kubernetes...

Alternatives to sum types in Go

· 25 January 2018

Many statically typed programming languages, such as Haskell or Swift, have a feature called “sum types”. Sum types (also known as tagged unions or variant types) allows a new type to be defined as the “union” of a set of other types and values, and allows users to “pattern match”...

Per-IP rate limiting with iptables

and · 19 September 2017

Every second, Pusher’s main pub-sub system handles 9,000 new WebSocket connections. No sweat. But earlier this year, when the system started receiving spikes of 20,000 new connections every second per server, the sweat began to bead on our foreheads. What, or who, were these new connections? Were they malicious or...

go tool trace

· 06 April 2017

Do you want to know what your Go program is really doing? go tool trace can show you: it visualizes all the runtime events over a run of your Go program, in exquisite detail. This under-documented tool is one of the most useful tools in the Go ecosystem for diagnosing...

Redis Pub/Sub under the hood

· 01 March 2017

Do you want to code a chat app? Here you’ll see how to do it the hard way. I show how Redis Pub/Sub works in detail, all the way down to the bits and bytes! This is the first part of a series of deep dives into Redis.

Golang’s Real-time GC in Theory and Practice

· 01 December 2016

Each day, Pusher sends billions of messages in real-time: source to destination in less than 100ms. How do we achieve this? A key factor is Go’s low-latency garbage collector.

What I Learned Building the PusherSwift Framework

· 17 November 2016

Creating a framework in the Apple developer ecosystem is a bit of a mixed bag. Some of the tools you come into contact with are intuitive, reliable, and a joy to use. Others are incredibly frustrating and you can lose hours (and days) to battling with them, before eventually emerging...

Announcing Go Interface Fuzzer

· 24 August 2016

We’re excited to announce the open source release of Go Interface Fuzzer, a tool for automating the boilerplate of writing a fuzz tester to compare implementations of an interface.

Low latency, large working set, and GHC’s garbage collector: pick two of three

· 12 May 2016

Pusher makes it easy for developers to reliably deliver data at scale. A few features we are working on required a rewrite of our internal message bus, but we knew anything we build has to perform as reliably as the existing platform. Our chosen language for this job was Haskell,...

3 lessons learned after 6 months in a startup

· 18 April 2016

There are thousands of articles and blogs out there about what it’s like working in a startup. Most of them extoll the virtues of Friday beers and remote working, but they struggle to give you much detail about what your work will actually entail.

Fuzz testing distributed systems with QuickCheck

· 31 March 2016

Here at Pusher we’re always trying new approaches to web development and engineering to uncover ways of making our software more reliable. That’s why I’ve recently been using QuickCheck to fuzz test our Raft implementation with some great results.

3 Approaches to Monadic API Design in Haskell

· 08 March 2016

Designing a good API for a library is a challenging problem. In Haskell getting the mix right between specialised and generic can be tricky. For example, just look at the controversy around FTP, where the functions that used to defined for Lists, were lifted to the Foldable and Traversable typeclasses....

My 5 favourite features of Go and how to use them

· 03 March 2016

Ever since I joined Pusher a little over a year ago, I’ve had the opportunity to work on some amazing projects and learn new programming languages. One of the languages I came across was Go.

Approaching Realtime as a Blank Slate

· 17 February 2016

When Pusher started as a company the main environment in which WebSockets were used was the web browser. Since the birth of WebSockets many new JavaScript runtimes have emerged. Now JavaScript runs everywhere, including servers and mobile applications which could benefit from a WebSocket connection. Unfortunately, those environments differ in...

Making Efficient use of memory in Haskell

· 10 February 2016

In part two of my series on Haskell we spoke about identifying memory leaks and tuning the GC to improve performance. In part three we’ll be looking at a number of other techniques and libraries we have found that maximise efficient use of memory in Haskell when writing performance critical...

Memory profiling in Haskell

· 29 January 2016

At Pusher we are currently writing a high performance system in Haskell and blogging about it along the way. In our first blog post we explained some tools and tips that we have used for improving CPU time of our program.

4 questions you need to ask before deploying Docker

Jonas Pfenniger Chevalier · 08 December 2015

Docker was released as open source in March 2013, so in software terms it’s relatively new. As always with shiny things, nerds like me see their potential and start thinking of ways of using them everywhere.

HackDays: Connect Pusher, Slack, Spotify and Sonos for office audio

· 29 October 2015

One of the things that makes working at Pusher great is our appetite for building. We’ve created various internal tools, members of the team work on various open source projects — and of course, Pusher itself is designed to help others in this way.

Top tips and tools for optimising Haskell

· 14 October 2015

(This blog post discusses CPU time profiling, but not space (memory) profiling. We’ll come back to that in Part 2.)

Here’s what I learned from 3 months of startup interviews

· 07 October 2015

At Pusher, we take great pride in making developers’ lives easier. Making messaging APIs simple means they can focus time and resources elsewhere. So it’s slightly ironic that our interview process is designed to make prospective engineers’ lives a little more difficult, if only briefly.

Unit testing IO in Haskell

· 29 September 2015

The generally side-effect free nature of Haskell code makes it convenient to test. Haskell programs can interact with the outside world – otherwise they would be useless – but these side-effects are only possible in the IO monad. It is still important to test this code that performs IO, but...

What the Hack?

Lauren Plews · 26 May 2015

Since joining Pusher last year I have participated in a few office hack days, each one unique, and a good opportunity to get our creative juices flowing. I love seeing the imaginative ways people use Pusher in their apps.

Useful online resources to learn Haskell

Alex Gevers · 23 April 2015

Such is the volume of books and online resources available that it can be difficult to decide where to start when learning Haskell. If in doubt, take a quick look at the monad tutorials timeline.

10 things I learnt diving in the functional programming deep end - with Haskell

· 21 April 2015

Having recently joined Pusher as a Junior Platform Engineer, I had my hands full from day one; cleaning up infrastructure code, writing integration tests, implementing client certificates and a fully blown Haskell project. Yes, Haskell! My colleague, Will, and I were tasked with replacing our existing integration test framework written...

Porting the Pusher integration tests to Haskell

· 08 April 2015

Those of you who use Pusher will know that the APIs we expose are small and straightforward to use. But this hides the complexity of what is going on behind the scenes. The sheer volume of messages (around 5 billion messages a day and approaching 1.25 trillion since we started...