How to load-test anything

Here at Squid, performance matters. We need to support hundreds of applications with thousands of users, all subscribed to queries on different documents and performing many updates per second, and we need all of this to happen fast. Of course, you can’t make something perform well without being able to measure how well it performs, and so I was tasked with measuring how our system performs under various types of load.

Sandbox instance

You know how every Squid app has a dev instance and a prod instance? Well, this is true for Squid as a whole. We have a sandbox instance of Squid (we’d initially called it "dev" but everyone got confused), which acts as a completely separate entity from the Squid our customers use; it has its own set of apps, its own backend runners, even its own console!

Because our load tests are designed to be some of the heaviest traffic we ever see, we only ever run load tests against our sandbox instance. That way, if it turns out our load tests are too heavy, only the sandbox instance suffers and our customers never know. This also has the advantage that we can test brand new code under load; we wouldn’t want to subject our customers to code if we don’t know its performance characteristics, so we deploy the code to our sandbox instance and test it there.

Running tests

I started out by writing a command-line utility that just ran the same query multiple times in rapid succession. When it was done it would log how much time it took (and also how many queries were successful; ideally they would all be).

const start = Date.now();
for (let i = 0; i < total_count; i++) {
  const data = await testSquid.collection('foo').query().dereference().snapshot();
  assert(data.my_field === 'test value');
}
console.log(`${total_count} queries completed in ${Date.now() - start}ms`);

I tried running this on my own computer, but it turns out my home network connection was just finicky enough that I couldn’t get any useful data out of it, so I put it into a Docker container and ran it in our Kubernetes cluster. This removes the networking almost completely from the equation. The queries are run from the same k8s cluster that the sandbox is running in, which despite being unrealistic allows us to isolate the parts of the system’s performance we care about.

Next up, I wanted to run multiple queries in parallel. At first I was running multiple threads on the same Docker container, but we realized quite quickly that this runs into networking issues on the pod itself. Rather than try to troubleshoot the kernel, we decided to simply run multiple Kubernetes pods, all running this query in a loop.

Sensing the increase in complexity, I decided to write a driver program that would start a job up for me. I wrote it in Typescript, using the @kubernetes/client-node to create a job with the batch API.

More tests

Simply running a query in a loop is a far cry from what Squid can do. The next thing I wanted to test was live updates. Having a driver script that runs everything for me allowed me to set up three separate jobs:

• Before all tests: delete everything from the foocollection.
• Update process: insert four documents into the collection, where each document contains a timestamp for when it was inserted on the client side.

for (let i = 0; i < 4; i++) {
  await testSquid
      .collection('foo')
      .doc(`document_${i}`)
      .insert({my_field: 'test value', insertion_time: Date.now()});;
  await sleep(5000);
}

• Test process over multiple processes and threads: set up a listener on the collection and measure how long it takes for each update to make its way from the update process.

const snapshots = testSquid
  .collection('foo')
  .query()
  .dereference()
  .snapshots();  // Note snapshots, plural
const results = await firstValueFrom(snapshots.pipe(
  map((foo) => Date.now() - foo.insertion_time),
  take(4),
  toArray(),
));
console.log(`Four updates received, timings (ms): ${results}`);

Since I wrote a script to set up the Kubernetes job, it was quite simple to add these two features: the "before all tests" action, and the separate processes for updates vs. queries. I got to a place where running a single command (ts-node createJob.ts) would generate a ton of very interesting performance data. It wasn’t hard to decide where to put all this data…

A Squid app

I wanted to create a web interface for viewing the results of a load test, and what better way to do that than to use Squid? All the data from our load tests lives in the built-in database of a Squid app (on the "real" Squid instance, not on the sandbox), and the interface is a single-page application written in React. Yes, we use Squid internally as much as we can. If it doesn’t work for us, we can’t expect it to work for you!

The app looked absolutely horrible before the design team lent us a few hours of their time

What’s next?

While the results of testing have exceeded our standards, we are always looking forward for other ways we can continue to improve, so we continue to test Squid’s performance from multiple scenarios and push the envelope on what’s possible.

To check out Squid’s blazing fast Client SDK for yourself, start with our quick-start tutorial where you build a CRUD app or check out our samples repo.