Speedshop - Ruby on Rails performance consulting

We Made Puma Faster With Sleep Sort

Thu, 17 Sep 2020 00:00:00 +0000

Puma 5 (codename Spoony Bard¹(When Puma gets a new ‘supercontributor’ that submits lots of important work to the project, we let them name the next release. This release features a lot of code from Will Jordan, who named this release ‘Spoony Bard’. Will said: ‘Final Fantasy IV is especially nostalgic for me, the first big open-source project I ever worked on was a fan re-translation of the game back in the late 90s.’)¹ When Puma gets a new ‘supercontributor’ that submits lots of important work to the project, we let them name the next release. This release features a lot of code from Will Jordan, who named this release ‘Spoony Bard’. Will said: ‘Final Fantasy IV is especially nostalgic for me, the first big open-source project I ever worked on was a fan re-translation of the game back in the late 90s.’) was released today (my birthday!). There’s a lot going on in this release, so I wanted to talk about the different features and changes to give Puma users confidence in upgrading.

Experimental Performance Features For Cluster Mode on MRI

This is probably the headline of the release - two features for reducing memory usage, and one for reducing latency.

Puma 5 contains 3 new experimental performance features:

wait_for_less_busy_worker config. This may reduce latency on MRI through inserting a small delay (sleep sort!) before re-listening on the socket if worker is busy. Intended result: If enabled, should reduce latency in high-load (>50% utilization) Puma clusters.
fork_worker option and refork command for reduced memory usage by forking from a worker process instead of the master process. Intended result: If enabled, should reduce memory usage.
Added nakayoshi_fork config option. Reduce memory usage in preloaded cluster-mode apps by GCing before fork and compacting, where available. Intended result: If enabled, should reduce memory usage.

All of these experiments are only for cluster mode Puma configs running on MRI.

We’re calling them experimental because we’re not sure if they’ll actually have any benefit. We’re pretty sure they’re stable and won’t break anything, but we’re not sure they’re actually going to have big benefits in the real world. People’s workloads are often not what we anticipate, and synthetic benchmarks are usually not of any help in figuring out if a change will be beneficial or not.

We do not believe any of the new features will have a negative effect or impact the stability of your application. This is either a “it works” or “it does nothing” experiment.

If any of the features turn out to be particularly beneficial, we may make them defaults in future versions of Puma.

If you upgrade and try any of the 3 new features, please post before and after results or screenshots to this Github issue. “It didn’t do anything” is still a useful report in this case. Posting ~24 hours of “before” and ~24 hours of “after” data would be most helpful.

wait_for_less_busy_worker: sleep sort for faster apps?!

This feature was contributed to Puma by Gitlab. Turn it on by adding wait_for_less_busy_worker to your Puma config.

When a request comes in to a Puma cluster, the operating system randomly selects a listening, free Puma worker process to pick up the request. “Listening” and “free” being the key words - a Puma process will only listen to the socket (and pick up more requests) if it has nothing else to do. However, when running Puma with multiple threads, Puma will also listen on the socket when all of its busy threads are waiting on I/O or have otherwise released the Global VM Lock.

When Gitlab investigated switching from Unicorn to Puma, they encountered an issue with this behavior. Under high load with moderate thread settings (a max pool size of 5 in their case), average request latency increased. Why?

Remember, I said that the operating system randomly assigns a request to a listening worker process. So, it will never send a request to a worker process that’s busy doing other things, but what about a worker process that’s got 4 threads that are processing other requests, but all 4 of those threads happen to be waiting on I/O right now?

Imagine a Puma cluster with 3 workers:

Worker 1: 0/5 threads busy.
Worker 2: 1/5 threads busy.
Worker 3: 4/5 threads busy.

If Worker 3’s 4 active threads happen to all have released the GVL, allowing that worker to listen to the socket, and a new request comes in - which worker process should we assign the request to, ideally? Worker 1, right? Unfortunately, most operating systems will assign the request to Worker 3 33% of the time.

So, what do we do? We want the operating system to prefer less-loaded workers. It would be really cool if we could sort the list of workers listening on the socket so that the operating system would give requests to the least-loaded worker. Well, we can’t really do that easily, but we can do something else.

wait_for_less_busy_worker causes a worker to wait to re-listen on the socket if it’s thread pool isn’t completely empty. This means that in high-load scenarios, the operating system will assign requests to less-loaded workers.

This is basically sleep-sorting our workers. We’re kind of doing doing this:

[].tap { |a| workers.map { |e| Thread.new{ sleep worker_busyness.to_f/1000; a << e} }.each{|t| t.join} }

… and hiding “more loaded” workers from the operating system by letting less-loaded workers listen first!

Originally the proposal was for a more complicated sort - processes slept longer if they had more busy threads - but that was removed when it was found that a simpler on/off sleep was just as effective.

The net effect is that in high-load scenarios, request latency decreases. This is because workers with more busy threads are slower than workers with no busy threads. We’re assuring that requests get assigned to the faster workers. Prior to this patch, Gitlab saw an increase in latency using Puma compared to Unicorn - after this patch, latency was the same (they also were able to reduce their fleet size by almost 30% thanks to Puma’s memory-saving multithreaded design).

There may be even more efficient ways for us to implement this behavior in the future. There’s some magic you can do with libev, I’m pretty sure, or we can just implement a different sleep/wait strategy.

fork_worker

Adding fork_worker to your puma.rb config file (or --fork-worker from the CLI) turns on this feature. This mode causes Puma to fork additional workers from worker 0, instead of directly from the master process:

 \_ puma 5.0.0 (tcp://0.0.0.0:9292) [puma]
     \_ puma: cluster worker 0: 10000 [puma]
         \_ puma: cluster worker 1: 10000 [puma]
         \_ puma: cluster worker 2: 10000 [puma]
         \_ puma: cluster worker 3: 10000 [puma]

Similar to the preload_app! option, the fork_worker option allows your application to be initialized only once for copy-on-write memory savings, and it has two additional advantages:

Compatible with phased restart. Because the master process itself doesn’t preload the application, this mode works with phased restart (SIGUSR1 or pumactl phased-restart), unlike preload_app!. When worker 0 reloads as part of a phased restart, it initializes a new copy of your application first, then the other workers reload by forking from this new worker already containing the new preloaded application.

This allows a phased restart to complete as quickly as a hot restart (SIGUSR2 or pumactl restart), while still minimizing downtime by staggering the restart across cluster workers.

‘Refork’ for additional copy-on-write improvements in running applications. Fork-worker mode introduces a new refork command that re-loads all nonzero workers by re-forking them from worker 0.

This command can potentially improve memory utilization in large or complex applications that don’t fully pre-initialize on startup, because the re-forked workers can share copy-on-write memory with a worker that has been running for a while and serving requests.

You can trigger a refork by sending the cluster the SIGURG signal or running the pumactl refork command at any time. A refork will also automatically trigger once, after a certain number of requests have been processed by worker 0 (default 1000). To configure the number of requests before the auto-refork, pass a positive integer argument to fork_worker (e.g., fork_worker 1000), or 0 to disable.

nakayoshi_fork

Add nakayoshi_fork to your puma.rb config to try this option.

Nakayoshi means “friendly”, so this is a “friendly fork”. The concept was originally implemented by MRI supercontributor Koichi Sasada in a gem, but we wanted to see if we could bring a simpler version into Puma.

Basically, we just do the following before forking a worker:

4.times { GC.start }
GC.compact # if available

The concept here is that we’re trying to get as clean of a Ruby heap as possible before forking to maximize copy-on-write benefits. That should, in turn, lead to reduced memory usage.

Other New Features

A few more things in the grab-bag:

You can now compile Puma on machines where OpenSSL is not installed.
There is now a thread-backtraces command in pumactl to print all active threads backtraces. This has been available via SIGINFO on Darwin, but now it works on Linux via this new command.
Puma.stats now has a requests_count counter.
lowlevel_error_handler got some enhancements - we also pass the status code to it now.
Phased restarts and worker timeouts should be faster.
Puma.stats_hash provides Puma statistics as a hash, rather than as JSON.

Loads of Bugfixes

The number of bugfixes in this release is pretty huge. Here’s the most important ones:

Shutdowns should be more reliable.
Issues surrounding socket closing on shutdown have been resolved.
Fixed some concurrency bugs in the Reactor.
out_of_band should be much more reliable now.
Fixed an issue users were seeing with ActionCable and not being able to start a server.
Many stability improvements to prune_bundler.

Nicer Internals and Tests

This release has seen a massive improvement to our test coverage. We’ve pretty much doubled the size of the test suite since 4.0, and it’s way more stable and reproducible now too.

A number of breaking changes come with this major release. For the complete list, see the HISTORY file.

Thanks to Our Contributors!

This release is our first major or minor release with new maintainer MSP-Greg on the team. Greg has been doing tons of work on the test suite to make it more reliable, as well as a lot of work on our SSL features to bring them up-to-date and more extendable. Greg is also our main Windows expert.

The following people contributed more than 10 commits to this release:

Tim Morgan
Vyacheslav Alexeev
Will Jordan
Jeff Levin
Patrik Ragnarsson, who’s also been very helpful in our Issues tracker.

If you’ve like to make a contribution to Puma, please see our Contributors Guide. We’re always looking for more help and try to make it as easy as possible to contribute.

Enjoy Puma 5!

The Practical Effects of the GVL on Scaling in Ruby

Mon, 11 May 2020 00:00:00 +0000

The Global Virtual Machine Lock confuses many Rubyists. Most Rubyists I’ve met have a vague sense that the GVL is somehow bad, and has something to do concurrency or parallelism.

(‘CRuby’ refers to the mainline Ruby implementation, written in C. Sometimes people call this ‘MRI’.)‘CRuby’ refers to the mainline Ruby implementation, written in C. Sometimes people call this ‘MRI’. The GVL (formerly known as GIL, as you’re about to learn) is a unique feature to CRuby, and doesn’t exist in JRuby or TruffleRuby.

JavaScript’s popular V8 virtual machine also has a VM lock. CPython also has a global VM lock. That’s three of the most popular dynamic languages in the world! VM locks in dynamic languages are very common.

(Instead of removing the GVL, Ruby core has signaled that it will take an approach similar to V8 Isolates with inspiration from the Actor concurrency model (discussed at the end).)Instead of removing the GVL, Ruby core has signaled that it will take an approach similar to V8 Isolates with inspiration from the Actor concurrency model (discussed at the end).

Understanding CRuby’s Global VM Lock is important when thinking about scaling Ruby applications. It will probably never be removed from CRuby completely, and its behavior changes how we scale Ruby apps efficiently.

Understanding what the GVL is and why the current GVL is “global” will help you to answer questions like these:

What should I set my Sidekiq concurrency to?
How many threads should I use with Puma?
Should I switch to Puma or Sidekiq from Unicorn, Resque, or DelayedJob?
What are the advantages of event-driven concurrency models, like Node?
What are the advantages of a global-lock-less language VM, like Erlang’s BEAM or Java’s JVM?
How will Ruby’s concurrency story change in Ruby 3?

We’ll deal with these questions and more in this article.

What we’re locking: the language virtual machine

(Most descriptions of the GVL immediately dive into concepts like atomicity and thread-safety. This description will start from a more basic premise and work up to that.)Most descriptions of the GVL immediately dive into concepts like atomicity and thread-safety. This description will start from a more basic premise and work up to that.

(YARV was essentially Koichi Sasada’s graduate thesis.)YARV was essentially Koichi Sasada’s graduate thesis.

Wait: isn’t it the GIL? What’s the GVL?

GIL stands for Global Interpreter Lock, and it’s something that was removed from Ruby (or just mutated, depending on how you look at it) in Ruby 1.9, when Koichi Sasada introduced YARV (Yet Another Ruby VM) to Ruby. YARV changed CRuby’s internal structure so that the lock existed around the Ruby virtual machine, not an interpreter. The correct terminology for over a decade now has been GVL, not GIL.

(You can interact with instruction sequences via the InstructionSequence class. Everything is an object in Ruby!)You can interact with instruction sequences via the InstructionSequence class. Everything is an object in Ruby!

How does an interpreter differ from a virtual machine?

A virtual machine is a little like a CPU-within-a-CPU. Virtual machines are computer programs that usually take simple instructions, and those instructions manipulate some internal state. A Turing machine, if it was implemented in software, would be a kind of virtual machine. We call them virtual machines and not machines because they’re implemented in software, rather than in hardware, like a CPU is.

Before Ruby 1.9, Ruby didn’t really have a separate virtual machine step - it just had an interpreter. As your Ruby program ran, it actually interpreted each line of Ruby as it went. Now, we just interpret the code once, turn it into a series of VM instructions, and then execute those instructions. This is much faster than interpreting Ruby constantly.

A Turing machine, implemented in software, would be a kind of virtual machine. Wikimedia Commons by RosarioVanTuple

The Ruby Virtual Machine understands a simple instruction set. Those instructions are generated from the Ruby code you write by the interpreter, and then the virtual machine instructions are fed into the Ruby VM.

Let’s watch this in action. First, in case you didn’t know, you can execute Ruby from the command line using the -e option:

$ ruby -e "puts 1 + 1"
2

Now, you can then dump the instructions for this simple program by calling --dump=insns:

$ ruby --dump=insns -e "puts 1 + 1"
== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,10)> (catch: FALSE)
putself                                                          (   1)[Li]
putobject_INT2FIX_1_
putobject_INT2FIX_1_
opt_plus                     <callinfo!mid:+, argc:1, ARGS_SIMPLE>, <callcache>
opt_send_without_block       <callinfo!mid:puts, argc:1, FCALL|ARGS_SIMPLE>, <callcache>
leave

Ruby is a “stack-based” VM. You can see how this works by looking at the generated instructions here - we add the integer 1 to the stack two times, than call plus. When plus is called, there are two integers on the stack. Those two integers are replaced by the result, 2, which is then on the stack.

So, what does the Ruby VM have to do with threading, concurrency and parallelism?

Concurrency and Paralellism

You may be aware that there’s a difference between concurrency and parallelism. Imagine a grocery store. At this grocery store, we have a line and some checkout clerks working to pull customers from the line and get their groceries checked out.

Each of our grocery store checkout clerks works in parallel. They don’t need to talk to each other to do their job, and what one clerk is doing doesn’t affect the other in any way. They’re working 100% in parallel.

Now, a clerk can work on multiple customers concurrently. This would look like a clerk grabbing multiple customers from the line, working on one customer’s groceries for a moment, then switching to another customer’s groceries, and so on. This would be working concurrently.

Let’s take a more concrete example. Compare three grocery store clerks working in parallel with a single one working concurrently. To check out a customer, we must perform two operations: scanning their groceries, and then bagging them. Imagine each customer’s groceries take the exact same amount of time to scan and bag.

Three customers arrive. Let’s say scanning takes time A and bagging takes time B. Our three parallel clerks will process these three customers in time (A + B).

The parallel case.

What about our concurrent clerk? All three of her customers arrive at the same time. The clerk scans each customer’s groceries, then bags each customer’s groceries. Each customer is worked on concurrently, but never in parallel.

The concurrent case.

In the concurrent case, our first customer experiences a total service time of (3A + B). They had to wait for everyone else’s groceries to be checked out for their own groceries to get bagged. The second customer will exerience a total service time of (3A + 2B), and the final customer will experience a service time of (3A + 3B).

Notice how the customers who got the concurrent checkout clerk experienced a longer total service time than the customers who used our three parallel clerks.

In short: concurrency is interesting, but parallelism is what speeds up systems and allows them to handle increased load.

Performing two operations concurrently means that the start and end times of those operations overlapped at some point. For example, you and I sit down to a sign a contract. However, there is only one pen. I sign where I’m supposed to, hand the pen to you, and then you sign. Then, you hand the pen back to me and I initial a few lines. You might say that we signed the contract concurrently, but never in parallel - there was only one pen, so we couldn’t sign the contract at the exact same time.

Peforming operations in parallel means that we are doing those operations at the exact same instant. In my contract example, a parallel contract signing would involve two pens (and probably two copies of the contract, otherwise it would get a little crowded).

Concurrency and Paralellism on a Computer

On a modern operating system, programs are run with a combination of processes and threads. Processes have at least one thread, and can have up to thousands.

To extend the grocery store analogy, processes are like the checkout counters that our clerks use. They contain tools and common resources, like the point-of-sale terminal and the barcode scanner, but they don’t actually do anything. A process usually contains a memory allocation (the heap), file descriptors (sockets, files, etc), and other such computer resources.

Threads actually run our code. Each process has at least one thread. In our analogy, they’re like the store clerks. They also hold a small amount of information. For example, if we’re adding two local variables in a Rails application, our thread contains information about those two variables (thread-local storage) and also what line of code we’re currently running (the stack).

Threads run the code when they are scheduled to by the operating system’s kernel. The Ruby runtime itself doesn’t actually manage when threads are executed - the operating system decides that.

When Ruby was written in the 90s, all processes had just one thread. This started to change in the early 2000s, necessitating the rewrite of the language VM in Ruby 1.9 (YARV), which is what gave us the GVL as we know it today.

What the GVL actually does

As mentioned earlier, the Ruby Virtual Machine is what actually turns Ruby virtual machine instructions (generated by the interpreter) into CPU instructions.

The Ruby Virtual Machine is not internally thread-safe. If two threads try to access the Ruby VM at the same time, really Bad Things would happen. This is a bit like the point of sale terminal at our grocery store checkout counters. If two checkout clerks tried to use the same POS terminal, they would interrupt each other and probably keep losing their work or corrupting each other’s work. You would end up paying for someone else’s groceries!

So, because it isn’t safe for multiple threads to access the Ruby Virtual Machine at the same moment, instead we use a global lock around it so that only one thread can access it in parallel.

(One caveat of the Javascript GVL is that it isn’t actually global: you can create additional Isolates. Koichi Sasada’s proposal for Ractors (formerly Guilds) would be similar.)One caveat of the Javascript GVL is that it isn’t actually global: you can create additional Isolates. Koichi Sasada’s proposal for Ractors (formerly Guilds) would be similar.

It is extremely common for dynamic language VMs to not be thread-safe. As mentioned, CPython and V8 are the most prominent examples. Java is probably the best example of a semi-dynamic language that does have a threadsafe VM. It’s also why so many languages are written on top of the JVM: writing your own threadsafe VM is really hard.

TFW you realize that there’s always going to be locks, the only difference is what level they’re implemented at and who implements them

There’s a few good reasons that having a GVL is so popular:

It’s faster. Single-threaded performance improves because you don’t have to constantly lock and unlock internals.
Integrating with extensions, such as C extensions, is easier.
It’s easier to write a lockless VM than one with a lot of locks.

Each Ruby process has its own Global VM Lock, so it might be more accurate to say that it’s a “process-wide VM lock”. Its “global” in the same sense that a “global variable” is global.

Only one thread in any Ruby process can hold the global VM lock at any given time. Since a thread needs access to the Ruby Virtual Machine to actually run any Ruby code, effectively only one thread can run Ruby code at any given time.

Let me play you the song of my people: “GGVVVVLLLLLLLLLLLLLLLL”

Think of the GVL like the conch shell in the Lord of the Flies - if you have it, you get to speak (or execute Ruby code in this case). If the GVL is already locked by a different thread, other threads must wait for the GVL to be released before they can hold the GVL.

Amdahl’s Law: Why 1 Sidekiq Process Can Be 2x as Efficient as DelayedJob or Resque

Your programs actually do many things that don’t need to access the Ruby Virtual Machine. The most important is waiting on I/O, such as database and network calls. These actions are executed in C, and the GVL is explicitly released by the thread waiting on that I/O to return. When the I/O returns, the thread attempts to reacquire the GVL and continue to do whatever the program says.

This has enormous real-world performance impacts.

Imagine you have a stack of satellite image data you have to process (with Ruby). You have written a Sidekiq job, called SatelliteDataProcessorJob, and each job works on a small fraction of all of the satellite data.

class SatelliteDataProcessorJob
  include Sidekiq::Worker

  def perform(some_satellite_data)
    process(some_satellite_data)
    touch_external_service(some_satellite_data)
    add_data_to_database(some_satellite_data)
  end
end

Let’s imagine that process is a 100% Ruby method, which does not call C extensions or external services. Further, let’s imagine that touch_external_service and add_data_to_database are effectively 100% I/O methods that spend all of their time waiting on the network.

First, an easy question: if each run of SatelliteDataProcessorJob takes 1 second, and you have 100 enqueued jobs and just 1 Sidekiq process with 1 thread, how long will it take to process all the jobs? Assume infinite CPU and memory resources.

100 seconds.

How about if you two processes? 50 seconds. And 25 seconds for 4 processes and so on. That’s parallelism.

Now, let’s say you have 1 Sidekiq process with 10 threads. How long will it take to process all of those jobs?

The answer is it depends. If you’re on JRuby or TruffleRuby, it will take about 10 seconds, because each thread is fully parallel with all the other threads.

But on MRI, we have the GVL. Does adding threads increase concurrency?

From Daniels 220 @ Wikipedia

It turns out, this exact problem interested computer scientist Gene Amdahl back in 1967. He proposed something called Amdahl’s Law, which gives the theoretical speedup in latency for the execution of tasks with fixed workloads when resources increase.

Amdahl figured out that the speedup you got from additional parallelism was related to the proportion of execution time that could be done in parallel. Sound familiar?

Amdahl’s Law is simply 1 / (1 - p + p/s), where p is the percentage of the task that could be done in parallel, and s is the speedup factor from the part of the task that gained improved resources (the parallel part).

So, in our example, let’s say that half of SatelliteDataProcessorJob is GVL-bound and half is IO-bound. In this case, p is 0.5 and s is 10, because we can wait for IO in parallel and there are 10 threads. In this case, Amdahl’s Law shows that a Sidekiq process would go through our jobs up to 1.81x faster than a single-threaded Resque or DelayedJob process.

Many background jobs in Ruby spend at least 50% of their time waiting on IO. For those jobs, Sidekiq can lead to a 2x decrease in resource usage, because 1 Sidekiq process can do the work of what used to take 2 single-threaded processes.

So, even with a GVL, adding threads to applications increases throughput-per-process, which in turn lowers memory consumption.

Threads, Puma and GVL-caused Latency

This also means that “how many threads does my Sidekiq or Puma process need” is a question answered by “how much time does that thread spend in non-GVL execution?” or “how much time does my program spend waiting on I/O?” Workloads with high percentages of time spent in I/O (75%+ or more) often benefit from 16 threads or even more, but more typical workloads see benefit from just 3 to 5 threads.

It’s possible to configure your thread pools to be too large. Setting Puma or Sidekiq to thread settings higher than 5 can lead to contention for the GVL if the work is not parallelizable enough. This increases service latency.

While total time to process all of the units of work remains the same, the latency experienced by each individual unit of work increases.

Imagine a grocery store where a checkout clerk grabbed 16 people off of the checkout queue and checked those 16 people’s groceries concurrently, scanning one item per person before scanning one item from the next person’s cart. Rather than experiencing checkout time as (A + B), they experience a checkout time of 16(A+B).

(This effect is generally present in a concurrent-but-not-100%-parallel system where overall utilization is not extremely high. We’re mitigating this effect slightly in Puma 5 by having Puma workers with more than one thread delay listening to the socket, so less-loaded workers pick up requests first.)This effect is generally present in a concurrent-but-not-100%-parallel system where overall utilization is not extremely high. We’re mitigating this effect slightly in Puma 5 by having Puma workers with more than one thread delay listening to the socket, so less-loaded workers pick up requests first.

Some people misidentify this additional latency as “context switching” costs. However, latency experienced by the individual unit of work is increasing without additional switching cost.

In any case, context switching on modern machines and operating systems is pretty cheap relative to the time it takes to service a typical web app request or background job. It does not add hundreds of milliseconds to response times - but oversaturating the GVL can.

If adding threads to a CRuby process can increase latency, why is it still useful?

(Shouldn’t adding an additonal thread only increase memory usage by 8MB, which is the size of the thread’s stack allocation? Ah, if only memory usage was that simple. Learn more about the complexities of RSS and thread-induced fragmentation here.)Shouldn’t adding an additonal thread only increase memory usage by 8MB, which is the size of the thread’s stack allocation? Ah, if only memory usage was that simple. Learn more about the complexities of RSS and thread-induced fragmentation here.

Adding more threads to a Ruby process helps us to improve CPU utilization at less memory cost than an entire additional process. Adding 1 process might use 512MB of memory, but adding 1 thread will probably cause less than 64MB of additional memory usage. With 2 threads instead of 1, when the first thread releases the GVL and listens on I/O, our 2nd thread can either pick up new work to do, increasing throughput and utilization of our server.

GitLab switched from Unicorn (single-thread model) to Puma (multi-thread model) and saw a 30% decrease in memory usage across their fleet. If you’re memory-constrained on your host, this allows you to run 30% more throughput for the same money. That’s awesome.

The Future

For a decade now, bystanders have declared that Ruby is dead because it “doesn’t have a proper concurrency story”.

I think we’ve shown that there is a concurrency story in Ruby. First, we have process-based concurrency. We multiply GVLs by multiplying processes. This works perfectly fine, if you have enough memory.

If you’re out of memory, you can use Sidekiq or Puma, which provides a threaded container for our apps, and then let pre-emptive threading do its thing.

Ruby has proven that process-based concurrency (which is really what the GVL forces us to do) scales well. It’s not much more expensive than other models, especially these days when memory is so cheap on cloud providers. Think critically about what an Actor-style approach or an Erlang Process-style approach would actually change about your deployment at the end of the day: you would use less memory per CPU. But on large deployments, most web applications are already CPU-bottlenecked, not memory!

Ractor

Koichi Sasada, author of YARV, is proposing a new concurrency abstraction for Ruby 3 called Ractors. It’s a proposal based on the Actor concurrency model (hence Ruby Actor -> Ractor). Basically, Actors are boxes for objects to go into, and each actor can only touch its own objects, but can send and receive objects to/from other Actors. Here’s an example written by Koichi Sasada:

r = Ractor.current
rs = (1..10).map{|i|
  r = Ractor.new r, i do |r, i|
    r.send Ractor.recv + "r#{i}"
  end
}
r.send "r0"
p Ractor.recv #=> "r0r10r9r8r7r6r5r4r3r2r1"

Eventually (not yet in the current implementation), each Ractor will get their own VM lock. That means the example code above will execute in parallel.

This is made possible because Ractors don’t share mutable state. Instead, they only share immutable objects, and can send mutable objects between each other. This should mean that we don’t need a VM lock inside of a Ractor.

Koichi Sasada’s Ractor proposal is now public, though as of this writing the docs are mostly in Japanese, and “each Ractor gets its own VM lock” has not yet been implemented. Ractors will essentially allow us to “multiply” GVLs in a process, which would make the GVL no longer “global”, although the lock will still exist in each Ractor. The Global VM Lock will become a Ractor VM Lock.

TL:DR;

Thanks for listening to me whinge. Here’s what you need to remember:

If you are memory bottlenecked on Ruby, you need to saturate the GVL by adding more threads, which will allow you to get more CPU work done with less memory use.
The GVL means that parallelism is limited to I/O in Ruby, so switch to a multithreaded background job processor before you switch to a multithreaded web server. Also, you’ll probably use much higher threadpool sizes with your background jobs than with your web server.
Ruby 3 might make the GVL no longer global by allowing you to multiply VMs using Ractors. Application servers and background job processors will probably change their backend to take advantage of this, you won’t really have to change much of your code at all, but you will no longer have to worry about thread safety (yay).
Process based concurrency scales very well, and while it might lose a few microseconds to other concurrency models, these concurrency switching costs generally don’t matter for the typical Rails application. Instead, the important thing is saturating CPU, which is the most scarce resource in today’s computing environments.

The World Follows Power Laws: Why Premature Optimization is Bad

Sun, 22 Dec 2019 07:00:00 +0000

(This post is a sample of the content available in the Complete Guide to Rails Performance. It’s actually the first lesson - there are 30+ more lessons and 18 hours of video in the course itself.)This post is a sample of the content available in the Complete Guide to Rails Performance. It’s actually the first lesson - there are 30+ more lessons and 18 hours of video in the course itself.

I want to tell you about a physicist from Schenectady, a Harvard linguist, and an Italian economist.

Pareto’s poltiical views are bit suspect, unfortunately, because he chose to see the way things are (unequal) as the way they ought to be.

The Italian economist you may already have heard of - Vilifredo Pareto. He became famous for something called The Pareto Principle, the idea that for most things, 80% of the effect comes from just 20% of the causes. The Pareto Principle is fundamental to performance work because it reminds us why premature optimization is so inefficient and useless. While you’ve probably heard of the Pareto Principle, I want you to understand why it actually works. And to do that, we’re going to have to talk about probability distributions.

Benford - the physicist

Frank Benford was an American electrical engineer and physicist who worked for General Electric. It was the early 20th century, when you had a job for life rather than a startup gig for 18 months, so he worked there from the day he graduated from the University of Michigan until his death 38 years later in 1948.
A page from Henry Briggs’ first table of common logarithms, Logarithmorum Chilias Prima, from 1617. Wikipedia

Back in that time, before calculators, if you wanted to know the logarithm of a number - say, 12 - you looked it up in a book. The books were usually organized by the leading digit. If you wanted to know the logarithm of 330, you first went to the section for 3, then looked for 330. Benford noticed that the first pages of the book were far more worn out than the last pages. Benford realized this meant that the numbers looked up in the table began more often with 1 than with 9.

Most people would have noticed that and thought nothing of it. But Benford pooled 20,000 numbers from widely divergent sources (he used the numbers in newspaper stories) and found that the leading digit of all those numbers followed a power law too.

This became known as Benford’s Law. Here are some other sets of numbers that conform to this power law:

Physical constants of the universe (pi, the molar constant, etc.)
Surface areas of rivers
Fibonacci numbers
Powers of 2
Death rates
Population censuses

That the physical constants of the universe follow this distribution is probably the most mind-blowing revelation of Benford’s Law, for me, anyway.

Benford’s Law is so airtight that it’s been admitted in US courts as evidence of accounting fraud (someone used RAND in their Excel sheet!). It’s been used to identify other types of fraud too - elections, scientific and even macroeconomic data.

What would cause numbers that have (seemingly) little relationship with each other to conform so perfectly to this non-random distribution?

Zipf - the linguist

A plot of the rank versus frequency for the first 10 million words in 30 different languages of Wikipedia. Note the logarithmic scales. Licensed CC-BY-SA by SergioJimenez.

At almost exactly the same time as Benford was looking at first leading digits, George Kingsley Zipf was studying languages at Harvard. Uniquely, George was applying the techniques of a new and interesting field - statistics - to the study of language. This landed him an astonishing insight: in nearly every language, some words are used a lot, but most (nearly all) words are used hardly at all.

Only a few words account for most of our use of language. The Brown Corpus is a collection of literature used by linguistics researchers. It consists of 500 samples of English-language text comprising 1 million words. Just 135 unique words are needed to account for 50% of those million words. That’s insane.

Zipf’s probability distribution is discrete. Discrete distributions are comprised of whole integers. Continuous distributions can take on any value. If you take Zipf’s distribution and make it continuous instead of discrete, you get the Pareto distribution.

Pareto - the economist

Pareto initially noticed a curious distribution when he was thinking about wealth in society - he noticed that 80% of the wealth and income came from 20% of the people in it.

The Pareto distribution, pictured, has been found to hold for a scary number of completely different and unrelated fields in the sciences. For example, here are some natural phenomena that exhibit a Pareto (power law) distribution:

Wealth inequality
Sizes of rocks on a beach
Hard disk drive error rates (!)
File size distribution of Internet traffic (!!!)

We tend to think of the natural world as random or chaotic. In schools, we’re taught the bell curve/normal distribution. But reality isn’t normally distributed. It’s log-normal. Many probability distributions, in the wild, support the Pareto Principle:

80% of the output will come from 20% of the input

Normal distributions are taught in schools because they’re quite easy to talk about mathematically, not because they’re particularly good descriptions of the natural world.

While you may have heard this before, what I’m trying to get across to you is that it isn’t made up. The Pareto distribution is used in hundreds of otherwise completely unrelated scientific fields - and we can use its ubiquity to our advantage.

It doesn’t matter what area you’re working in - if you’re applying equal effort to all areas, you are wasting your time. What the Pareto distribution shows us is that most of the time, our efforts would be better spent finding and identifying the crucial 20% that accounts for 80% of the output.

Allow me to reformulate and apply this to web application performance:

80% of an application’s work occurs in 20% of its code.

There are other applications in our performance realm too:

80% of an application’s traffic will come from 20% of its features.

80% of an application’s memory usage will come from 20% of its allocated objects.

The ratio isn’t always 80/20. Actually, usually it’s way more severe - 90/10, 95/5, 99/1. Sometimes it’s less severe. So long as it isn’t 50/50 we’re talking about a non-normal distribution.

This is why premature optimization is so bad and why performance monitoring, profiling and benchmarking are so important. The world is full of power-law distributions, not normal distributions. Spreading your effort evenly across a power-law distribution is a massive waste of effort.

What the Pareto Principle reveals to us is that optimizing any random line of code in our application is in fact unlikely to speed up our application at all! 80% of the “slowness” in any given app will be hidden away in a minority of the code.

So instead of optimizing blindly, applying principles at random that we read from blog posts, or engaging in Hacker-News-Driven-Development by using the latest and “most performant” web technologies, we need to measure where the bottlenecks and problem areas are in our application.

An Optimization Story - Measurement, Profiling and Benchmarking

There’s only one skill in performance work that you need to understand completely and deeply - how to measure your application’s performance. Once you have that skill mastered, knowing every possible thing about performance might be a waste of time. Your problems are not other’s problems. There are going to be lessons to learn that solve problems you don’t have (or don’t comprise that crucial 20% of the causes of slowness in your application).

On the flip side, you should realize that the Pareto Principle is extremely liberating. You don’t need to fix every performance issue in your application. You don’t need to go line-by-line to look for problems under every rock. You need to measure the actual performance of your application, and focus on the 20% of your code that is the worst performance offender.

My first conference talk ever, actually.

I once gave a conference talk that was a guided read-through of Minitest, the Ruby testing framework. Minitest is a great read if you’ve got a spare hour or two - it’s fairly short at just 1,500 lines. As I was reading Minitest’s code, I came across this funny line:

def self.runnable_methods
  methods = methods_matching(/^test_/)

  case self.test_order
  when :random, :parallel then
    max = methods.size
    methods.sort.sort_by { rand max }
  when :alpha, :sorted then
    methods.sort
  else
    raise "Unknown test_order: #{self.test_order.inspect}"
  end
end

This code is extremely readable as to what’s going on; we determine which methods on a class are runnable with a regex (“starts with test_”), and then sort them depending upon this test class’s test_order. Minitest uses the return value to execute all of the runnable_methods on all the test classes you give it. Usually this is a randomized array of method names, because the default test order is :random.

What I was honing in on was this line, which is run when :test_order is :random or :parallel:

max = methods.size
methods.sort.sort_by { rand max }

This seemed like a really roundabout way to do methods.shuffle to me. Maybe Ryan (Minitest’s author) was doing some weird thing to ensure deterministic execution given a seed. Minitest runs your tests in the same order given the same seed to the random number generator. It turns out methods.shuffle is deterministic, though, just like the code as written. So, I decided to benchmark it, mostly out of curiosity.

Whenever I need to write a micro benchmark of Ruby code, I reach for benchmark/ips.¹(The reason I use benchmark/ips rather than the stdlib benchmark is because the stdlib version requires you to run a certain line of code X number of times and tells you how long that took. The problem with that is that I don’t usually know how fast the code is to begin with, so I have no idea how to set X. Usually I run the code a few times, guess at a number of X that will make the benchmark take 10 seconds to run, and then move on. benchmark/ips does that work for me by running my benchmark for 10 seconds and calculating iterations-per-second.)¹ The reason I use benchmark/ips rather than the stdlib benchmark is because the stdlib version requires you to run a certain line of code X number of times and tells you how long that took. The problem with that is that I don’t usually know how fast the code is to begin with, so I have no idea how to set X. Usually I run the code a few times, guess at a number of X that will make the benchmark take 10 seconds to run, and then move on. benchmark/ips does that work for me by running my benchmark for 10 seconds and calculating iterations-per-second. ips stands for iterations-per-second. The gem is an extension of the Benchmark module, something we get in the Ruby stdlib.

Here’s that benchmark:

require "benchmark/ips"

class TestBench
  def methods
    @methods ||= ("a".."z").to_a
  end

  def fast
    methods.shuffle
  end

  def slow
    max = methods.size
    methods.sort.sort_by { rand max }
  end
end

test = TestBench.new

Benchmark.ips do |x|
  x.report("faster alternative") { test.fast }
  x.report("current minitest code") { test.slow }
  x.compare!
end

This benchmark suggested that shuffle was 12x faster than sort.sort_by { rand methods.size }. This makes sense - shuffle randomizes the array with C, which will always be faster than randomizing it with pure Ruby. In addition, Ryan was actually sorting the array twice - once in alphabetical order, followed by a random shuffle based on the output of rand.

Ryan’s conference talks are pretty good, too.

I asked Ryan Davis, minitest author, what was up with this. He gave me a great reply: “you benchmarked it, but did you profile it?”

What did he mean by this? Well, first, you have to know the difference between benchmarking and profiling - the two fundamental performance measurement tools.

There are a lot of different ways to define this difference. Here’s my attempt:

Benchmarking

A benchmark is a test of one or many different pieces of code that measures how fast they execute or how many resources they consume.

When we benchmark, we take two competing pieces of code and compare them. It could be as simple as a one liner, like in my story, or as complex as an entire web framework. Then, we put them up against each other (usually comparing them in terms of iterations/second) using a simple, contrived task. At the end of the task, we come up with a single metric - a score. We use the score to compare the two competing options.

In my example above, it was just how fast each line could shuffle an array. If you were benchmarking web frameworks, you might test how fast a framework can return a simple “Hello World” response. Benchmarks put the competing alternatives on exactly equal footing by coming up with a contrived, simple, non-real-world example.

I wrote a v v long post once about why this benchmark doesn’t mean much for Rails

It’s usually too difficult to benchmark real-world code because the alternatives aren’t doing exactly the same thing. For example, comparing Rails against Sinatra isn’t entirely fair because Rails has many features that Sinatra does not - even for a simple Hello World response, the Rails application is, for example, performing many security checks that the Sinatra app doesn’t. Comparing these frameworks in a 1-to-1 benchmark will always be slightly misleading for that reason.

Profiling

Profiles are a accounting of all the sub-steps required to run a given piece of code. When we profile, we’re usually examining the performance characteristics of an entire, real-world application. For example, this might be a web application or a test suite. Because profiling works with real-world code, we can’t really use it to compare competing alternatives, because the alternative usually doesn’t exactly match what we’re profiling. Profiling doesn’t usually produce a comparable “score” at the end with which to measure these alternatives, either. But that’s not to say profiling is useless - it can tell us a lot of valuable things, like what percentage of CPU time was used where, where memory was allocated, and what lines of code are important and which ones aren’t.

What Ryan was asking me was - “Yeah, that way is faster on this one line, but does it really matter in the grand scheme of Minitest”? How much time does a Minitest test run actually spend shuffling the methods? 1%? 10%? 0.001%? Profiling can tells us that.

You said that this one-line change would speed up minitest. A higher-level benchmark determined that was a lie.

Is this one line really part of Pareto’s “20%”? We can assume, based on the Principle, that 80% of Minitest’s execution time will come from just 20% of its code. Was this line part of that 20%?

I’ve already shown you how to benchmark on the micro scale. But before we get to profiling, I’m going to do a quick macro-benchmark to test my assumption that using shuffle instead of sort.sort_by will speed up Minitest.

Minitest is used to run tests, so we’re going to benchmark a whole test suite. Rubygems.org, an open-source Rails application with a Minitest suite, will make a good example test suite.

When micro-benchmarking, I reach for benchmark-ips. When macro-benchmarking (and especially in this case, with a test suite), I usually reach first for the simplest tool available: the unix utility time! We’re going to run the tests 10 times, and then divide the total time by 10.

$ time for i in {1..10}; do bundle exec rake; done

...

real	15m59.384s
user	11m39.100s
sys	1m15.767s

When using time, we’re usually only going to pay attention the user statistic. real gives the actual total time (as if you had used a stopwatch), sys gives the time spent in the kernel (in a test run, this would be things like shelling out to I/O), and user will be the closest approximation to time actually spent running Ruby. You’ll notice that user and sys don’t add up to real - the difference is time spent waiting on the CPU while other operations (like running my web browser, etc) block.

With stock minitest, the whole thing takes 11 minutes and 39 seconds, for an average of 69.9 seconds per run. Now, let’s alter the Gemfile to point to a modified version (with shuffle on the line in question) of minitest on my local machine:

gem 'minitest', require: false, path: '../minitest'

To make sure the test is 100% fair, I only make the change to my local version after I check out minitest to the same version that Rubygems.org is running (5.8.1).

Even so-called performance experts mess this shit up sometimes.

The result? 11 minutes 56 seconds. Longer than the original test! We know my code is faster in micro, but the macro benchmark told me that it actually takes longer. A lot of things can cause this (the most likely being other stuff running on my machine), but what’s clear is this - my little patch doesn’t seem to be making a big difference to the big picture of someone’s test suite. While making this change would, in theory, speed up someone’s suite, in reality, the impact is so minuscule that it didn’t really matter.

So, while a benchmark told me one thing - X is 10x faster than Y! - a higher-level benchmark told me another (make your change and this thing didn’t really matter.) Not only does this show the value of profiling (which would have told me before that the sorting didn’t take much of the total time) but also how microbenchmarks and relative comparisons can mislead.

Performance measurement is a critical skill. Anywhere along the way, I could have been mislead by a single number or a rogue measurement. By applying a scientific, empirical approach, I was able to put my benchmark in context of a larger program.

Premature optimization is ignoring these lessons and optimizing “when we feel like it”, or optimizing constantly all the time. Hopefully I’ve convinced you: it’s a guaranteed waste of time.

Repeat after me: I will not optimize anything in my application until my metrics tell me so.

Why Your Rails App is Slow: Lessons Learned from 3000+ Hours of Teaching

Mon, 17 Jun 2019 07:00:00 +0000

“What I do have is a particular set of skills, a set of skills which makes me a nightmare for slow Rails applications like you.”

For the last 4 years, I’ve been working on making Rails applications faster and more scalable. I teach workshops, I sell a course, and I consult. If you do anything for a long period of time, you start to see patterns. I’ve noticed four different factors that prevent software organizations from improving the performance of their Rails applications, and I’d like to share them here.

Performance becomes a luxury good, especially when no one is watching

Often times at my Rails Performance Workshop, I discover that an attendee simply has no visiblity into what their application is doing in production - they either don’t understand their dashboards, they don’t have them, or they’re not allowed to even access them (“DevOps team only”).

Performance metrics are often just not tracked. No one is aware if the app is over or underscaled, no one knows if the app is “slow” or “fast”. Is it any wonder, then, that no one spends any time working on it?

Performance is rarely the first priority of any organization, and often gets “trickled down” hours and resources.

Some of this is actually a good thing. There’s a reason that the classic programming mantra of “make it work, make it clean, make it fast” is in that order and not the opposite way around. People pay for software that does stuff. If it does that stuff quickly and in a pleasantly performant way, then that’s great, but it’s not always required (especially if the organization is first to market in their space and customers have no other options).

Often, my consulting clients are at a point in their organization where they’re no longer scraping by on ramen and cheeto dust, but have a solid business that’s expanding (slowly or quickly). They’ve finally gotten their heads above water and they’re ready to start thriving, not just surviving. People don’t come to me when they’re still trying to achieve product market fit unless things have become untenable, and then that’s more of a rescue job.

When the time comes to set the budget on performance instead of feature velocity.

This is a natural and correct progression. It also means organizations accumulate performance debt during that initial period of building and obtaining product-market fit. I think there may be some that believe that all kinds of technical debt are some kind of sinful black stain on any organization, and that if you just Coded The Right Way or were a Software Craftsperson™ this would not have happened. I think that’s probably wrong. Before achieving ramen profitability, businesses must take out technical debt as a kind of financing of their own product development runway. This will happen regardless of one’s coding techniques or knowledge level.

However, performance is not a luxury good. It isn’t something that can simply be ignored until one’s organization has a spare four or five figures in the couch cushions. Like technical debt, there is a point when feature work grinds to a halt because the organization is too busy maintaining the performance debt that has accrued. Requests are timing out. Customers are complaining about slow the app feels and switching to competitors. You’re scared to check the AWS bill.

Ideally, organizations monitor and sensibly take out performance debt when required, and understand the full extent of the work that must be done in the future.

To do this sort of “sensible debt accrual”, you need performance monitoring/metrics and you need to understand how to present numbers to management. I find that while most people know subscribe to a performance monitoring service, such as New Relic, Skylight or Scout, they often have no idea how to read it and extract useful insights from it, making it a very expensive average latency monitor. Being able to actually use your APM is a critical performance skill that I cover in great detail in my workshops and course.

If you can’t draw insights from this, you’re just throwing cash out the door.

Monitoring these metrics allows you to assess where you’re at and to figure out what parts of the application have accrued performance debt. It also helps you to make decisions on the “cost/benefit” of future work.

It also means you need to be able to “speak manager” or “speak business”. The business case for adding more features is obvious to the non-technical side of your organization. There is a great business case for performance, fortunately, both from the side of the customer and from the cost side as well - reducing average latency by 50% means you can spend 50% less on your application’s servers thanks to queueing theory and something called Little’s Law.

At my workshops, I spend a lot of time simply discussing terminology, like request queueing, latency, throughput, tracing and profiling. Giving people the vocabulary they need to understand the tools out there seems to be half the battle of getting everyone comfortable reading their own metrics.

Complex apps and complex problems, with little training

This leads me to the second cause of performance problems in software - a simple lack of knowledge. We can’t optimize what we don’t understand and we can’t fix what we can’t see.

I wrote the Complete Guide to Rails Performance simply because there was so much information about this topic that had simply never been compiled before into one place.

“What’s request queueing?”

This shows itself most when scaling for throughput. Most organizations simply aren’t tracking critical scaling metrics or even know what they are, often because they believe the platform-as-a-service that they’re using should “take care of this” for them. By the time I’ve been called in, they’re spending thousands of dollars a month more than they need to, and could have fixed this months or even years ago with some simple autoscaling policies and a bit of organizational knowledge around scaling. Or, the flipside is happening and they’re massive under-scaled, with 25-50% of their total request latency being just time spent queueing for resources.

Performance work is not rocket science. However, unlike a lot of other areas in software¹(The only other area in software that requires an even wider base of knowledge is security. Consider Rowhammer - basically an electrical engineering exploit in very particular configurations of DRAM.)¹ The only other area in software that requires an even wider base of knowledge is security. Consider Rowhammer - basically an electrical engineering exploit in very particular configurations of DRAM., it can require an extremely broad base of knowledge. When your customer says the site “feels slow”, the problem can quite reasonably be almost anywhere between the pixels on the user’s screen (say, an issue with the customer’s client machine) and the electrons running through the silicon on your cloud service provider (for example, a mitigation for a recent Intel security issue puts your servers above capacity). Feature work and even to a large extent refactoring work generally only requires knowledge of the language and frameworks in use. Performance work often needs esoteric knowledge from other fields (such as queueing theory) in addition to highly in-depth knowledge in your frameworks and language.²(I wrote a 3000+ word blog about the critical performance differences between English-language synonyms .present? and .exists? in Rails, for example, but my Rails performance course spends the majority of the time talking about things which are not Ruby-specific.)² I wrote a 3000+ word blog about the critical performance differences between English-language synonyms .present? and .exists? in Rails, for example, but my Rails performance course spends the majority of the time talking about things which are not Ruby-specific.

Looking at a flamegraph of a Rails app for the first time often leads to this reaction.

This depth of knowledge simply isn’t present in many organizations, especially those who place sprint velocity before the development of engineering capacity and skills in the organization.

The workshops I’ve been doing have really allowed me to go in deep on complex problems and help people deal with the “wrinkles” introduced by their application. Getting to look over people’s shoulders while they experience an error or something I hadn’t anticipated has been very rewarding, both for them and for me as an educator.

Also, during those workshops, I don’t emphasize “pre-baked” problem/solutions, but instead have the attendees bring their real world applications, and we immediately try to apply what we’ve learned on their actual apps right then and there. I don’t want anyone to go home and run into a problem caused by the complexity of their app - rather, I’d like that to happen while we’re both in the same room!

Boiling frogs - even when tracked, performance slips without fix

In the Slack channel for the Complete Guide to Rails Performance, we’ve had a few conversations about managing performance work in the software organization.

Walking into the office on Monday like

An organizational culture that always places completeness over quality inevitably runs into issues. Often when I get new clients, they’re experiencing not just performance issues but have problems with all the various dimensions of software quality: low correctness (an excess of bugs and lack of test coverage), high complexity (“technical debt”, spaghetti organization), and a poor deployment pipeline (broken builds, janky deploys). These aspects of software quality tend to either all be good or all be bad. Project management can (and often should) sacrifice quality for a period of time to prioritize completeness and features, but when it’s done pathologically, it inevitably leads to ruin.

I find that the lack of software quality culture often arises because no one is measuring it³(I actually really don’t vibe with the ‘software craftsperson’ aesthetic that people like Uncle Bob try to push. Quality is great but it isn’t everything. It’s possible to turn this into navelgazing, and building ivory towers.)³ I actually really don’t vibe with the ‘software craftsperson’ aesthetic that people like Uncle Bob try to push. Quality is great but it isn’t everything. It’s possible to turn this into navelgazing, and building ivory towers.. Feature velocity is measured, or at least vaguely tracked, with things like pull request counts, sprint points, or user stories. We shipped 5 stories last week, so management expects us to ship 5 this week.

Fortunately, many software quality measures are actually very easy to track. How many bugs were reported or experienced by customers last week? How much downtime did we have? How many deploys were there? Are these numbers rising or falling?

In terms of performance, most organizations would benefit from setting simple thresholds that, if exceeded, move performance work into the “bug fixing” pipeline that the organization employs. For example, an organization can commit to a maximum 95th percentile latency of 1 second. If a transaction⁴(In New Relic parlance - a single controller action is a ‘transaction’.)⁴ In New Relic parlance - a single controller action is a ‘transaction’. exceeds that threshold, a new bug is recorded.

For organizations that want to improve the customer’s experience and perceived performance of the application, other budgets may be necessary. For example, a first-page-load time of 5 seconds. This page load target has implications that flow down throughout the stack, as one simply cannot ship 10 megabytes of JavaScript and also have a page load in 5 seconds⁵(In fact, I would estimate that to keep page load times below 5 seconds on the average connection and hardware, you can probably ship only a few hundred KB)⁵ In fact, I would estimate that to keep page load times below 5 seconds on the average connection and hardware, you can probably ship only a few hundred KB.

Software engineers are often poor communicators, and they very often fail to communicate to other parts of the organization that prioritizing feature velocity at all costs is not sustainable.

Think of it this way: how do you think the project managers in your organization would answer the following questions?

Bezos showering in your AWS bill

Is your tolerance for the slowness of our application infinite? (i.e. can the app just beachball for all customers all the time?)
Do you have infinite money to spend on our EC2 instance bills?

If the answer to either of those questions is “no”, then it is your job as a software developer to find and make explicit those tolerances. They will be different for every organization. These performance requirements can be easily translated into automated alerts and thresholds.

You just have to have the conversation beforehand. The difference between “Hey boss - we’ve been shipping 12 points a week for the last 8 weeks and now we can’t ship anything for 6 weeks because we need to write tests and make the homepage load time somewhat bearable” and “we exceeded the limit for page load time that we all agreed upon 6 months ago, and we’ll need to reduce velocity for a while to compensate” is miles apart.

As a result of seeing this pattern often enough, I’ve changed how I phrase my consulting deliverables, as I now realize I need to provide ammunition for the engineers when bringing back my recommendations to the “business side”.

It’s not Ruby, and it isn’t (really) Rails

LEAVE MATZ ALONE

And, finally, here’s what isn’t the reason why your web application is slow: your framework or language choice. Once 90th percentile latency is lower than 500 milliseconds and median latency is below 100 milliseconds, most web application backends are no longer the bottleneck in their customer’s experience (if they ever were to begin with, which, in the age of 10 megabyte JavaScript bundles, they are usually not).

It’s 2017 and web applications don’t return flat HTML files anymore⁵(CNN.com took 5MB of resources and 112 requests to render for me, today. R.I.P. the old light web.)⁵ CNN.com took 5MB of resources and 112 requests to render for me, today. R.I.P. the old light web.. Websites are gargantuan, with JavaScript bundles stretching into the size of megabytes and stylesheets that couldn’t fit in ten Apollo Guidance Computers. So how much of a difference does a web application which responds in 1 millisecond or less make in this environment?

Vanishingly little. Nowadays, the average webpage takes 5 seconds to render. Some JavaScript single-page-applications can take 12 seconds or more on initial render.

Server response times simply make up a minority part of the actual user experience of loading and interacting with a webpage - cutting 99 milliseconds off the server response time just doesn’t make a difference.

Not to mention: if Ruby on Rails, frequently maligned “as too slow” or “can’t scale”, can run several of the top 1000 websites in the world by traffic, including that little fly-by-night outfit called GitHub, then it’s a fine choice for whatever your application is. Rails is just an example here - there are many comparable frameworks in comparable languages that you could substitute like Python and Django. There are some web applications for which 100 milliseconds of latency is an unacceptable eon (advertising is the most common case), but for the vast majority of us delivering HTML or JSON to a client, that’s zippy quick.

Whither Rails Today?

I’m doing a Rails Perf workshop tour this summer in the US of A.

One of the questions I ask in my post-workshop survey is “How do you feel writing Ruby on Rails? Would you like to keep doing it?”.

The answers I get back are always astoundingly positive. For all the FUD on the web at large, people writing Ruby are incredibly happy doing it. And that’s what keeps me writing and teaching: as long as people feel that performance concerns are keeping them from enjoying Ruby or choosing it as their tech stack, I’ll keep doing what I do.

3 ActiveRecord Mistakes That Slow Down Rails Apps: Count, Where and Present

Thu, 10 Jan 2019 07:00:00 +0000

“When does ActiveRecord execute queries? No one knows!”

ActiveRecord is great. Really, it is. But it’s an abstraction, intended to insulate you from the actual SQL queries being run on your database. And, if you don’t understand how ActiveRecord works, you may be causing SQL queries to run that you didn’t intend to.

Unfortunately, the performance costs of many features of ActiveRecord means we can’t afford to ignore unnecessary usage or treat our ORM as just an implementation detail. We need to understand exactly what queries are being run on our performance-sensitive endpoints. Freedom isn’t free, and neither is ActiveRecord.

One particular case of ActiveRecord misuse that I find is common amongst my clients is that ActiveRecord is executing SQL queries that aren’t really necessary. Most of my clients are completely unaware that this is even happening.

Unnecessary SQL is a common cause of overly slow controller actions, especially when the unnecessary query appears in a partial which is rendered for every element in a collection. This is common in search actions or index actions. This is one of the most common problems I encounter in my performance consulting. It’s a problem in nearly every app I’ve ever worked on.

One way to eliminate unnecessary queries is to poke our heads into ActiveRecord and understand its internals, and know exactly how certain methods are implemented. Today, we’re going to look at the implementation and usage of three methods which cause lots of unnecessary queries in Rails applications: count, where and present?.

How Do I Know if a Query is Unnecessary?

I have a rule of thumb to judge whether or not any particular SQL query is unnecessary. Ideally, a Rails controller action should execute one SQL query per table. If you’re seeing more than one SQL query per table, you can usually find a way to reduce that to one or two queries. If you’ve got more than a half-dozen or so queries on a single table, you almost definitely have unnecessary queries. ¹(Please don’t email or tweet with me with ‘Well ackshually…’ on this one. It’s a guideline, not a rule, and I understand there are circumstances where more than one query per table is a good idea.)¹ Please don’t email or tweet with me with ‘Well ackshually…’ on this one. It’s a guideline, not a rule, and I understand there are circumstances where more than one query per table is a good idea.

The number of SQL queries per table can be easily seen on NewRelic, for example, if you have that installed.

I keep an eyewash station next to my desk for really bad N+1s

Another rule of thumb is that most queries should execute during the first half of a controller action’s response, and almost never during partials. Queries executed during partials are usually unintentional, and are often N+1s. These are easy to spot during a controller’s execution if you just read the logs in development mode. For example, if you see this:

User Load (0.6ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT 1  [["id", 2]]
Rendered posts/_post.html.erb (23.2ms)
User Load (0.3ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT 1  [["id", 3]]
Rendered posts/_post.html.erb (15.1ms)

… you have an N+1 in this partial.

Usually, when a query is executed halfway through a controller action (somewhere deep in a partial, for example) it means that you haven’t preloaded the data that you needed.

So, let’s look specifically at the count, where and present? methods, and why they cause unnecessary SQL queries.

.count executes a COUNT every time

I see this one at almost every company I contract for. It seems to be little-known that calling count on an ActiveRecord relation will always try to execute a SQL query, every time. This is inappropriate in most scenarios, but, in general, only use count if you want to always execute a SQL COUNT right now.

“How many queries do we want per table?”

The most common cause of unnecessary count queries is when you count an association you will use later in the view (or have already used):

# _messages.html.erb
# Assume @messages = user.messages.unread, or something like that

<h2>Unread Messages: <%= @messages.count %></h2>

<% @messages.each do |message| %>
blah blah blah
<% end %>

This executes 2 queries, a COUNT and a SELECT. The COUNT is executed by @messages.count, and @messages.each executes a SELECT to load all the messages. Changing the order of the code in the partial and changing count to size eliminates the COUNT query completely and keeps the SELECT:

<% @messages.each do |message| %>
blah blah blah
<% end %>

<h2>Unread Messages: <%= @messages.size %></h2>

Why is this the case? We need not look any further than the actual method definition of size on ActiveRecord::Relation:

# File activerecord/lib/active_record/relation.rb, line 210
def size
  loaded? ? @records.length : count(:all)
end

If the relation is loaded (that is, the query that the relation describes has been executed and we have stored the result), we call length on the already loaded record array. That’s just a simple Ruby method on Array. If the ActiveRecord::Relation isn’t loaded, we trigger a COUNT query.

On the other hand, here’s how count is implemented (in ActiveRecord::Calculations):

def count(column_name = nil)
  if block_given?
    # ...
    return super()
  end

  calculate(:count, column_name)
end

And, of course, the implementation of calculate doesn’t memoize or cache anything, and executes a SQL calculation every time it is called.

Simply changing count to size in our original example would have still triggered a COUNT. The record’s wouldn’t be loaded? when size was called, so ActiveRecord will still attempt a COUNT. Moving the method after the records are loaded eliminates the query. Now, moving our header to the end of the partial doesn’t really make any logical sense. Instead, we can use the load method.

<h2>Unread Messages: <%= @messages.load.size %></h2>

<% @messages.each do |message| %>
blah blah blah
<% end %>

load just causes all of the records described by @messages to load immediately, rather than lazily. It returns the ActiveRecord::Relation, not the records. So, when size is called, the records are loaded? and a query is avoided. Voilà.

What if, in that example, we used messages.load.count? We’d still trigger a COUNT query!

When doesn’t count trigger a query? Only if the result has been cached by ActiveRecord::QueryCache.²(I have some Opinions on the use of QueryCache, but that’s a post for another day.)² I have some Opinions on the use of QueryCache, but that’s a post for another day. This could occur by trying to run the same SQL query twice:

<h2>Unread Messages: <%= @messages.count %></h2>

... lots of other view code, then later:

<h2>Unread Messages: <%= @messages.count %></h2>

Every time you use count when you could have used size

In my opinion, most Rails developers should be using size in most of the places that they use count. I’m not sure why everyone seems to write count instead of size. size uses count where it is appropriate, and it doesn’t when the records are already loaded. I think it’s because when you’re writing an ActiveRecord relation, you’re in the “SQL” mindset. You think: “This is SQL, I should write count because I want a COUNT!”

So, when do you actually want to use count? Use it when you won’t actually ever be loading the full association that you’re counting. For example, take this view on Rubygems.org, which displays a single gem:

In the “versions” list, the view does a count to get the total number of releases (versions) of this gem.

Here’s the actual code:

<% if show_all_versions_link?(@rubygem) %>
  <%= link_to t('.show_all_versions', :count => @rubygem.versions.count), rubygem_versions_url(@rubygem), :class => "gem__see-all-versions t-link--gray t-link--has-arrow" %>
<% end %>

The thing is, this view never loads all of the Rubygem’s versions. It only loads five of the most recent ones, in order to show that versions list.

So, a count makes perfect sense here. Even though size would be logically equivalent (it would just execute a COUNT as well because @versions is not loaded?), it states the intent of the code in a clear way.

My advice is to grep through your app/views directory for count calls and make sure that they actually make sense. If you’re not 100% sure that you really need a real SQL COUNT right then and there, switch it to size. Worst case, ActiveRecord will still execute a COUNT if the association isn’t loaded. If you’re going to use the association later in the view, change it to load.size.

.where means filtering is done by the database

What’s the problem with this code (let’s say its _post.html.erb)

<% @posts.each do |post| %>
  <%= post.content %>
  <%= render partial: :comment, collection: post.active_comments %>
<% end %>

and in Post.rb:

class Post < ActiveRecord::Base
  def active_comments
    comments.where(soft_deleted: false)
  end
end

If you said, “this causes a SQL query to be executed on every rendering of the post partial”, you’re correct! where always causes a query. I didn’t even bother to write out the controller code, because it doesn’t matter. You can’t use includes or other preloading methods to stop this query. where will always try to execute a query!

This also happens when you call scopes on associations. Imagine instead our Comment model looked like this:

class Comment < ActiveRecord::Base
  belongs_to :post

  scope :active, -> { where(soft_deleted: false) }
end

Allow me to sum this up with two rules: Don’t call scopes on associations when you’re rendering collections and don’t put query methods, like where, in instance methods of an ActiveRecord::Base class.

Calling scopes on associations means we cannot preload the result. In the example above, we can preload the comments on a post, but we can’t preload the active comments on a post, so we have to go back to the database and execute new queries for every element in the collection.

This isn’t a problem when you only do it once, and not on every element of a collection (like every post, as above). Feel free to use scopes galore in those situations - for example, if this was a PostsController#show action that only displayed one post and its associated comments. But in collections, scopes on associations cause N+1s, every time.

The best way I’ve found to fix this particular problem is to create a new association. Justin Weiss, of “Practicing Rails”, taught me this in this blog post about preloading Rails scopes. The idea is that you create a new association, which you can preload:

class Post
  has_many :comments
  has_many :active_comments, -> { active }, class_name: "Comment"
end

class Comment
  belongs_to :post
  scope :active, -> { where(soft_deleted: false) }
end

class PostsController
  def index
    @posts = Post.includes(:active_comments)
  end
end

The view is unchanged, but now executes just 2 SQL queries, one on the Posts table and one on the Comments table. Nice!

<% @posts.each do |post| %>
  <%= post.content %>
  <%= render partial: :comment, collection: post.active_comments %>
<% end %>

The second rule of thumb I mentioned, don’t put query methods, like where, in instance methods of an ActiveRecord::Base class, may seem less obvious. Here’s an example:

class Post < ActiveRecord::Base
  belongs_to :post

  def latest_comment
    comments.order('published_at desc').first
  end

What happens if the view looks like this?

<% @posts.each do |post| %>
  <%= post.content %>
  <%= render post.latest_comment %>
<% end %>

That’s a SQL query on every post, regardless of what you preloaded. In my experience, every instance method on an ActiveRecord::Base class will eventually get called inside a collection. Someone adds a new feature and isn’t paying attention. Maybe it’s by a different developer than the one who wrote the method originally, and they didn’t fully read the implementation. Ta-da, now you’ve got an N+1. The example I gave could be rewritten as an association, like I described earlier. That can still cause an N+1, but at least it can be fixed easily with the correct preloading.

Which ActiveRecord methods should we avoid inside of our ActiveRecord model instance methods? Generally, it’s pretty much everything in the QueryMethods, FinderMethods, and Calculations. Any of these methods will usually try to run a SQL query, and are resistant to preloading. where is the most frequent offender, however.

any?, exists? and present?

Rails programmers have been struck by a major affliction - they’re adding a particular predicate method to just about every variable in their applications. present? has spread across Rails codebases faster than the plague in 13th century Europe. The vast majority of the time, the predicate adds nothing but verbosity, and really, all the author needed was a truthy/falsey check, which they could have done by just writing the variable name.

Here’s an example from CodeTriage, a free and open-source Rails application written by my friend Richard Schneeman:

class DocComment < ActiveRecord::Base
  belongs_to :doc_method, counter_cache: true

  # ... things removed for clarity...

  def doc_method?
    doc_method_id.present?
  end
end

What is present? doing here? One, it transforms the value of doc_method_id from either nil or an Integer into true or false. Some people have Strong Opinions about whether predicates should return true/false or can return truthy/falsey. I don’t. But adding present? also does something else, and we have to look at the implementation to figure out what:

class Object
  def present?
    !blank?
  end
end

blank? is a more complicated question than “is this object truthy or falsey”. Empty arrays and hashes are truthy, but blank, and empty strings are also blank?. In the example above from CodeTriage, however, the only things that doc_method_id will ever be is nil or Integer, meaning present? is logically equivalent to !!:

def doc_method?
  !!doc_method_id
  # same as doc_method_id.present?
end

Using present? in cases like this is the wrong tool for the job. If you don’t care about “emptiness” in the value you’re calling the predicate on (i.e. the value cannot be [] or {}), use the simpler (and much faster) language features available to you. I sometimes see people even do this on values which are already boolean, which means you’re just adding verbosity and making me wonder if there’s some weird edge cases I’m not seeing.

Alright, that’s my style gripe. I understand that you may not agree. present? makes more sense when dealing with strings, which can frequently be empty ("").

Where people get into trouble is calling predicates, such as present?, on ActiveRecord::Relation objects. Let’s say you need to know if an ActiveRecord::Relation has any records. You can use the English-language synonyms any?/present?/exists? or their negations none?/blank?/empty?. Surely it doesn’t matter which method you choose, right? Just pick the one that sounds the most natural when read aloud? Nope.

What SQL queries do you think the following code will execute? Assume @comments is an ActiveRecord::Relation.

- if @comments.any?
  h2 Comments on this Post
  - @comments.each do |comment|

The answer is two. One will be an existence check, triggered by @comments.any? (SELECT 1 AS one FROM ... LIMIT 1), then the @comments.each line will trigger a loading of the entire relation (SELECT "comments".* FROM "comments" WHERE ...).

What about this?

- unless @comments.load.empty?
  h2 Comments on this Post
  - @comments.each do |comment|

This one only executes one query - @comments.load loads the entire relation right away with SELECT "comments".* FROM "comments" WHERE ....

And this one?

- if @comments.exists?
  This post has
  = @comments.size
  comments
- if @comments.exists?
  h2 Comments on this Post
  - @comments.each do |comment|

Four! exists? doesn’t memoize itself and it doesn’t load the relation. exists? here triggers a SELECT 1 ..., .size triggers a COUNT because the relation hasn’t been loaded yet, and then the next exists? triggers ANOTHER SELECT 1 ... and finally @comments loads the entire relation! Yay! Isn’t this fun? You could reduce this down to just 1 query with the following:

- if @comments.load.any?
  This post has
  = @comments.size
  comments
- if @comments.any?
  h2 Comments on this Post
  - @comments.each do |comment|

And it just gets better - this behavior changes depending if you’re on Rails 4.2, Rails 5.0 or Rails 5.1+.

Here’s how it works in Rails 5.1+:

method	SQL generated	memoized?	implementation	Runs query if `loaded?`
present?	SELECT “users”.* FROM “users”	yes (`load`)	Object (!blank?)	no
blank?	SELECT “users”.* FROM “users”	yes (`load`)	`load`; `blank?`	no
any?	SELECT 1 AS one FROM “users” LIMIT 1	no unless `loaded`	`!empty?`	no
empty?	SELECT 1 AS one FROM “users” LIMIT 1	no unless `loaded`	`exists?` if !`loaded?`	no
none?	SELECT 1 AS one FROM “users” LIMIT 1	no unless `loaded`	`empty?`	no
exists?	SELECT 1 AS one FROM “users” LIMIT 1	no	ActiveRecord::Calculations	yes

Here’s how it works in Rails 5.0:

method	SQL generated	memoized?	implementation	Runs query if `loaded?`
present?	SELECT “users”.* FROM “users”	yes (`load`)	Object (!blank?)	no
blank?	SELECT “users”.* FROM “users”	yes (`load`)	`load`; `blank?`	no
any?	SELECT COUNT(*) FROM “users”	no unless `loaded`	`!empty?`	no
empty?	SELECT COUNT(*) FROM “users”	no unless `loaded`	count(:all) > 0	no
none?	SELECT COUNT(*) FROM “users”	no unless `loaded`	`empty?`	no
exists?	SELECT 1 AS one FROM “users” LIMIT 1	no	ActiveRecord::Calculations	yes

Here’s how it works in Rails 4.2:

method	SQL generated	memoized?	implementation	Runs query if `loaded?`
present?	SELECT “users”.* FROM “users”	yes	Object (!blank?)	no
blank?	SELECT “users”.* FROM “users”	yes	to_a.blank?	no
any?	SELECT COUNT(*) FROM “users”	no unless `loaded`	`!empty?`	no
empty?	SELECT COUNT(*) FROM “users”	no unless `loaded`	count(:all) > 0	no
none?	SELECT “users”.* FROM “users”	yes (`load` called)	Array	no
exists?	SELECT 1 AS one FROM “users” LIMIT 1	no	ActiveRecord::Calculations	yes

any?, empty? and none? remind me of the implementation of size - if the records are loaded? do a simple method call on a basic Array, if they’re not loaded, always run a SQL query. exists? has no caching or memoization built in, just like other ActiveRecord::Calculations. This means that exists?, which is another method people like to write in these circumstances, is actually much worse than present? in some cases!

These six predicate methods, which are English-language synonyms all asking the same question, have completely different implementations and performance implications, and these consequences depend on which version of Rails you are using. So, let me distill all of the above into some concrete advice:

present? and blank? should not be used if the ActiveRecord::Relation will never be used in its entirety after you call present? or blank?. For example, @my_relation.present?; @my_relation.first(3).each.
any?, none? and empty? should probably be replaced with present? or blank? unless you will only take a section of the ActiveRecord::Relation using first or last. They will generate an extra existence SQL check if you’re just going to use the entire relation if it exists. In essence, change @users.any?; @users.each... to @users.present?; @users.each... or @users.load.any?; @users.each..., but @users.any?; @users.first(3).each is fine.
exists? is a lot like count - it is never memoized, and always executes a SQL query. Most people probably do not actually want this behavior, and would be better off using present? or blank?

Conclusion

As your app grows in size and complexity, unnecessary SQL can become a real drag on your application’s performance. Each SQL query involves a round-trip back to the database, which entails, usually, at least a millisecond, and sometimes much more for complex WHERE clauses. Even if one extra exists? check isn’t a big deal, if it suddenly happens in every row of a table or a partial in a collection, you’ve got a big problem!

ActiveRecord is a powerful abstraction, but since database access will never be “free”, we need to be aware of how ActiveRecord works internally so that we can avoid database access in unnecessary cases.

App Checklist

Look for uses of present?, none?, any?, blank? and empty? on objects which may be ActiveRecord::Relations. Are you just going to load the entire array later if the relation is present? If so, add load to the call (e.g. @my_relation.load.any?)
Be careful with your use of exists? - it ALWAYS executes a SQL query. Only use it in cases where that is appropriate - otherwise use present? or any other the other methods which use empty?
Be extremely careful using where in instance methods on ActiveRecord objects - they break preloading and often cause N+1s when used in rendering collections.
count always executes a SQL query - audit its use in your codebase, and determine if a size check would be more appropriate.

The Complete Guide to Rails Performance, Version 2

Fri, 01 Jun 2018 07:00:00 +0000

Today, the Complete Guide to Rails Performance has been updated to version 2.0. You can purchase it here.

All existing purchasers have had their copies updated on Gumroad. When I started this project, I always believed that a digital course should be better than a typical paperback programming book. That’s why I don’t include any DRM or proprietary video codecs. That’s why I think, like most software, updates should be free.

“Version 2.0” isn’t quite as drastic a change as a software v2.0, though. The world of Rails performance has actually changed very little since I wrote the course 2 years ago. The apps I consult on still have many of the same problems. The V2 update reflects this: I have revised the content for clarity, and updated a few places to reflect changes in Ruby 2.5 and Rails 5.2, but it is mostly still the same. I have also added four lessons: memory fragmentation, application server config, GC tuning, and PGBouncer config. These lessons were added based on new problems and thinking I’ve had since the course was released. Web-Scale Package purchasers will also get a new interview with Noah Gibbs of Appfolio next week.

So, what does it mean that not much has changed in the Rails performance world?

This tweet put me in an introspective mood this morning:

It is profoundly sad how Rails has institutionalized a "nobody cares" attitude toward performance. https://t.co/UhzvxyLjuz
— Jeff Atwood (@codinghorror) June 1, 2018

To summarize, Jeff’s cofounder, Sam Saffron (who I interviewed for the CGRP), wrote a great, in-depth blog post about memory use in ActiveRecord. In short, Sam finds that ActiveRecord creates excessive amounts of objects, even when doing simple and supposedly “optimized” work. Sam posted a proof-of-concept patch which improves this quite a bit.

Jeff’s tweet diminishes the work of many Rails contributors. Aaron Patterson has spent the last two years working on Rails performance and a compacting garbage collector. Richard Schneeman has improved Sprockets’ performance a great deal. Sam Saffron himself has contributed over a dozen performance improvements to Rails, which, as far I can tell, have all been accepted. I know also that Andrew White, Eileen Uchitelle, and Rafael Franca are all Rails core members that care deeply about performance (probably because all of them have day-jobs running large Rails applications!). So any idea that Rails’ contributors or core members “don’t care” about performance is laughingly misguided, and is an opinion that can only really be held by someone outside the community. The way Jeff tried to turn it around in the replies into a “hot take” that people should “get angry” and “punk rock” about the “status quo” just made it more obvious.

It’s pretty easy to take potshots at a mature framework like Ruby on Rails. It has almost 13 years of history behind it. There’s going to be cruft, baggage, and outdated decisions baked in. That’s what happens. But there’s also tremendous productivity, something gained from the thousands of contributors who have all contributed their “lessons learned” back to the framework. But if you forget about that history, it’s easy to craft a benchmark to make it look like that history has overtaken it’s usefulness in the present.

This is the gap I’ve tried to bridge in my writing and in publishing The Complete Guide to Rails Performance. I believe that performance problems in Rails are pedagogical, not technical. It’s not because we don’t have enough people working on performance (though it helps!). It’s not because we don’t value it as a community (how many times do I have to cite all of the top 10,000 websites that run Rails at speed?). It’s because Rails (and Ruby) optimizes for programmer happiness, and that means we provide sharp tools which are easy to cut yourself on. Rather than throw the tools out, I think we need to teach people to use them safely.

ActiveRecord is probably the best example of what I’m talking about. It’s an extremely productive tool. It works very well for 80% of web-app use-cases. But every year, someone wants to throw it out and thinks that some other Rubygem or pattern (e.g. DataMapper) will save them. It’s so easy to craft a line of code with ActiveRecord that will slow your application to a crawl if you’re not thinking through through the consequences, as anyone who has written User.all.each can tell you.

There is no One True Pattern or One True Framework. But there is a Thing Which Works For Most People. And if you end up being one of the 20% for whom it doesn’t work so well, or the tool’s productivity preference means that it’s easier to make performance mistakes, I don’t think that’s the tool or framework’s fault.

In this way, I think publishing the Complete Guide to Rails Performance was placing my faith in the developer community of Rails. If I didn’t think that people could make their Rails apps faster through knowledge and skills, and instead they had to wait until the framework or the language itself got faster, I would have gone to work at Github or Shopify and made a bunch of patches to Rails and Ruby. I might have started an alternative, “lightweight” framework or ORM that prioritized performance over usefulness and productivity. Instead, I think that teaching Rails developers how to find and fix performance problems will make a bigger dent in the average Rails app’s response time than improving the language or framework’s performance by even 2-3x, or by removing “dangerous” features.

As I think we’ve slowly discovered over the course of trying to make Ruby 3x faster, there is no “waste” or “bloat” that can be cut out of a framework or language without cost that suddenly makes the whole thing faster. It’s sort of like how politicians always promise to “cut waste in government spending”, but no-one can ever tell you exactly where or how which programs will be cut. Everything was implemented for a reason. There is no magic wand or amount of man-hours that can be waved at these problems. I’ve discovered this in my consulting and writing as well. I wish it was that easy. But it isn’t.

However, far from Jeff’s doomsday attitude, I believe that the macro picture for Ruby and Rails performance looks good, as it always had. Ruby 2.0 to 2.5 made a number of incremental performance improvements, particularly in garbage collection. I feel like the community has become more mature and performance-savvy over the last few years too. We’re waking up the mainstream Rails developer to things like jemalloc and teaching them how to use ActiveRecord and avoid performance issues.

The technical future of Ruby looks strong, too. Ruby 2.6 will contain a JIT compiler. How cool is that? TruffleRuby has made great progress to becoming useable enough to run a Rails application. JRuby continues to truck along with more performance improvements and compatibility fixes all the time. The technical future of the language hardly looks dim - in fact, I think it’s much brighter than it was in 2011, when I got started in Ruby and Rails.

I’ll continue to do my part for the Rails performance community by publishing and writing, to improve the technical skills and capacity of the average Rails developer so that they can make their apps faster. Here’s to you, developers!

A New Ruby Application Server: NGINX Unit

Wed, 28 Mar 2018 07:00:00 +0000

There’s a new application server on the block for Rubyists - NGINX Unit. As you could probably guess by the name, it’s a project of NGINX Inc., the for-profit open-source company that owns the NGINX web server. In fall of 2017, they announced the NGINX Unit project. It’s essentially an application server designed to replace all of the various application servers used with NGINX. In Ruby’s case, that’s Puma, Unicorn, and Passenger.¹(For a far more in-depth comparison of these application servers, read my article about configuring Puma, Passenger and Unicorn)¹ For a far more in-depth comparison of these application servers, read my article about configuring Puma, Passenger and Unicorn NGINX Unit also runs Python, Go, PHP and Perl.

The overarching idea seems to be to make microservice administration a lot easier. One NGINX Unit process can run any number of applications running any number of languages - for example, one NGINX Unit server can manage a half-dozen different Ruby applications, each running a different version of the Ruby runtime. Or you can run a Ruby application and a Python application side-by-side. The combinations are only limited by your system resources.

Unfortunately, the “microservice” space is quite prone to buzzword-laden marketing pages.²(I really don’t like when software projects advertise themselves as “modern”. It’s like “subtweeting” all pre-existing software projects in this problem space and saying they’re all old and busted, and this is the New Way To Do Things. Why it’s better than the “old busted ways” is never explicitly stated. This kind of marketing preys on software developer’s fear of becoming obsolete in their skillset, rather than making any substantive point.)² I really don’t like when software projects advertise themselves as “modern”. It’s like “subtweeting” all pre-existing software projects in this problem space and saying they’re all old and busted, and this is the New Way To Do Things. Why it’s better than the “old busted ways” is never explicitly stated. This kind of marketing preys on software developer’s fear of becoming obsolete in their skillset, rather than making any substantive point. Words like “dynamic”, “modular”, “lightweight” are mixed in with “service mesh”, “seamless” and “graceful”. This article is going to be about cutting through the marketing and getting into what NGINX Unit means for those of us running production Ruby applications.

Before I move on to more about NGINX Unit’s architecture and what makes it unique, let’s make sure we all understand the difference between an application server and a web server. A web server connects to clients over HTTP, and usually serves static files or proxies to other HTTP-enabled servers, and acts as a middleman. An application server is the thing which actually starts and runs the language runtime. In Ruby, these functions are sometimes combined. For example, all of the major Ruby application servers also are web servers. However, many web servers, such as Nginx and Apache, are not also application servers. Nginx UNIT is both a web and application server.

NGINX Unit runs four different types of processes: main, router, controller, and application. Application processes are the self-explanatory ones - this would just be the Ruby runtime running your Rails application. The router and controller processes, and how they interact with each other and the application processes, is what defines how NGINX Unit works.

The main process creates the router and application processes. That’s really all it does. Application processes in NGINX Unit are dynamic, however - the number of processes running can be changed at any time, Ruby versions can be changed, or even entire new applications can be added while the server is running. The thing that tells the main process what application processes to run is the controller process.

The controller process (like the main process, there’s only one) has two jobs: expose a JSON configuration API over HTTP, and configure the router and main processes. This is probably the most novel and interesting part of NGINX Unit for Rubyists. Rather than working with configuration files, you POST JSON objects to the controller process to tell it what to do. For example, with this json file:

{
    "listeners": {
        "*:3000": {
            "application": "rails"
        }
    },

    "applications": {
        "rails": {
            "type": "ruby",
            "processes": 5,
            "script": "/www/railsapp/config.ru"
        }
    }
}

… we can PUT it to an NGINX Unit controller process with this (assuming our NGINX Unit server is listening on port 8443):

curl -d "myappconfig.json" -X PUT '127.0.0.1:8443'

… and create a new Ruby application.

NGINX Unit’s JSON configuration object is divided into listeners and applications. Applications are the actual apps you want to run. Listeners are where those apps are exposed to the world (i.e. what port they’re on).

Changes in application and listener configuration are supposed to be seamless. For example, a “hot deploy” of a new version of your application would be accomplished by adding a new application to the configuration:

{
  "rails-new": {
      "type": "ruby",
      "processes": 5,
      "script": "/www/rails-new-app/config.ru"
  }
}

curl -d "mynewappconfig.json" -X PUT

and then switching the listener to the new application:

curl -X PUT -d '"rails-new"' '127.0.0.1:8443/listeners/*:3000/application`

This transition is (supposedly) seamless, and clients won’t notice. This is similar to a Puma “phased restart”. In a phased restart in Puma, each worker process is restarted one at a time, which means that the other works processes are up and available to take requests. Puma accomplishes this using a control server (managed by the pumactl utility). However, unlike Puma, NGINX Unit “hot restarts” will not have two versions of the application taking requests at the same time.

In a Puma phased restart, say your application has six workers. Halfway through the phased restart, 3 works will be running the old code, and half will be running new code. This can cause some problems with database schema changes, for example. NGINX Unit restarts happen “all at once”, so while two versions of the code will be running at once, only one version will be taking requests at any point in time.

This functionality seems quite useful to those who are running their own Ruby applications on a service such as AWS, where you have to manage your own deployment. However, Heroku users won’t find any of this useful, as you’ve already had this sort of “hot deploy” functionality using Heroku’s preboot system. However, these two features aren’t doing exactly the same thing. Heroku creates an entirely new virtual server and hot-swaps the whole thing, whereas NGINX Unit is just changing processes on a single machine, but they’re completely the same from a client perspective.

The router process is pretty much what it sounds like - the thing which turns HTTP connections from clients into requests to the web application processes. NGINX claims a single Unit router can handle thousands of simultaneous connections. The router works a lot like an NGINX web server, and has a number of worker threads to accept, buffer and parse incoming connections.

To me, this is one of the most exciting parts of NGINX Unit for Rubyists. It is very difficult for Ruby application servers to deal with HTTP connections without some kind of reverse proxy in front of the app server. Unicorn, for example, is recommended for use only behind a reverse proxy because it cannot buffer requests. That is, if a client sends one byte of their request and then stops (due to network conditions, a bad cellphone connection perhaps), then the Unicorn process just stops all work and cannot continue until that request has finished buffering. Using NGINX, for example, in front of Unicorn allows NGINX to buffer that request before it reaches Unicorn. Since NGINX is written in highly optimized C and it’s not restricted by Ruby’s GVL, it can buffer hundreds of connections for Unicorn. Passenger solves this problem by basically just being an addon for NGINX or Apache³(Now you know why it’s called Passenger!)³ Now you know why it’s called Passenger! (mod_ruby!) and offloading all of the connection-related work to the webserver. In this way, NGINX Unit is more similar to Passenger than it is to Unicorn.

The application configuration has a processes key. This key can have a minimum number and maximum number of processes:

{
  "rails-new": {
      "type": "ruby",
      "processes": {
        "spare": 5
        "max": 10
      },
      "script": "/www/rails-new-app/config.ru"
  }
}

For some reason, the “minimum” number of processes is called “spare”. The config above will start 5 processes immediately, and will scale to 10 if the load requires it.

No word yet on if any settings like Puma’s preload_app! and similar settings in Passenger and Unicorn are available so you will be able to start up processes before they are needed and take advantage of copy-on-write memory.

This leaves the application processes. The interesting and novel thing here is that the router does not communicate with the application processes via HTTP - it uses Unix sockets and shared memory. This looks like an optimization aimed at microservice architectures, as communicating between services on the same machine will be considerably faster without any HTTP in between. I have yet to see any Ruby code examples of how this could work, however.

It is unclear to me in the long-term if it is intended for you to run NGINX in front of NGINX Unit, or if NGINX Unit can run on it’s own without anything in front of it. As of right now (Q1 2018), you should probably be running NGINX in front of NGINX Unit as a reverse proxy, because NGINX Unit lacks static file serving, HTTPS (TLS), and HTTP/2. Obviously, the integration is pretty seamless.

NGINX Unit is approaching a stable 1.0 release. You can’t really run it in production right now for Ruby applications: As I write this sentence, the Ruby module is literally 5 days old. It’s still under very active development right now - minor versions are released every few weeks. TLS and HTTP-related features seem like the next “big features” to come down the pipe, with static file serving being next. There is some discussion about support for Java, which could probably be turned into support for JRuby and TruffleRuby as well.

There is no Windows support, and I don’t think I would hold my breath for any in the future. NGINX Unit only supports Ruby 2.0 and above.

I will not be benchmarking NGINX Unit in this post. It’s Ruby module is extremely new and probably not ready for any kind of benchmarking. However, the real reason I won’t be benchmarking NGINX Unit against Puma, Unicorn or Passenger is because application server choice in Ruby is not a matter of speed (techincally, latency) but throughput. Application servers tend to differ in how many requests they can serve in parallel, rather than how quickly they do it. Application servers impose very little latency overhead on the applications they serve, probably on the order of a couple of milliseconds.

The most important Ruby application server setting which affects throughput is threading. The reason is that it is the only application server setting which can increase the number of requests served concurrently. A multithreaded Ruby application server can make greater and more efficient use of the available CPU and memory resources and serve more requests-per-minute than a single-threaded Ruby application process.

Currently, the only free application server which runs Ruby web applications in multiple threads is Puma. Passenger Enterprise will do it, but you must pay for a license.

NGINX Unit plans support for multiple threads in Python applications, so it is not inconceivable that it will support Ruby applications in multiple threads sometime in the future.

So, how does NGINX Unit currently “shake out” in comparison to Unicorn, Passenger and Puma? I think that the traditional Rails application setup: one monolithic application, run on a Plaform-as-a-Service provider like Heroku will probably not see any benefit at all from NGINX Unit’s current features and planned roadmap. Puma already serves these users very well.

NGINX Unit may be interesting for Unicorn users who want to stop using a reverse proxy. Once NGINX Unit’s HTTP features are fleshed out, it could replace a Unicorn/NGINX setup with just a single NGINX Unit server.

NGINX Unit is probably most directly comparable to Phusion Passenger, which also recently went into the “microservice” realm by supporting Javascript and Python as well as Ruby applications. NGINX Unit currently supports more languages and will probably support even more in the future, so those that need greater language support will probably switch. However, Phusion is a Ruby-first company, so I expect Passenger to always “support” Ruby in a better, more complete way than NGINX Unit ever will. And, as mentioned above, Phusion Passenger Enterprise supports multithreaded execution today.

So, what is the ideal NGINX Unit app? If you’re running your own cloud (that is, not on a service which manages the routing for you, like Heroku) and you have many Ruby applications running on different Ruby versions or many services in many different languages and those services/apps need to talk to each other, quickly, it looks like NGINX Unit was designed for you. If you don’t fit that profile, though, it’s probably best to stick to the existing top three options (Puma, Passenger, and Unicorn).

Malloc Can Double Multi-threaded Ruby Program Memory Usage

Mon, 04 Dec 2017 07:00:00 +0000

Sometimes, it really is that simple.

It’s not every day that a simple configuration change can completely solve a problem.

I had a client whose Sidekiq processes were using a lot of memory - about a gigabyte each. They would start at about 300MB each, then slowly grow over the course of several hours to almost a gigabyte, where they would start to level off.

I asked him to change a single environment variable: MALLOC_ARENA_MAX. “Please set it to 2.”

His processes restarted, and immediately the slow growth was eliminated. Processes settled at about half the memory usage they had before - around 512MB each.

Actually, it’s not that simple. There are no free lunches. Though this one might be close to free. Like a ten cent lunch.

Now, before you go copy-pasting this “magical” environment variable into all of your application environments, know this: there are drawbacks. You may not be suffering the problem it solves. There are no silver bullets.

Ruby is not known for being a language that’s light on memory use. Many Rails applications suffer from up to a gigabyte of memory use per process. That’s approaching Java levels. Sidekiq, the popular Ruby background job processor, has processes which can get just as large or even larger. The reasons are many, but one reason in particular is extremely difficult to diagnose and debug: fragmentation.

Typical Ruby memory growth looks logarithmic.

The problem manifests itself as a slow, creeping memory growth in Ruby processes. It is often mistaken for a memory leak. However, unlike a memory leak, memory growth due to fragmentation is logarithmic, while memory leaks are linear.

A memory leak in a Ruby program is usually caused by a C-extension bug. For example, if your Markdown parser leaks 10kb every time you call it, your memory growth will continue forever at a linear rate, since you tend to call the markdown parser at a regular frequency.

Memory fragmentation causes logarithmic growth in memory. It looks like a long curve, approaching some unseen limit. All Ruby processes experience some memory fragmentation. It’s an inevitable consequence of how Ruby manages memory.

In particular, Ruby cannot move objects in memory. Doing so would potentially break any C language extensions which are holding raw pointers to a Ruby object. If we can’t move objects in memory, fragmentation is an inevitable result. It’s a fairly common issue in C programs, not just Ruby.

Actual client graph. This is what fragmentation looks like. Note the enormous drop after MALLOC_ARENA_MAX changed to 2.

However, fragmentation can sometimes cause Ruby programs to twice as much memory as they would otherwise, sometimes as much as four times more!

Ruby programmers aren’t used to thinking about memory, especially not at the level of malloc. And that’s OK: the entire language is designed to abstract memory away from the programmer. It’s right in the manpage. But while Ruby can guarantee memory safety, it cannot provide perfect memory abstraction. One cannot be completely ignorant of memory. Because Ruby programmers are often inexperienced with how computer memory works, when problems occur, they often have no idea where to even start with debugging it, and may dismiss it as an intrinsic feature of a dynamic, interpreted language like Ruby.

“And underneath 4 layers of memory abstraction, she noticed some fragmentation!”

What makes it worse is that memory is abstracted away from Rubyists through four separate layers. First is the Ruby virtual machine itself, which has its own internal organization and memory tracking features (sometimes called the ObjectSpace). Second is the allocator, which differs greatly in behavior depending on the particular implementation you’re using. Third is the operating system, which abstracts actual physical memory addresses away into virtual memory addresses. The way it does this varies significantly depending on the kernel - Mach does this much differently than Linux, for example. Finally, there’s the actual hardware itself, which uses several strategies to keep frequently-accessed data in “hot” locations where it can be more quickly accessed. There are even special parts of the CPU involved here, such as the translation lookaside buffer.

This is what makes memory fragmentation so difficult for Rubyists to deal with. It’s a problem that generally happens at the level of the virtual machine and the allocator, parts of the Ruby language that 95% of Rubyists are probably unfamiliar with.

Some fragmentation is inevitable, but it can also get so bad that it doubles the memory usage of your Ruby processes. How can you know if you’re suffering the latter rather than the former? What causes critical levels of memory fragmentation? Well, I have one thesis about a cause of memory fragmentation which affects multithreaded Ruby applications, like webapps running on Puma or Passenger Enterprise, and multithreaded job processors such as Sidekiq or Sucker Punch.

Per-Thread Memory Arenas in glibc Malloc

It all boils down to a particular feature of the standard glibc malloc implementation called “per-thread memory arenas”.

To understand why, I need to explain how garbage collection works in CRuby really quickly.

ObjectSpace visualization by Aaron Patterson. Each pixel is an RVALUE. Green is “new”, red is “old”. See heapfrag.

All objects have a entry in the ObjectSpace. The ObjectSpace is a big list which contains an entry for every Ruby object currently alive in the process. The list entries take the form of RVALUEs, which are 40-byte C structs that contain some basic data about the object. The exact contents of these structs varies depending on the class of the object. As an example, if it is a very short String like “hello”, the actual bits that contain the character data are embedded directly in the RVALUE. However, we only have 40 bytes - if the string is 23 bytes or longer, the RVALUE contains only a raw pointer to where the object data actually lies in memory, outside the RVALUE.

RVALUEs are further organized in the ObjectSpace into 16KB “pages”. Each page contains about 408 RVALUEs.

These numbers can be confirmed by looking at the GC::INTERNAL_CONSTANTS constant in any Ruby process:

GC::INTERNAL_CONSTANTS
=> {
:RVALUE_SIZE=>40,
:HEAP_PAGE_OBJ_LIMIT=>408,
# ...
}

Creating a long string (let’s say it’s a 1000-character HTTP response for example) looks like this:

Add an RVALUE to the ObjectSpace list. If we are out of free slots in the ObjectSpace, we lengthen the list by 1 heap page, calling malloc(16384).
Call malloc(1000) and receive a address to a 1000-byte memory location.¹(Actually, Ruby will request an area slightly larger than it needs in case the string is added to or resized.)¹ Actually, Ruby will request an area slightly larger than it needs in case the string is added to or resized. This is where we’ll put our HTTP response.

The malloc calls here are what I want to bring your attention to. All we’re doing is asking for a memory location of a particular size, somewhere. Actually, malloc’s contiguity is undefined, that is, it makes no guarantees about where that memory location will actually be. This means that, from the perspective of the Ruby VM, fragmentation (which is fundamentally a problem about where memory is) is a problem of the allocator.²(However, allocation patterns and sizes can definitely make things harder for the allocator.)² However, allocation patterns and sizes can definitely make things harder for the allocator.

Ruby can, in a way, measure the fragmentation of its own ObjectSpace. A method in the GC module, GC.stat, provides a wealth of information about the current memory and GC state. It’s a little overwhelming and is under-documented, but the output is a hash that looks like this:

GC.stat
=> {
:count=>12,
:heap_allocated_pages=>91,
:heap_sorted_length=>91,
# ... way more keys ...
}

There are two keys in this hash that I want to point your attention to: GC.stat[:heap_live_slots] and GC.stat[:heap_eden_pages].

:heap_live_slots refers to the number of slots in the ObjectSpace currently occupied by live (not marked for freeing) RVALUE structs. This is roughly the same as “currently live Ruby objects”.

The Eden heap

:heap_eden_pages is the number of ObjectSpace pages which currently contain at least one live slot. ObjectSpace pages which have at least one live slot are called eden pages. ObjectSpace pages which contain no live objects are called tomb pages. This distinction is important from the GC’s perspective, because tomb pages can be returned back to the operating system. Also, the GC will put new objects into eden pages first, and then tomb pages after all the eden pages have filled up. This reduces fragmentation.

If you divide the number of live slots by the number of slots in all eden pages, you get a measure of the current fragmentation of the ObjectSpace. As an example, here’s what I get in a fresh irb process:

5.times { GC.start }
GC.stat[:heap_live_slots] # 24508
GC.stat[:heap_eden_pages] # 83
GC::INTERNAL_CONSTANTS[:HEAP_PAGE_OBJ_LIMIT] # 408

# live_slots / (eden_pages * slots_per_page)
# 24508 / (83 * 408) = 72.3%

About 28% of my eden page slots are currently unoccupied. A high percentage of free slots indicates that the ObjectSpace’s RVALUEs are spread across many more heap pages than they would be if we could move them around. This is a kind of internal memory fragmentation.

Another measure of internal fragmentation in the Ruby VM comes from GC.stat[:heap_sorted_length]. This key is the “length” of the heap. If we have three ObjectSpace pages, and I free the 2nd one (the one in the middle), I only have two heap pages remaining. However, I cannot move heap pages around in memory, so the “length” of the heap (essentially the highest index of the heap pages) is still 3.

Yes, this heap is fragmented, but it looks really tasty.

Dividing GC.stat[:heap_eden_pages] by GC.stat[:heap_sorted_length] gives a measure of internal fragmentation at the level of ObjectSpace pages - a low percentage here would indicate a lot of heap-page-sized “holes” in the ObjectSpace list.

While these measures are interesting, most memory fragmentation (and most allocation) doesn’t happen in the ObjectSpace - it happens in the process of allocating space for objects which don’t fit inside a single RVALUE. It turns out that’s most of them, according to experiments performed by Aaron Patterson and Sam Saffron. A typical Rails app’s memory usage will be 50%-80% in these malloc calls to get space for objects larger than a few bytes.

Well this sucks. Looks like only 15% of the heap in a basic Rails app is managed by the GC. 85% is just mallocs pic.twitter.com/sPbtAq4g8j
— Aaron Patterson (@tenderlove) June 28, 2017

When Aaron says “managed by the GC” here, he means “inside the ObjectSpace list”.

Ok, so let’s talk about where per-thread memory arenas come in.

The per-thread memory arena was an optimization introduced in glibc 2.10, and lives today in arena.c. It’s designed to decrease contention between threads when accessing memory.

In a naive, basic allocator design, the allocator makes sure only one thread can request a memory chunk from the main arena at a time. This ensures that two threads don’t accidentally get the same chunk of memory. If they did, that would cause some pretty nasty multi-threading bugs. However, for programs with a lot of threads, this can be slow, since there’s a lot of contention for the lock. All memory access for all threads is gated through this lock, so you can see how this could be a bottleneck.

Removing this lock has been an area of major effort in allocator design because of its performance impact. There are even a few lockless allocators out there.

The per-thread memory arena implementation alleviates lock contention with the following process (paraphrased from this article by Siddhesh Poyarekar):

We call malloc in a thread. The thread attempts to obtain the lock for the memory arena it accessed previously (or the main arena, if no other arenas have been created).
If that arena is not available, try the next memory arena (if there are any other memory arenas).
If none of the memory arenas are available, create a new arena and use that. This new arena is linked to to the last arena in a linked list.

In this way, the main arena is basically extended into a linked list of arenas/heaps. The number of arenas is limited by mallopt, specifically the M_ARENA_MAX parameter (documented here, note the “environment variables” section). By default, the limit on the number of per-thread memory arenas that can be created is 8 times the number of available cores. Most Ruby web applications run about 5 threads per core, and Sidekiq clusters can often run far more than that. In practice, this means that many, many per-thread memory arenas can get created by a Ruby application.

Let’s take a look at exactly how this would play out in a multithreaded Ruby application.

You are running a Sidekiq process with the default setting of 25 threads.
Sidekiq begins running 5 new jobs. Their job is to communicate with an external credit card processor - so they POST a request via HTTPS and receive a response ~3 seconds later.
Each job (which is running a separate thread in Rubyland) sends an HTTP request and waits for a response using the IO module. Generally, almost all IO in CRuby releases the Global VM lock, which means that these threads are working in parallel and may contend for the main memory arena lock, causing the creation of new memory arenas.

If multiple CRuby threads are running but not doing I/O, it is pretty much impossible for them to contend for the main memory arena because the Global VM Lock prevents two Ruby threads from executing Ruby code at the same time. Thus, per-thread-memory arenas only affect CRuby applications which are both multithreaded and performing I/O.

How does this lead to memory fragmentation?

Bin-packing can be fun, too!

Memory fragmentation is essentially a bin packing problem - how can we efficiently distribute oddly-sized items between multiple bins so that they take up the least amount of space? Bin-packing is made much more difficult for the allocator because a) Ruby never moves memory locations around (once we allocate a location, the object/data stays there until it is freed) b) per-thread memory arenas essentially create a lot of different bins, which cannot be combined or “packed” together. Bin-packing is already NP-hard, and these constraints just make it even more difficult to achieve an optimal solution.

Per-thread memory arenas leading to large amounts of RSS use over time is something of a known issue on the glibc malloc tracker. In fact, the MallocInternals wiki says specifically:

As pressure from thread collisions increases, additional arenas are created via mmap to relieve the pressure. The number of arenas is capped at eight times the number of CPUs in the system (unless the user specifies otherwise, see mallopt), which means a heavily threaded application will still see some contention, but the trade-off is that there will be less fragmentation.

There you have it - lowering the number of available memory arenas reduces fragmentation. There’s an explicit tradeoff here: fewer arenas decreases memory use, but may slow the program down by increasing lock contention.

Heroku discovered this side-effect of per-thread memory arenas when they created the Cedar-14 stack, which upgraded glibc to version 2.19.

Heroku customers reported greater memory consumption of their applications when upgrading their apps to the new stack. Testing by Terrence Hone of Heroku produced some interesting results:

Configuration	Memory Use
Base (unlimited arenas)	1.73x
Base (before arenas introduced)	1x
MALLOC_ARENA_MAX=1	0.86
MALLOC_ARENA_MAX=2	0.87

Basically, the default memory arena behavior in libc 2.19 reduced execution time by 10%, but increased memory use by 75%! Reducing the maximum number of memory arenas to 2 essentially eliminated the speed gains, but reduced memory usage over the old Cedar-10 stack by 10% (and reduced memory usage by about 2X over the default memory arena behavior!).

Configuration	Response Times
Base (unlimited arenas)	0.9x
Base (before arenas introduced)	1x
MALLOC_ARENA_MAX=1	1.15x
MALLOC_ARENA_MAX=2	1.03x

For almost all Ruby applications, a 75% memory gain for 10% speed gain is not an appropriate tradeoff. But let’s get some more real-world results in here.

A Replicating Program

I wrote a demo application, which is a Sidekiq job which generates some random data and writes the response to a database.

After switching MALLOC_ARENA_MAX to 2, memory usage was 15% lower after 24 hours.

I’ve noticed that real-world workloads magnify this effect greatly, which means I don’t fully understand the allocation pattern which can cause this fragmentation yet. I’ve seen plenty of memory graphs on the Complete Guide to Rails Performance Slack channel that show 2-3x memory savings in production with MALLOC_ARENA_MAX=2.

Fixing the Problem

There are two main solutions for this problem, along with one possible solution for the future.

Fix 1: Reduce Memory Arenas

One fairly obvious fix would be to reduce the maximum number of memory arenas available. We can do this by changing the MALLOC_ARENA_MAX environment variable. As mentioned before, this increases lock contention in the allocator and will have a negative impact on the performance of your application across the board.

It’s impossible to recommend a generic setting here, but it seems like 2 to 4 arenas is appropriate for most Ruby applications. Setting MALLOC_ARENA_MAX to 1 seems to have a high negative impact on performance with only a very marginal improvement to memory usage (1-2%). Experiment with these settings and measure the results both in memory use reduction and performance reduction until you’ve made a tradeoff appropriate for your app.

Fix 2: Use `jemalloc`

This is CodeTriage's Sidekiq worker memory use with and without jemalloc. I'm really starting to wonder how much of Ruby's memory problems are just caused by the allocator. pic.twitter.com/FD0fVbJCLt
— Nate Berkopec (@nateberkopec) December 1, 2017

Another possible solution is to simply use a different allocator. jemalloc also implements per-thread arenas, but their design seems to avoid the fragmentation issues present in malloc.

The above tweet was from when I removed jemalloc from CodeTriage’s background job processes. As you can see, the effect was pretty drastic. I also experimented with using malloc with MALLOC_ARENA_MAX=2, but memory usage was still almost 4 times greater than memory usage with jemalloc. If you can switch to jemalloc with Ruby, do it. It seems to have the same or better performance than malloc with far less memory use.

This isn’t a jemalloc blog post, but some finer points on using jemalloc with Ruby:

You can use it on Heroku with this buildpack.
Do not use jemalloc 4.x with Ruby. It has a bad interaction with Transparent Huge Pages that reduces the memory savings you’ll see. Instead, use jemalloc 3.6. 5.0’s performance with Ruby is currently unknown.
You do not need to compile Ruby with jemalloc (though you can). You can dynamically load it with LD_PRELOAD.

Fix 3: Compacting GC

Fragmentation can generally be reduced if one can move locations in memory around. We can’t do that in CRuby because C-extensions may use raw pointers to refer to Ruby’s memory - moving that location would cause a segfault or incorrect data to be read.

Aaron Patterson has been working on a compacting garbage collector for a while now. The work looks promising, but perhaps a ways off in the future.

TL;DR:

Multithreaded Ruby programs may be consuming 2 to 4 times the amount of memory that they really need, due to fragmentation caused by per-thread memory arenas in malloc. To fix this, you can reduce the maximum number of arenas by setting the MALLOC_ARENA_MAX environment variable or by switching to an allocator with better performance, such as jemalloc.

The potential memory savings here are so great and the penalties so minor that I would recommend that if you are using Ruby and Puma or Sidekiq in production, you should always use jemalloc.

While this effect is most pronounced in CRuby, it may also affect the JVM and JRuby.

Configuring Puma, Unicorn and Passenger for Maximum Efficiency

Thu, 12 Oct 2017 07:00:00 +0000

In Ruby, web application servers are like gasoline in a car: the fancy stuff won’t make your car go any faster, but the nasty stuff will bring you grinding to a halt. Application servers can’t actually make your app significantly faster - no, they’re all pretty much the same and changing from one to the other won’t improve your throughput or response times by much. But it is easy to shoot yourself in the foot with a bad setting or misconfigured server. It’s one of the most common problems I see on client applications.

This post will be about optimizing resource usage (memory and CPU) and maximizing throughput (that is, requests-per-second) from the three major Ruby application servers: Puma, Unicorn and Passenger. I’m going to use the terms “server” and “container” interchangeably, because nothing here is specific to a virtualized environment.

I can cover all three of the popular application servers in a single guide because they all use, fundamentally, the same design. With the fork system call, these application servers create several child processes, which then do the job of serving requests. ¹(In all three app servers, the ‘master’ process that creates the child processes does not actually answer any requests. Passenger will actually shut down the ‘master’ preload process after a while if you haven’t forked recently.)¹ In all three app servers, the ‘master’ process that creates the child processes does not actually answer any requests. Passenger will actually shut down the ‘master’ preload process after a while if you haven’t forked recently. Most of the differences between these servers lie in the finer details (which I’ll also cover here, where important for maximum performance).

Throughout this guide, we’re going to try to maximize our throughput-per-server-dollar. We want to serve the most number of requests per second for the lowest amount of server resources (and therefore, cash).

The most important configuration settings for performance

Timeouts are fairly important too, but they’re not really throughput-related. I’ll leave them for another day.

There are 4 fundamental settings on your application server that determine its performance and resource consumption:

Number of child processes.
Number of threads.
Copy-on-write.
Container size.

Let’s go through each in turn.

Child process count

Unicorn, Puma and Passenger all use a forking design.²(JRuby people can probably skip to the next section.)² JRuby people can probably skip to the next section. This means that they create one application process and call fork a number of times to create copies of that application process. We call these copies child processes. The number of child processes we have on each server is probably the most important setting for maximizing throughput-per-server-dollar. ³(This is because of CRuby’s Global VM Lock. Only one thread can execute Ruby code at a time, so the only way to achieve parallel Ruby work is to run multiple processes.)³ This is because of CRuby’s Global VM Lock. Only one thread can execute Ruby code at a time, so the only way to achieve parallel Ruby work is to run multiple processes. We want to run as many processes per server as possible without exceeding the resources of the server.

I recommend that all Ruby webapps run at least 3 processes per server or container. This maximizes routing performance. Puma and Unicorn both use a design where the child processes listen directly on a single socket, and then let the operating system balance load between the processes. Passenger uses a reverse proxy (nginx or Apache) to route requests to a child process.⁵(Passenger’s least-busy-process-first routing is actually one of my favorite features of theirs.)⁵ Passenger’s least-busy-process-first routing is actually one of my favorite features of theirs. Both approaches are pretty efficient and mean that a request will be quickly routed to a worker that is idle. Routing at higher layers (that is, at the load balancer or Heroku’s HTTP mesh) is far more difficult to do so efficiently, because the load balancer usually has no idea whether or not the servers its routing to are busy or not.⁶(For one client I had, moving from 30 servers with 2 processes each to 3 servers with 20 processes each almost completely eliminated the timeout errors they were having (which were being caused by fast requests piling up behind slow ones).)⁶ For one client I had, moving from 30 servers with 2 processes each to 3 servers with 20 processes each almost completely eliminated the timeout errors they were having (which were being caused by fast requests piling up behind slow ones).

Consider a setup with 3 servers, each running 1 processes (so a total of 3 processes). How does the load balancer optimally route a request to one of the three servers? It could do so randomly or in a round-robin fashion, but this does not guarantee that the request will be routed to a server with an idle, waiting process. For example, with a round-robin strategy, let’s say Request A is routed to server #1. Request B is then routed to server #2, and Request C to server #3.
My face when you give me a request but all my children are busy. Now here comes a fourth request, Request D. What happens if Request B and C have already been successfully served and those servers (2 and 3) are idle, but Request A was somebody’s CSV export and will take 20 seconds to complete? The load balancer will continue to give requests to server #1 even though its busy and won’t process them until it’s done with Request A. All load balancers have ways of knowing if a server is completely dead, but most of these methods have a long lag time (i.e. 30 seconds or more of delay). Running higher numbers of processes per server insulates us from the risk of long-lived requests “hogging” the majority of a server’s child processes, because at the server level, requests will never be given to an already-busy process. Instead, they’ll back up at the socket level or the reverse proxy until a worker is free. From experience, I find that 3 processes per server is a good minimum to achieve this. If you can’t run at least 3 processes per server due to resource constraints, get a bigger server (more on that later).

So, we should run at least 3 child processes per container. But what’s the maximum? That’s constrained by our memory and CPU resources.

Let’s start with memory. Each child process uses a certain amount of memory. Obviously, we shouldn’t add more child processes than our server’s RAM can support!

Actual memory usage of Ruby processes is logarithmic. Due to memory fragmentation, memory usually doesnt level off, but only approaches a limit. Measuring the actual memory usage of a single Ruby application process can be tricky, however. It’s not enough to just start up a process on your computer or production environment and check the number right away.
After a while, Puma workers can get rather… large. For a number of reasons, Ruby web application processes increase in memory usage over time, even as much as doubling or tripling their memory usage from when they are spawned. To get an accurate measurement of how much memory your Ruby application processes are using, disable all process restarts (worker killers) and wait 12-24 hours to take a measurement with ps. If you’re on Heroku, you can use the new Heroku Exec to use ps on a running dyno, or simply divide Heroku’s memory usage metric by the number of processes you are running per dyno. Most Ruby applications will use between 200 and 400 MB per process, but some can use as much as 1GB.

1 upvote = 1 prayer

Be sure to give yourself some headroom on the memory number - if you want an equation, set your child process count to something like (TOTAL_RAM / (RAM_PER_PROCESS * 1.2))

Exceeding the available memory capacity of a server/container can cause major slowdowns as memory is overcommitted and swapping starts to occur. This is why you want your application’s memory usage to be predictable and consistent with no sudden spikes. Sudden increases in memory usage are a condition I call memory bloat. Solving this is a topic for another day or post, but the topic is covered in The Complete Guide to Rails Performance

Second, we don’t want to exceed the available CPU capacity of our server. Ideally, we don’t spend more than 5% of our total deployed time at 100% CPU usage - more than that means that we’re being bottlenecked by the available CPU capacity. Most Ruby and Rails applications tend to be memory-bottlenecked on most cloud providers, but sometimes CPU can be the bottlenecking resource too. How do you know? Just use your favorite server monitoring tool - AWS’s built in tools are probably good enough for figuring out if CPU usage is frequently maxing out.

You said that OS context switching was expensive. Actual production use determined that was a lie. It’s frequently said that you shouldn’t have more child processes per server than CPUs. This is only partly true. It’s a good starting point, but actual CPU usage is the metric you should watch and optimize. In practice, most applications will probably settle at a process count that is 1.25-1.5x the number of available hyperthreads.

On Heroku, use log-runtime-metrics to get a CPU load metric written to your logs. I would look at the 5 and 15 minute load averages - if they are consistently close to or higher than 1, you are maxing out CPU and need to reduce child process counts.

Setting child process counts is pretty easy in every application server:

# Puma
$ puma -w 3 # Command-line option
workers 3 # in your config/puma.rb

# Unicorn
worker_processes 3 # config/unicorn.rb

# Passenger (nginx/Standalone)
# Passenger can automatically scale workers up and down - I don't find this
# super useful. Instead, just run a constant number by setting the max and min:
passenger_max_pool_size 3;
passenger_min_instances 3;

Instead of setting this to a hard number, you may want to set it to an environment variable such as WEB_CONCURRENCY:

workers Integer(ENV["WEB_CONCURRENCY"] || 3)

In summary, most applications will want to use 3-8 processes per server, depending on available resources. Highly memory-constrained applications or apps which have high 95th percentile times (5-10 seconds or more) may want to run higher numbers, up to 4x the available hyperthread count. Most app’s child process counts should not exceed 1.5x the amount of available hyperthreads.

Thread count

Puma and Passenger Enterprise support multi-threading your application, so this discussion is aimed at those servers.

Threads can be a resource-light way of improving your application’s concurrency (and, therefore, throughput). Rails is already threadsafe, and most applications aren’t doing weird things like creating their own threads or using globals to access shared resource, like database connections (looking at you, $redis!) So, most Ruby web-applications are thread-safe. The only real way to know is to actually give it a shot. Ruby applications tend to surface threading bugs in loud, exception-raising ways, so it’s easy to give it a shot and see the results.

So how many threads should we use? The speedup you can gain from additional parallelism depends on the portion of your program’s execution which can be done in parallel. This is known as Amdahl’s Law. In MRI/C Ruby, we can only parallelize waiting on IO (waiting on a database result, for example). For most web applications, this is probably 10-25% of their total time. You can check for your own application by looking at the amount of time you spend “in the database” per request. Unfortuantely, what Amdahl’s law reveals is that for programs that have small parallel portions (less than 50%), there is little to no benefit past a handful of threads. This matches my own experience: on client applications, thread settings of more than 5 have no effect. Noah Gibbs also tested this against the Discourse homepage benchmark and settled on a thread count of 6.

Unlike process count, where I advise you to constantly check the metrics against your settings and tune appropriately, with threads, it’s usually OK to just “set it and forget it” to 5 threads per application server process.

In MRI/C Ruby, threads can have a surprisingly large memory impact. This is due to a host of complicated reasons (which I’ll probably get into in a future post). Be sure to check memory consumption before and after adding threads to the application. Do not expect that each thread will only consume an additional 8MB of stack space, they will often increase total memory usage by far more than that.

Here’s how to set thread counts:

# Puma. Again, I don't really use the "automatic" spin-up/spin-down features, so
# I set the max and min to the same number.
$ puma -t 5:5 # Command-line option
threads 5, 5 # in your config/puma.rb

# Passenger (nginx/Standalone)
passenger_concurrency_model thread;
passenger_thread_count 5;

For JRuby people - threads are fully parallelizable, so you can take the full benefit of Amdahl’s law here. Setting thread counts for you will be more like setting process counts under MRI (described above) - increase them until you run out of memory or CPU resources.

Copy-on-write behavior

All Unix-based operating systems implement copy-on-write memory behavior. It’s pretty simple: when a process forks and creates a child, that child process’ memory is shared, completely, with the parent process. All memory reads from the child process will simply read from the parent’s memory. However, modifying a memory location creates a copy, solely for the private use of the child process. It’s extremely useful for reducing the memory usage of forking webservers, since child processes should, in theory, be able to share things like shared libraries and other “read-only” memory with the parent, rather than creating their own copy.

Copy-on-write just happens. ⁶(You can’t really ‘support’ copy-on-write so much as just ‘make it more effective at saving you memory’.)⁶ You can’t really ‘support’ copy-on-write so much as just ‘make it more effective at saving you memory’. It can’t be “turned off”, but you can make it more effective. Basically, we want to load all of our application before forking. Most Ruby webapp servers call this “preloading”. All it does is change when fork is called - before or after your application is initialized.

You’ll also need to re-connect to any databases you’re using after forking. For example, with ActiveRecord:

# Puma
preload_app!
on_worker_boot do
  # Valid on Rails 4.1+ using the `config/database.yml` method of setting `pool` size
  ActiveRecord::Base.establish_connection
end

# Unicorn
preload_app true
after_fork do |server, worker|
	ActiveRecord::Base.establish_connection
end

# Passenger uses preloading by default, so no need to turn it on.
# Passenger automatically establishes connections to ActiveRecord,
# but for other DBs, you will have to:
PhusionPassenger.on_event(:starting_worker_process) do |forked|
  if forked
    reestablish_connection_to_database # depends on the DB
  end
end

In theory, you have to do this for every database your application uses. However, in practice, Sidekiq doesn’t try to connect to Redis until you actually try to do something, so unless you’re running Sidekiq jobs during application boot, you don’t have to reconnect after fork.

Unfortunately, there are limits to the gains of copy-on-write. Transparent Huge Pages can cause even a 1-bit memory modification to copy an entire 2MB page, and fragmentation can also limit savings. But it doesn’t hurt, so turn on preloading anyway.

Container size

Gimme some of that memory, boi In general, we want to make sure we’re utilizing 70-80% of our server’s available CPU and memory. These needs will differ between applications, and the ratio between CPU cores and GB of memory will differ in turn. One application might be happiest on a 4 vCPU / 4 GB of RAM server with 6 Ruby processes, while another less-memory-hungry and more CPU-heavy application might do well with 8 vCPUs and 2GB of RAM. There’s no one perfect container size, but the ratio between CPU and memory should be chosen based on your actual production metrics.

The amount of memory available to our server or container is probably one of the most important resources we can tune. On many providers, this number is exceedingly low - 512MB on the standard Heroku dyno, for example. Ruby applications, especially sufficiently complex and mature ones, are memory hungry, and the amount of memory we have to work with is probably one of our most important resources.

Because most Rails applications use ~300MB of RAM and I think everyone should be running at least 3 processes per server, most Rails applications will need a server with at least 1 GB of RAM.

Our server’s CPU resources are another important lever we can tune. We need to know how many CPU cores are available to us, and how many threads we can execute at a single time (basically, does this server support Hyper-Threading or not?).

As I mentioned in the discussion of child process counts, your container should support at least 3 child processes. Even better would be 8 or more processes per server/container. Higher process counts per container improves request routing and decreases latency.

TL;DR

This was an overview of how to best maximize the throughput of your Ruby web application servers. In a short, list format, here’s the steps:

Figure out how much memory 1 worker with 5 threads uses. If you’re using Unicorn, obviously no threads required. Run just a few workers on a single server under production load for at least 12 hours without restarting. Use ps to get the memory usage of a typical worker.
Choose a container size with memory equal to at least 3X that number. Most Rails applications will use ~300-400MB of RAM per worker. So, most Rails apps will need at least 1 GB container/server. This gives us enough memory headroom to run at least 3 processes per server. You can run a number of child processes equal to (TOTAL_RAM / (RAM_PER_PROCESS * 1.2)).
Check CPU core/hyperthread counts If your container has fewer hyperthreads (vCPUs on AWS) than your memory can support, you can either choose a container size with less memory or more CPU. Ideally, the number of child processes you run should equal 1.25-1.5x the number of hyperthreads.
Deploy and watch CPU and memory consumption. Tune child process count and container size as appropriate to maximize usage.

Is Ruby Too Slow For Web-Scale?

Tue, 11 Jul 2017 07:00:00 +0000

¹(Okay, okay, I know. Betteridge’s Law of Headlines. Of course Ruby and Rails are fast enough for big websites - Shopify makes it work and they’re one of the largest in the world. But some people genuinely do seem to think that Rails ‘isn’t fast enough’. That’s what this article is about.)¹ Okay, okay, I know. Betteridge’s Law of Headlines. Of course Ruby and Rails are fast enough for big websites - Shopify makes it work and they’re one of the largest in the world. But some people genuinely do seem to think that Rails ‘isn’t fast enough’. That’s what this article is about. How does one choose a framework or programming language for a new web application?

You almost certainly need one, unless you’re doing something pretty trivial. All web applications have a lot of boilerplate they need to get running: security, object-relational mapping, templating and testing. So how do you know which one to choose?

This is what Rails is, right? Well, you certainly don’t want to pick a slow framework, do you? That wouldn’t be good - we want a fast, modern, and lightweight web framework, not some heavy, old, slow, web framework. Heavy, old, and slow…like Ruby on Rails, right? Ruby on Rails, the king of the all-in-one web framework space for the last 10 years, is constantly under assault by faster, nippier, lighter competitors. Is Rails a dinosaur that can no longer compete?

Well, we could look at some benchmarks to find out. Surely a “fast” and “lightweight” framework would do well on a benchmark, while a old, busted framework would do poorly.

Yeah! Rails sucks! You would be forgiven for thinking that Ruby on Rails was somehow irretrievably graveyard-bound if you looked at the benchmarks posted by sites such as TechEmpower. Sequel author Jeremy Evans recently pointed out that even other Ruby frameworks can bury Rails in these comparisons. You look at those benchmarks at think: “Wow, Sequel is ten times faster than ActiveRecord and Rails!”

And in a narrow sense, you’d be right. Benchmarks are like statistics - it’s easy to give the right answer to the wrong question, and allow the reader to draw a conclusion which isn’t supported by the data. If you looked at those benchmarks and thought: “If I take my Rails application and rewrite it in Sequel and Sinatra, it will be ten times faster than it is now!”, you would be wrong.

And, even if it was faster, would it matter? Is there such a thing as a fast enough web application? Just how important is performance when choosing a web framework or even a programming language for a web application?

Latency and Throughput

Let’s start with some definitions.

Servers are like funnels: latency is how long it takes one molecule of water to pass through the funnel, throughput is how much water passes throught the funnel every second. A high-latency high-throughput server would be something like a long, wide tube, and a low-latency low-throughput server would look like a short, wide disc with a narrow opening. In server application design, latency and throughput are king. Latency is the amount of time it takes our server to respond to a single request. Throughput is how many requests we can serve at the same time, usually measured in a unit like responses/second.

Throughput of a web application is generally governed by CPU and parallelism - how many CPU cycles does it take to respond to a web request, and how efficiently can you saturate all the CPU cores of the host machine? The amount of CPU cycles is governed by the application’s domain, framework, and language - complicated apps take more time, and dynamic languages like Ruby generate more CPU instructions than compiled languages like C or Rust. Efficiently using all the available CPU resources varies depending on the language - Go’s goroutines, Elixir’s “processes”, multi-process servers to get around global VM locks like Python and Ruby, event-driven architectures like Node, or true threading like Java.

Latency, however, is even more important. This is because latency is inversely proportional to throughput. If we halve the latency of our web application, we double its maximum throughput. Latency also affects the end-user experience - a 500 millisecond response time manifests as an extra 500 milliseconds the user must spend waiting for the webpage to load.

Benchmark Trip-Ups

TechEmpower’s servers Let’s take a look at the TechEmpower web framework benchmarks. TechEmpower measures latency and maximum throughput across six synthetic benches. These benchmarks are run on pretty fat servers - they’ve got 4 cpus with 10 cores and 20 threads each (so, 40 cores and 80 hyperthreads in total). Oh yeah, and 528 GB of RAM.

Rails implementation of the benchmark One of the more relevant benchmarks is the multiple-queries benchmark. It’s pretty simple - it executes 20 queries, sequentially, against a SQL database, and then returns the result. This is a pretty common web application workload - most Rails applications I’ve worked on roughly look like this. As we render the template, we execute a few SQL queries to get the results to populate the template, and return it.

In Round 14, the typical Rails setup (puma-mri-rails) clocks in at a measly ~531 requests per second. Roda, an extreme lightweight Ruby web framework, when used with Sequel, clocks in at about 7000 requests/second, depending on the webserver used.

So does that mean Rails is more than 10 times slower than Roda and Sequel? On an 80 core machine, is 531 requests/second really all you can get out of Ruby on Rails?

TechEmpower’s Rails setup is unbelievably crippled compared to their Roda setup. Their Puma server is configured to run just 8 processes, while Roda auto-tunes itself, ending up with around 100 processes. So the Rails benchmark is using, at best, about 15-20% of the available hyperthreads, while the Roda benchmark is using all of them. So that’s at least a 5-8x throughput penalty for the Rails benchmark out of the gate. But that’s fixable - TechEmpower is open source and we can just open a pull request and fix this, and we’ll get better results for Round 15.

Let’s take a look at another TechEmpower measurement - average request latency. Focusing on request latency allows to put all languages and frameworks on a somewhat more even footing, because things like global VM locks and other concurrency features usually don’t really matter when processing a single request.²(Concurrency features generally increase throughput, not decrease latency.)² Concurrency features generally increase throughput, not decrease latency. On the multiple-query database test, Puma and Rails clock in at 129 milliseconds. The Roda/Sequel/Puma stack clocks in at 31.3 milliseconds.

Now, as I said, the Puma settings for Rails on TechEmpower are incredibly crippled compared to the Roda settings, so Rails could probably still shave a lot off of that time, but let’s take it as it is. Let’s just say Rails adds one hundred milliseconds of latency to the average web application response over a microframework or other competing platform like Phoenix. (Actually, Phoenix is slower on this test than Rails. The framework creators dispute this result though, and I don’t doubt it if the Rails benchmark is this gimped too).

The Computer Changes, But The Human Does Not

The funny thing about computers is that although they keep getting faster, squishy human beings stay the same speed. Just how fast a human-computer interaction has to be has been studied since the 1960s. You can understand their interest in this, back in the times when computers were the size of rooms and computations took hours rather than microseconds. If the computer was going to move out of the mainframe and the science lab and into public life, it was going to have to be faster. But how much faster?

Jakob Nielsen summarized the results in 1993:

Jakob Nielsen. I am glad that this photo exists.

0.1 second: Limit for users feeling that they are directly manipulating objects in the UI. (…)

1 second: Limit for users feeling that they are freely navigating the command space without having to unduly wait for the computer. (…)

10 seconds: Limit for users keeping their attention on the task. (…)

You can read his full article on the topic here.

On the web, how fast is fast enough?

Let’s assume that all our little web application does is return an HTML response with no JavaScript or CSS. It’s just a flat, HTML document with the default browser styling.³(Imagine if you would, for a moment, a website whose styling is even more boring than this one.)³ Imagine if you would, for a moment, a website whose styling is even more boring than this one. How long would it take for a user to visit www.oursite.com and receive a response?

Well, if our user is on a desktop computer in the same country as our servers, it will take about 20 milliseconds for their packets to get from their computer to our servers, and another 20 milliseconds back. This is a best case scenario: if they’re on the other side of the world, this could easily be 100 milliseconds each way. If they’re on a mobile cellular connection, we’re talking ~300-400 milliseconds. My home DSL connection fluctuates from 50-150 milliseconds to most US servers.

150 milliseconds time-to-first-byte? That’s Brent Rambo Approved. So, if we’ve already got ~40 milliseconds of round-trip network latency in the first place, will our users be able to perceive the difference in a web application which renders a response in 1 millisecond or 100 milliseconds? That is, one application will take 41 milliseconds in total and the other 141. The answer is emphatically no. Both applications will appear almost instantaneous to the user. And in the worst cases of network conditions, the difference will completely vanish. So minor latency differences (100 milliseconds or less, as in the difference between web frameworks) only matter in their contribution to improving throughput.

Your Server is Just a Small Part of the User Experience

WELCOME TO THE MODERN WEB, BITCH. It’s 2017 and web applications don’t return flat HTML files anymore. Websites are gargantuan, with JavaScript bundles stretching into the size of megabytes and stylesheets that couldn’t fit in ten Apollo Guidance Computers. So how much of a difference does a web application which responds in 1 millisecond or less make in this environment?

Vanishingly little. Nowadays, the average webpage takes 5 seconds to render. Some JavaScript single-page-applications can take 12 seconds or more on initial render.

Server response times simply make up a teeny-tiny part of the actual user experience of loading and interacting with a webpage - cutting 99 milliseconds off the server response time just doesn’t make a difference.

There’s a Ceiling: Web Apps Aren’t Video Games

In the video gaming world, speed matters. Faster languages can mean more polygons on the screen per frame. There’s really no upper limit for this - more polygons will always be good, so a faster language will always help with increasing the fidelity of the simulation.
for those unfamiliar with the meme

Web applications are not like this. Fundamentally, 90% of them are simple CRUD applications. A faster language does not open more possibilities for functionality or features, it just takes the same HTML webform we’ve been rendering and renders it a few milliseconds faster. There’s a ceiling on the usefulness of reduced request latency.

Ruby is Slow, so More Ruby is Slower

Mike Perham. And, ultimately, most of Ruby’s internals boil down to hash tables, so… Ruby isn’t a fast language. So, if you execute less of it, you’ll have a faster benchmark result.

Feature-rich frameworks like Rails have a lot of code, and execute a lot more on each request because they are doing more stuff.

This seems like 101-level stuff, but again, TechEmpower and other benchmarks typically do not make the difference in features obvious. On TechEmpower, all you get is this impossible-to-skim array of tags.

Yes, this is is an easy-to-understand feature comparison which humans can read. On throughput microbenchmarks like TechEmpower, where differences are measured in milliseconds (or even microseconds), what you’re really measuring is how many CPU instructions a particular language runtime generates in response to a particular request. And since there’s no real way to compare featuresets between frameworks on TechEmpower, all frameworks are placed on an “equal footing” and you’ll think that Rails is the slowest web framework in the world.

The truth is that Rails does a lot on every request. Just create a new Rails app and look at the middleware stack (rake middleware). There’s a lot of work being done here that every good web application should do but many frameworks do not do for you, at least by default.

Performance is More Complicated than CPU or Maximum Throughput

While on TechEmpower CPU usage is the bottleneck, in the real world, the CPU performance of language or the framework is almost never the bottleneck for a web application’s performance. Web applications are fairly I/O heavy, especially as they grow more complicated. The modern Rails application may interact with three separate databases or more - their SQL database, Redis for their backend job processor, and Memcache for caching. Often, time spent interacting with these databases can make up 25% or more of a response.

In addition, as a Ruby on Rails performance consultant, I’ve seen so many problems with application deployments that have nothing to do with the CPU performance of the framework or language: poor server configurations, memory leaks or bloat, or poor use of caching. Programmers, mysteriously, seem to find a way to completely degrade the performance of their application all on their own!

Finally, most mature web applications spend at most 50% of their execution time in the framework itself, and far more time in the actual application code and other added dependencies. This is pretty easy to see in Ruby - take a look at a stacktrace and count how many of the top frames are from your framework. It won’t be many. If your application could be rewritten in a faster framework in the same language, you would halve its response times at best.

Rewrite Your Entire Application to Save $1,000/month

What I worry about is what people do with the information presented in relative benchmarks like TechEmpower. Do they go home and rewrite their applications in the flavor-of-the-week framework or stack? Or, when choosing a stack for a new product or service, do people choose the “faster” stack over the “slower” one?

Heck, Pinterest rewrote it’s Ads API in Elixir and now they have response times of less than a millisecond. Surely, that’s just better, right?

The question is, why? As we’ve already established, there’s no difference for the end-user experience. So there’s really only two reasons to choose a framework over another: a) it’s faster and therefore I’ll spend less on server costs to host it b) it’s easy to develop with, and helps me ship quality features faster.

Let’s take a look at that server cost one, for a second.

The majority of web applications handle far less than 1000 requests per second. I’d go as far as to say that most web application developers are employed by a company whose entire webapp does far less than 1000 requests/second. Most of them do less than 1000 requests/minute.

Let’s say you have a Rails application which serves 20,000 RPM (request/minute, or about 300 req/sec) at an average response time of 250 milliseconds. That’s a pretty average profile for a large, mature Rails application. Such an application will take about 200 Puma processes to serve properly. That’s equal to roughly a dozen Performance-L dynos on Heroku, or $6,000/month.

Now, let’s say you rewrite it in Phoenix, Node, or whatever flavor of the week you want and reduce that to 125 milliseconds. Before you jump out of your seat, remember that you’re not going to reduce latency to 12 milliseconds or some other stupid-low amount: you’re still going to be limited by I/O to the databases that back this application.

Halving our application’s latency means we need about half the amount of servers we needed before. So, congratulations: you rewrote your application (or chose your framework) to save $3,000/month. The load on the relational database backing this application won’t change, so those costs will remain the same. When your application is big enough to be doing 20,000 RPM, you will have anywhere from a half-dozen to even fifty engineers, depending on your application’s domain. A single software engineer costs a company at least $10,000/month in employee benefits and salary. So we’re choosing our frameworks based on saving one-third of an engineer per month? And if that framework caused your development cycles to slow down by even one third of a mythical man-month, you’ve increased your costs, not decreased them. Choosing a web framework based on server costs is clearly a sucker’s game.

Why cargo-cult engineering practices from huge companies where a few milliseconds can save tens of thousands per month? You’re not Pinterest (or Netflix, or…), you have different problems, and that’s OK.

It Isn’t Getting Worse

Computers aren’t getting slower. While Wirth’s Law⁴(Software is getting slower more rapidly than hardware becomes faster.)⁴ Software is getting slower more rapidly than hardware becomes faster. certainly holds for most end-user applications like your mobile phone apps, it doesn’t really hold for your typical web application. Ruby web applications (and any web application) will continue to get faster because the slow grind of progress in hardware will continue to find ways to jam more CPU instructions into a clock cycle, or to make those clock cycles even faster, or to cram more cores onto a die.

And the language isn’t getting slower, either. Noah Gibbs of Appfolio has shown that each minor version of Ruby decreases average response times by about 5-10%.

Let’s Talk About Happiness

The performance doomsayers have always been wrong, and will continue to be wrong. Take this gentleman from 2007:

No matter what implementation becomes the next de-facto Ruby platform, one thing is clear: People are interested in taking advantage of their newer, more powerful multi-core systems (as the recent surge in interest in Erlang in recent RailsConf and RubyConfs has shown). As Ruby becomes increasingly part of solutions that deal in high volumes of data processing, this demand can only increase.

Ten years later, and scaling across multiple cores through preforking webservers like Puma and Unicorn is still plenty Good Enough. Ruby still isn’t dead. I’m excited for the possibilities afforded by the proposed Guild model, but is the language unusable until then? Nope.

What I want is for the conversation around web frameworks and programming languages to change. There’s too much talk of performance and concurrency, when in reality the margins are narrow and the costs minimal and getting lower. Languages aren’t dying based on their concurrency or performance features alone.

The better conversation, the more meaningful and impactful one, is which framework helps me write software faster, with more quality, and with more happiness. I know what the answer to that question is for me, and maybe the answer is different for you.

“Polyglots” and “The New Hotness Stack”

There’s a subset of engineers who will never be happy writing software which isn’t on the “new hotness stack”. Engineers are always looking for a new problem to solve, something new to learn - and that’s great! I’ve never related. GORUCO, the NYC Ruby conference started calling itself a “polyglot conference” this year, and the speaker schedule features talks on Python, Elixir, Rust, React and static typing. Conference organizer Mike Dalessio’s blog post announcing this reads like a tombstone.

Benchmarks are often waved around in this “is X dead” discussion. As I hope I’ve shown above, there really is no benchmark which can prove that any language or framework is not suitable for writing web applications. Performance isn’t the concern.

Instead, the performance discussion regarding web applications is mostly FUD, spread by those trying to justify the engineering time they just spent rewriting their entire stack or what they’re telling management so that they get to play with the coolest new toy they saw on Hacker News.

Programmers are perpetually terrified of career obsolescence. Some are afraid of intellectual stagnation - that they’ll become the crusty old person in the back office writing RPG to keep a truck parts company’s order system running. But almost all of them are afraid of unemployment. They’re worried that the world will move on from their particular stack, leaving their salaries and jobs in jeopardy. These fears are real - but let’s realize that most of the discussion around “is stack X dead?!” are driven by fear, not concerns for the requirements of web applications.

Fun and Games

Let’s be clear - performance still matters. Most organizations can and should save on server costs by focusing on speeding up their endpoints, and particularly slow endpoints probably do impact the customer experience or the bottom line and should be sped up. What I’ve talked about above is just how little framework choice matters in the performance of your web application.

Also, I’m not ragging on TechEmpower. It’s a massive project, and they depend on domain experts creating PRs that fix any problems with the results. They’re genuinely good people in my opinion, and aren’t trying to push an agenda or participate in benchmarketing in favor of any particular stack.

In conclusion, JavaScript, Go, Elixir and Python all suck, write Ruby :) No, of course not - write what you’re productive in. If you’re a web programmer, count your lucky stars that you get to choose your tools based on ergonomics, not on performance.

The highest rule of computing: computers SHOULD exist to accommodate their creators, never the other way around.
— Gary Bernhardt (@garybernhardt) June 28, 2017