Speedshop - Ruby on Rails performance consulting

Organization for Transformative Works Performance Audit

Fri, 29 May 2026 00:00:00 +0000

What follows is the complete text of an audit report I produced for the Organization for Transformative Works, a non-profit which was for many years run one of the largest fan-fiction archives on the Internet.

This was produced as part of my Ruby on Rails performance retainer service. Since the app is open source, OTW has kindly agreed to let me publish the report online for all to see.

The Archive Of Our Own, also known as “AO3”, is a Rails application that is more than 16 years old. It was the focus of my engagement.

Without any further ado, here’s what I handed over to OTW and their crack team of volunteers. I think I am one of a very, very small number of people who has ever been paid to work on AO3. The sheer amount of volunteer love and sweat in this project is truly astounding.

The Report

It’s an honor to work on behalf of such an engaged community like the one surrounding AO3. It’s also a tremendous honor to be one of the few people (maybe ever?) to get paid to work on this application. What you’ve accomplished with the volunteer time you have is truly incredible.

For some context, AO3 is certainly one of the largest scale applications I’ve worked on in terms of requests per second. Certainly, in terms of just requests per dollar spent on the server rack, it’s one of the most efficient I’ve ever worked on.

I see three main areas for improvement:

Removing footguns which could potentially result in incidents in the future
Make the app feel faster for your users
Targeted reduction of technical debt

To do this, I strongly suggest tracking and improving these two numbers:

The number of active, non-idle database connections to the primary.
Controller’s p95 web transaction service time.

AI Disclosure: In compiling this report, I made extensive use of LLMs for research. I did not use LLMs to generate any of the text in this report.

This document is organized at the top level by our desired Outcomes, which are my goals for your performance improvements over the next six months. Underneath that are specific Recommendations to achieve those outcomes. Each Recommendation has an associated cost and benefit, rated subjectively on a 5 point scale.

Ratings are designed mostly to be relative to each other, i.e. a 5 is always harder or more valuable than a 4, etc. Even cost/benefit roughly means I think it’s a toss-up whether or not you should do it, while a cost higher than the benefit rating means I think it’s not worth doing in the near term future.

I hope you enjoy this document and find it a useful guide for the next 6-12 months of performance work on your application.

Outcome: Remove Footguns

In the intro, I said you should focus on reducing the number of active, non-idle database connections to the primary. I see this as your primary scale bottleneck. If that number consistently stays at or below ~1/2 the number of CPUs on the DB (which, IIRC, was like 196 or something?) you’ll be golden! Easy as that, right?

This section is mostly about reducing opportunities for downtime (though not all directly relate to that DB metric).

Recommendation: Remove puma_worker_killer. Cost: 1 Benefit: 2

You have puma_worker_killer in the Gemfile. There’s no other config in the gem for it, but this gem is frequently configured with env variables.

PWK’s job is to SIGTERM processes which go over a particular memory threshold. This sounds like a nice-to-have “safety” feature, but it’s a footgun in waiting.

Several years ago I was consulting on a client project and the app was just really slow. I was looking at the response times and thinking, this app is about twice as slow as it probably should be. What’s going on? We removed puma_worker_killer and watched response times drop in half instantly.

It turned out that someone had installed and configured puma_worker_killer years ago and had forgotten about it. As the app grew over time, it continued to work, but the baseline memory usage of the app just kept increasing, leading to more and more frequent process restarts triggered by puma_worker_killer. In the end, it turned out Puma processes were processing about 20 requests on average before being restarted.

Old processes are fast processes. We want processes to stick around for as long as possible.

puma_worker_killer combines two things which are almost always very difficult to observe as an SRE:

Configuration in ENV variables. ENV variables are almost always forgotten about. No one knows what they are and what they are set to, because they often contain secrets. Access to them is restricted as a result, and they’re not easily checked or in the front of anyone’s mind.
The “failure mode” is high process restarts, which is almost never properly observed. If PWK’s thresholds are set too low, the only way you’re going to find out is if you look at how often process restarts are happening and think: hm, that’s a bit high (which, you may not even realize after looking at it). Your app could be restarting after serving only a single request and you might not notice it, if load is low enough and you’ve provisioned enough puma processes.

puma_worker_killer is sort of a relic from a Heroku-dominant era, where you needed to fit processes into teeny-tiny dynos of like 512MB. Nowadays, most people buy cloud hardware with 4GB of RAM per CPU, and an application which cannot fit within that constraint reliably is far more sick than something puma_worker_killer can even address.

For that reason, I think puma_worker_killer should be removed from the setup.

Recommendation: Change send_data to a background job powered workflow. Cost: 2 Benefit: 3

You have a couple places in the codebase where you’re using send_data to send files of fairly significant size down to the client:

DownloadsController#show
send_csv_data helper, used by ChallengeSignupsController, TagWranglers controller, AdminUsers controller.

There are two big issues with what’s happening here.

Memory usage. You’re loading ~several mb of stuff into memory at once to put some of these CSVs together. In a web process, that memory usage sticks around and is difficult to get rid of. In a background process, usually this kind of “dirtiness” in the heap is already occurring so you’re not making things much worse, and it’s in general easier to stop and restart background job processes.
Transmitting large files to the client is slow Ruby web servers aren’t really designed for this. Puma does not buffer client responses and so you’re essentially locking up the thread for as long as it takes to stream this response back to the client. It’s worse for single-threaded situations (which you’re in I think, for web today).

Both of these I think could get out of hand and cause some minor incidents.

For me, the best path forward would be to reorient the whole thing around a background job:

Controller kicks off a job to generate the CSV.
Controller returns to the web client a URL and says “your CSV will be here when it’s done”.
Background job does the CSV work and uploads to S3.
S3 serves the CSV.

This would solve both problems for you at once. You could optionally try to work to stream the CSV upload as well to reduce memory usage in the worker, but honestly I think just getting it out of the web process is enough.

Recommendation: Install some kind of database monitoring product Cost: 1 Benefit: 3

I think of observability stacks for Rails app as having three main components:

Infrastructure. Most of the time this is coupled to your provider, e.g. AWS Cloudwatch, but can include 3rd parties like Datadog.
APM/RUM. Options include Sentry, Datadog, New Relic, etc.
Database. Options include pganalyze, pghero, Datadog’s Database Monitor.

I feel like we’re missing the last one, particularly for an app like this where performance and stability is so sensitive to DB conditions.

I’m flying a bit in the dark here because I think there are really good options here for Postgres but less so for InnoDB/MySQL. It is, of course, possible to use only parts of Datadog and not the whole hog, and I believe DD’s database monitoring is like $70/80 a month. I think the only open source option here is Percona PMM? But I’ve never used it and can’t say if it’s any good, and it doesn’t have an index advisor.

A good database monitoring solution does the following:

Index suggestion. Based on actual query data, can do things like gauge how much impact a new index has on writes, how big it would be, what queries it would alleviate.
Capture and visualize EXPLAIN plans.
CPU, Memory, active connection count metrics.

I’m not really sure what the best direction to go here is, but I was definitely feeling the absence of this tooling when preparing this report.

Recommendation: Move load to readers with queues and GET requests. Cost: 3 Benefit: 3

I see a lot of people create replicas and then basically struggle to direct any meaningful usage to them. You’re bottlenecked pretty much entirely by your SQL primary, so moving more usage to them would be helpful.

I’ve only seen two strategies really work here.

The first is to create read only job queues. These are job queues which are duplicates of an existing queue, like high, but only allow jobs in that queue to talk to the replica database using ActiveRecord’s multi-db/role support.

So, instead of just a utilities queue, you might also have a utilities-read-only queue. You’d have an around_perform hook in ApplicationJob that does something like:

def use_replica_if_read_only
  if read_only_queue?
    begin
      ActiveRecord::Base.connected_to(role: :reading) { yield }
    rescue ActiveRecord::ReadOnlyError
      self.class.set(queue: writable_queue_name).perform_later(*arguments)
    end
  else
    yield
  end
end

def read_only_queue?
  queue_name.include?("read-only")
end

def writable_queue_name
  queue_name.gsub(/[-_]?read-only[-_]?/, "")
end

This pattern makes it easy to transition significant amounts of background jobs to only touch the reader.

Second, you can make GET requests only talk to the reader. In theory, GET requests should not have any side effects or writes, so this should be possible. In addition, Rails makes it easy to “read your own writes”:

Rails guarantees “read your own write” and will send your GET or HEAD request to the writer if it’s within the delay window. By default the delay is set to 2 seconds. You should change this based on your database infrastructure. Rails doesn’t guarantee “read a recent write” for other users within the delay window and will send GET and HEAD requests to the replicas unless they wrote recently.

I took a brief look around and don’t see a lot of major issues with GET side effects. Lots of stuff writes to Redis but obviously that’s not a problem.

Recommendation: Improve the ES failure handling. Cost: 2 Benefit: 4

Currently, the app has a pretty hard failure path when Elastic goes down.

If ES is unreachable, searches, connection refusal, or brownouts/timeouts end up in 500s, which will cascade and take down the app as the service degrades. It’s my impression that this has happened in the past.

I think it could be instructive to look at how the Rubygems.org codebase tends to hold Elasticsearch much more at a distance, which means the service is a lot more resilient to ES outages.

Here’s a few things I think you could improve:

Add a query timeout in every query body.
Add a tighter client timeout - probably something like ~2s. I saw a few timeouts in the Sentry data that showed 30s timeouts in use. If you don’t get a response in 2 seconds, you’re not gonna get one in 30.
Add more layered error handling. When you catch a connection failure, timeout or other Elasticsearch error, return what you can and the rest can 200.

The app current treats ES as implicitly available, which I think the past shows is not a great assumption. If ES was treated as a non-critical read-only feature, you could degrade the site into a no-search mode that I think would greatly improve your overall uptime.

I would also install timeouts in queries and clients so that brownouts affect you far less - better not to tie up a Ruby process for that long.

Recommendation: Move audits to a different database or truncate. Cost: 1 Benefit: 2

Like most people who have a “paper trail” feature in the database, I think you’re just starting to feel some pain on this one, given the size of this table.

It’s unclear to me what your obligations are regarding how long these audits need to be kept. Obviously, the easiest thing to do is have a policy to only keep them around for short periods of time and dump the rest.

Multi-db has a number of weird risks that just aren’t that important for audits: transactional integrity, cascading deletes. The “best-effort” compromises you’ll make by doing multi-db here I think are fine for the use-case.

If you can’t do that, I think audits make good sense to move to a separate database. You only read from them rarely, and the read performance isn’t a big deal for them (since only admins can view) so any performance loss on “I can’t JOIN natively on these tables anymore” is basically not an issue.

Since user model changes and admin activity updates don’t happen that often, I kinda doubt there’s much write load being added here. It would just be an improvement to the overall database size, making it easier to backup and possibly improving cache hitrates a little.

Recommendation: Shard a “works” database (workers, chapters, kudos) by work_id, and “activity” database (readings, inbox_comments) by user_id. Cost: 4 Benefit: 5

You asked me in Slack about shards.

I do see two pretty clear groupings of models that could shard:

Work-owned objects, like workers, chapters, kudos.
Activity objects, owned by users, like readings, inbox_comments.

You could potentially split comments as well into two models, one for comments on Works and the other for comments on tags and admin posts somewhere else. The polymorphism means it wouldn’t work to shard in my scheme today.

There’s still some important relations though that wouldn’t survive this scheme:

Reading.visible joins directly between works and readings.
Work.update_stat_counter might need a rewrite if you don’t also shard comments.

It would also mean you’d need to probably do some kind of event/async system to get things to sync across databases. Comments created on a work would have to emit an event or job which then means inbox rows get created for users, etc. This is kind of already how Reading creation works via SADD, so maybe this isn’t that hard actually? And since you already cache so much, perhaps moving the source of those caches from 1 db to several just doesn’t feel that painful in the view layer?

Of course it introduces eventual consistency, but I don’t think that’s actually a huge concern, given what I’ve seen in the app’s usage?

It may be that audits are the easiest way to get your feet wet on ActiveRecord multi-db, and then readings and inbox_comments in a new (sharded) db are the second step.

Outcome: Top controllers, by traffic, should all have a p95 of less than 1 second.

I think the second most important thing I’d like to accomplish, beyond just keeping the website up more, would be to make it feel faster for your users.

Currently, Sentry reports the overall p95 of all web transactions as 200ms, which is actually quite good. However, today, you have several transactions where the p95 is significantly higher:

BookmarksController#index, 2.58 seconds.
AutocompleteController#tag, 2.98 seconds. Most autocomplete controller actions have similar p95s.
DownloadsController#show, 7.72 seconds

These three are what I would like to focus effort on, because the rest feel pretty good (or at least, additional effort here would not cause significant user-noticeable gains in performance).

Recommendation: Swap Bullet for Prosopite, default to fail in tests. Cost: 2 Benefit: 3

I’ve never been all that high on Bullet, but recently I’ve had a lot of success with prosopite. It’s highly accurate, with basically zero false positives. That means you can set up a workflow where Prosopite is configured to raise by default in the test environment. That means you can’t ship new features with proven N+1s! That’s an extremely powerful thing to be always checking for in the background without any additional effort.

Usually what I do with clients is work along the following axis:

Install prosopite, log in dev and test.
Add prosopite as an RSpec context tag, like :check_for_n_plus_ones. Add it to all test classes which don’t have N+1s in them today (add them everywhere they wouldn’t fail a test today).
Gradually burn through the remaining bad examples/classes, adding the context one by one, until you have just a few left. At that point, you can…
Make prosopite raise by default in test and remove the remaining N+1s as global ignores in prosopite’s config.

It’s extremely useful!

Recommendation: Turn YJIT on. Cost: 1 Benefit: 2

You have YJIT disabled in application.rb. I recommend trying it. Pretty consistently, what I see is that it makes that app ~10-15% faster for 20-30% higher memory use.

You’ll have to look at your current server provisioning to decide if that’s worth it for you.

If you have any memory left, might as well use it and reduce load on your servers by ~10-15%. If you don’t have any memory left, then I guess you’re stuck with keeping it off.

In my experience, Ruby 3.4+ does quite well with YJIT on. Of course it’s also quite stable, since basically everyone has it turned on now.

Recommendation: Refactor Rails.cache.fetch into view-layer fragment caches. Cost: 2 Benefit: 3

Looking through the traces, my overall picture is that you are over-using cache, peppering sometimes hundreds of cache calls all over the place. I can see how you got here. You’re trying to remove load from the SQL database and put it somewhere else. But from the perspective of total latency of the entire transaction, you haven’t improved things much (Redis isn’t magically faster than a hot, in-the-buffers hit from a MySQL db).

If you look at, for example, _intro_module.html.erb, there’s several cache calls happening right in a row.

I think this could be improved by focusing less on Rails.cache manual usage, but instead focusing on more cache (and cache-reuse) of entire HTML fragments at the view layer. You’re caching more activity (not just the query but also the HTML generation) and also across potentially multiple queries and datasources.

Another example: on the homepage, for each marked for later work, there are 5 cache round trips to render the blurb, even though the fragment itself is already cached. If the entire readings section was single fragment keyed per-user, you could turn 30 cache ops into 1.

Recommendation: Use Turbo Drive or Turbo Frames. Cost: 4 Benefit: 5.

I harp on this one on social media a lot but it’s true. The biggest performance impact you can have on a web app user is to turn a full page navigation into an SPA-style route change. For “golden path” Rails apps like AO3, that means using Turbo.

The reason why these kinds of requests are so much faster is because the CSSOM and Javascript VM are re-used. You don’t need to completely relayout everything, recalculate the CSSOM and re-execute all your JS. You can just move on! It’s truly the only thing that remove seconds from a user waiting on the page to do something, rather than milliseconds.

For AO3, I think the most realistic transition would be to use Turbo Frames. It asks the least amount from the current setup in order to make the transition. I’ve done this migration myself on a legacy ActiveAdmin panel with lots of custom JQuery stuff, and I was able to make it happen pretty quickly.

A more ambitious transition is Turbo Drive. I find that pretty much every jQuery plugin’s assumption about how pageloads work will break, however. It would require an extensive inventory of all your JavaScript and the behaviors you expect to work, and probably adding a lot more tests for that level of integration/browser behavior than what you have today.

It’s a lot of work, but there’s really nothing that makes pageloads faster. If you can turn a full-page-nav into something else, people really feel it.

Recommendation: Turn on Cloudflare Polish Cost: 1 Benefit: 3

Cloudflare Polish automatically serves webp and avif formatted images to compatible clients. You have Cloudflare on, but Polish is not.

These image formats can save ~10-50% filesize, depending on the image. I love it, it’s probably one of the easiest frontend optimizations you can make. Underneath, since it’s working based on the Accept header, it literally can’t break anything for anyone because the client has to tell you they accept the format before Cloudflare decides to serve it to them!

I recommend the lossy setting because in practice I haven’t seen any visible degradation.

Recommendation: Change CSRF policy to :header_or_legacy_token Cost: 2 Benefit: 3

You’ve got this somewhat-unusual token_dispenser.json route which you use to generate a CSRF token, which then gets provided to any form that needs it. You need to do this because you want to be able to cache logged-out pages with HTTP, which means of course you can’t put a CSRF token in <head>.

Since Rails 8.2, you now have the option to change how CSRF authentication works.

In Rails 8.2, you can configure Rails to check the Sec-Fetch-Site header, which is sent by browsers on every request.

From MDN:

this header tells a server whether a request for a resource is coming from the same origin, the same site, a different site, or is a “user initiated” request. The server can then use this information to decide if the request should be allowed.

Possible values:

cross-site The request initiator and the server hosting the resource have a different site (i.e., a request by “potentially-evil.com” for a resource at “example.com”).

same-origin The request initiator and the server hosting the resource have the same origin (same scheme, host and port).

same-site The request initiator and the server hosting the resource have the same site, including the scheme.

none This request is a user-originated operation. For example: entering a URL into the address bar, opening a bookmark, or dragging-and-dropping a file into the browser window.

So, Rails allows updates from same-origin or same-site, disallows all cross-site (unless in Rails’ trusted_origins) or header missing (except GET of course). Since browsers themselves set Sec-Fetch-Site, it cannot be forged.

With the header, the following all becomes unnecessary:

HomeController#token_dispenser
updatedCachedTokens()
loadedCSRF
csrf_meta_tags in layouts, authentication_token in forms

Of course, this is not supported in every browser ever.

It is supported in:

Chrome 76+ (2019)
Firefox 90+ (2021)
Safari 16.4+ (2023)
Edge 79+ (2020)

I don’t think AO3 has any official compatibility targets (e.g. https://browsersl.ist/) (maybe it’s time to officially come up with some?).

You can test this out by switching to header_or_legacy_token, which will allow browsers which present the header to do so.

The “feature check” though on the browser side is a bit weird. How do you know if you’re sending that header or not? You don’t! So, the approach (if you can’t drop incompatible clients) would be to probably modify the token dispenser so that after the first request, it notices if the browser sent Sec-Fetch-Site, and if it did, it tells the frontend “hey, you don’t have to come back for any more CSRF tokens. Just present sec-fetch-site.” I think you’d probably want to store that in the session cookie or something.

The primary benefit I see here would be then that form actions can happen immediately without waiting for CSRF tokens on a network roundtrip.

Recommendation: Reduce cache fetches on Bookmarks#index Cost: 3 Benefit: 3

I took a look at a random Bookmarks#index trace and saw the following:

179 cache spans
148 reads
31 writes

That’s pretty high. Ideally we only go to the cache a handful of times. These were split between the MemcacheStore and LocalStore. Since the MemCacheStore is the only one that costs us $$$ to use and has network latency, I’ll focus on that.

Reads: 114
Hits: 84
Misses: 30
Hit rate: 73.7%

That’s a pretty poor hitrate. If we look in a bit more detail, here’s how that breaks down by key shape:

ao3-v8.0:views/bookmark-blurb-bookmarks/…-v3
- 20 reads
- 19 hits
- 1 miss
- 1 put
ao3-v8.0:…/blurb_css_classes
- 20 reads
- 20 hits
ao3-v8.0:…/bookmark_count
- 40 reads total
- 18 remote misses!
- 22 local hits
- 18 puts ### Work metadata / fragments
ao3-v8.0:byline_data/…
- 17 reads
- 17 hits
ao3-v8.0:/v4/work_blurb_tag_cache_key/…
- 7 reads
- 7 hits
ao3-v8.0:views/works/…-showwarn-showfreeform-v11
- 7 reads
- 7 hits
ao3-v8.0:views/works/…/stats-v4
- 7 reads
- 2 hits
- 5 misses
- 5 puts
ao3-v8.0:works/…/count_visible_comments
- 5 reads
- 5 misses
- 5 puts
ao3-v8.0:works/…/kudos_count-v2
- 5 reads
- 5 hits
ao3-v8.0:/v1/public_bookmarks_count/…
- 5 reads
- 5 hits

I basically see these as a kind of “fast” N+1 problem. Ideally I think we’d only be going to each keyspace one time. It’s a sign of an “insufficiently Russian-dolled-cache”. I think it also gets back to the observation I had earlier that you are in a lot of caches replacing single database queries with single cache lookups, which doesn’t do much for you.

I’d try to roll these up a single bookmark_row fragment, which should be able to wrap around ~8 of those keys (kudos, blurb, views, byline data, etc etc). Then you can MGET for all the bookmarks you want.

It’s also possible the cache keys like byline_data and blurb_css_classes are just too small to help to begin with and should be removed completely.

I think this transition is difficult because it requires a deep domain understanding of the cache keys and ERD involved here, but it doesn’t look impossible to me.

Recommendation: Always preload current_user roles. Cost: 1 Benefit: 2

I noticed a lot of Sentry traces with roles lookups peppered all over the place. You already do the right thing in permit_yo:

def has_role?(role_name)
  return self.roles.any? { |role| role.name == role_name.to_s } if self.roles.loaded?

  role = Role.find_by(name: role_name)
  self.roles.include?(role)
end

So that’s good. The fact that these queries are happening then is a sign that the association wasn’t loaded.

I’ve encountered this permissions thing a ton of times. I think it’s almost always worth it to just load all user roles upfront rather than try to be clever about which you load when.

Since you’re using Devise, you override serialize_from_session:

def self.serialize_from_session(key, salt)
  record = includes(:roles).where(primary_key => key).first
  record if record && record.authenticatable_salt == salt
end

There are also a handful of places in views where you inspect user.roles directly, e.g. @user.roles.any?. I’d replace all these with has_role? and then make sure the association is loaded.

Recommendation: Timeout Autocompletes, add a hard LIMIT, and consider LRU caches around common 3+ char words. Cost: 2 Benefit: 4

AutocompleteSource.autocomplete_lookup is a very performance sensitive method that powers all the various autocomplete controllers. It routinely has a p95 of 2sec+.

There are a couple of things I’d do to improve it:

Add a hard LIMIT on zrevrangebyscore for 3+ character terms.. Currently the ZREVRANGEBYSCORE can take 2 seconds or more when the search term is something really common. Adding a limit: [0,50] here would, I think, really help bring that P95 down a lot, because the score key of autocomplete_tag_all_score_some_popular_word could be thousands of tags, which you then just truncate down to 15 in the end. Ouch.

Get a small LRU cache together for popular 3+ character words. After trying the previous fix, if I still wanted more “juice”, I think I’d try to create a separate CacheStore with a given size limit (so that I can use LRU behavior to only cache “the most important” stuff) for 3+ character search parameters. I think this could even be process-local using MemoryStore and a size setting of like 10.megabytes.

Add a timeout to these queries, particularly the zrevrangebyscore. One thing I’m always thinking about with Redis is that it’s single-threaded. It’s really not designed to service long-running queries, because those queries may end up blocking other concurrent operations. Unlike a SQL database, which parallelizes quite well across CPU cores, your Redis DB is just locked to a single core and so a very slow query can end up affecting others.

I would also consider a circuit breaker and/or open timeout to the autocomplete Redis. If it goes down, the site should be able to quickly recover and not go down completely.

Recommendation: Use a test queue to make builds faster, more reliably fast Cost: 2 Benefit: 2

You currently have 13 Cucumber jobs working as explicit directory splits, and 3 RSpec jobs also statically configured. So, if you get unlucky and one of those Cucumber jobs takes 14 minutes, well, that’s how long the build takes. There’s no balancing happening there.

I tell basically everyone these days to balance tests between workers using a queue. This allows the maximum amount of redistribution to occur, so that the minimum and maximum total execution time per worker (process on a single-machine approach, or per agent/runner/VM as well) to be as close together as possible.

The “enterprisey” way is to pay for Knapsack Pro. However, there are a number of open source projects that can do similar stuff.

I’m going to take as a constraint here that we have to use the default, free Github Actions Runner hardware. Those machines have 4 vCPU and 16GB of RAM per.

test-queue by the prolific tmm1, author of stackprof. I’m pretty sure the networked mode wouldn’t work in Github Actions, but the fork-based model would. That would allow you to use the remaining 3 idle cores on each Github Actions worker. Has cucumber support though maybe it’s not very heavily used.
parallel_tests by grosser. Popular, I’ve seen this one in use a few times. Not networked, but supports Cucumber.
spec-wrk by Daniel Westendorf. No Cucumber support here but does support networked runners in Github Actions. Could be cool if you added a Cucumber adapter!

With some networked/parallel runners I think you could easily take builds down to 5 minutes or less.

Recommendation: Remove resque inline in tests. Cost: 2 Benefit: 3

You have the following in your resque config:

  Resque.inline = ENV["RAILS_ENV"] == "test"

And in test.rb:

config.active_job.queue_adapter = :inline

So in tests, all Resque enqueue calls execute synchronously.

In my experience (which is more with the equivalent setting in Sidekiq), this ends up creating two big problems for you:

Tests now have a ton of extra state and state modification flying around that’s hard to reason about because it’s all implicit. This makes the tests harder to understand.
You will also be doing a lot of work which is not necessary to make the assertions pass.

For both of those reasons, I prefer that any background queue draining be done explicitly instead of implictly. If you use the traditional arrange/act/assert framework, I think tests are much clearer if you include these queue drain/execute job calls in the Act or Arrange phase.

It looks like you kind of already started to do this with suspend_resque_workers in spec_helper.rb.

For ActiveJob, switching is easy. You just change the queue_adapter to :test and then drain only when necessary. Some specs already are written this way.

Resque doesn’t really have a Sidekiq fake mode built in. I think we basically could extend suspend_resque_workers to be the default.

Outcome: Reduce technical debt

These are just a couple of things I noticed as I was looking around.

Recommendation: Complete the migration to Puma. Cost: 1 Benefit: 1

Of course I’m gonna say this, as I’m the maintainer. But I really do think it’s a great application server! It looks like you’re about halfway to transitioning.

Moving from single-threaded Unicorn to single-threaded Puma should just be a “straight swap”. When you use the servers in this way they don’t really differ that much in terms of behavior.

I think probably this application never becomes multi-threaded. It’s probably just more hassle than it’s worth. I usually tell people: what changing your concurrency model gives you (from processes to threads, threads to fibers, etc) is allows you to run more open, idle connections with low switching cost and low memory use. If your workload doesn’t have that need, process based concurrency probably works just fine for you. Almost every major Rails app, the real giga-shops like Shopify, Github, Intercom, Gusto: they’re all running single-threaded. They decided the threading bugs aren’t worth the ~30% memory savings. I suspect, given the limited volunteer resources and the age of this app, that the calculus is the same for you.

Recommendation: load_admin_banner doesn’t need caching. Cost: 1 Benefit: 1

This is just one of a couple of places I noticed unnecessary caching. From what I can see in the traces, this takes about 1ms whether or not it’s hot or cold. Since it’s on every page load via before_action, that stood out to me.

  before_action :load_admin_banner
  def load_admin_banner
    if Rails.env.development?
      @admin_banner = AdminBanner.where(active: true).last
    else
      # http://stackoverflow.com/questions/12891790/will-returning-a-nil-value-from-a-block-passed-to-rails-cache-fetch-clear-it
      # Basically we need to store a nil separately.
      @admin_banner = Rails.cache.fetch("v1/admin_banner") do
        banner = AdminBanner.where(active: true).last
        banner.nil? ? "" : banner
      end
      @admin_banner = nil if @admin_banner == ""
    end
  end

Fetching the exact same row every time in SQL is just not any slower than reading it out of Redis. Just a bit of unnecessary complexity.

Recommendation: Move JQuery/JQueryUI to first party serving. Cost: 1 Benefit: 1

I noticed you’re serving the following:

https://ajax.googleapis.com/ajax/libs/jquery/1.9.0/jquery.min.js
https://ajax.googleapis.com/ajax/libs/jqueryui/1.10.0/jquery-ui.min.js

I think in general, for a site with the kinds of concerns about privacy that AO3 has, you generally just want to “talk” to as few third parties as possible. So just for user privacy reasons, I think it’s better to serve these first-party. And they’re like 15 year old versions, so who knows, maybe one day Google axes them.

It used to be thought in the frontend communities that 3rd party CDNs were a performance “boost” because “oh, everyone will use the same CDN cached version of this library” but that became less true over the years as people realized this could be used to fingerprint clients and so browsers ended up changing how caching worked and breaking this behavior completely. And of course, not a lot of people use 13-year-old versions of jQuery anymore.

Recommendation: Remove active_record_query_trace and replace with native Rails verbose query logs Cost: 1 Benefit: 1

You currently have three overlapping query source trace mechanisms.

active_record_query_trace gem. Appends a colorized multi-line call stack after each query in dev logs.
config.active_record.verbose_query_logs = true. Rails 5.2+ built-in, appends ↳ file:line after each query
config/initializers/active_record_log_subscriber.rb — a custom LogQuerySource module that also prepends ↳ file:line

While I love this feature and of course recommend everyone use it, it is kinda baked into Rails now and I prefer to use the Rails mainline stuff where I can rather than bring in dependencies. Probably we only need one of these right?

It is a little bit of a bummer but verbose_query_logs only displays a single line:

Before (with gem):
User Load (0.4ms)  SELECT "users".* FROM "users" WHERE ...
  app/models/concerns/is_active.rb:11:in `active?'
  app/models/user.rb:67:in `active?'
  app/controllers/users_controller.rb:42:in `index'

After (without gem):
User Load (0.4ms)  SELECT "users".* FROM "users" WHERE ...
  ↳ app/controllers/users_controller.rb:42

I think you can just keep verbose_query_logs and delete the other two.

Recommendation: Move to SLO queues Cost: 3 Benefit: 5

SLO queues are a concept introduced in my book, Sidekiq in Practice. It was adopted successfully at Gusto and the community at large has really picked up and run with the idea.

The basic idea is that all queues in your system are named after their queue time SLO - how long they promise that a job will be in the queue until it is a executed.

For example, a common setup is:

within_0_seconds
within_5_minutes
within_15_minutes
within_1_hour

You then launch any number of processes in Resque which listen to one of those queues (and one queue only).

This allows you to intelligently autoscale: we don’t need to start scaling up the within 1 hour queue until the latency of that queue is above ~45 minutes or so.

Using many queues which are based on just the “domain” or “subject” of the queue (e.g. imports, exports, orders, etc) leads to an explosion of queue types, whose SLOs are not clear. You don’t know when to page someone or wake them up because the queue time expectation is not clear.

For a volunteer-run shop, I think getting completely crystal-clear about how long it’s OK to wait in the queue until someone needs to be notified would be really helpful. It also makes it easy for app developers in the future to “sign up” for what level of queue time they need (it’s right in the name!).

Announcing the Rails Performance Apocrypha

Thu, 14 Jan 2021 00:00:00 +0000

Hello Rubyists!

Today, I’m launching a new product: The Ruby on Rails Performance Apocrypha.

Over the last four years, I’ve written a lot of stuff to this newsletter. Until now, none of that stuff has been publicly accessible if you wanted to go back and read it again. If something useful was posted to the newsletter before you subscribed, you were just sort of screwed.

So, I’ve compiled 4 years of writing to this newsletter into a book. It covers my usual topics: performance science and engineering, frontend performance, Ruby performance, and scaling. It’s a fun ramble around all of these topics with a lot of tidbits and useful information scattered about. Each chapter is quite short, so it’s easy to pick up and put down again.

It’s available now on Gumroad for just $10. As always, it’s DRM-free and available in PDF, e-reader and even HTML and plain-text formats.

I called this book the “apocrypha”, because I consider my “main-line” of Rails performance instruction, the canonical “scripture”, to be my Rails Performance Workshop. By contrast, this book is a bit of an all-over-the-place ramble, and it covers some things that I didn’t cover in great detail in my other instructional books and workshops, such as HTTP/2 resource prioritization, and a detailed how-to on how to use New Relic’s Ruby VM information.

Here’s the chapter titles, in case you’re wondering what’s covered:

What I Value
Performance Science
Why Performance?
You Are Not a Compiler
What does 10% faster really mean?
Benchmarks for Rails Apps
Build-your-own APM
Reading Flamegraphs
DRM: Database, Ruby, Memory
Performance in the Design Space
Microservices and Trends
On Minitest
Corporate Support for Ruby
Why is Ruby Slow?
Popularity
Stinky Dependencies
Why Cache?
Software Quality at Startups
Frontend
Simple Frontend Config Changes
What is TTFB?
Always Use a CDN
Page Weights and Frontend Load Times
Lazy-loading
What’s Resource Prioritization?
HTML on the Wire
Exceptions: Silent, not Free
On Thread-Safety
What is the GVL?
Timeslicing the GVL
The GVL and C
Bloat
Minimum Viable Rails
The Weird Setting No One Used
Object Allocation
You Should Always Use a Production Profiler
Reproducing Issues Locally
Worker Killers
What’s QueryCache?
Reading New Relic’s Ruby VM Tab
Test Setup
What is Time Consumed?
Request Queue Times
Amdahl’s Law
Threads
CPU-bound or IO-bound?
What is Swap?
Database Pools
Single-thread Performance
Read Replicas
Why Lambda?
Never use Perf-M
Daily Restarts

Check it out on Gumroad.

We Made Puma Faster With Sleep Sort

Thu, 17 Sep 2020 00:00:00 +0000

Puma 5 (codename Spoony Bard (When Puma gets a new ‘supercontributor’ that submits lots of important work to the project, we let them name the next release. This release features a lot of code from Will Jordan, who named this release ‘Spoony Bard’. Will said: ‘Final Fantasy IV is especially nostalgic for me, the first big open-source project I ever worked on was a fan re-translation of the game back in the late 90s.’)) was released today (my birthday!). There’s a lot going on in this release, so I wanted to talk about the different features and changes to give Puma users confidence in upgrading.

Experimental Performance Features For Cluster Mode on MRI

This is probably the headline of the release - two features for reducing memory usage, and one for reducing latency.

Puma 5 contains 3 new experimental performance features:

wait_for_less_busy_worker config. This may reduce latency on MRI through inserting a small delay (sleep sort!) before re-listening on the socket if worker is busy. Intended result: If enabled, should reduce latency in high-load (>50% utilization) Puma clusters.
fork_worker option and refork command for reduced memory usage by forking from a worker process instead of the master process. Intended result: If enabled, should reduce memory usage.
Added nakayoshi_fork config option. Reduce memory usage in preloaded cluster-mode apps by GCing before fork and compacting, where available. Intended result: If enabled, should reduce memory usage.

All of these experiments are only for cluster mode Puma configs running on MRI.

We’re calling them experimental because we’re not sure if they’ll actually have any benefit. We’re pretty sure they’re stable and won’t break anything, but we’re not sure they’re actually going to have big benefits in the real world. People’s workloads are often not what we anticipate, and synthetic benchmarks are usually not of any help in figuring out if a change will be beneficial or not.

We do not believe any of the new features will have a negative effect or impact the stability of your application. This is either a “it works” or “it does nothing” experiment.

If any of the features turn out to be particularly beneficial, we may make them defaults in future versions of Puma.

If you upgrade and try any of the 3 new features, please post before and after results or screenshots to this Github issue. “It didn’t do anything” is still a useful report in this case. Posting ~24 hours of “before” and ~24 hours of “after” data would be most helpful.

wait_for_less_busy_worker: sleep sort for faster apps?!

This feature was contributed to Puma by Gitlab. Turn it on by adding wait_for_less_busy_worker to your Puma config.

When a request comes in to a Puma cluster, the operating system randomly selects a listening, free Puma worker process to pick up the request. “Listening” and “free” being the key words - a Puma process will only listen to the socket (and pick up more requests) if it has nothing else to do. However, when running Puma with multiple threads, Puma will also listen on the socket when all of its busy threads are waiting on I/O or have otherwise released the Global VM Lock.

When Gitlab investigated switching from Unicorn to Puma, they encountered an issue with this behavior. Under high load with moderate thread settings (a max pool size of 5 in their case), average request latency increased. Why?

Remember, I said that the operating system randomly assigns a request to a listening worker process. So, it will never send a request to a worker process that’s busy doing other things, but what about a worker process that’s got 4 threads that are processing other requests, but all 4 of those threads happen to be waiting on I/O right now?

Imagine a Puma cluster with 3 workers:

Worker 1: 0/5 threads busy.
Worker 2: 1/5 threads busy.
Worker 3: 4/5 threads busy.

If Worker 3’s 4 active threads happen to all have released the GVL, allowing that worker to listen to the socket, and a new request comes in - which worker process should we assign the request to, ideally? Worker 1, right? Unfortunately, most operating systems will assign the request to Worker 3 33% of the time.

So, what do we do? We want the operating system to prefer less-loaded workers. It would be really cool if we could sort the list of workers listening on the socket so that the operating system would give requests to the least-loaded worker. Well, we can’t really do that easily, but we can do something else.

wait_for_less_busy_worker causes a worker to wait to re-listen on the socket if it’s thread pool isn’t completely empty. This means that in high-load scenarios, the operating system will assign requests to less-loaded workers.

This is basically sleep-sorting our workers. We’re kind of doing doing this:

[].tap { |a| workers.map { |e| Thread.new{ sleep worker_busyness.to_f/1000; a << e} }.each{|t| t.join} }

… and hiding “more loaded” workers from the operating system by letting less-loaded workers listen first!

Originally the proposal was for a more complicated sort - processes slept longer if they had more busy threads - but that was removed when it was found that a simpler on/off sleep was just as effective.

The net effect is that in high-load scenarios, request latency decreases. This is because workers with more busy threads are slower than workers with no busy threads. We’re assuring that requests get assigned to the faster workers. Prior to this patch, Gitlab saw an increase in latency using Puma compared to Unicorn - after this patch, latency was the same (they also were able to reduce their fleet size by almost 30% thanks to Puma’s memory-saving multithreaded design).

There may be even more efficient ways for us to implement this behavior in the future. There’s some magic you can do with libev, I’m pretty sure, or we can just implement a different sleep/wait strategy.

fork_worker

Adding fork_worker to your puma.rb config file (or --fork-worker from the CLI) turns on this feature. This mode causes Puma to fork additional workers from worker 0, instead of directly from the master process:

 \_ puma 5.0.0 (tcp://0.0.0.0:9292) [puma]
     \_ puma: cluster worker 0: 10000 [puma]
         \_ puma: cluster worker 1: 10000 [puma]
         \_ puma: cluster worker 2: 10000 [puma]
         \_ puma: cluster worker 3: 10000 [puma]

Similar to the preload_app! option, the fork_worker option allows your application to be initialized only once for copy-on-write memory savings, and it has two additional advantages:

Compatible with phased restart. Because the master process itself doesn’t preload the application, this mode works with phased restart (SIGUSR1 or pumactl phased-restart), unlike preload_app!. When worker 0 reloads as part of a phased restart, it initializes a new copy of your application first, then the other workers reload by forking from this new worker already containing the new preloaded application.

This allows a phased restart to complete as quickly as a hot restart (SIGUSR2 or pumactl restart), while still minimizing downtime by staggering the restart across cluster workers.

‘Refork’ for additional copy-on-write improvements in running applications. Fork-worker mode introduces a new refork command that re-loads all nonzero workers by re-forking them from worker 0.

This command can potentially improve memory utilization in large or complex applications that don’t fully pre-initialize on startup, because the re-forked workers can share copy-on-write memory with a worker that has been running for a while and serving requests.

You can trigger a refork by sending the cluster the SIGURG signal or running the pumactl refork command at any time. A refork will also automatically trigger once, after a certain number of requests have been processed by worker 0 (default 1000). To configure the number of requests before the auto-refork, pass a positive integer argument to fork_worker (e.g., fork_worker 1000), or 0 to disable.

nakayoshi_fork

Add nakayoshi_fork to your puma.rb config to try this option.

Nakayoshi means “friendly”, so this is a “friendly fork”. The concept was originally implemented by MRI supercontributor Koichi Sasada in a gem, but we wanted to see if we could bring a simpler version into Puma.

Basically, we just do the following before forking a worker:

4.times { GC.start }
GC.compact # if available

The concept here is that we’re trying to get as clean of a Ruby heap as possible before forking to maximize copy-on-write benefits. That should, in turn, lead to reduced memory usage.

Other New Features

A few more things in the grab-bag:

You can now compile Puma on machines where OpenSSL is not installed.
There is now a thread-backtraces command in pumactl to print all active threads backtraces. This has been available via SIGINFO on Darwin, but now it works on Linux via this new command.
Puma.stats now has a requests_count counter.
lowlevel_error_handler got some enhancements - we also pass the status code to it now.
Phased restarts and worker timeouts should be faster.
Puma.stats_hash provides Puma statistics as a hash, rather than as JSON.

Loads of Bugfixes

The number of bugfixes in this release is pretty huge. Here’s the most important ones:

Shutdowns should be more reliable.
Issues surrounding socket closing on shutdown have been resolved.
Fixed some concurrency bugs in the Reactor.
out_of_band should be much more reliable now.
Fixed an issue users were seeing with ActionCable and not being able to start a server.
Many stability improvements to prune_bundler.

Nicer Internals and Tests

This release has seen a massive improvement to our test coverage. We’ve pretty much doubled the size of the test suite since 4.0, and it’s way more stable and reproducible now too.

A number of breaking changes come with this major release. For the complete list, see the HISTORY file.

Thanks to Our Contributors!

This release is our first major or minor release with new maintainer MSP-Greg on the team. Greg has been doing tons of work on the test suite to make it more reliable, as well as a lot of work on our SSL features to bring them up-to-date and more extendable. Greg is also our main Windows expert.

The following people contributed more than 10 commits to this release:

Tim Morgan
Vyacheslav Alexeev
Will Jordan
Jeff Levin
Patrik Ragnarsson, who’s also been very helpful in our Issues tracker.

If you’ve like to make a contribution to Puma, please see our Contributors Guide. We’re always looking for more help and try to make it as easy as possible to contribute.

Enjoy Puma 5!

The Practical Effects of the GVL on Scaling in Ruby

Mon, 11 May 2020 00:00:00 +0000

The Global Virtual Machine Lock confuses many Rubyists. Most Rubyists I’ve met have a vague sense that the GVL is somehow bad, and has something to do concurrency or parallelism.

(‘CRuby’ refers to the mainline Ruby implementation, written in C. Sometimes people call this ‘MRI’.) The GVL (formerly known as GIL, as you’re about to learn) is a unique feature to CRuby, and doesn’t exist in JRuby or TruffleRuby.

JavaScript’s popular V8 virtual machine also has a VM lock. CPython also has a global VM lock. That’s three of the most popular dynamic languages in the world! VM locks in dynamic languages are very common.

(Instead of removing the GVL, Ruby core has signaled that it will take an approach similar to V8 Isolates with inspiration from the Actor concurrency model (discussed at the end).)

Understanding CRuby’s Global VM Lock is important when thinking about scaling Ruby applications. It will probably never be removed from CRuby completely, and its behavior changes how we scale Ruby apps efficiently.

Understanding what the GVL is and why the current GVL is “global” will help you to answer questions like these:

What should I set my Sidekiq concurrency to?
How many threads should I use with Puma?
Should I switch to Puma or Sidekiq from Unicorn, Resque, or DelayedJob?
What are the advantages of event-driven concurrency models, like Node?
What are the advantages of a global-lock-less language VM, like Erlang’s BEAM or Java’s JVM?
How will Ruby’s concurrency story change in Ruby 3?

We’ll deal with these questions and more in this article.

What we’re locking: the language virtual machine

(Most descriptions of the GVL immediately dive into concepts like atomicity and thread-safety. This description will start from a more basic premise and work up to that.)

(YARV was essentially Koichi Sasada’s graduate thesis.)

Wait: isn’t it the GIL? What’s the GVL?

GIL stands for Global Interpreter Lock, and it’s something that was removed from Ruby (or just mutated, depending on how you look at it) in Ruby 1.9, when Koichi Sasada introduced YARV (Yet Another Ruby VM) to Ruby. YARV changed CRuby’s internal structure so that the lock existed around the Ruby virtual machine, not an interpreter. The correct terminology for over a decade now has been GVL, not GIL.

(You can interact with instruction sequences via the InstructionSequence class. Everything is an object in Ruby!)

How does an interpreter differ from a virtual machine?

A virtual machine is a little like a CPU-within-a-CPU. Virtual machines are computer programs that usually take simple instructions, and those instructions manipulate some internal state. A Turing machine, if it was implemented in software, would be a kind of virtual machine. We call them virtual machines and not machines because they’re implemented in software, rather than in hardware, like a CPU is.

Before Ruby 1.9, Ruby didn’t really have a separate virtual machine step - it just had an interpreter. As your Ruby program ran, it actually interpreted each line of Ruby as it went. Now, we just interpret the code once, turn it into a series of VM instructions, and then execute those instructions. This is much faster than interpreting Ruby constantly.

A Turing machine, implemented in software, would be a kind of virtual machine. Wikimedia Commons by RosarioVanTuple

The Ruby Virtual Machine understands a simple instruction set. Those instructions are generated from the Ruby code you write by the interpreter, and then the virtual machine instructions are fed into the Ruby VM.

Let’s watch this in action. First, in case you didn’t know, you can execute Ruby from the command line using the -e option:

$ ruby -e "puts 1 + 1"
2

Now, you can then dump the instructions for this simple program by calling --dump=insns:

$ ruby --dump=insns -e "puts 1 + 1"
== disasm: #<ISeq:<main>@-e:1 (1,0)-(1,10)> (catch: FALSE)
putself                                                          (   1)[Li]
putobject_INT2FIX_1_
putobject_INT2FIX_1_
opt_plus                     <callinfo!mid:+, argc:1, ARGS_SIMPLE>, <callcache>
opt_send_without_block       <callinfo!mid:puts, argc:1, FCALL|ARGS_SIMPLE>, <callcache>
leave

Ruby is a “stack-based” VM. You can see how this works by looking at the generated instructions here - we add the integer 1 to the stack two times, than call plus. When plus is called, there are two integers on the stack. Those two integers are replaced by the result, 2, which is then on the stack.

So, what does the Ruby VM have to do with threading, concurrency and parallelism?

Concurrency and Paralellism

You may be aware that there’s a difference between concurrency and parallelism. Imagine a grocery store. At this grocery store, we have a line and some checkout clerks working to pull customers from the line and get their groceries checked out.

Each of our grocery store checkout clerks works in parallel. They don’t need to talk to each other to do their job, and what one clerk is doing doesn’t affect the other in any way. They’re working 100% in parallel.

Now, a clerk can work on multiple customers concurrently. This would look like a clerk grabbing multiple customers from the line, working on one customer’s groceries for a moment, then switching to another customer’s groceries, and so on. This would be working concurrently.

Let’s take a more concrete example. Compare three grocery store clerks working in parallel with a single one working concurrently. To check out a customer, we must perform two operations: scanning their groceries, and then bagging them. Imagine each customer’s groceries take the exact same amount of time to scan and bag.

Three customers arrive. Let’s say scanning takes time A and bagging takes time B. Our three parallel clerks will process these three customers in time (A + B).

The parallel case.

What about our concurrent clerk? All three of her customers arrive at the same time. The clerk scans each customer’s groceries, then bags each customer’s groceries. Each customer is worked on concurrently, but never in parallel.

The concurrent case.

In the concurrent case, our first customer experiences a total service time of (3A + B). They had to wait for everyone else’s groceries to be checked out for their own groceries to get bagged. The second customer will exerience a total service time of (3A + 2B), and the final customer will experience a service time of (3A + 3B).

Notice how the customers who got the concurrent checkout clerk experienced a longer total service time than the customers who used our three parallel clerks.

In short: concurrency is interesting, but parallelism is what speeds up systems and allows them to handle increased load.

Performing two operations concurrently means that the start and end times of those operations overlapped at some point. For example, you and I sit down to a sign a contract. However, there is only one pen. I sign where I’m supposed to, hand the pen to you, and then you sign. Then, you hand the pen back to me and I initial a few lines. You might say that we signed the contract concurrently, but never in parallel - there was only one pen, so we couldn’t sign the contract at the exact same time.

Peforming operations in parallel means that we are doing those operations at the exact same instant. In my contract example, a parallel contract signing would involve two pens (and probably two copies of the contract, otherwise it would get a little crowded).

Concurrency and Paralellism on a Computer

On a modern operating system, programs are run with a combination of processes and threads. Processes have at least one thread, and can have up to thousands.

To extend the grocery store analogy, processes are like the checkout counters that our clerks use. They contain tools and common resources, like the point-of-sale terminal and the barcode scanner, but they don’t actually do anything. A process usually contains a memory allocation (the heap), file descriptors (sockets, files, etc), and other such computer resources.

Threads actually run our code. Each process has at least one thread. In our analogy, they’re like the store clerks. They also hold a small amount of information. For example, if we’re adding two local variables in a Rails application, our thread contains information about those two variables (thread-local storage) and also what line of code we’re currently running (the stack).

Threads run the code when they are scheduled to by the operating system’s kernel. The Ruby runtime itself doesn’t actually manage when threads are executed - the operating system decides that.

When Ruby was written in the 90s, all processes had just one thread. This started to change in the early 2000s, necessitating the rewrite of the language VM in Ruby 1.9 (YARV), which is what gave us the GVL as we know it today.

What the GVL actually does

As mentioned earlier, the Ruby Virtual Machine is what actually turns Ruby virtual machine instructions (generated by the interpreter) into CPU instructions.

The Ruby Virtual Machine is not internally thread-safe. If two threads try to access the Ruby VM at the same time, really Bad Things would happen. This is a bit like the point of sale terminal at our grocery store checkout counters. If two checkout clerks tried to use the same POS terminal, they would interrupt each other and probably keep losing their work or corrupting each other’s work. You would end up paying for someone else’s groceries!

So, because it isn’t safe for multiple threads to access the Ruby Virtual Machine at the same moment, instead we use a global lock around it so that only one thread can access it in parallel.

(One caveat of the Javascript GVL is that it isn’t actually global: you can create additional Isolates. Koichi Sasada’s proposal for Ractors (formerly Guilds) would be similar.)

It is extremely common for dynamic language VMs to not be thread-safe. As mentioned, CPython and V8 are the most prominent examples. Java is probably the best example of a semi-dynamic language that does have a threadsafe VM. It’s also why so many languages are written on top of the JVM: writing your own threadsafe VM is really hard.

TFW you realize that there’s always going to be locks, the only difference is what level they’re implemented at and who implements them

There’s a few good reasons that having a GVL is so popular:

It’s faster. Single-threaded performance improves because you don’t have to constantly lock and unlock internals.
Integrating with extensions, such as C extensions, is easier.
It’s easier to write a lockless VM than one with a lot of locks.

Each Ruby process has its own Global VM Lock, so it might be more accurate to say that it’s a “process-wide VM lock”. Its “global” in the same sense that a “global variable” is global.

Only one thread in any Ruby process can hold the global VM lock at any given time. Since a thread needs access to the Ruby Virtual Machine to actually run any Ruby code, effectively only one thread can run Ruby code at any given time.

Let me play you the song of my people: “GGVVVVLLLLLLLLLLLLLLLL”

Think of the GVL like the conch shell in the Lord of the Flies - if you have it, you get to speak (or execute Ruby code in this case). If the GVL is already locked by a different thread, other threads must wait for the GVL to be released before they can hold the GVL.

Amdahl’s Law: Why 1 Sidekiq Process Can Be 2x as Efficient as DelayedJob or Resque

Your programs actually do many things that don’t need to access the Ruby Virtual Machine. The most important is waiting on I/O, such as database and network calls. These actions are executed in C, and the GVL is explicitly released by the thread waiting on that I/O to return. When the I/O returns, the thread attempts to reacquire the GVL and continue to do whatever the program says.

This has enormous real-world performance impacts.

Imagine you have a stack of satellite image data you have to process (with Ruby). You have written a Sidekiq job, called SatelliteDataProcessorJob, and each job works on a small fraction of all of the satellite data.

class SatelliteDataProcessorJob
  include Sidekiq::Worker

  def perform(some_satellite_data)
    process(some_satellite_data)
    touch_external_service(some_satellite_data)
    add_data_to_database(some_satellite_data)
  end
end

Let’s imagine that process is a 100% Ruby method, which does not call C extensions or external services. Further, let’s imagine that touch_external_service and add_data_to_database are effectively 100% I/O methods that spend all of their time waiting on the network.

First, an easy question: if each run of SatelliteDataProcessorJob takes 1 second, and you have 100 enqueued jobs and just 1 Sidekiq process with 1 thread, how long will it take to process all the jobs? Assume infinite CPU and memory resources.

100 seconds.

How about if you two processes? 50 seconds. And 25 seconds for 4 processes and so on. That’s parallelism.

Now, let’s say you have 1 Sidekiq process with 10 threads. How long will it take to process all of those jobs?

The answer is it depends. If you’re on JRuby or TruffleRuby, it will take about 10 seconds, because each thread is fully parallel with all the other threads.

But on MRI, we have the GVL. Does adding threads increase concurrency?

From Daniels 220 @ Wikipedia

It turns out, this exact problem interested computer scientist Gene Amdahl back in 1967. He proposed something called Amdahl’s Law, which gives the theoretical speedup in latency for the execution of tasks with fixed workloads when resources increase.

Amdahl figured out that the speedup you got from additional parallelism was related to the proportion of execution time that could be done in parallel. Sound familiar?

Amdahl’s Law is simply 1 / (1 - p + p/s), where p is the percentage of the task that could be done in parallel, and s is the speedup factor from the part of the task that gained improved resources (the parallel part).

So, in our example, let’s say that half of SatelliteDataProcessorJob is GVL-bound and half is IO-bound. In this case, p is 0.5 and s is 10, because we can wait for IO in parallel and there are 10 threads. In this case, Amdahl’s Law shows that a Sidekiq process would go through our jobs up to 1.81x faster than a single-threaded Resque or DelayedJob process.

Many background jobs in Ruby spend at least 50% of their time waiting on IO. For those jobs, Sidekiq can lead to a 2x decrease in resource usage, because 1 Sidekiq process can do the work of what used to take 2 single-threaded processes.

So, even with a GVL, adding threads to applications increases throughput-per-process, which in turn lowers memory consumption.

Threads, Puma and GVL-caused Latency

This also means that “how many threads does my Sidekiq or Puma process need” is a question answered by “how much time does that thread spend in non-GVL execution?” or “how much time does my program spend waiting on I/O?” Workloads with high percentages of time spent in I/O (75%+ or more) often benefit from 16 threads or even more, but more typical workloads see benefit from just 3 to 5 threads.

It’s possible to configure your thread pools to be too large. Setting Puma or Sidekiq to thread settings higher than 5 can lead to contention for the GVL if the work is not parallelizable enough. This increases service latency.

While total time to process all of the units of work remains the same, the latency experienced by each individual unit of work increases.

Imagine a grocery store where a checkout clerk grabbed 16 people off of the checkout queue and checked those 16 people’s groceries concurrently, scanning one item per person before scanning one item from the next person’s cart. Rather than experiencing checkout time as (A + B), they experience a checkout time of 16(A+B).

(This effect is generally present in a concurrent-but-not-100%-parallel system where overall utilization is not extremely high. We’re mitigating this effect slightly in Puma 5 by having Puma workers with more than one thread delay listening to the socket, so less-loaded workers pick up requests first.)

Some people misidentify this additional latency as “context switching” costs. However, latency experienced by the individual unit of work is increasing without additional switching cost.

In any case, context switching on modern machines and operating systems is pretty cheap relative to the time it takes to service a typical web app request or background job. It does not add hundreds of milliseconds to response times - but oversaturating the GVL can.

If adding threads to a CRuby process can increase latency, why is it still useful?

(Shouldn’t adding an additonal thread only increase memory usage by 8MB, which is the size of the thread’s stack allocation? Ah, if only memory usage was that simple. Learn more about the complexities of RSS and thread-induced fragmentation here.)

Adding more threads to a Ruby process helps us to improve CPU utilization at less memory cost than an entire additional process. Adding 1 process might use 512MB of memory, but adding 1 thread will probably cause less than 64MB of additional memory usage. With 2 threads instead of 1, when the first thread releases the GVL and listens on I/O, our 2nd thread can either pick up new work to do, increasing throughput and utilization of our server.

GitLab switched from Unicorn (single-thread model) to Puma (multi-thread model) and saw a 30% decrease in memory usage across their fleet. If you’re memory-constrained on your host, this allows you to run 30% more throughput for the same money. That’s awesome.

The Future

For a decade now, bystanders have declared that Ruby is dead because it “doesn’t have a proper concurrency story”.

I think we’ve shown that there is a concurrency story in Ruby. First, we have process-based concurrency. We multiply GVLs by multiplying processes. This works perfectly fine, if you have enough memory.

If you’re out of memory, you can use Sidekiq or Puma, which provides a threaded container for our apps, and then let pre-emptive threading do its thing.

Ruby has proven that process-based concurrency (which is really what the GVL forces us to do) scales well. It’s not much more expensive than other models, especially these days when memory is so cheap on cloud providers. Think critically about what an Actor-style approach or an Erlang Process-style approach would actually change about your deployment at the end of the day: you would use less memory per CPU. But on large deployments, most web applications are already CPU-bottlenecked, not memory!

Ractor

Koichi Sasada, author of YARV, is proposing a new concurrency abstraction for Ruby 3 called Ractors. It’s a proposal based on the Actor concurrency model (hence Ruby Actor -> Ractor). Basically, Actors are boxes for objects to go into, and each actor can only touch its own objects, but can send and receive objects to/from other Actors. Here’s an example written by Koichi Sasada:

r = Ractor.current
rs = (1..10).map{|i|
  r = Ractor.new r, i do |r, i|
    r.send Ractor.recv + "r#{i}"
  end
}
r.send "r0"
p Ractor.recv #=> "r0r10r9r8r7r6r5r4r3r2r1"

Eventually (not yet in the current implementation), each Ractor will get their own VM lock. That means the example code above will execute in parallel.

This is made possible because Ractors don’t share mutable state. Instead, they only share immutable objects, and can send mutable objects between each other. This should mean that we don’t need a VM lock inside of a Ractor.

Koichi Sasada’s Ractor proposal is now public, though as of this writing the docs are mostly in Japanese, and “each Ractor gets its own VM lock” has not yet been implemented. Ractors will essentially allow us to “multiply” GVLs in a process, which would make the GVL no longer “global”, although the lock will still exist in each Ractor. The Global VM Lock will become a Ractor VM Lock.

TL:DR;

Thanks for listening to me whinge. Here’s what you need to remember:

If you are memory bottlenecked on Ruby, you need to saturate the GVL by adding more threads, which will allow you to get more CPU work done with less memory use.
The GVL means that parallelism is limited to I/O in Ruby, so switch to a multithreaded background job processor before you switch to a multithreaded web server. Also, you’ll probably use much higher threadpool sizes with your background jobs than with your web server.
Ruby 3 might make the GVL no longer global by allowing you to multiply VMs using Ractors. Application servers and background job processors will probably change their backend to take advantage of this, you won’t really have to change much of your code at all, but you will no longer have to worry about thread safety (yay).
Process based concurrency scales very well, and while it might lose a few microseconds to other concurrency models, these concurrency switching costs generally don’t matter for the typical Rails application. Instead, the important thing is saturating CPU, which is the most scarce resource in today’s computing environments.

The World Follows Power Laws: Why Premature Optimization is Bad

Sun, 22 Dec 2019 07:00:00 +0000

(This post is a sample of the content available in the Complete Guide to Rails Performance. It’s actually the first lesson - there are 30+ more lessons and 18 hours of video in the course itself.)

I want to tell you about a physicist from Schenectady, a Harvard linguist, and an Italian economist.

Pareto’s poltiical views are bit suspect, unfortunately, because he chose to see the way things are (unequal) as the way they ought to be.

The Italian economist you may already have heard of - Vilifredo Pareto. He became famous for something called The Pareto Principle, the idea that for most things, 80% of the effect comes from just 20% of the causes. The Pareto Principle is fundamental to performance work because it reminds us why premature optimization is so inefficient and useless. While you’ve probably heard of the Pareto Principle, I want you to understand why it actually works. And to do that, we’re going to have to talk about probability distributions.

Benford - the physicist

Frank Benford was an American electrical engineer and physicist who worked for General Electric. It was the early 20th century, when you had a job for life rather than a startup gig for 18 months, so he worked there from the day he graduated from the University of Michigan until his death 38 years later in 1948.
A page from Henry Briggs’ first table of common logarithms, Logarithmorum Chilias Prima, from 1617. Wikipedia

Back in that time, before calculators, if you wanted to know the logarithm of a number - say, 12 - you looked it up in a book. The books were usually organized by the leading digit. If you wanted to know the logarithm of 330, you first went to the section for 3, then looked for 330. Benford noticed that the first pages of the book were far more worn out than the last pages. Benford realized this meant that the numbers looked up in the table began more often with 1 than with 9.

Most people would have noticed that and thought nothing of it. But Benford pooled 20,000 numbers from widely divergent sources (he used the numbers in newspaper stories) and found that the leading digit of all those numbers followed a power law too.

This became known as Benford’s Law. Here are some other sets of numbers that conform to this power law:

Physical constants of the universe (pi, the molar constant, etc.)
Surface areas of rivers
Fibonacci numbers
Powers of 2
Death rates
Population censuses

That the physical constants of the universe follow this distribution is probably the most mind-blowing revelation of Benford’s Law, for me, anyway.

Benford’s Law is so airtight that it’s been admitted in US courts as evidence of accounting fraud (someone used RAND in their Excel sheet!). It’s been used to identify other types of fraud too - elections, scientific and even macroeconomic data.

What would cause numbers that have (seemingly) little relationship with each other to conform so perfectly to this non-random distribution?

Zipf - the linguist

A plot of the rank versus frequency for the first 10 million words in 30 different languages of Wikipedia. Note the logarithmic scales. Licensed CC-BY-SA by SergioJimenez.

At almost exactly the same time as Benford was looking at first leading digits, George Kingsley Zipf was studying languages at Harvard. Uniquely, George was applying the techniques of a new and interesting field - statistics - to the study of language. This landed him an astonishing insight: in nearly every language, some words are used a lot, but most (nearly all) words are used hardly at all.

Only a few words account for most of our use of language. The Brown Corpus is a collection of literature used by linguistics researchers. It consists of 500 samples of English-language text comprising 1 million words. Just 135 unique words are needed to account for 50% of those million words. That’s insane.

Zipf’s probability distribution is discrete. Discrete distributions are comprised of whole integers. Continuous distributions can take on any value. If you take Zipf’s distribution and make it continuous instead of discrete, you get the Pareto distribution.

Pareto - the economist

Pareto initially noticed a curious distribution when he was thinking about wealth in society - he noticed that 80% of the wealth and income came from 20% of the people in it.

The Pareto distribution, pictured, has been found to hold for a scary number of completely different and unrelated fields in the sciences. For example, here are some natural phenomena that exhibit a Pareto (power law) distribution:

Wealth inequality
Sizes of rocks on a beach
Hard disk drive error rates (!)
File size distribution of Internet traffic (!!!)

We tend to think of the natural world as random or chaotic. In schools, we’re taught the bell curve/normal distribution. But reality isn’t normally distributed. It’s log-normal. Many probability distributions, in the wild, support the Pareto Principle:

80% of the output will come from 20% of the input

Normal distributions are taught in schools because they’re quite easy to talk about mathematically, not because they’re particularly good descriptions of the natural world.

While you may have heard this before, what I’m trying to get across to you is that it isn’t made up. The Pareto distribution is used in hundreds of otherwise completely unrelated scientific fields - and we can use its ubiquity to our advantage.

It doesn’t matter what area you’re working in - if you’re applying equal effort to all areas, you are wasting your time. What the Pareto distribution shows us is that most of the time, our efforts would be better spent finding and identifying the crucial 20% that accounts for 80% of the output.

Allow me to reformulate and apply this to web application performance:

80% of an application’s work occurs in 20% of its code.

There are other applications in our performance realm too:

80% of an application’s traffic will come from 20% of its features.

80% of an application’s memory usage will come from 20% of its allocated objects.

The ratio isn’t always 80/20. Actually, usually it’s way more severe - 90/10, 95/5, 99/1. Sometimes it’s less severe. So long as it isn’t 50/50 we’re talking about a non-normal distribution.

This is why premature optimization is so bad and why performance monitoring, profiling and benchmarking are so important. The world is full of power-law distributions, not normal distributions. Spreading your effort evenly across a power-law distribution is a massive waste of effort.

What the Pareto Principle reveals to us is that optimizing any random line of code in our application is in fact unlikely to speed up our application at all! 80% of the “slowness” in any given app will be hidden away in a minority of the code.

So instead of optimizing blindly, applying principles at random that we read from blog posts, or engaging in Hacker-News-Driven-Development by using the latest and “most performant” web technologies, we need to measure where the bottlenecks and problem areas are in our application.

An Optimization Story - Measurement, Profiling and Benchmarking

There’s only one skill in performance work that you need to understand completely and deeply - how to measure your application’s performance. Once you have that skill mastered, knowing every possible thing about performance might be a waste of time. Your problems are not other’s problems. There are going to be lessons to learn that solve problems you don’t have (or don’t comprise that crucial 20% of the causes of slowness in your application).

On the flip side, you should realize that the Pareto Principle is extremely liberating. You don’t need to fix every performance issue in your application. You don’t need to go line-by-line to look for problems under every rock. You need to measure the actual performance of your application, and focus on the 20% of your code that is the worst performance offender.

My first conference talk ever, actually.

I once gave a conference talk that was a guided read-through of Minitest, the Ruby testing framework. Minitest is a great read if you’ve got a spare hour or two - it’s fairly short at just 1,500 lines. As I was reading Minitest’s code, I came across this funny line:

def self.runnable_methods
  methods = methods_matching(/^test_/)

  case self.test_order
  when :random, :parallel then
    max = methods.size
    methods.sort.sort_by { rand max }
  when :alpha, :sorted then
    methods.sort
  else
    raise "Unknown test_order: #{self.test_order.inspect}"
  end
end

This code is extremely readable as to what’s going on; we determine which methods on a class are runnable with a regex (“starts with test_”), and then sort them depending upon this test class’s test_order. Minitest uses the return value to execute all of the runnable_methods on all the test classes you give it. Usually this is a randomized array of method names, because the default test order is :random.

What I was honing in on was this line, which is run when :test_order is :random or :parallel:

max = methods.size
methods.sort.sort_by { rand max }

This seemed like a really roundabout way to do methods.shuffle to me. Maybe Ryan (Minitest’s author) was doing some weird thing to ensure deterministic execution given a seed. Minitest runs your tests in the same order given the same seed to the random number generator. It turns out methods.shuffle is deterministic, though, just like the code as written. So, I decided to benchmark it, mostly out of curiosity.

Whenever I need to write a micro benchmark of Ruby code, I reach for benchmark/ips. (The reason I use benchmark/ips rather than the stdlib benchmark is because the stdlib version requires you to run a certain line of code X number of times and tells you how long that took. The problem with that is that I don’t usually know how fast the code is to begin with, so I have no idea how to set X. Usually I run the code a few times, guess at a number of X that will make the benchmark take 10 seconds to run, and then move on. benchmark/ips does that work for me by running my benchmark for 10 seconds and calculating iterations-per-second.) ips stands for iterations-per-second. The gem is an extension of the Benchmark module, something we get in the Ruby stdlib.

Here’s that benchmark:

require "benchmark/ips"

class TestBench
  def methods
    @methods ||= ("a".."z").to_a
  end

  def fast
    methods.shuffle
  end

  def slow
    max = methods.size
    methods.sort.sort_by { rand max }
  end
end

test = TestBench.new

Benchmark.ips do |x|
  x.report("faster alternative") { test.fast }
  x.report("current minitest code") { test.slow }
  x.compare!
end

This benchmark suggested that shuffle was 12x faster than sort.sort_by { rand methods.size }. This makes sense - shuffle randomizes the array with C, which will always be faster than randomizing it with pure Ruby. In addition, Ryan was actually sorting the array twice - once in alphabetical order, followed by a random shuffle based on the output of rand.

Ryan’s conference talks are pretty good, too.

I asked Ryan Davis, minitest author, what was up with this. He gave me a great reply: “you benchmarked it, but did you profile it?”

What did he mean by this? Well, first, you have to know the difference between benchmarking and profiling - the two fundamental performance measurement tools.

There are a lot of different ways to define this difference. Here’s my attempt:

Benchmarking

A benchmark is a test of one or many different pieces of code that measures how fast they execute or how many resources they consume.

When we benchmark, we take two competing pieces of code and compare them. It could be as simple as a one liner, like in my story, or as complex as an entire web framework. Then, we put them up against each other (usually comparing them in terms of iterations/second) using a simple, contrived task. At the end of the task, we come up with a single metric - a score. We use the score to compare the two competing options.

In my example above, it was just how fast each line could shuffle an array. If you were benchmarking web frameworks, you might test how fast a framework can return a simple “Hello World” response. Benchmarks put the competing alternatives on exactly equal footing by coming up with a contrived, simple, non-real-world example.

I wrote a v v long post once about why this benchmark doesn’t mean much for Rails

It’s usually too difficult to benchmark real-world code because the alternatives aren’t doing exactly the same thing. For example, comparing Rails against Sinatra isn’t entirely fair because Rails has many features that Sinatra does not - even for a simple Hello World response, the Rails application is, for example, performing many security checks that the Sinatra app doesn’t. Comparing these frameworks in a 1-to-1 benchmark will always be slightly misleading for that reason.

Profiling

Profiles are a accounting of all the sub-steps required to run a given piece of code. When we profile, we’re usually examining the performance characteristics of an entire, real-world application. For example, this might be a web application or a test suite. Because profiling works with real-world code, we can’t really use it to compare competing alternatives, because the alternative usually doesn’t exactly match what we’re profiling. Profiling doesn’t usually produce a comparable “score” at the end with which to measure these alternatives, either. But that’s not to say profiling is useless - it can tell us a lot of valuable things, like what percentage of CPU time was used where, where memory was allocated, and what lines of code are important and which ones aren’t.

What Ryan was asking me was - “Yeah, that way is faster on this one line, but does it really matter in the grand scheme of Minitest”? How much time does a Minitest test run actually spend shuffling the methods? 1%? 10%? 0.001%? Profiling can tells us that.

You said that this one-line change would speed up minitest. A higher-level benchmark determined that was a lie.

Is this one line really part of Pareto’s “20%”? We can assume, based on the Principle, that 80% of Minitest’s execution time will come from just 20% of its code. Was this line part of that 20%?

I’ve already shown you how to benchmark on the micro scale. But before we get to profiling, I’m going to do a quick macro-benchmark to test my assumption that using shuffle instead of sort.sort_by will speed up Minitest.

Minitest is used to run tests, so we’re going to benchmark a whole test suite. Rubygems.org, an open-source Rails application with a Minitest suite, will make a good example test suite.

When micro-benchmarking, I reach for benchmark-ips. When macro-benchmarking (and especially in this case, with a test suite), I usually reach first for the simplest tool available: the unix utility time! We’re going to run the tests 10 times, and then divide the total time by 10.

$ time for i in {1..10}; do bundle exec rake; done

...

real	15m59.384s
user	11m39.100s
sys	1m15.767s

When using time, we’re usually only going to pay attention the user statistic. real gives the actual total time (as if you had used a stopwatch), sys gives the time spent in the kernel (in a test run, this would be things like shelling out to I/O), and user will be the closest approximation to time actually spent running Ruby. You’ll notice that user and sys don’t add up to real - the difference is time spent waiting on the CPU while other operations (like running my web browser, etc) block.

With stock minitest, the whole thing takes 11 minutes and 39 seconds, for an average of 69.9 seconds per run. Now, let’s alter the Gemfile to point to a modified version (with shuffle on the line in question) of minitest on my local machine:

gem 'minitest', require: false, path: '../minitest'

To make sure the test is 100% fair, I only make the change to my local version after I check out minitest to the same version that Rubygems.org is running (5.8.1).

Even so-called performance experts mess this shit up sometimes.

The result? 11 minutes 56 seconds. Longer than the original test! We know my code is faster in micro, but the macro benchmark told me that it actually takes longer. A lot of things can cause this (the most likely being other stuff running on my machine), but what’s clear is this - my little patch doesn’t seem to be making a big difference to the big picture of someone’s test suite. While making this change would, in theory, speed up someone’s suite, in reality, the impact is so minuscule that it didn’t really matter.

So, while a benchmark told me one thing - X is 10x faster than Y! - a higher-level benchmark told me another (make your change and this thing didn’t really matter.) Not only does this show the value of profiling (which would have told me before that the sorting didn’t take much of the total time) but also how microbenchmarks and relative comparisons can mislead.

Performance measurement is a critical skill. Anywhere along the way, I could have been mislead by a single number or a rogue measurement. By applying a scientific, empirical approach, I was able to put my benchmark in context of a larger program.

Premature optimization is ignoring these lessons and optimizing “when we feel like it”, or optimizing constantly all the time. Hopefully I’ve convinced you: it’s a guaranteed waste of time.

Repeat after me: I will not optimize anything in my application until my metrics tell me so.

Why Your Rails App is Slow: Lessons Learned from 3000+ Hours of Teaching

Mon, 17 Jun 2019 07:00:00 +0000

“What I do have is a particular set of skills, a set of skills which makes me a nightmare for slow Rails applications like you.”

For the last 4 years, I’ve been working on making Rails applications faster and more scalable. I teach workshops, I sell a course, and I consult. If you do anything for a long period of time, you start to see patterns. I’ve noticed four different factors that prevent software organizations from improving the performance of their Rails applications, and I’d like to share them here.

Performance becomes a luxury good, especially when no one is watching

Often times at my workshops, I discover that an attendee simply has no visiblity into what their application is doing in production - they either don’t understand their dashboards, they don’t have them, or they’re not allowed to even access them (“DevOps team only”).

Performance metrics are often just not tracked. No one is aware if the app is over or underscaled, no one knows if the app is “slow” or “fast”. Is it any wonder, then, that no one spends any time working on it?

Performance is rarely the first priority of any organization, and often gets “trickled down” hours and resources.

Some of this is actually a good thing. There’s a reason that the classic programming mantra of “make it work, make it clean, make it fast” is in that order and not the opposite way around. People pay for software that does stuff. If it does that stuff quickly and in a pleasantly performant way, then that’s great, but it’s not always required (especially if the organization is first to market in their space and customers have no other options).

Often, my consulting clients are at a point in their organization where they’re no longer scraping by on ramen and cheeto dust, but have a solid business that’s expanding (slowly or quickly). They’ve finally gotten their heads above water and they’re ready to start thriving, not just surviving. People don’t come to me when they’re still trying to achieve product market fit unless things have become untenable, and then that’s more of a rescue job.

When the time comes to set the budget on performance instead of feature velocity.

This is a natural and correct progression. It also means organizations accumulate performance debt during that initial period of building and obtaining product-market fit. I think there may be some that believe that all kinds of technical debt are some kind of sinful black stain on any organization, and that if you just Coded The Right Way or were a Software Craftsperson™ this would not have happened. I think that’s probably wrong. Before achieving ramen profitability, businesses must take out technical debt as a kind of financing of their own product development runway. This will happen regardless of one’s coding techniques or knowledge level.

However, performance is not a luxury good. It isn’t something that can simply be ignored until one’s organization has a spare four or five figures in the couch cushions. Like technical debt, there is a point when feature work grinds to a halt because the organization is too busy maintaining the performance debt that has accrued. Requests are timing out. Customers are complaining about slow the app feels and switching to competitors. You’re scared to check the AWS bill.

Ideally, organizations monitor and sensibly take out performance debt when required, and understand the full extent of the work that must be done in the future.

To do this sort of “sensible debt accrual”, you need performance monitoring/metrics and you need to understand how to present numbers to management. I find that while most people know subscribe to a performance monitoring service, such as New Relic, Skylight or Scout, they often have no idea how to read it and extract useful insights from it, making it a very expensive average latency monitor. Being able to actually use your APM is a critical performance skill that I cover in great detail in my workshops and course.

If you can’t draw insights from this, you’re just throwing cash out the door.

Monitoring these metrics allows you to assess where you’re at and to figure out what parts of the application have accrued performance debt. It also helps you to make decisions on the “cost/benefit” of future work.

It also means you need to be able to “speak manager” or “speak business”. The business case for adding more features is obvious to the non-technical side of your organization. There is a great business case for performance, fortunately, both from the side of the customer and from the cost side as well - reducing average latency by 50% means you can spend 50% less on your application’s servers thanks to queueing theory and something called Little’s Law.

At my workshops, I spend a lot of time simply discussing terminology, like request queueing, latency, throughput, tracing and profiling. Giving people the vocabulary they need to understand the tools out there seems to be half the battle of getting everyone comfortable reading their own metrics.

Complex apps and complex problems, with little training

This leads me to the second cause of performance problems in software - a simple lack of knowledge. We can’t optimize what we don’t understand and we can’t fix what we can’t see.

I wrote the Complete Guide to Rails Performance simply because there was so much information about this topic that had simply never been compiled before into one place.

“What’s request queueing?”

This shows itself most when scaling for throughput. Most organizations simply aren’t tracking critical scaling metrics or even know what they are, often because they believe the platform-as-a-service that they’re using should “take care of this” for them. By the time I’ve been called in, they’re spending thousands of dollars a month more than they need to, and could have fixed this months or even years ago with some simple autoscaling policies and a bit of organizational knowledge around scaling. Or, the flipside is happening and they’re massive under-scaled, with 25-50% of their total request latency being just time spent queueing for resources.

Performance work is not rocket science. However, unlike a lot of other areas in software (The only other area in software that requires an even wider base of knowledge is security. Consider Rowhammer - basically an electrical engineering exploit in very particular configurations of DRAM.), it can require an extremely broad base of knowledge. When your customer says the site “feels slow”, the problem can quite reasonably be almost anywhere between the pixels on the user’s screen (say, an issue with the customer’s client machine) and the electrons running through the silicon on your cloud service provider (for example, a mitigation for a recent Intel security issue puts your servers above capacity). Feature work and even to a large extent refactoring work generally only requires knowledge of the language and frameworks in use. Performance work often needs esoteric knowledge from other fields (such as queueing theory) in addition to highly in-depth knowledge in your frameworks and language. (I wrote a 3000+ word blog about the critical performance differences between English-language synonyms .present? and .exists? in Rails, for example, but my Rails performance course spends the majority of the time talking about things which are not Ruby-specific.)

Looking at a flamegraph of a Rails app for the first time often leads to this reaction.

This depth of knowledge simply isn’t present in many organizations, especially those who place sprint velocity before the development of engineering capacity and skills in the organization.

The workshops I’ve been doing have really allowed me to go in deep on complex problems and help people deal with the “wrinkles” introduced by their application. Getting to look over people’s shoulders while they experience an error or something I hadn’t anticipated has been very rewarding, both for them and for me as an educator.

Also, during those workshops, I don’t emphasize “pre-baked” problem/solutions, but instead have the attendees bring their real world applications, and we immediately try to apply what we’ve learned on their actual apps right then and there. I don’t want anyone to go home and run into a problem caused by the complexity of their app - rather, I’d like that to happen while we’re both in the same room!

Boiling frogs - even when tracked, performance slips without fix

In the Slack channel for the Complete Guide to Rails Performance, we’ve had a few conversations about managing performance work in the software organization.

Walking into the office on Monday like

An organizational culture that always places completeness over quality inevitably runs into issues. Often when I get new clients, they’re experiencing not just performance issues but have problems with all the various dimensions of software quality: low correctness (an excess of bugs and lack of test coverage), high complexity (“technical debt”, spaghetti organization), and a poor deployment pipeline (broken builds, janky deploys). These aspects of software quality tend to either all be good or all be bad. Project management can (and often should) sacrifice quality for a period of time to prioritize completeness and features, but when it’s done pathologically, it inevitably leads to ruin.

I find that the lack of software quality culture often arises because no one is measuring it (I actually really don’t vibe with the ‘software craftsperson’ aesthetic that people like Uncle Bob try to push. Quality is great but it isn’t everything. It’s possible to turn this into navelgazing, and building ivory towers.). Feature velocity is measured, or at least vaguely tracked, with things like pull request counts, sprint points, or user stories. We shipped 5 stories last week, so management expects us to ship 5 this week.

Fortunately, many software quality measures are actually very easy to track. How many bugs were reported or experienced by customers last week? How much downtime did we have? How many deploys were there? Are these numbers rising or falling?

In terms of performance, most organizations would benefit from setting simple thresholds that, if exceeded, move performance work into the “bug fixing” pipeline that the organization employs. For example, an organization can commit to a maximum 95th percentile latency of 1 second. If a transaction (In New Relic parlance - a single controller action is a ‘transaction’.) exceeds that threshold, a new bug is recorded.

For organizations that want to improve the customer’s experience and perceived performance of the application, other budgets may be necessary. For example, a first-page-load time of 5 seconds. This page load target has implications that flow down throughout the stack, as one simply cannot ship 10 megabytes of JavaScript and also have a page load in 5 seconds (In fact, I would estimate that to keep page load times below 5 seconds on the average connection and hardware, you can probably ship only a few hundred KB).

Software engineers are often poor communicators, and they very often fail to communicate to other parts of the organization that prioritizing feature velocity at all costs is not sustainable.

Think of it this way: how do you think the project managers in your organization would answer the following questions?

Bezos showering in your AWS bill

Is your tolerance for the slowness of our application infinite? (i.e. can the app just beachball for all customers all the time?)
Do you have infinite money to spend on our EC2 instance bills?

If the answer to either of those questions is “no”, then it is your job as a software developer to find and make explicit those tolerances. They will be different for every organization. These performance requirements can be easily translated into automated alerts and thresholds.

You just have to have the conversation beforehand. The difference between “Hey boss - we’ve been shipping 12 points a week for the last 8 weeks and now we can’t ship anything for 6 weeks because we need to write tests and make the homepage load time somewhat bearable” and “we exceeded the limit for page load time that we all agreed upon 6 months ago, and we’ll need to reduce velocity for a while to compensate” is miles apart.

As a result of seeing this pattern often enough, I’ve changed how I phrase my consulting deliverables, as I now realize I need to provide ammunition for the engineers when bringing back my recommendations to the “business side”.

It’s not Ruby, and it isn’t (really) Rails

LEAVE MATZ ALONE

And, finally, here’s what isn’t the reason why your web application is slow: your framework or language choice. Once 90th percentile latency is lower than 500 milliseconds and median latency is below 100 milliseconds, most web application backends are no longer the bottleneck in their customer’s experience (if they ever were to begin with, which, in the age of 10 megabyte JavaScript bundles, they are usually not).

It’s 2017 and web applications don’t return flat HTML files anymore (CNN.com took 5MB of resources and 112 requests to render for me, today. R.I.P. the old light web.). Websites are gargantuan, with JavaScript bundles stretching into the size of megabytes and stylesheets that couldn’t fit in ten Apollo Guidance Computers. So how much of a difference does a web application which responds in 1 millisecond or less make in this environment?

Vanishingly little. Nowadays, the average webpage takes 5 seconds to render. Some JavaScript single-page-applications can take 12 seconds or more on initial render.

Server response times simply make up a minority part of the actual user experience of loading and interacting with a webpage - cutting 99 milliseconds off the server response time just doesn’t make a difference.

Not to mention: if Ruby on Rails, frequently maligned “as too slow” or “can’t scale”, can run several of the top 1000 websites in the world by traffic, including that little fly-by-night outfit called GitHub, then it’s a fine choice for whatever your application is. Rails is just an example here - there are many comparable frameworks in comparable languages that you could substitute like Python and Django. There are some web applications for which 100 milliseconds of latency is an unacceptable eon (advertising is the most common case), but for the vast majority of us delivering HTML or JSON to a client, that’s zippy quick.

Whither Rails Today?

One of the questions I ask in my post-workshop survey is “How do you feel writing Ruby on Rails? Would you like to keep doing it?”.

The answers I get back are always astoundingly positive. For all the FUD on the web at large, people writing Ruby are incredibly happy doing it. And that’s what keeps me writing and teaching: as long as people feel that performance concerns are keeping them from enjoying Ruby or choosing it as their tech stack, I’ll keep doing what I do.

3 ActiveRecord Mistakes That Slow Down Rails Apps: Count, Where and Present

Thu, 10 Jan 2019 07:00:00 +0000

“When does ActiveRecord execute queries? No one knows!”

ActiveRecord is great. Really, it is. But it’s an abstraction, intended to insulate you from the actual SQL queries being run on your database. And, if you don’t understand how ActiveRecord works, you may be causing SQL queries to run that you didn’t intend to.

Unfortunately, the performance costs of many features of ActiveRecord means we can’t afford to ignore unnecessary usage or treat our ORM as just an implementation detail. We need to understand exactly what queries are being run on our performance-sensitive endpoints. Freedom isn’t free, and neither is ActiveRecord.

One particular case of ActiveRecord misuse that I find is common amongst my clients is that ActiveRecord is executing SQL queries that aren’t really necessary. Most of my clients are completely unaware that this is even happening.

Unnecessary SQL is a common cause of overly slow controller actions, especially when the unnecessary query appears in a partial which is rendered for every element in a collection. This is common in search actions or index actions. This is one of the most common problems I encounter in my performance consulting. It’s a problem in nearly every app I’ve ever worked on.

One way to eliminate unnecessary queries is to poke our heads into ActiveRecord and understand its internals, and know exactly how certain methods are implemented. Today, we’re going to look at the implementation and usage of three methods which cause lots of unnecessary queries in Rails applications: count, where and present?.

How Do I Know if a Query is Unnecessary?

I have a rule of thumb to judge whether or not any particular SQL query is unnecessary. Ideally, a Rails controller action should execute one SQL query per table. If you’re seeing more than one SQL query per table, you can usually find a way to reduce that to one or two queries. If you’ve got more than a half-dozen or so queries on a single table, you almost definitely have unnecessary queries. (Please don’t email or tweet with me with ‘Well ackshually…’ on this one. It’s a guideline, not a rule, and I understand there are circumstances where more than one query per table is a good idea.)

The number of SQL queries per table can be easily seen on NewRelic, for example, if you have that installed.

I keep an eyewash station next to my desk for really bad N+1s

Another rule of thumb is that most queries should execute during the first half of a controller action’s response, and almost never during partials. Queries executed during partials are usually unintentional, and are often N+1s. These are easy to spot during a controller’s execution if you just read the logs in development mode. For example, if you see this:

User Load (0.6ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT 1  [["id", 2]]
Rendered posts/_post.html.erb (23.2ms)
User Load (0.3ms)  SELECT  "users".* FROM "users" WHERE "users"."id" = $1 LIMIT 1  [["id", 3]]
Rendered posts/_post.html.erb (15.1ms)

… you have an N+1 in this partial.

Usually, when a query is executed halfway through a controller action (somewhere deep in a partial, for example) it means that you haven’t preloaded the data that you needed.

So, let’s look specifically at the count, where and present? methods, and why they cause unnecessary SQL queries.

.count executes a COUNT every time

I see this one at almost every company I contract for. It seems to be little-known that calling count on an ActiveRecord relation will always try to execute a SQL query, every time. This is inappropriate in most scenarios, but, in general, only use count if you want to always execute a SQL COUNT right now.

“How many queries do we want per table?”

The most common cause of unnecessary count queries is when you count an association you will use later in the view (or have already used):

# _messages.html.erb
# Assume @messages = user.messages.unread, or something like that

<h2>Unread Messages: <%= @messages.count %></h2>

<% @messages.each do |message| %>
blah blah blah
<% end %>

This executes 2 queries, a COUNT and a SELECT. The COUNT is executed by @messages.count, and @messages.each executes a SELECT to load all the messages. Changing the order of the code in the partial and changing count to size eliminates the COUNT query completely and keeps the SELECT:

<% @messages.each do |message| %>
blah blah blah
<% end %>

<h2>Unread Messages: <%= @messages.size %></h2>

Why is this the case? We need not look any further than the actual method definition of size on ActiveRecord::Relation:

# File activerecord/lib/active_record/relation.rb, line 210
def size
  loaded? ? @records.length : count(:all)
end

If the relation is loaded (that is, the query that the relation describes has been executed and we have stored the result), we call length on the already loaded record array. That’s just a simple Ruby method on Array. If the ActiveRecord::Relation isn’t loaded, we trigger a COUNT query.

On the other hand, here’s how count is implemented (in ActiveRecord::Calculations):

def count(column_name = nil)
  if block_given?
    # ...
    return super()
  end

  calculate(:count, column_name)
end

And, of course, the implementation of calculate doesn’t memoize or cache anything, and executes a SQL calculation every time it is called.

Simply changing count to size in our original example would have still triggered a COUNT. The record’s wouldn’t be loaded? when size was called, so ActiveRecord will still attempt a COUNT. Moving the method after the records are loaded eliminates the query. Now, moving our header to the end of the partial doesn’t really make any logical sense. Instead, we can use the load method.

<h2>Unread Messages: <%= @messages.load.size %></h2>

<% @messages.each do |message| %>
blah blah blah
<% end %>

load just causes all of the records described by @messages to load immediately, rather than lazily. It returns the ActiveRecord::Relation, not the records. So, when size is called, the records are loaded? and a query is avoided. Voilà.

What if, in that example, we used messages.load.count? We’d still trigger a COUNT query!

When doesn’t count trigger a query? Only if the result has been cached by ActiveRecord::QueryCache. (I have some Opinions on the use of QueryCache, but that’s a post for another day.) This could occur by trying to run the same SQL query twice:

<h2>Unread Messages: <%= @messages.count %></h2>

... lots of other view code, then later:

<h2>Unread Messages: <%= @messages.count %></h2>

Every time you use count when you could have used size

In my opinion, most Rails developers should be using size in most of the places that they use count. I’m not sure why everyone seems to write count instead of size. size uses count where it is appropriate, and it doesn’t when the records are already loaded. I think it’s because when you’re writing an ActiveRecord relation, you’re in the “SQL” mindset. You think: “This is SQL, I should write count because I want a COUNT!”

So, when do you actually want to use count? Use it when you won’t actually ever be loading the full association that you’re counting. For example, take this view on Rubygems.org, which displays a single gem:

In the “versions” list, the view does a count to get the total number of releases (versions) of this gem.

Here’s the actual code:

<% if show_all_versions_link?(@rubygem) %>
  <%= link_to t('.show_all_versions', :count => @rubygem.versions.count), rubygem_versions_url(@rubygem), :class => "gem__see-all-versions t-link--gray t-link--has-arrow" %>
<% end %>

The thing is, this view never loads all of the Rubygem’s versions. It only loads five of the most recent ones, in order to show that versions list.

So, a count makes perfect sense here. Even though size would be logically equivalent (it would just execute a COUNT as well because @versions is not loaded?), it states the intent of the code in a clear way.

My advice is to grep through your app/views directory for count calls and make sure that they actually make sense. If you’re not 100% sure that you really need a real SQL COUNT right then and there, switch it to size. Worst case, ActiveRecord will still execute a COUNT if the association isn’t loaded. If you’re going to use the association later in the view, change it to load.size.

.where means filtering is done by the database

What’s the problem with this code (let’s say its _post.html.erb)

<% @posts.each do |post| %>
  <%= post.content %>
  <%= render partial: :comment, collection: post.active_comments %>
<% end %>

and in Post.rb:

class Post < ActiveRecord::Base
  def active_comments
    comments.where(soft_deleted: false)
  end
end

If you said, “this causes a SQL query to be executed on every rendering of the post partial”, you’re correct! where always causes a query. I didn’t even bother to write out the controller code, because it doesn’t matter. You can’t use includes or other preloading methods to stop this query. where will always try to execute a query!

This also happens when you call scopes on associations. Imagine instead our Comment model looked like this:

class Comment < ActiveRecord::Base
  belongs_to :post

  scope :active, -> { where(soft_deleted: false) }
end

Allow me to sum this up with two rules: Don’t call scopes on associations when you’re rendering collections and don’t put query methods, like where, in instance methods of an ActiveRecord::Base class.

Calling scopes on associations means we cannot preload the result. In the example above, we can preload the comments on a post, but we can’t preload the active comments on a post, so we have to go back to the database and execute new queries for every element in the collection.

This isn’t a problem when you only do it once, and not on every element of a collection (like every post, as above). Feel free to use scopes galore in those situations - for example, if this was a PostsController#show action that only displayed one post and its associated comments. But in collections, scopes on associations cause N+1s, every time.

The best way I’ve found to fix this particular problem is to create a new association. Justin Weiss, of “Practicing Rails”, taught me this in this blog post about preloading Rails scopes. The idea is that you create a new association, which you can preload:

class Post
  has_many :comments
  has_many :active_comments, -> { active }, class_name: "Comment"
end

class Comment
  belongs_to :post
  scope :active, -> { where(soft_deleted: false) }
end

class PostsController
  def index
    @posts = Post.includes(:active_comments)
  end
end

The view is unchanged, but now executes just 2 SQL queries, one on the Posts table and one on the Comments table. Nice!

<% @posts.each do |post| %>
  <%= post.content %>
  <%= render partial: :comment, collection: post.active_comments %>
<% end %>

The second rule of thumb I mentioned, don’t put query methods, like where, in instance methods of an ActiveRecord::Base class, may seem less obvious. Here’s an example:

class Post < ActiveRecord::Base
  belongs_to :post

  def latest_comment
    comments.order('published_at desc').first
  end

What happens if the view looks like this?

<% @posts.each do |post| %>
  <%= post.content %>
  <%= render post.latest_comment %>
<% end %>

That’s a SQL query on every post, regardless of what you preloaded. In my experience, every instance method on an ActiveRecord::Base class will eventually get called inside a collection. Someone adds a new feature and isn’t paying attention. Maybe it’s by a different developer than the one who wrote the method originally, and they didn’t fully read the implementation. Ta-da, now you’ve got an N+1. The example I gave could be rewritten as an association, like I described earlier. That can still cause an N+1, but at least it can be fixed easily with the correct preloading.

Which ActiveRecord methods should we avoid inside of our ActiveRecord model instance methods? Generally, it’s pretty much everything in the QueryMethods, FinderMethods, and Calculations. Any of these methods will usually try to run a SQL query, and are resistant to preloading. where is the most frequent offender, however.

any?, exists? and present?

Rails programmers have been struck by a major affliction - they’re adding a particular predicate method to just about every variable in their applications. present? has spread across Rails codebases faster than the plague in 13th century Europe. The vast majority of the time, the predicate adds nothing but verbosity, and really, all the author needed was a truthy/falsey check, which they could have done by just writing the variable name.

Here’s an example from CodeTriage, a free and open-source Rails application written by my friend Richard Schneeman:

class DocComment < ActiveRecord::Base
  belongs_to :doc_method, counter_cache: true

  # ... things removed for clarity...

  def doc_method?
    doc_method_id.present?
  end
end

What is present? doing here? One, it transforms the value of doc_method_id from either nil or an Integer into true or false. Some people have Strong Opinions about whether predicates should return true/false or can return truthy/falsey. I don’t. But adding present? also does something else, and we have to look at the implementation to figure out what:

class Object
  def present?
    !blank?
  end
end

blank? is a more complicated question than “is this object truthy or falsey”. Empty arrays and hashes are truthy, but blank, and empty strings are also blank?. In the example above from CodeTriage, however, the only things that doc_method_id will ever be is nil or Integer, meaning present? is logically equivalent to !!:

def doc_method?
  !!doc_method_id
  # same as doc_method_id.present?
end

Using present? in cases like this is the wrong tool for the job. If you don’t care about “emptiness” in the value you’re calling the predicate on (i.e. the value cannot be [] or {}), use the simpler (and much faster) language features available to you. I sometimes see people even do this on values which are already boolean, which means you’re just adding verbosity and making me wonder if there’s some weird edge cases I’m not seeing.

Alright, that’s my style gripe. I understand that you may not agree. present? makes more sense when dealing with strings, which can frequently be empty ("").

Where people get into trouble is calling predicates, such as present?, on ActiveRecord::Relation objects. Let’s say you need to know if an ActiveRecord::Relation has any records. You can use the English-language synonyms any?/present?/exists? or their negations none?/blank?/empty?. Surely it doesn’t matter which method you choose, right? Just pick the one that sounds the most natural when read aloud? Nope.

What SQL queries do you think the following code will execute? Assume @comments is an ActiveRecord::Relation.

- if @comments.any?
  h2 Comments on this Post
  - @comments.each do |comment|

The answer is two. One will be an existence check, triggered by @comments.any? (SELECT 1 AS one FROM ... LIMIT 1), then the @comments.each line will trigger a loading of the entire relation (SELECT "comments".* FROM "comments" WHERE ...).

What about this?

- unless @comments.load.empty?
  h2 Comments on this Post
  - @comments.each do |comment|

This one only executes one query - @comments.load loads the entire relation right away with SELECT "comments".* FROM "comments" WHERE ....

And this one?

- if @comments.exists?
  This post has
  = @comments.size
  comments
- if @comments.exists?
  h2 Comments on this Post
  - @comments.each do |comment|

Four! exists? doesn’t memoize itself and it doesn’t load the relation. exists? here triggers a SELECT 1 ..., .size triggers a COUNT because the relation hasn’t been loaded yet, and then the next exists? triggers ANOTHER SELECT 1 ... and finally @comments loads the entire relation! Yay! Isn’t this fun? You could reduce this down to just 1 query with the following:

- if @comments.load.any?
  This post has
  = @comments.size
  comments
- if @comments.any?
  h2 Comments on this Post
  - @comments.each do |comment|

And it just gets better - this behavior changes depending if you’re on Rails 4.2, Rails 5.0 or Rails 5.1+.

Here’s how it works in Rails 5.1+:

method	SQL generated	memoized?	implementation	Runs query if `loaded?`
present?	SELECT “users”.* FROM “users”	yes (`load`)	Object (!blank?)	no
blank?	SELECT “users”.* FROM “users”	yes (`load`)	`load`; `blank?`	no
any?	SELECT 1 AS one FROM “users” LIMIT 1	no unless `loaded`	`!empty?`	no
empty?	SELECT 1 AS one FROM “users” LIMIT 1	no unless `loaded`	`exists?` if !`loaded?`	no
none?	SELECT 1 AS one FROM “users” LIMIT 1	no unless `loaded`	`empty?`	no
exists?	SELECT 1 AS one FROM “users” LIMIT 1	no	ActiveRecord::Calculations	yes

Here’s how it works in Rails 5.0:

method	SQL generated	memoized?	implementation	Runs query if `loaded?`
present?	SELECT “users”.* FROM “users”	yes (`load`)	Object (!blank?)	no
blank?	SELECT “users”.* FROM “users”	yes (`load`)	`load`; `blank?`	no
any?	SELECT COUNT(*) FROM “users”	no unless `loaded`	`!empty?`	no
empty?	SELECT COUNT(*) FROM “users”	no unless `loaded`	count(:all) > 0	no
none?	SELECT COUNT(*) FROM “users”	no unless `loaded`	`empty?`	no
exists?	SELECT 1 AS one FROM “users” LIMIT 1	no	ActiveRecord::Calculations	yes

Here’s how it works in Rails 4.2:

method	SQL generated	memoized?	implementation	Runs query if `loaded?`
present?	SELECT “users”.* FROM “users”	yes	Object (!blank?)	no
blank?	SELECT “users”.* FROM “users”	yes	to_a.blank?	no
any?	SELECT COUNT(*) FROM “users”	no unless `loaded`	`!empty?`	no
empty?	SELECT COUNT(*) FROM “users”	no unless `loaded`	count(:all) > 0	no
none?	SELECT “users”.* FROM “users”	yes (`load` called)	Array	no
exists?	SELECT 1 AS one FROM “users” LIMIT 1	no	ActiveRecord::Calculations	yes

any?, empty? and none? remind me of the implementation of size - if the records are loaded? do a simple method call on a basic Array, if they’re not loaded, always run a SQL query. exists? has no caching or memoization built in, just like other ActiveRecord::Calculations. This means that exists?, which is another method people like to write in these circumstances, is actually much worse than present? in some cases!

These six predicate methods, which are English-language synonyms all asking the same question, have completely different implementations and performance implications, and these consequences depend on which version of Rails you are using. So, let me distill all of the above into some concrete advice:

present? and blank? should not be used if the ActiveRecord::Relation will never be used in its entirety after you call present? or blank?. For example, @my_relation.present?; @my_relation.first(3).each.
any?, none? and empty? should probably be replaced with present? or blank? unless you will only take a section of the ActiveRecord::Relation using first or last. They will generate an extra existence SQL check if you’re just going to use the entire relation if it exists. In essence, change @users.any?; @users.each... to @users.present?; @users.each... or @users.load.any?; @users.each..., but @users.any?; @users.first(3).each is fine.
exists? is a lot like count - it is never memoized, and always executes a SQL query. Most people probably do not actually want this behavior, and would be better off using present? or blank?

Conclusion

As your app grows in size and complexity, unnecessary SQL can become a real drag on your application’s performance. Each SQL query involves a round-trip back to the database, which entails, usually, at least a millisecond, and sometimes much more for complex WHERE clauses. Even if one extra exists? check isn’t a big deal, if it suddenly happens in every row of a table or a partial in a collection, you’ve got a big problem!

ActiveRecord is a powerful abstraction, but since database access will never be “free”, we need to be aware of how ActiveRecord works internally so that we can avoid database access in unnecessary cases.

App Checklist

Look for uses of present?, none?, any?, blank? and empty? on objects which may be ActiveRecord::Relations. Are you just going to load the entire array later if the relation is present? If so, add load to the call (e.g. @my_relation.load.any?)
Be careful with your use of exists? - it ALWAYS executes a SQL query. Only use it in cases where that is appropriate - otherwise use present? or any other the other methods which use empty?
Be extremely careful using where in instance methods on ActiveRecord objects - they break preloading and often cause N+1s when used in rendering collections.
count always executes a SQL query - audit its use in your codebase, and determine if a size check would be more appropriate.

The Complete Guide to Rails Performance, Version 2

Fri, 01 Jun 2018 07:00:00 +0000

Today, the Complete Guide to Rails Performance has been updated to version 2.0. You can purchase it here.

All existing purchasers have had their copies updated on Gumroad. When I started this project, I always believed that a digital course should be better than a typical paperback programming book. That’s why I don’t include any DRM or proprietary video codecs. That’s why I think, like most software, updates should be free.

“Version 2.0” isn’t quite as drastic a change as a software v2.0, though. The world of Rails performance has actually changed very little since I wrote the course 2 years ago. The apps I consult on still have many of the same problems. The V2 update reflects this: I have revised the content for clarity, and updated a few places to reflect changes in Ruby 2.5 and Rails 5.2, but it is mostly still the same. I have also added four lessons: memory fragmentation, application server config, GC tuning, and PGBouncer config. These lessons were added based on new problems and thinking I’ve had since the course was released. Web-Scale Package purchasers will also get a new interview with Noah Gibbs of Appfolio next week.

So, what does it mean that not much has changed in the Rails performance world?

This tweet put me in an introspective mood this morning:

It is profoundly sad how Rails has institutionalized a "nobody cares" attitude toward performance. https://t.co/UhzvxyLjuz

— Jeff Atwood (@codinghorror), June 1, 2018

To summarize, Jeff’s cofounder, Sam Saffron (who I interviewed for the CGRP), wrote a great, in-depth blog post about memory use in ActiveRecord. In short, Sam finds that ActiveRecord creates excessive amounts of objects, even when doing simple and supposedly “optimized” work. Sam posted a proof-of-concept patch which improves this quite a bit.

Jeff’s tweet diminishes the work of many Rails contributors. Aaron Patterson has spent the last two years working on Rails performance and a compacting garbage collector. Richard Schneeman has improved Sprockets’ performance a great deal. Sam Saffron himself has contributed over a dozen performance improvements to Rails, which, as far I can tell, have all been accepted. I know also that Andrew White, Eileen Uchitelle, and Rafael Franca are all Rails core members that care deeply about performance (probably because all of them have day-jobs running large Rails applications!). So any idea that Rails’ contributors or core members “don’t care” about performance is laughingly misguided, and is an opinion that can only really be held by someone outside the community. The way Jeff tried to turn it around in the replies into a “hot take” that people should “get angry” and “punk rock” about the “status quo” just made it more obvious.

It’s pretty easy to take potshots at a mature framework like Ruby on Rails. It has almost 13 years of history behind it. There’s going to be cruft, baggage, and outdated decisions baked in. That’s what happens. But there’s also tremendous productivity, something gained from the thousands of contributors who have all contributed their “lessons learned” back to the framework. But if you forget about that history, it’s easy to craft a benchmark to make it look like that history has overtaken it’s usefulness in the present.

This is the gap I’ve tried to bridge in my writing and in publishing The Complete Guide to Rails Performance. I believe that performance problems in Rails are pedagogical, not technical. It’s not because we don’t have enough people working on performance (though it helps!). It’s not because we don’t value it as a community (how many times do I have to cite all of the top 10,000 websites that run Rails at speed?). It’s because Rails (and Ruby) optimizes for programmer happiness, and that means we provide sharp tools which are easy to cut yourself on. Rather than throw the tools out, I think we need to teach people to use them safely.

ActiveRecord is probably the best example of what I’m talking about. It’s an extremely productive tool. It works very well for 80% of web-app use-cases. But every year, someone wants to throw it out and thinks that some other Rubygem or pattern (e.g. DataMapper) will save them. It’s so easy to craft a line of code with ActiveRecord that will slow your application to a crawl if you’re not thinking through through the consequences, as anyone who has written User.all.each can tell you.

There is no One True Pattern or One True Framework. But there is a Thing Which Works For Most People. And if you end up being one of the 20% for whom it doesn’t work so well, or the tool’s productivity preference means that it’s easier to make performance mistakes, I don’t think that’s the tool or framework’s fault.

In this way, I think publishing the Complete Guide to Rails Performance was placing my faith in the developer community of Rails. If I didn’t think that people could make their Rails apps faster through knowledge and skills, and instead they had to wait until the framework or the language itself got faster, I would have gone to work at Github or Shopify and made a bunch of patches to Rails and Ruby. I might have started an alternative, “lightweight” framework or ORM that prioritized performance over usefulness and productivity. Instead, I think that teaching Rails developers how to find and fix performance problems will make a bigger dent in the average Rails app’s response time than improving the language or framework’s performance by even 2-3x, or by removing “dangerous” features.

As I think we’ve slowly discovered over the course of trying to make Ruby 3x faster, there is no “waste” or “bloat” that can be cut out of a framework or language without cost that suddenly makes the whole thing faster. It’s sort of like how politicians always promise to “cut waste in government spending”, but no-one can ever tell you exactly where or how which programs will be cut. Everything was implemented for a reason. There is no magic wand or amount of man-hours that can be waved at these problems. I’ve discovered this in my consulting and writing as well. I wish it was that easy. But it isn’t.

However, far from Jeff’s doomsday attitude, I believe that the macro picture for Ruby and Rails performance looks good, as it always had. Ruby 2.0 to 2.5 made a number of incremental performance improvements, particularly in garbage collection. I feel like the community has become more mature and performance-savvy over the last few years too. We’re waking up the mainstream Rails developer to things like jemalloc and teaching them how to use ActiveRecord and avoid performance issues.

The technical future of Ruby looks strong, too. Ruby 2.6 will contain a JIT compiler. How cool is that? TruffleRuby has made great progress to becoming useable enough to run a Rails application. JRuby continues to truck along with more performance improvements and compatibility fixes all the time. The technical future of the language hardly looks dim - in fact, I think it’s much brighter than it was in 2011, when I got started in Ruby and Rails.

I’ll continue to do my part for the Rails performance community by publishing and writing, to improve the technical skills and capacity of the average Rails developer so that they can make their apps faster. Here’s to you, developers!

A New Ruby Application Server: NGINX Unit

Wed, 28 Mar 2018 07:00:00 +0000

There’s a new application server on the block for Rubyists - NGINX Unit. As you could probably guess by the name, it’s a project of NGINX Inc., the for-profit open-source company that owns the NGINX web server. In fall of 2017, they announced the NGINX Unit project. It’s essentially an application server designed to replace all of the various application servers used with NGINX. In Ruby’s case, that’s Puma, Unicorn, and Passenger. (For a far more in-depth comparison of these application servers, read my article about configuring Puma, Passenger and Unicorn) NGINX Unit also runs Python, Go, PHP and Perl.

The overarching idea seems to be to make microservice administration a lot easier. One NGINX Unit process can run any number of applications running any number of languages - for example, one NGINX Unit server can manage a half-dozen different Ruby applications, each running a different version of the Ruby runtime. Or you can run a Ruby application and a Python application side-by-side. The combinations are only limited by your system resources.

Unfortunately, the “microservice” space is quite prone to buzzword-laden marketing pages. (I really don’t like when software projects advertise themselves as “modern”. It’s like “subtweeting” all pre-existing software projects in this problem space and saying they’re all old and busted, and this is the New Way To Do Things. Why it’s better than the “old busted ways” is never explicitly stated. This kind of marketing preys on software developer’s fear of becoming obsolete in their skillset, rather than making any substantive point.) Words like “dynamic”, “modular”, “lightweight” are mixed in with “service mesh”, “seamless” and “graceful”. This article is going to be about cutting through the marketing and getting into what NGINX Unit means for those of us running production Ruby applications.

Before I move on to more about NGINX Unit’s architecture and what makes it unique, let’s make sure we all understand the difference between an application server and a web server. A web server connects to clients over HTTP, and usually serves static files or proxies to other HTTP-enabled servers, and acts as a middleman. An application server is the thing which actually starts and runs the language runtime. In Ruby, these functions are sometimes combined. For example, all of the major Ruby application servers also are web servers. However, many web servers, such as Nginx and Apache, are not also application servers. Nginx UNIT is both a web and application server.

NGINX Unit runs four different types of processes: main, router, controller, and application. Application processes are the self-explanatory ones - this would just be the Ruby runtime running your Rails application. The router and controller processes, and how they interact with each other and the application processes, is what defines how NGINX Unit works.

The main process creates the router and application processes. That’s really all it does. Application processes in NGINX Unit are dynamic, however - the number of processes running can be changed at any time, Ruby versions can be changed, or even entire new applications can be added while the server is running. The thing that tells the main process what application processes to run is the controller process.

The controller process (like the main process, there’s only one) has two jobs: expose a JSON configuration API over HTTP, and configure the router and main processes. This is probably the most novel and interesting part of NGINX Unit for Rubyists. Rather than working with configuration files, you POST JSON objects to the controller process to tell it what to do. For example, with this json file:

{
    "listeners": {
        "*:3000": {
            "application": "rails"
        }
    },

    "applications": {
        "rails": {
            "type": "ruby",
            "processes": 5,
            "script": "/www/railsapp/config.ru"
        }
    }
}

… we can PUT it to an NGINX Unit controller process with this (assuming our NGINX Unit server is listening on port 8443):

curl -d "myappconfig.json" -X PUT '127.0.0.1:8443'

… and create a new Ruby application.

NGINX Unit’s JSON configuration object is divided into listeners and applications. Applications are the actual apps you want to run. Listeners are where those apps are exposed to the world (i.e. what port they’re on).

Changes in application and listener configuration are supposed to be seamless. For example, a “hot deploy” of a new version of your application would be accomplished by adding a new application to the configuration:

{
  "rails-new": {
      "type": "ruby",
      "processes": 5,
      "script": "/www/rails-new-app/config.ru"
  }
}

curl -d "mynewappconfig.json" -X PUT

and then switching the listener to the new application:

curl -X PUT -d '"rails-new"' '127.0.0.1:8443/listeners/*:3000/application`

This transition is (supposedly) seamless, and clients won’t notice. This is similar to a Puma “phased restart”. In a phased restart in Puma, each worker process is restarted one at a time, which means that the other works processes are up and available to take requests. Puma accomplishes this using a control server (managed by the pumactl utility). However, unlike Puma, NGINX Unit “hot restarts” will not have two versions of the application taking requests at the same time.

In a Puma phased restart, say your application has six workers. Halfway through the phased restart, 3 works will be running the old code, and half will be running new code. This can cause some problems with database schema changes, for example. NGINX Unit restarts happen “all at once”, so while two versions of the code will be running at once, only one version will be taking requests at any point in time.

This functionality seems quite useful to those who are running their own Ruby applications on a service such as AWS, where you have to manage your own deployment. However, Heroku users won’t find any of this useful, as you’ve already had this sort of “hot deploy” functionality using Heroku’s preboot system. However, these two features aren’t doing exactly the same thing. Heroku creates an entirely new virtual server and hot-swaps the whole thing, whereas NGINX Unit is just changing processes on a single machine, but they’re completely the same from a client perspective.

The router process is pretty much what it sounds like - the thing which turns HTTP connections from clients into requests to the web application processes. NGINX claims a single Unit router can handle thousands of simultaneous connections. The router works a lot like an NGINX web server, and has a number of worker threads to accept, buffer and parse incoming connections.

To me, this is one of the most exciting parts of NGINX Unit for Rubyists. It is very difficult for Ruby application servers to deal with HTTP connections without some kind of reverse proxy in front of the app server. Unicorn, for example, is recommended for use only behind a reverse proxy because it cannot buffer requests. That is, if a client sends one byte of their request and then stops (due to network conditions, a bad cellphone connection perhaps), then the Unicorn process just stops all work and cannot continue until that request has finished buffering. Using NGINX, for example, in front of Unicorn allows NGINX to buffer that request before it reaches Unicorn. Since NGINX is written in highly optimized C and it’s not restricted by Ruby’s GVL, it can buffer hundreds of connections for Unicorn. Passenger solves this problem by basically just being an addon for NGINX or Apache (Now you know why it’s called Passenger!) (mod_ruby!) and offloading all of the connection-related work to the webserver. In this way, NGINX Unit is more similar to Passenger than it is to Unicorn.

The application configuration has a processes key. This key can have a minimum number and maximum number of processes:

{
  "rails-new": {
      "type": "ruby",
      "processes": {
        "spare": 5
        "max": 10
      },
      "script": "/www/rails-new-app/config.ru"
  }
}

For some reason, the “minimum” number of processes is called “spare”. The config above will start 5 processes immediately, and will scale to 10 if the load requires it.

No word yet on if any settings like Puma’s preload_app! and similar settings in Passenger and Unicorn are available so you will be able to start up processes before they are needed and take advantage of copy-on-write memory.

This leaves the application processes. The interesting and novel thing here is that the router does not communicate with the application processes via HTTP - it uses Unix sockets and shared memory. This looks like an optimization aimed at microservice architectures, as communicating between services on the same machine will be considerably faster without any HTTP in between. I have yet to see any Ruby code examples of how this could work, however.

It is unclear to me in the long-term if it is intended for you to run NGINX in front of NGINX Unit, or if NGINX Unit can run on it’s own without anything in front of it. As of right now (Q1 2018), you should probably be running NGINX in front of NGINX Unit as a reverse proxy, because NGINX Unit lacks static file serving, HTTPS (TLS), and HTTP/2. Obviously, the integration is pretty seamless.

NGINX Unit is approaching a stable 1.0 release. You can’t really run it in production right now for Ruby applications: As I write this sentence, the Ruby module is literally 5 days old. It’s still under very active development right now - minor versions are released every few weeks. TLS and HTTP-related features seem like the next “big features” to come down the pipe, with static file serving being next. There is some discussion about support for Java, which could probably be turned into support for JRuby and TruffleRuby as well.

There is no Windows support, and I don’t think I would hold my breath for any in the future. NGINX Unit only supports Ruby 2.0 and above.

I will not be benchmarking NGINX Unit in this post. It’s Ruby module is extremely new and probably not ready for any kind of benchmarking. However, the real reason I won’t be benchmarking NGINX Unit against Puma, Unicorn or Passenger is because application server choice in Ruby is not a matter of speed (techincally, latency) but throughput. Application servers tend to differ in how many requests they can serve in parallel, rather than how quickly they do it. Application servers impose very little latency overhead on the applications they serve, probably on the order of a couple of milliseconds.

The most important Ruby application server setting which affects throughput is threading. The reason is that it is the only application server setting which can increase the number of requests served concurrently. A multithreaded Ruby application server can make greater and more efficient use of the available CPU and memory resources and serve more requests-per-minute than a single-threaded Ruby application process.

Currently, the only free application server which runs Ruby web applications in multiple threads is Puma. Passenger Enterprise will do it, but you must pay for a license.

NGINX Unit plans support for multiple threads in Python applications, so it is not inconceivable that it will support Ruby applications in multiple threads sometime in the future.

So, how does NGINX Unit currently “shake out” in comparison to Unicorn, Passenger and Puma? I think that the traditional Rails application setup: one monolithic application, run on a Plaform-as-a-Service provider like Heroku will probably not see any benefit at all from NGINX Unit’s current features and planned roadmap. Puma already serves these users very well.

NGINX Unit may be interesting for Unicorn users who want to stop using a reverse proxy. Once NGINX Unit’s HTTP features are fleshed out, it could replace a Unicorn/NGINX setup with just a single NGINX Unit server.

NGINX Unit is probably most directly comparable to Phusion Passenger, which also recently went into the “microservice” realm by supporting Javascript and Python as well as Ruby applications. NGINX Unit currently supports more languages and will probably support even more in the future, so those that need greater language support will probably switch. However, Phusion is a Ruby-first company, so I expect Passenger to always “support” Ruby in a better, more complete way than NGINX Unit ever will. And, as mentioned above, Phusion Passenger Enterprise supports multithreaded execution today.

So, what is the ideal NGINX Unit app? If you’re running your own cloud (that is, not on a service which manages the routing for you, like Heroku) and you have many Ruby applications running on different Ruby versions or many services in many different languages and those services/apps need to talk to each other, quickly, it looks like NGINX Unit was designed for you. If you don’t fit that profile, though, it’s probably best to stick to the existing top three options (Puma, Passenger, and Unicorn).

Malloc Can Double Multi-threaded Ruby Program Memory Usage

Mon, 04 Dec 2017 07:00:00 +0000

Sometimes, it really is that simple.

It’s not every day that a simple configuration change can completely solve a problem.

I had a client whose Sidekiq processes were using a lot of memory - about a gigabyte each. They would start at about 300MB each, then slowly grow over the course of several hours to almost a gigabyte, where they would start to level off.

I asked him to change a single environment variable: MALLOC_ARENA_MAX. “Please set it to 2.”

His processes restarted, and immediately the slow growth was eliminated. Processes settled at about half the memory usage they had before - around 512MB each.

Actually, it’s not that simple. There are no free lunches. Though this one might be close to free. Like a ten cent lunch.

Now, before you go copy-pasting this “magical” environment variable into all of your application environments, know this: there are drawbacks. You may not be suffering the problem it solves. There are no silver bullets.

Ruby is not known for being a language that’s light on memory use. Many Rails applications suffer from up to a gigabyte of memory use per process. That’s approaching Java levels. Sidekiq, the popular Ruby background job processor, has processes which can get just as large or even larger. The reasons are many, but one reason in particular is extremely difficult to diagnose and debug: fragmentation.

Typical Ruby memory growth looks logarithmic.

The problem manifests itself as a slow, creeping memory growth in Ruby processes. It is often mistaken for a memory leak. However, unlike a memory leak, memory growth due to fragmentation is logarithmic, while memory leaks are linear.

A memory leak in a Ruby program is usually caused by a C-extension bug. For example, if your Markdown parser leaks 10kb every time you call it, your memory growth will continue forever at a linear rate, since you tend to call the markdown parser at a regular frequency.

Memory fragmentation causes logarithmic growth in memory. It looks like a long curve, approaching some unseen limit. All Ruby processes experience some memory fragmentation. It’s an inevitable consequence of how Ruby manages memory.

In particular, Ruby cannot move objects in memory. Doing so would potentially break any C language extensions which are holding raw pointers to a Ruby object. If we can’t move objects in memory, fragmentation is an inevitable result. It’s a fairly common issue in C programs, not just Ruby.

Actual client graph. This is what fragmentation looks like. Note the enormous drop after MALLOC_ARENA_MAX changed to 2.

However, fragmentation can sometimes cause Ruby programs to twice as much memory as they would otherwise, sometimes as much as four times more!

Ruby programmers aren’t used to thinking about memory, especially not at the level of malloc. And that’s OK: the entire language is designed to abstract memory away from the programmer. It’s right in the manpage. But while Ruby can guarantee memory safety, it cannot provide perfect memory abstraction. One cannot be completely ignorant of memory. Because Ruby programmers are often inexperienced with how computer memory works, when problems occur, they often have no idea where to even start with debugging it, and may dismiss it as an intrinsic feature of a dynamic, interpreted language like Ruby.

“And underneath 4 layers of memory abstraction, she noticed some fragmentation!”

What makes it worse is that memory is abstracted away from Rubyists through four separate layers. First is the Ruby virtual machine itself, which has its own internal organization and memory tracking features (sometimes called the ObjectSpace). Second is the allocator, which differs greatly in behavior depending on the particular implementation you’re using. Third is the operating system, which abstracts actual physical memory addresses away into virtual memory addresses. The way it does this varies significantly depending on the kernel - Mach does this much differently than Linux, for example. Finally, there’s the actual hardware itself, which uses several strategies to keep frequently-accessed data in “hot” locations where it can be more quickly accessed. There are even special parts of the CPU involved here, such as the translation lookaside buffer.

This is what makes memory fragmentation so difficult for Rubyists to deal with. It’s a problem that generally happens at the level of the virtual machine and the allocator, parts of the Ruby language that 95% of Rubyists are probably unfamiliar with.

Some fragmentation is inevitable, but it can also get so bad that it doubles the memory usage of your Ruby processes. How can you know if you’re suffering the latter rather than the former? What causes critical levels of memory fragmentation? Well, I have one thesis about a cause of memory fragmentation which affects multithreaded Ruby applications, like webapps running on Puma or Passenger Enterprise, and multithreaded job processors such as Sidekiq or Sucker Punch.

Per-Thread Memory Arenas in glibc Malloc

It all boils down to a particular feature of the standard glibc malloc implementation called “per-thread memory arenas”.

To understand why, I need to explain how garbage collection works in CRuby really quickly.

ObjectSpace visualization by Aaron Patterson. Each pixel is an RVALUE. Green is “new”, red is “old”. See heapfrag.

All objects have a entry in the ObjectSpace. The ObjectSpace is a big list which contains an entry for every Ruby object currently alive in the process. The list entries take the form of RVALUEs, which are 40-byte C structs that contain some basic data about the object. The exact contents of these structs varies depending on the class of the object. As an example, if it is a very short String like “hello”, the actual bits that contain the character data are embedded directly in the RVALUE. However, we only have 40 bytes - if the string is 23 bytes or longer, the RVALUE contains only a raw pointer to where the object data actually lies in memory, outside the RVALUE.

RVALUEs are further organized in the ObjectSpace into 16KB “pages”. Each page contains about 408 RVALUEs.

These numbers can be confirmed by looking at the GC::INTERNAL_CONSTANTS constant in any Ruby process:

GC::INTERNAL_CONSTANTS
=> {
:RVALUE_SIZE=>40,
:HEAP_PAGE_OBJ_LIMIT=>408,
# ...
}

Creating a long string (let’s say it’s a 1000-character HTTP response for example) looks like this:

Add an RVALUE to the ObjectSpace list. If we are out of free slots in the ObjectSpace, we lengthen the list by 1 heap page, calling malloc(16384).
Call malloc(1000) and receive a address to a 1000-byte memory location. (Actually, Ruby will request an area slightly larger than it needs in case the string is added to or resized.) This is where we’ll put our HTTP response.

The malloc calls here are what I want to bring your attention to. All we’re doing is asking for a memory location of a particular size, somewhere. Actually, malloc’s contiguity is undefined, that is, it makes no guarantees about where that memory location will actually be. This means that, from the perspective of the Ruby VM, fragmentation (which is fundamentally a problem about where memory is) is a problem of the allocator. (However, allocation patterns and sizes can definitely make things harder for the allocator.)

Ruby can, in a way, measure the fragmentation of its own ObjectSpace. A method in the GC module, GC.stat, provides a wealth of information about the current memory and GC state. It’s a little overwhelming and is under-documented, but the output is a hash that looks like this:

GC.stat
=> {
:count=>12,
:heap_allocated_pages=>91,
:heap_sorted_length=>91,
# ... way more keys ...
}

There are two keys in this hash that I want to point your attention to: GC.stat[:heap_live_slots] and GC.stat[:heap_eden_pages].

:heap_live_slots refers to the number of slots in the ObjectSpace currently occupied by live (not marked for freeing) RVALUE structs. This is roughly the same as “currently live Ruby objects”.

The Eden heap

:heap_eden_pages is the number of ObjectSpace pages which currently contain at least one live slot. ObjectSpace pages which have at least one live slot are called eden pages. ObjectSpace pages which contain no live objects are called tomb pages. This distinction is important from the GC’s perspective, because tomb pages can be returned back to the operating system. Also, the GC will put new objects into eden pages first, and then tomb pages after all the eden pages have filled up. This reduces fragmentation.

If you divide the number of live slots by the number of slots in all eden pages, you get a measure of the current fragmentation of the ObjectSpace. As an example, here’s what I get in a fresh irb process:

5.times { GC.start }
GC.stat[:heap_live_slots] # 24508
GC.stat[:heap_eden_pages] # 83
GC::INTERNAL_CONSTANTS[:HEAP_PAGE_OBJ_LIMIT] # 408

# live_slots / (eden_pages * slots_per_page)
# 24508 / (83 * 408) = 72.3%

About 28% of my eden page slots are currently unoccupied. A high percentage of free slots indicates that the ObjectSpace’s RVALUEs are spread across many more heap pages than they would be if we could move them around. This is a kind of internal memory fragmentation.

Another measure of internal fragmentation in the Ruby VM comes from GC.stat[:heap_sorted_length]. This key is the “length” of the heap. If we have three ObjectSpace pages, and I free the 2nd one (the one in the middle), I only have two heap pages remaining. However, I cannot move heap pages around in memory, so the “length” of the heap (essentially the highest index of the heap pages) is still 3.

Yes, this heap is fragmented, but it looks really tasty.

Dividing GC.stat[:heap_eden_pages] by GC.stat[:heap_sorted_length] gives a measure of internal fragmentation at the level of ObjectSpace pages - a low percentage here would indicate a lot of heap-page-sized “holes” in the ObjectSpace list.

While these measures are interesting, most memory fragmentation (and most allocation) doesn’t happen in the ObjectSpace - it happens in the process of allocating space for objects which don’t fit inside a single RVALUE. It turns out that’s most of them, according to experiments performed by Aaron Patterson and Sam Saffron. A typical Rails app’s memory usage will be 50%-80% in these malloc calls to get space for objects larger than a few bytes.

Well this sucks. Looks like only 15% of the heap in a basic Rails app is managed by the GC. 85% is just mallocs pic.twitter.com/sPbtAq4g8j
— Aaron Patterson (@tenderlove) June 28, 2017

When Aaron says “managed by the GC” here, he means “inside the ObjectSpace list”.

Ok, so let’s talk about where per-thread memory arenas come in.

The per-thread memory arena was an optimization introduced in glibc 2.10, and lives today in arena.c. It’s designed to decrease contention between threads when accessing memory.

In a naive, basic allocator design, the allocator makes sure only one thread can request a memory chunk from the main arena at a time. This ensures that two threads don’t accidentally get the same chunk of memory. If they did, that would cause some pretty nasty multi-threading bugs. However, for programs with a lot of threads, this can be slow, since there’s a lot of contention for the lock. All memory access for all threads is gated through this lock, so you can see how this could be a bottleneck.

Removing this lock has been an area of major effort in allocator design because of its performance impact. There are even a few lockless allocators out there.

The per-thread memory arena implementation alleviates lock contention with the following process (paraphrased from this article by Siddhesh Poyarekar):

We call malloc in a thread. The thread attempts to obtain the lock for the memory arena it accessed previously (or the main arena, if no other arenas have been created).
If that arena is not available, try the next memory arena (if there are any other memory arenas).
If none of the memory arenas are available, create a new arena and use that. This new arena is linked to to the last arena in a linked list.

In this way, the main arena is basically extended into a linked list of arenas/heaps. The number of arenas is limited by mallopt, specifically the M_ARENA_MAX parameter (documented here, note the “environment variables” section). By default, the limit on the number of per-thread memory arenas that can be created is 8 times the number of available cores. Most Ruby web applications run about 5 threads per core, and Sidekiq clusters can often run far more than that. In practice, this means that many, many per-thread memory arenas can get created by a Ruby application.

Let’s take a look at exactly how this would play out in a multithreaded Ruby application.

You are running a Sidekiq process with the default setting of 25 threads.
Sidekiq begins running 5 new jobs. Their job is to communicate with an external credit card processor - so they POST a request via HTTPS and receive a response ~3 seconds later.
Each job (which is running a separate thread in Rubyland) sends an HTTP request and waits for a response using the IO module. Generally, almost all IO in CRuby releases the Global VM lock, which means that these threads are working in parallel and may contend for the main memory arena lock, causing the creation of new memory arenas.

If multiple CRuby threads are running but not doing I/O, it is pretty much impossible for them to contend for the main memory arena because the Global VM Lock prevents two Ruby threads from executing Ruby code at the same time. Thus, per-thread-memory arenas only affect CRuby applications which are both multithreaded and performing I/O.

How does this lead to memory fragmentation?

Bin-packing can be fun, too!

Memory fragmentation is essentially a bin packing problem - how can we efficiently distribute oddly-sized items between multiple bins so that they take up the least amount of space? Bin-packing is made much more difficult for the allocator because a) Ruby never moves memory locations around (once we allocate a location, the object/data stays there until it is freed) b) per-thread memory arenas essentially create a lot of different bins, which cannot be combined or “packed” together. Bin-packing is already NP-hard, and these constraints just make it even more difficult to achieve an optimal solution.

Per-thread memory arenas leading to large amounts of RSS use over time is something of a known issue on the glibc malloc tracker. In fact, the MallocInternals wiki says specifically:

As pressure from thread collisions increases, additional arenas are created via mmap to relieve the pressure. The number of arenas is capped at eight times the number of CPUs in the system (unless the user specifies otherwise, see mallopt), which means a heavily threaded application will still see some contention, but the trade-off is that there will be less fragmentation.

There you have it - lowering the number of available memory arenas reduces fragmentation. There’s an explicit tradeoff here: fewer arenas decreases memory use, but may slow the program down by increasing lock contention.

Heroku discovered this side-effect of per-thread memory arenas when they created the Cedar-14 stack, which upgraded glibc to version 2.19.

Heroku customers reported greater memory consumption of their applications when upgrading their apps to the new stack. Testing by Terrence Hone of Heroku produced some interesting results:

Configuration	Memory Use
Base (unlimited arenas)	1.73x
Base (before arenas introduced)	1x
MALLOC_ARENA_MAX=1	0.86
MALLOC_ARENA_MAX=2	0.87

Basically, the default memory arena behavior in libc 2.19 reduced execution time by 10%, but increased memory use by 75%! Reducing the maximum number of memory arenas to 2 essentially eliminated the speed gains, but reduced memory usage over the old Cedar-10 stack by 10% (and reduced memory usage by about 2X over the default memory arena behavior!).

Configuration	Response Times
Base (unlimited arenas)	0.9x
Base (before arenas introduced)	1x
MALLOC_ARENA_MAX=1	1.15x
MALLOC_ARENA_MAX=2	1.03x

For almost all Ruby applications, a 75% memory gain for 10% speed gain is not an appropriate tradeoff. But let’s get some more real-world results in here.

A Replicating Program

I wrote a demo application, which is a Sidekiq job which generates some random data and writes the response to a database.

After switching MALLOC_ARENA_MAX to 2, memory usage was 15% lower after 24 hours.

I’ve noticed that real-world workloads magnify this effect greatly, which means I don’t fully understand the allocation pattern which can cause this fragmentation yet. I’ve seen plenty of memory graphs on the Complete Guide to Rails Performance Slack channel that show 2-3x memory savings in production with MALLOC_ARENA_MAX=2.

Fixing the Problem

There are two main solutions for this problem, along with one possible solution for the future.

Fix 1: Reduce Memory Arenas

One fairly obvious fix would be to reduce the maximum number of memory arenas available. We can do this by changing the MALLOC_ARENA_MAX environment variable. As mentioned before, this increases lock contention in the allocator and will have a negative impact on the performance of your application across the board.

It’s impossible to recommend a generic setting here, but it seems like 2 to 4 arenas is appropriate for most Ruby applications. Setting MALLOC_ARENA_MAX to 1 seems to have a high negative impact on performance with only a very marginal improvement to memory usage (1-2%). Experiment with these settings and measure the results both in memory use reduction and performance reduction until you’ve made a tradeoff appropriate for your app.

Fix 2: Use `jemalloc`

This is CodeTriage's Sidekiq worker memory use with and without jemalloc. I'm really starting to wonder how much of Ruby's memory problems are just caused by the allocator. pic.twitter.com/FD0fVbJCLt
— Nate Berkopec (@nateberkopec) December 1, 2017

Another possible solution is to simply use a different allocator. jemalloc also implements per-thread arenas, but their design seems to avoid the fragmentation issues present in malloc.

The above tweet was from when I removed jemalloc from CodeTriage’s background job processes. As you can see, the effect was pretty drastic. I also experimented with using malloc with MALLOC_ARENA_MAX=2, but memory usage was still almost 4 times greater than memory usage with jemalloc. If you can switch to jemalloc with Ruby, do it. It seems to have the same or better performance than malloc with far less memory use.

This isn’t a jemalloc blog post, but some finer points on using jemalloc with Ruby:

You can use it on Heroku with this buildpack.
Do not use jemalloc 4.x with Ruby. It has a bad interaction with Transparent Huge Pages that reduces the memory savings you’ll see. Instead, use jemalloc 3.6. 5.0’s performance with Ruby is currently unknown.
You do not need to compile Ruby with jemalloc (though you can). You can dynamically load it with LD_PRELOAD.

Fix 3: Compacting GC

Fragmentation can generally be reduced if one can move locations in memory around. We can’t do that in CRuby because C-extensions may use raw pointers to refer to Ruby’s memory - moving that location would cause a segfault or incorrect data to be read.

Aaron Patterson has been working on a compacting garbage collector for a while now. The work looks promising, but perhaps a ways off in the future.

TL;DR:

Multithreaded Ruby programs may be consuming 2 to 4 times the amount of memory that they really need, due to fragmentation caused by per-thread memory arenas in malloc. To fix this, you can reduce the maximum number of arenas by setting the MALLOC_ARENA_MAX environment variable or by switching to an allocator with better performance, such as jemalloc.

The potential memory savings here are so great and the penalties so minor that I would recommend that if you are using Ruby and Puma or Sidekiq in production, you should always use jemalloc.

While this effect is most pronounced in CRuby, it may also affect the JVM and JRuby.