This is Article #4 of a 4-part series. For a good primer, check out the first two articles listed below. For the reasoning and analysis behind the “Recommended” option in this article, check out Part 3, How to Combine GZip + CDN for Fastest Page Loads. Otherwise, jump right in!

• The Importance of Page Load Speed
• Improve Page Load Speed (by 80%) by Improving Component Load Speed
• How to Combine GZip + CDN for Fastest Page Loads
• Caching, Zipping, and (Amazon CloudFront) CDN For A Rails App
  • Prerequisites
    1. Cached stylesheets and javascripts
    2. Creating an Amazon AWS Account
  • Setup S3 Buckets
  • Setup CloudFront Distributions
  • Create CNAME records (optional)
  • Install Rails S3 Synch Plugin
    1. Installing AWS-S3 Gem
    2. Configure S3 Synch Plugin
    3. Add S3 Synch to Deployment
  • Option A: Compressible Assets from App Server, Images from CloudFront (recommended)
    1. Configure Rails Asset Host
    2. Create A-name Record
    3. Configure Apache
  • Option B: Serve Everything from CloudFront (easier, but not recommended)
    1. Configure Rails Asset Host
    2. Pre-compile Cached Stylesheet and Javascript File
  • Conclusion

In this article, we’re going to speed up our Rails application by up to 75%, simply by optimizing our Rails asset host. We’re going to serve our components (stylesheets, javascripts, images, etc.) from a combination of our app’s server and Amazon CloudFront (Option A, recommended), or entirely from CloudFront (Option B – easier).

The best option for you may depend on your specific needs, but I’ll cover both processes below. For a an in-depth analysis of why Option A is recommended over Option B, see the last article in this series, How to Combine GZip + CDN for Fastest Page Loads.

Prerequisites

Cached Stylesheets and Javascripts

Another way to reduce page load time is to combine all of your components into as few files as possible. In other words, combine all of your stylesheets into a single css file, and likewise with your javascripts. Remember from the last article, that each request takes 50-150ms, not including the response and download time. If you have 10 separate javascripts, this equates to 0.5-1.5 seconds just to request the files (not to mention all the time to download them). If you can combine all of the files into one, that means you need just one request to get the same amount of data.

Luckily in Rails, this is easy, simply add :cache => 'cached-file-name' to your stylesheet_link_tag and javascript_include_tag in your application layout. For example:

Now, as long as the following line is set to true in your environment.rb, or more likely in production.rb, Rails will either load your combined files in the layout, or create and load them if they don’t already exist.

Simply packing all stylesheets and javascripts into one file each reduced page load time of one of our production applications from 10.1 to 8.3 seconds (an 18% reduction in load time alone).

Creating an Amazon AWS Account

If you do not yet have an Amazon AWS account, you will need to create that and enable S3 and CloudFront services. See this writeup on creating and setting up an Amazon S3 account. for helpful instructions.

Setup S3 Buckets

Once you’ve signed up for your Amazon AWS account and activated S3 and CloudFront, you’ll want to setup 4 S3 buckets for your application, using Amazons S3 management console.

We’re going to setup 4 buckets and CDN distributions because some old browsers still have an artificial limitation that only allow 2 concurrent connections to each domain, meaning our components will take longer to download from Amazon if they can only be downloaded 2 at a time. By creating 4 different domains pointing to 4 different buckets/distributions, we’re allowing our components to download up to 8 at a time from those browsers that still enforce this limitation.

Setup CloudFront Distributions

Once your S3 buckets are created, click over to the CloudFront tab and create one distribution for each S3 bucket as shown. You can type any comment to help you quickly identify each distribution.

Create CNAME Records (optional)

Once you’ve created your 4 CloudFront distributions, you may create a CNAME record for each distribution. This allows you to serve files from CloudFront using your own asset subdomains, like cdn0.yourapp.com, instead of raNDomString1234.cloudfront.net. We’ll use the following format of cdn%d.yourapp.com, where %d stands for digits 0-3:

Install Rails S3 Synch Plugin

This plugin adds some Capistrano recipes to synch our application’s public directory with our four S3 buckets automatically every time we deploy our app. See Spatten Design’s documentation for more information. I’ve made some updates to their original plugin to properly set the Cache-control and Expires headers for our assets on S3, as well as to properly set the Content-encoding header for Gzipped assets.

Update: I’ve updated the S3 Synch Plugin further; it can now handle unique S3 buckets for different Rails environments (e.g. one set of buckets for production and another for staging). Be sure to update your synch_s3_asset_host.yml file as shown below.

Installing AWS-S3 Gem

The synch_s3_asset_host plugin requires the AWS-S3 gem, so add the following to your environment.rb:

…and then run the following from the terminal to install the S3 Synch plugin’s gem dependency:

Configure S3 Synch Plugin

Create a config/synch_s3_asset_host.yml file like this:

Update: The “production” part in the file above has been added for my latest update of the S3 Asset Synch Plugin.

Add S3 Synch to Deployment

Now, in your Capistrano deploy.rb script, add the following line to the :deploy namespace:

…and then add the :asset_host_syncher => true flag to the :web role:

Option A: Compressible Assets from App Server, Images from CloudFront (recommended)

For more detail about why this method is recommended, see the last article in this series.

Configure Rails Asset Host

Use the following configuration in your production.rb file to configure the way Rails writes the URLs for asset_tags:

Create A-name Record

We will also need to create an A-name record for the cache.yourapp.com subdomain, which points to your application server’s IP address.

Configure Apache

Now we need to configure Apache to accept incoming requests to our “cache” subdomain, setting the appropriate far-future Expires and Cache-control headers. We also need to tell Apache to automatically compress and serve any compressible filetype on the fly. Add this to your site’s Apache conf file:

Now we need to make sure the appropriate Apache modules are enabled and restart Apache.

Option B: Serve Everything from CloudFront (easier, but not recommended)

For more detail about why this is not recommended, see the last article in this series. Basically, though, it’s because it requires you to make one of the following compromises:

• a) Serve all files uncompressed, resulting in file sizes up to 4x bigger than necessary.
• b) Serve Gzipped assets from CloudFront without first detecting whether or not the visitor’s browser support Gzip encoding.

That being said, if this is acceptable for you, this method is simpler to set up and configure.

Configure Rails Asset Host

Add the following to your production.rb:

Pre-compile Cached Stylesheet and Javascript File

If you’re serving every component from CloudFront, you will need to pre-compile your stylesheets and javascripts on every deploy. Otherwise, Rails will try to compile and save the files to your application server, but try to serve them from S3 (where they won’t exist).

To solve this, we’ll add some Capistrano scripts to our deploy.rb to compile our files for us before the synch_s3_asset_host plugin copies our public directory over to our S3 buckets. But this means, we’d have to copy the list of asset files to be compiled into our Capistrano script, as well as having them listed in our application.html.erb layout. To DRY things up a little, we’re going to create some project-wide constants:

lib/assets_for_cache.rb

And then replace your javascript_include_tag and stylesheet_link_tag in your application layout with the following:

Add this to your deploy.rb script:

…and then add this to the :deploy namespace in your deploy.rb file, before calling the s3_asset_host sync script:

Conclusion

Now simply save your project and deploy it! The first deploy will take quite a while, as your entire /public directory will be copied to all 4 buckets on Amazon S3, one at a time. But after that, it’s a painless process.

Whether you chose the “recommended” or the “easier” option, you should immediately notice a significant increase in the performance of your Rails app. Thanks for sticking with me through this 4-part series! Please let me know if you have any thoughts, questions, or feedback in the comments.

This is Article #3 of a 4-part series. For a good primer, check out the first two articles listed below. Otherwise, jump right in!

• The Importance of Page Load Speed
• Improve Page Load Speed (by 80%) by Improving Component Load Speed
• How to Combine GZip + CDN for Fastest Page Loads
  • Pitfall of Amazon S3 + CloudFront as a CDN: No GZipping
  • Possible Solutions (that don’t work)
    • Intercept CloudFront requests with app server and rewrite
    • Detect requests with Rails and write asset URLs accordingly
  • The Solution: Hybrid Gzipping/CloudFront Depending on Asset-type

In my last post, I discussed the three techniques used to improve asset load speed. In this post, I will discuss how to combine the use of GZipping and a Content Delivery Network (CDN) for the fastest possible page loads.

Pitfall of Amazon S3 + CloudFront

Everyone’s favorite CDN these days is Amazon’s CloudFront service, which serves files directly from Amazon’s scalable “simple storage system”, Amazon S3. It is very easy to work with, has widespread support in Ruby gems and plugins (and countless other libraries), and is very inexpensive with it’s pay-as-you-go billing.

However, there is one large pitfall to using Amazon S3 + CloudFront, and that is that neither S3 nor CloudFront support GZip detecting and encoding. It would seem that we need to now decide whether we’ll do without GZipping or using a CDN. Not so! There is another way.

Possible Solutions (that don’t work)

Amazon S3 and CloudFront servers do not detect whether the incoming requests accept GZip encoding, and so they are not able to Gzip and serve components on the fly. Then, it’s simply a matter of figuring out whether we should link to the compressed or the uncompressed components when the user visits the page.

Detect requests with application and write asset URLs accordingly

This solution is similar to the last, except that it attacks the problem one step earlier in the workflow. So, let’s take a step back. Instead of linking to the asset through our own server, this time, we’ll revert to linking directly to CloudFront:

However, this time we’ll have our application (whether it be Ruby, PHP, Python, or whatever) detect if the request header accepts GZip encoding, and rewrite the asset tag accordingly.

I won’t go into detail about how to actually accomplish this, because the truth is, this won’t work either.

Why this doesn’t work

This will only work as long as your code is run dynamically every time a user loads the page. That means, once you implement this strategy, you no longer have the option to cache the page. Ever.

Sure, you could probably come up with some system that creates two versions of each cached page (one with gzipped links and one without), but that will add a lot of complexity to your server setup and filesystem, and it’s just too much trouble. So, let’s move on to another solution.

Intercept CloudFront requests with app server and rewrite

Now this first solution may seem clever, but let’s see if you can figure out why it won’t work. The idea here is that rather link to a stylesheet, for example, on CloudFront like this:

…we’ll instead link to our own server, which will read the request and redirect to either the compressed or the uncompressed stylesheet on CloudFront as appropriate.

And then the Apache configuration for the compressed.yourdomain.com virtual host would look like this:

Why this doesn’t work

Remember in the last article, one of the added benefits of off-loading your assets to a CDN is that your server no longer must listen for and respond to asset requests. This solution rescinds that benefit; even though the asset download still takes place between the CDN server and the user’s browser, the initial request must still go through your server to be resolved to the CDN.

Furthermore, each component request now requires two DNS lookups instead of one, which adds to the request latency (though the request is latency is small compared to the download time in the request/download cycle).

But the real reason this won’t work well is because it disrupts the magic that make CDNs fast. A CDN is beneficial primarily because serves files faster by choosing, for each request, the server (or “service node”) that is closest in proximity to the user. CDNs are able to estimate the closest server in the CDN using a variety of techniques, including reactive probing, proactive probing, and connection monitoring.  (See Content Networking Techniques for more info)

By inserting your server (acting as a proxy) into the request cycle between the user’s computer and the CDN, you may cause the CDN to choose a sub-optimal service node for the delivery of content directly to the user. If the CDN probes the network from the request side, it will most likely choose the edge node location closest to your server rather than to the user’s computer, completely negating the benefit of using  the CDN in the first place.

To illustrate this point, consider the typical request/download cycle for a javascript file served from your application’s server:

Below is a simplified diagram of the typical request/response cycle for a javascript file when using a CDN to serve the component.

This final diagram depicts the request/response cycle when delivery components through the CDN with your application server acting as a proxy (so that your app server can read the request and tell the CDN whether to serve the unzipped or the zipped component).

Notice in the diagram above, the CDN should have chosen the service node closest to the User, so that the javascript file would have less distance to travel and would thus download the fastest. Instead, it chose the node closer to the application server that proxied the request to the CDN.

The graph below compares download times for the user from my server (located in St. Louis, Missouri), from a server in Amazon CloudFront’s CDN, and from CloudFront with my server acting as a proxy. I performed this comparison from my own computer here in Ann Arbor, MI, while my buddy, Dave Leal, downloaded the file from his computer in Portugal.

The Solution: Hybrid Gzipping/CloudFront Depending on Asset-type

At this point, it may seem like we can choose between two alternatives:

• Move all assets to the Amazon CloudFront and forget about GZipping
• Keep components self-hosted and configure our server to detect incoming requests and perform on-the-fly GZipping as appropriate

In our last post, we saw that Gzipping our components can compress them down to ~25% of their original size, which means they’ll transfer 4X faster. And in this post, we see that serving components from Amazon CloudFront can transfer component files ~2X faster*.

*This last statement depends entirely upon your CDN and your own server’s location and capabilities, so you might want to do a little homework and verify the difference in download times for your own setup.

The following is the simple formula for Download Time. You can see that File Size is directly proportional to Download Time (so reducing File Size by 1/2 reduces Download Time by 1/2). And Download Speed is indirectly proportional to Download Time (so increasing Download Speed by 2 reduces Download Time by 1/2):

Ideally we’d be able to do both (and some other more expensive Content Delivery Networks actually will allow you to). But if we must choose, compressible file-types gain much more by way of serving them compressed, than by serving them uncompressed from a CDN edge location. So, we will serve compressible file-types (stylesheets, javascripts, and static HTML files) from our own server, GZipped.

However, images are already compressed in the image encoding; image file size is unaffected by Gzipping them on our server. So, we may as well allow images to benefit from the 2X speed improvement by serving them straight from our Amazon CloudFront CDN.

Using this solution for hosting/serving components, we’ve been able to reduce page load time by 75% on several of our sites.

This is Article #2 of a 4-part series. This article (along with Article #1) serves as a primer for the next two entries in this series, which discuss the most efficient way to put these concepts into practice in your web application. For more a more in-depth look at these concepts, see Yahoo!’s Best Practices for Speeding Up Your Web Site and Google’s Speed Tracer tutorial

• The Importance of Page Load Speed
• Improve Page Load Speed by Improving Asset Load Speed
  • 3 Techniques to Speed Up Asset Loading
    • Better Caching with Expires header
    • Zipping
    • Content Delivery Network (CDN)

In my last post, I discussed why it’s becoming increasingly important to ensure your website loads quickly for users (and Googlebots). At the end of the article, I mentioned a quote from Yahoo!’s article, Best Practices for Speeding Up Your Web Site:

80-90% of the end-user response time is spent downloading all the components in the page: images, stylesheets, scripts, Flash, etc. This is the Performance Golden Rule.

In other words, you can get the most “bang for your buck” when it comes to page load optimization by speeding up the loading of your site components (a.k.a. “assets”, such as javascripts, CSS stylesheets, images, etc.). In this post, I’ll discuss the 3 main techniques used to improve asset load speed.

Asset load optimization often results in a reduction in asset-load-time by 75-90%. Since asset-load-time accounts for 80-90% of your total page-load-time, this equates to an overall reduction in page-load-time by up to 80%!

Below are graphs showing the load time of RateMyStudentRental.com before and after (respectively) implementing the asset-optimization shown here. Using Google’s Speed Tracer extension, we can see that total load time decreased from 40 milliseconds down to 15ms.

3 Techniques to Speed Up Asset Loading

So we know that when a user visits your site, their browser must download your javascripts and CSS stylesheets so that your site will display properly. Luckily this only happens once per session by default, because the user’s browser will then cache those assets on the user’s computer. This means that each subsequent page load, as the user navigates through your site, will simply reload the assets from the user’s computer without needing to download them again. But this isn’t good enough, we can make it faster.

Better Caching with Expires Header

Remember when I said the browser will cache assets once per session by default? That’s not good enough for our purposes. We need to setup our server to explicitly tell users’ browsers to cache assets until further notice.

This is difficult to do reliably across all browsers/environments, so the way we accomplish this is by actually telling users’ browsers to cache everything for a year (or some other arbitrarily long duration). Then Rails’s built-in asset helpers append timestamps to the asset URLs every time you update and deploy your assets. This fools users’ browsers into thinking they’re looking at a different asset (since the filename has a different timestamp), which makes the browsers re-download and cache the new files when you deploy updates.

We’ve now discussed telling users’ browsers to keep cached versions of our assets for longer than one session. But we still need to speed up that first page visit, by speeding up the user’s first download of our assets.

Zipping

Have you ever zipped a file before emailing it to someone in order to make the file-size smaller? Well, most modern browsers actually know how to read a zipped file directly without needing to unzip it. This means that instead of serving full javascript files or CSS stylesheets, you can serve zipped files (often 1/3 or less of the original file-size) and make the page load much faster, since smaller files take less time to download.

Even better is that servers, such as Apache, can check if the requesting browser accepts GZip encoding (the standard zipping algorithm), and automatically serve zipped files if accepted. All you have to do is enable it in your server.

Content Delivery Networks (CDNs)

So far, we’ve discussed making the files stay cached by users’ browsers for longer than one session, and then speeding up the first download by zipping the assets, making their file-sizes smaller. There is still one more thing we can do to make the assets download faster, and that is use a Content Delivery Network, or CDN. A CDN is a group of servers physically scattered all around the world that host and serve copies of your assets for you.

Even though it may seem like data transmission is instantaneous, it still takes time for electrons to travel through wires (even optical networks “only” transmit data at the speed of light, which is not instantaneous). Thus, the shorter the distance we need to transmit data, the faster it will be transmitted. If your existing asset server (by default, the server from which your site is hosted) is located in the US, then your assets will download more quickly for me (located in the US) than it will for my buddy, Dave, in Portugal. The purpose of a CDN is to serve your assets on a per-user-basis from the server in the CDN that is physically closest to that user.

Don’t believe me? Just check out the difference in ping time (the amount of time it takes for a computer to contact the server and for the server to respond) for my buddy, Dave, in Portugal pinging my server in the US compared to pinging a CDN server in Europe.

Putting This Knowledge To Use

What have we learned here? We’ve learned that we can reduce download time by reducing the size of the files being downloaded. GZip compression can reduce text-based files to 1/3rd their original size, which reduces download time by 67%. Then we can move assets to a CDN, which can reduce transmission time by another 67%. Doing the math, we see that component download time is reduced by 90%, which equates to an overall reduction in page load time of 80%.

Now that we’ve covered the importance of page load speed, and the 3 techniques used to speed up the loading of assets (or site components), we can figure out how best to combine this knowledge into the optimal configuration, and then finally, how to implement this into our own sites.

This is article #1 of a 4-part series. This article (along with Article #2) serves as a primer for the last two entries in this series, which discuss the most efficient way to put these concepts into practice in your web application. For more a more in-depth look at these concepts, see Yahoo!’s Best Practices for Speeding Up Your Web Site and Google’s Speed Tracer tutorial

Page load speed is becoming increasingly important as rich web applications become more interactive. It’s not just about usability anymore; it can now directly affect your placement in search engine results, now that Google uses page load speed in their ranking algorithm. Are you ready for a reality check? Get Google Webmaster Tools for your site, and go to the Labs >> Site Performance to view your average page load time, as seen by Google’s web crawlers. That’s right, Google is already tracking your site’s performance history.

Google Webmaster Tools is even kind enough to tell you how you stack up against the rest of the web. Here is what Webmaster Tools had to say about one of our sites before optimizing it for quick page loading:

On average, pages in your site take 4.5 seconds to load (updated on Feb 21, 2010). This is slower than 70% of sites. These estimates are of low accuracy (fewer than 100 data points). The chart below shows how your site’s average page load time has changed over the last few months. For your reference, it also shows the 20th percentile value across all sites, separating slow and fast load times.

Ouch. Did I mention this would be a painful reality check?

Now to be fair, there’s a very reasonable explanation for this. Google claims that the majority of users will click “back” to the search results page if a link takes too long to load. So, if a webpage is too slow for the visitor to read it, the relevance of the content is…well, irrelevant. I should point out, however, that it’s unknown precisely how much page load speed affects your placement in search results.

What is Included in Page Load Time?

At first, you may think that Google crawlers only index the initial load-time of the HTML. You’d be wrong. They actually include the time it takes to load all Javascript files, CSS stylesheets, images, etc. Now it’s time to see how your site performs. For this, you’ll need either the Page Speed extension for Firefox, or for an even better look into your site’s performance, get Speed Tracer for Chrome. And of course, there’s always the ever-popular YSlow extension for Firefox from Yahoo!

Watch Your Assets

Your website’s assets include all of the files the visitor’s browser must download to render your webpage. This includes Javascript files, CSS stylesheets, and images. And according you Yahoo!:

80-90% of the end-user response time is spent downloading all the components in the page: images, stylesheets, scripts, Flash, etc.

For instance, it is very difficult to place your tabs below the tabbed content with jQuery-UI tabs.. You must perform some crazy CSS trickery, because jQuery-UI tabs forces you to place your <ul> before your content <div>s. Not to mention all of the jQuery-UI styling it does to your tabs by default, making it difficult to override and apply custom stylization to your tabs.

This new plugin, jQuery EasyTabs, is very flexible for the required markup and structure of your tabs and content. Also, it handles only the functionality of creating in-page tabs, leaving all of the styling up to you and your own CSS.

To check it out, see our documentation and demos. Then when you’re ready, you can download it from the jQuery plugins website.

It has become common practice to use JQuery as the javascript library of choice for Ruby on Rails applications. If you have a Rails app already using Prototype, it usually isn’t feasible to replace it in one swoop. And sometimes, you just simply need both to work at the same time. Whether you want to use both in unison (not recommended as a long-term strategy), or you want to gradually migrate from Prototype to JQuery, it’s not as scary as you may think.

The secret is that it is difficult to get Prototype to play nicely with JQuery, while it is rather trivial to get JQuery to play nicely with Prototype. This may seem like a minor distinction, but it is one that will save you a lot of headaches.

Use Prototype/Scriptaculous with JQuery

First, in our application layout, we’re going to get rid of <%=javascript_include_tag :defaults %> and include each javascript file explicitly just to make things a little less magicky and a little more straight forward.

Now, we’ll add in JQuery (and JQuery UI while we’re at it, just for fun). You’ll want to grab the latest release from the JQuery download page (and the latest JQuery UI as well).

Right under that line, in your Rails application layout, we’ll insert the one-liner that makes JQuery play nicely with Prototype, YUI, etc.

Try doing this with Prototype (especially if you’re using other scripts built on top of Prototype and Scriptaculous), and you’ll soon find yourself pulling out your hair.

If you simply want the ability to use both libraries together in your app, then congratulations, you’re done. You can now do this:

Notice the first line uses the Prototype element id selector, $ with a Scriptaculous effect, and the second line uses the new noConflict JQuery selector $j.

Make Rails Use JQuery Instead of Prototype

Let’s assume that we’d like to phase Prototype out of our app in favor of JQuery. In addition to going through your application and replacing all of your Prototype code with JQuery code, you’ll need to take care of the Rails Prototype helper magic (note that when Rails 3 becomes the standard, this won’t be necessary).

This is really easy, thanks to the JRails plugin, which can be found on Github. Install it in your Rails project from your root directory.

Now, your Rails helpers, such as link_to_remote, form_remote_tag, etc. are using JQuery to build the AJAX code instead of Prototype on the back end. Isn’t Rails magic neat?

Caveat: Delayed Observer

There is one more step if you’re using the observe_form helper method. This helper relies on Prototype’s Element.observe method, which does not exist in the standard JQuery library.

No worries, someone has done the work for you already (not me). Just download and include the JQuery delayed observer plugin in your layout, and you’re all set.

Boot Prototype Once and For All

Once you’ve eliminated all Prototype/Scriptaculous code from your project (meaning there should now be no $ references in your entire project, only $j), you can simply remove it Prototype0+Scriptaculous from your project.

And just to celebrate, remove the var $j = jQuery.noConflict(); line from your application layout, and do a Find and Replace All $j with $ for your project. You have now ditched Prototype once and for all.

Feel free to skip straight to our “Meat” Algorithm, as we’ve so endearingly named it, if you’re not interested in the specifics of implementing it.

The Tools: Ruby and Nokogiri

Thanks to Ruby and a Ruby gem, called Nokogiri, it’s far easier to create this printable view than you may think. If you haven’t heard of it before, Nokogiri is a gem that reads and parses HTML, XML, and SAX, and allows you to easily search and manipulate these documents based on CSS selectors and XPATH.

I should also note that Nokogiri also requires Open-URI, which is included in the standard Ruby library.

Reading and Parsing in 5 Lines

Nokogiri is really straight forward and easy to use. In fact, it’s so easy, I’m going to show you how to open any webpage and create a print-formatted version of it in 5 lines of code!

Explanation: The Algorithm

Ok, so now you may be asking, what exactly is this code doing? Let’s take a look, piece-by-piece…

Setting Up the HTML with the Configuration Options

This config block for Nokogiri is simply stripping away blank nodes, entities, and suppressing any errors generated from malformed HTML. You can read more about these configuration options on Nokogiri’s site.

Stripping Out the Media

This line is simply removing all javascript, images, iframes, and embeded objects such as videos and flash. You can modify this to remove pretty much any elements you want from the page.

The “Meat” Algorithm: Determining What’s Important  

This, believe it or not, is our algorithm for determining what is the important part of the webpage to keep. This is how we determine what the “meat” of the page is, the part that needs to be kept.

All we are doing is searching the document for <p> paragraph tags. Then we collect the parent <div> blocks (or whatever the parent elements of the <p> tags happen to be) into an array. However, if there are multiple <p> tags in one block (and let’s face it, there will be), this creates duplicate parent blocks in the array (one for each <p> tag). So, we simply call the .uniq method on the array to get rid of the duplicates.

Note the reason we grab the parent element of all paragraph elements (rather than just grabbing all of the paragraph elements themselves) is so that we make sure not to exclude any headings, unordered lists, ordered lists, or any other elements that may be in the body of the page, but not wrapped in paragraph tags.

95% of the Time, It Works Every Time

This is not full-proof, as there could be a webpage that throws semantics and proper markup to the wind, and decides that plain text and <br /> tags are better than <p> tags. This may also exclude other important information elsewhere in the webpage. However, in our limited experience so far, it works very well in at least 95% of scenarios.

However, once you’ve launched your application and people start actually using it, you may find server experiencing excessive swapping once the traffic begins to pick up. Before getting too deep into scaling, caching, upgrading your server, etc., there are a few Passenger settings you can tune to wring out the best performance your existing stack. I’ll address these settings with a few rules of thumb we’ve gathered from the Passenger documentation and simple trial-and-error.

Before getting started, we highly recommend employing Munin (instructions for Munin with Passenger here) and NewRelic Performance Monitoring for your Passenger/Rails application, so that you can fine-tune your settings for the best performance.

Note: the following settings are specific for Apache, you will need to modify the syntax/file-names for Nginx.

Per-server Settings

These settings are set per-server, meaning if you have multiple applications running on the server, these should be set only once throughout all of your .conf files. If you have only one Rails/Passenger app on the server, then you may set these in that application’s .conf file. On Ubuntu, this may be found at /etc/apache2/sites-available/yourapp.conf.

A more appropriate place for these however would be in Apache’s .conf file.

PassengerMaxPoolSize and PassengerPoolIdleTime

/etc/apache2/apache2.conf:

You can essentially copy and paste this into your apache conf file. I’ve added some notes in there to help you remember how they should be tuned.

PassengerMaxPoolSize: This is the number of Passenger instances that can be spawned at once for your site. This basically relates directly to the amount of RAM that is available to your application. However, it is not a linear relationship. It’s more of an exponential relationship. Numbers that we and others have found to work well are:

Since this depends on available RAM, this will vary depending on what other processes are running on your server, such as MySQL, etc. If you have Munin installed with the Passenger plugins, you can monitor the Passenger Memory Stats and Passenger Status graphs to tweak this as necessary.

PassengerPoolIdleTime: This is essentially the amount of time (in seconds) that each Passenger instance continues to run after it becomes idle (i.e. someone stops clicking around the site). According to the documentation, this should be set to the average time a visitor views a page multiplied by 2. This of course prevents Passenger instances from shutting down (and thus having to re-spawn) while someone is viewing a page.

If your application has been in production long enough to have a few hundred viewers and you have Google Analytics installed, an easy way to get the ideal setting for your application is to take the Average Time on Site, divide by the Average Page Views per Visit, and multiply by 2.

PassengerMaxInstancesPerApp (optional)

If you do have more than one Rails/Passenger application on the server, you may optionally set PassengerMaxInstancesPerApp.

PassengerMaxInstancesPerApp: This is the maximum number of instances each application may use of the total PassengerMaxPoolSize. Therefore, PassengerMaxInstancesPerApp must be smaller than PassengerMaxPoolSize.

For example, if you have 2 applications running on one 256MB server, you may want to set PassengerMaxInstancesPerApp to 1, so that one app may never hog all of the rails instances at the expense of the other app. If you have 2 applications running on a 512MB server (PassengerMaxPoolSize 6), then you may want to set cci]PassengerMaxInstancesPerApp[/cci] to 4 or 5 to allow each app to better handle higher load (hopefully while the other app is under less load).

RailsAutoDetect

To get a little more performance out of your server, you can also turn the RailsAutoDetect setting of Passenger off. This will make it so that each request through Apache does not check to see if it is a Rails request that requires Passenger. However, this will now require that you explicitly tell Apache which requests are Rails and do require Passenger, using RailsBaseURI. This is easy though.

Again, in your apache2.conf file:

Now, within your the Vhost block in your site’s .conf file, you will need to set the RailsBaseURI:

Here is a complete example of the Passenger block in your apache2.conf file:

And a simple sites-available/yourapp.conf may look like this:

More Passenger Configuration Settings

These settings so far only scratch the surface of Passenger configuration. Give the Passenger documentation a good read-through for more information on configuring Passenger.

Further Reading: Apache Settings

At this point, if you still need to squeeze more performance out of your server (and you will eventually), you’ll want to configure Apache to utilize its resources most efficiently with Passenger. I recommend reading through this thread on the Slicehost forums to get a good idea just how much trial-and-error this will take. Keep a close eye on your server when you tweak these settings, you may end up making it slower several times before finding that sweet spot.

You may read the documentation for each setting in the Apache docs (note that these are the Apache 1.3 docs, but they still explain purpose of each setting well). For more information using these settings to reduce the RAM consumed by Apache, see this other Slicehost forum and this blog post on saving RAM with Rails and Passenger.

If you’re impatient and just want to get something better than default, here are some numbers that worked well for us on a recent application running on a 256MB VPS on Slicehost. These settings can be found in your apache2.conf file.