Building Fast Websites
Complex websites have a tendency to be slow, and can give a horrible user-experience leaving your visitors waiting for pages to load.
To prevent this you need to look at all aspects of a website, and how it all hangs together, from the web-server, down to the very last pixel of an image.
Let's break this down starting at the very front-end of any website.
Before making any optimisations to a website or page, you'll want to benchmark the site to get some stats on how fast or slow the page is to start with.
In the Apache software-suite you'll find ab (Apache Benchmark) to send requests to a web-server and produce some interesting stats. To send 50 requests, 5 at a time, use...
[rob@rob ~]$ ab -n50 -c5 http://www.example.com/
I also find timing each page is useful, so you can always see at a glance if a page is taking a bit too long. Time::HiRes is ideal for this, just put the time taken in the footer of the every page, like this one.
CSS Image Sprites
If a page uses several images, they are all downloaded as individual requests.
Firstly make sure you've optimised each image for use on the web, don't save JPGs with a quality of 12 unless you really need to, and check for profiles in your image-editor to "save-for-web".
With CSS Image Sprites, you build up a single image containing all the images for the page, and use CSS to specify where on the image each sub-image is located, with it's width and height.
Read more about CSS Image Sprites at W3Schools.
A really simple quick-win can be to simply enable compression on the web-server, since most browsers support compressed data being sent to them, reducing the size of data being transferred.
Caching is often used to speed up a website by keeping pages of a website in memory to re-serve quicker the next time they're requested, rather than going back to disk to load them. Squid is a common caching-proxy used for this purpose.
If you are building an application, you can also use caching to keep various objects/data-sets in memory. Maybe you have a large database query that takes over a second each time it's called, instead of doing that query everytime, you could cache the results for a set time period.
Memcached is a great tool for this type of caching, and can be configured across many servers in a cluster. It runs as a daemon and access is typically done over TCP, you can even query the cache over Telnet to inspect it.
Don't cache everything, you only really get benefits from caching large complex data-sets, if it's just a list of countries to use in a drop-down for example, a database hit each time will probably be faster since the database engine is no doubt caching queries too.
Indexing Database Tables
If your application makes use of a database, you can usually get huge speed improvements by putting an index on a table on the commonly used column used for looking up data.
Under MySQL you can take a query and ask the engine to explain how it will get the data. This becomes increasingly useful with joins, and should point you in the direction of which table/column to apply an index to.
mysql> explain select * from merchant where id=120; +----+-------------+----------+-------+---------------+---------+---------+-------+------+-------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +----+-------------+----------+-------+---------------+---------+---------+-------+------+-------+ | 1 | SIMPLE | merchant | const | PRIMARY | PRIMARY | 4 | const | 1 | | +----+-------------+----------+-------+---------------+---------+---------+-------+------+-------+ 1 row in set (0.02 sec)
Depending on the table size, adding an index can take a very long time, so make sure you test in a development environment before modifying a live database, and possibly look at doing the change out-of-hours and take the website offline temporarily.
If your data is extremely large and your app is simply too slow when searching it, such as free text search operations, consider a NoSQL solution such as Solr or ElasticSearch. Think of them as a single table database, you lose some structure from a standard database, but you get a blisteringly-fast search component.
If you're familiar with a simple CGI script, each time a request is made, the Perl (or PHP, Bash, Python, etc) interpreter is loaded, along with all necessary modules. Your script then runs, sends back a response to the browser, and then exits, unloading the interpreter and all modules.
When a second request comes in, the whole process is repeated.
When you begin to use larger modules, such as Moose and the Catalyst framework, this load-time can easily reach a second or two, making your website awfully slow.
Persistence is the technique of loading the Perl interpreter and required modules once at web-server startup, with each request taking advantage of the modules in-memory.
Apache provides mod_perl to enable persistence between requests. Modules are only loaded when the web-server is started or restarted. This can be a pain at times if you're modifying modules and manually testing, needing to restart the web-server every time for the changes to take effect (or you could look at Apache2::Reload).
FastCGI is another approach which is web-server independent. FastCGI is actually a process which wraps your application in a daemon that either listens on a TCP port or via a UNIX socket file. When a request comes into the web-server, it in-turn sends a FastCGI request to your FastCGI application which is always running, waiting for requests.
It's important to note that FastCGI is a protocol itself, much like HTTP, but different. Your FastCGI applications(s) can also reside on a different server to the web-server, so scaling your application is quite straight-forward. FastCGI processes can be restarted independently of the web-server, and vice-versa.
For larger web-applications it's a good idea to investigate what frameworks are available to make your life easier. One common framework in the Perl-space is Catalyst, which has been developed by the Perl community to solve many common problems, so you don't waste time re-inventing the wheel, authentication and sessions are just 2 examples of features that are all ready for you to switch on in your app.
For database applications it's also worth taking a look at DBIx::Class which will speed up the majority of your database interactions and provides a clean way to work with your data - no more writing SQL statements.
Here we have a real-world example which unfortunately (for the owner) went the wrong way. An e-commerce website originally developed using Catalyst and running under FastCGI with Lighttpd, using Memcached for session caching.
The website was re-developed recently using the open-source PHP e-commerce platform, Magento. By using the web developer tools in Chrome, it was noted that page load times went from around 0.5s to 1.5s.
Server Software: lighttpd/1.4.31 Server Hostname: gandys.fleetwebdesign.co.uk Server Port: 80 Document Path: / Document Length: 6185 bytes Concurrency Level: 5 Time taken for tests: 2.123 seconds Complete requests: 50 Failed requests: 0 Write errors: 0 Total transferred: 331194 bytes HTML transferred: 315435 bytes Requests per second: 23.55 [#/sec] (mean) Time per request: 212.335 [ms] (mean) Time per request: 42.467 [ms] (mean, concurrent) Transfer rate: 152.32 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 22 32 8.9 31 65 Processing: 83 177 98.0 126 390 Waiting: 52 146 97.3 103 354 Total: 116 209 96.3 166 413 Percentage of the requests served within ms 50% 166 66% 217 75% 275 80% 306 90% 395 95% 412 98% 413 99% 413 100% 413 (longest request)
Server Software: Apache/2.2.23 Server Hostname: www.gandysflipflops.com Server Port: 80 Document Path: / Document Length: 15311 bytes Concurrency Level: 5 Time taken for tests: 11.700 seconds Complete requests: 50 Failed requests: 0 Write errors: 0 Total transferred: 793050 bytes HTML transferred: 765550 bytes Requests per second: 4.27 [#/sec] (mean) Time per request: 1169.984 [ms] (mean) Time per request: 233.997 [ms] (mean, concurrent) Transfer rate: 66.19 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 12 18 5.5 16 39 Processing: 866 1113 89.4 1124 1304 Waiting: 839 1073 88.2 1088 1262 Total: 906 1131 88.7 1138 1336 Percentage of the requests served within ms 50% 1138 66% 1170 75% 1173 80% 1207 90% 1234 95% 1245 98% 1336 99% 1336 100% 1336 (longest request)
The above tests were ran using Apache Benchmark, a total of 50 requests, 5 concurrently. Even with twice the amount of HTML, this shouldn't happen, we can deduce that the backend system needs to be optmised, probably database queries and application-level caching.