Queuing cannot (even theoretically) speed up your average transaction rate. The best it can do is load-level, to ensure the peak burst transaction rate is increased, but only if there is time to clear up the queue during quiet periods. This is useful if you expect input transactions to come along in a bursty pattern.

It depends how we’re measuring transaction rates. In total system terms you’re write; in fact a queue adds overhead. But remember, my top priority is to minimise response time. Pushing data straight to a fast queue means the web server can immediately serve the next request rather than waiting on a database write.

Apart from evening out bursts (as you pointed out), the other reason queues get used a lot is to allow the web and database layers to be scaled independently without much jiggery-pokery.

MySQL is generally faster at simple operations than PostgreSQL.

Only on a single CPU, and only if you’re using MyISAM. I’d sooner trust my data to used toilet paper.

Postgres scales better, is much more reliable, has better inbuilt crypto, supports proper constraints, has decent VIEWs, allows me to use Lua as the language for stored procedures and is much more extensible in general. I haven’t yet decided whether to bulk load data and then process it, or process it and then do the bulk load. I will need to build prototypes for both and see which works best.

If you are buying virtual machine space, is there any good reason to put the database server on a separate virtual machine? If you buy two VMs, they are just as likely going to end up as two processes on the same hardware anyhow so may as well plonk everything into the same VM (and reduce the communication and context-switch overhead).

Firstly, it allows me to scale the web front end and the database separately. On the backend there will be a small pool of beefy VMs. On the front end a buzzing cloud of cheap, small VMs.

Secondly, and to me this is more important, it gives me an extra ringfence around the data. In a traditional setup (like my original arch above), you set up the DB servers so that they only respond to traffic from the web servers. In the second and third architectures you can go one better — they need not even know where the web servers are or vice versa, making it that much harder for an attacker to compromise the data.

Getting it running correctly is more important than getting it running fast, especially when you have no customers

True; but on the other hand, “measure twice, cut once”. Or should that be “design thrice, code once”?

And also, in my case, speed is a feature. It partly defines correctness for my requirements.

Getting it running correctly is more important than getting it running fast, especially when you have no customers :-)

Queuing cannot (even theoretically) speed up your average transaction rate. The best it can do is load-level, to ensure the peak burst transaction rate is increased, but only if there is time to clear up the queue during quiet periods. This is useful if you expect input transactions to come along in a bursty pattern.

MySQL is generally faster at simple operations than PostgreSQL. If write performance is the main issue then don’t bother indexing the data (or keep the index as minimal as you can). Also, for the single query at a time database operations, make sure it is not re-parsing the SQL each time (use the long-winded prepare and execute process, should be faster for repetitive operations because prepare only happens once). Probably still won’t be as fast as bulk-insert.

If you are buying virtual machine space, is there any good reason to put the database server on a separate virtual machine? If you buy two VMs, they are just as likely going to end up as two processes on the same hardware anyhow so may as well plonk everything into the same VM (and reduce the communication and context-switch overhead).

There are a few tricks with ext3 that the Squid guys have documented, like disabling updates of access times and similar. Something to try.

--- EXT3 RESULTS ---
Server Software:        lighttpd/1.4.19
Server Hostname:        10.1.1.6
Server Port:            80

Document Path:          /test_test_test
Document Length:        107 bytes

Concurrency Level:      253
Time taken for tests:   588.523 seconds
Complete requests:      1000000
Failed requests:        27
   (Connect: 0, Receive: 0, Length: 27, Exceptions: 0)
Write errors:           0
Total transferred:      252993169 bytes
HTML transferred:       106997111 bytes
Requests per second:    1699.17 [#/sec] (mean)
Time per request:       148.896 [ms] (mean)
Time per request:       0.589 [ms] (mean, across all concurrent requests)
Transfer rate:          419.80 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   43 259.6      5   35052
Processing:     1  105 596.4     70   97360
Waiting:        0  103 493.6     69   97360
Total:         26  148 718.9     77  101371

Percentage of the requests served within a certain time (ms)
  50%     77
  66%     83
  75%     89
  80%     93
  90%    114
  95%    269
  98%   1074
  99%   1963
 100%  101371 (longest request)

--- NILFS RESULTS ---
Server Software:        lighttpd/1.4.19
Server Hostname:        10.1.1.6
Server Port:            80

Document Path:          /test_test_test
Document Length:        107 bytes

Concurrency Level:      253
Time taken for tests:   484.517 seconds
Complete requests:      1000000
Failed requests:        36
   (Connect: 0, Receive: 0, Length: 36, Exceptions: 0)
Write errors:           0
Total transferred:      252991145 bytes
HTML transferred:       106996255 bytes
Requests per second:    2063.91 [#/sec] (mean)
Time per request:       122.583 [ms] (mean)
Time per request:       0.485 [ms] (mean, across all concurrent requests)
Transfer rate:          509.91 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   37 224.1      5    7038
Processing:     6   85 554.3     57   66351
Waiting:        0   82 397.8     56   66351
Total:         14  122 655.0     63   71375

Percentage of the requests served within a certain time (ms)
  50%     63
  66%     66
  75%     69
  80%     72
  90%     89
  95%    248
  98%   1047
  99%   1352
 100%  71375 (longest request)

Jacques, it strikes me that your distribution is not normal here and that a standard deviation (ie. variance) may not even be defined for the actual distribution – cauchy perhaps?

Other than that, good to see that Lua has some real world use. I looked at it a couple of years ago but couldn’t figure out if it was useful for anything much.

But good to read about anyway, thanks!