We're faster (but not resting)
Last Wednesday John brought live two new database servers, Alexander and Hannibal*.
Together, they more than doubled our database heft. Put another way, our servers, which were operating at near full capacity all day long, can finally rest a bit. They can do everything as fast as they're able, unencumbered by unsupportable amounts of work.
Performance. The effect on site performance has been positive. But problems remain. Profile pages are dramatically faster. Author, work, subject are faster and no longer slow down at peak times. Talk pages are essentially unchanged.
The catalog is faster. The page-generation averages now hover just over one second, not around two seconds. But I was hoping for more. The standard deviation of page-creation times remains high—people with huge libraries get hurt. Last night we I made a series of improvements which I hope will pay off. (The standard deviation is down, but will it stay down?)
The future. We will continue to improve. Until Wednesday the situation was desperate. When a box got behind, we had to turn off access to interior pages to all but signed-in members. That day is over, thank God**. And we can finally tease apart what was is itself slow, versus what was just slow because everything else was slowing it down. Lastly, John has long wanted to try out some low-level tweaks, but with no spare capacity, couldn't. I expect he will find ways to wring more out of what we have.
Whether he can or not, we are going to keep improving. We have laid aside the money to buy a number of other servers—up to ten, if needed. One or two will be database servers, probably removing administration and caching traffic from the live servers. A number will be memory machines—low-end boxes with tiny disk drives and obscene amounts of RAM. They'll help us use memory caching more effectively, reducing database load. The balance will be tasked in other ways—supporting LibraryThing for Libraries, serving secondary resources (covers, APIs, widgets) and providing redundancy, so we won't be skating along a cliff anymore.
Thanks to John for getting the new servers racked and running. Thanks to the members for hanging in with us as we grow, and grew and grew!
*Yes, I named them. Cliche, I know. But Alexander was my research interest in grad school, so I'm allowed! Anyway, at least they're consistent, and set a pattern we can follow (next up, Mithridates and Shapur). I'm still bothered that a previous sysadmin named our twin MyISAM databases Apollo and Athena, not Apollo and Artemis (who were twins). Then there's Plato and his bigger twin Mongo, which makes no sense, but feels right, and the one everyone hates, our backup machine, Mnemosyne.
** John adds "the upgrade has given our database servers more horsepower rather than more raw speed. While the new servers are faster, the biggest initial gain is in the amount of load we can take on without starting to slow down."
Together, they more than doubled our database heft. Put another way, our servers, which were operating at near full capacity all day long, can finally rest a bit. They can do everything as fast as they're able, unencumbered by unsupportable amounts of work.
Performance. The effect on site performance has been positive. But problems remain. Profile pages are dramatically faster. Author, work, subject are faster and no longer slow down at peak times. Talk pages are essentially unchanged.
The catalog is faster. The page-generation averages now hover just over one second, not around two seconds. But I was hoping for more. The standard deviation of page-creation times remains high—people with huge libraries get hurt. Last night we I made a series of improvements which I hope will pay off. (The standard deviation is down, but will it stay down?)
The future. We will continue to improve. Until Wednesday the situation was desperate. When a box got behind, we had to turn off access to interior pages to all but signed-in members. That day is over, thank God**. And we can finally tease apart what was is itself slow, versus what was just slow because everything else was slowing it down. Lastly, John has long wanted to try out some low-level tweaks, but with no spare capacity, couldn't. I expect he will find ways to wring more out of what we have.
Whether he can or not, we are going to keep improving. We have laid aside the money to buy a number of other servers—up to ten, if needed. One or two will be database servers, probably removing administration and caching traffic from the live servers. A number will be memory machines—low-end boxes with tiny disk drives and obscene amounts of RAM. They'll help us use memory caching more effectively, reducing database load. The balance will be tasked in other ways—supporting LibraryThing for Libraries, serving secondary resources (covers, APIs, widgets) and providing redundancy, so we won't be skating along a cliff anymore.
Thanks to John for getting the new servers racked and running. Thanks to the members for hanging in with us as we grow, and grew and grew!
*Yes, I named them. Cliche, I know. But Alexander was my research interest in grad school, so I'm allowed! Anyway, at least they're consistent, and set a pattern we can follow (next up, Mithridates and Shapur). I'm still bothered that a previous sysadmin named our twin MyISAM databases Apollo and Athena, not Apollo and Artemis (who were twins). Then there's Plato and his bigger twin Mongo, which makes no sense, but feels right, and the one everyone hates, our backup machine, Mnemosyne.
** John adds "the upgrade has given our database servers more horsepower rather than more raw speed. While the new servers are faster, the biggest initial gain is in the amount of load we can take on without starting to slow down."
posted by Tim at 10:49 PM
18 Comments:
Just wondering, not knowing your database setup: have you applied the usual database optimisation techniques?
Proper indexes, the right joins, no text field when a varchar field is possible (and very much faster!), &c. ?
Thanks for the update - the only thing I've found that's been slow today is the blog. (Thanks, Blogger!)
Mongo like from Flash Gordon? O_o
Also, why the hate for Mnemosyne?
I agree, what's wrong with "Mnemosyne"? I think it's apt for a backup server, and could only be more apt for a memory server. The namesake is, after all, the Greek personification of memory.
Was about to ask the same re: Mnemosyne. When I looked up Greek mythology so many years ago, she was my fave name! (even though I ended up picking this one because the first place I ever registered had a 7 char limit on names, for some reason >___>)
I've no idea what your finances are like, but you've built up such a large reservoir of goodwill among your members that you could easily raise money for new servers from a one-off appeal if necessary.
How about Thutmosis III?
You should have a naming contest.
Yes, the electronic database server that keeps a permanent history of every individual Library on Earth.
I vote for a server named:
POLYBIUS
re: database optimisation, we're not perfect but it is something we spend quite a bit of time on. We certainly pay attention to column types and indexes!
@Felius, sorry, I should have known :)
Apollo and Athena might just be Battlestar Galactica. :)
I known
So how does this effect Collections?
(Gilroy on LT, using Google log in)
So, does this have anything to do with the fact that LT has been largely inaccessible to me for most of this afternoon (the 25th)?
One of my first requests at a pharmaceutical manufacturing library was to find the names of Greek Gods to name the three components of a new HDL computer system in the late 1970s. Hercules, of course, was an easy one, and by the time I had come up with two more names, it was too late.
The computer guys had already named them after Donald Duck's HDL nephews: Huey, Dewey and Louie.
Lee Hadden
I realize this has nothing to do with servers, but it has been bugging me since I joined. Why can't authors with the same name be separated? Couldn't something simple be assigned to the name of duplicate authors like birthyears to separate them? John Doe 1950 versus John Doe 1929 ??? It's really disturbing to click on an author and see the wrong photo and a list of books that do not belong to that author. Thanks for your consideration.
So how come we're getting so many "failure to get a read connection to database" errors today? Are the new servers unhappy already?
Post a Comment
<< Home