Some books lost; apology and next steps
UPDATE: Many books found, but not all. Crash problem solved. See end of this post.
If you've been following the blog, you know that LibraryThing recently moved to a new, much better server. This move has made LibraryThing fast again, but the transition has been rocky. Yesterday, a glitch erased some recently-entered books, and the new server's backup failed.
The losses are from books added in the last four days (ie., since the server switch). Some books added since then were recovered from "bin" files. And all books added via import have been recovered, but only as a re-import, so any subsequent edits to those books have been lost. The site will go live this evening, once the import queue has finished and I've sent messages to users who lost books (see below).
Here are the steps I am taking for users:
The person who has been helping me with the database offered the pithy remark "this sort of thing never happens twice." It's not that lightning never strikes twice, but that once you've been hit you stop bringing stainless steel cutlery to picnics. LibraryThing's new backup regime—nightly backups and an enterprise level "hot backup" at a different location (in fact, on opposite sites of the continent)—will go a long way to ensuring future data security.
Thank you for your understanding.
UPDATE: Comments posted to this entry noted that some missing books predated the server transfer. This was a good catch. At the same time, the server could hardly stay up five minutes. The problems were related. (And related to the previously-mentioned crc32 change.)
I found all these older books, substantially reducing the number of users affected. (Over 100 users can keep their free accounts although they have fewer than 25 books missing—most have none.) As for books entered Monday night, few users managed to enter any, but those that were entered were lost—the crashes were just too severe. All this did have one salutary effect—I figured out the big problem. I think it will stay up now.
If you've been following the blog, you know that LibraryThing recently moved to a new, much better server. This move has made LibraryThing fast again, but the transition has been rocky. Yesterday, a glitch erased some recently-entered books, and the new server's backup failed.
The losses are from books added in the last four days (ie., since the server switch). Some books added since then were recovered from "bin" files. And all books added via import have been recovered, but only as a re-import, so any subsequent edits to those books have been lost. The site will go live this evening, once the import queue has finished and I've sent messages to users who lost books (see below).
Here are the steps I am taking for users:
- If you lost books, the system will send you a profile message indicating how many.
- Users who lost more than 25 books receive a free membership or membership upgrade. You can use it on your own account or gift it. Check your profile for information.
- All members who lost more than 25 books and paid in the last 30 days can receive a full refund—thirty days is the period PayPal gives me to do this. Send me your user name and either the email PayPal knows or the PayPal receipt.
- The import feature has been changed to screen out ISBN duplicates. This means that you can upload an exported file without duplicating or "writing over" books.
- A second server (the old one), which is housed separately, will act as a constant "live" backup, getting all the same information that the primary server does.
- The nightly backup has been fixed. (The new server's didn't have the right "permissions").
- I will shortly improve the CSV export and add a tab-delimited export.
The person who has been helping me with the database offered the pithy remark "this sort of thing never happens twice." It's not that lightning never strikes twice, but that once you've been hit you stop bringing stainless steel cutlery to picnics. LibraryThing's new backup regime—nightly backups and an enterprise level "hot backup" at a different location (in fact, on opposite sites of the continent)—will go a long way to ensuring future data security.
Thank you for your understanding.
UPDATE: Comments posted to this entry noted that some missing books predated the server transfer. This was a good catch. At the same time, the server could hardly stay up five minutes. The problems were related. (And related to the previously-mentioned crc32 change.)
I found all these older books, substantially reducing the number of users affected. (Over 100 users can keep their free accounts although they have fewer than 25 books missing—most have none.) As for books entered Monday night, few users managed to enter any, but those that were entered were lost—the crashes were just too severe. All this did have one salutary effect—I figured out the big problem. I think it will stay up now.
posted by Tim at 4:02 PM
66 Comments:
Tim -- I don't know yet if I've lost any records, but really, even if I have, it's not a huge deal. You've put in a lot of work here, and I think you've got an excellent product that will only get better.
Glitches come along with Beta software, like ants at a picnic.
I was on the site when the glitch happened yesterday, and noticed that books went missing - but I'm not entirely sure all of those were books I entered after the server swap on Wednesday. I could have sworn I was in the middle of inputting stuff, actually. I guess I'll find out when the site comes back up.
Don't be so hard on yourself. I'm not upset at the lost books, I'm more upset at the lack of communication over the past day. Warning people about downtimes as well as keeping people updated during long downtimes goes a long way toward keeping people placated.
~LB
These things happen, Tim. Nothing is really lost here except a bit of time re-entering. We still have our books, after all!
Thanks for letting us know what's happening. Bummer. I'll let you get on with the mopping up.
Thanks for the update and for all your hard work! We appreciate it! This is such an awesome site.
Stop saying nice things!
Actually, I'm going to bring it up shortly--without all the queue. There are over 10,000 items in the queue, and Amazon only lets me hit them once/second. So, it would be hours before the lamb went through the python.
Ugh, I think I lost my entire library. Goodbye Librarything, it has been fun.
Tim, nobody in their right mind is going to be mad at you for losses which are a) small or b) so graciously requited; and which are both in the nature of things, and so publicly, personally acknowledged and apologized for.
If I'd lost 73 books instead of 12, I might be frustrated, but I wouldn't, personally, feel it remotely fair to let you refund my money, (or, or, or)! I am firmly persuaded you haven't got enough of it, as it is!
Thank you for developing this wonderful Thing with such care - and responding to problems in ways that make us even more pleased to be part of it.
Thanks, especially, for letting us know what's going on, and what to expect.
Julie
Thank you for the response on this - rest assured that I, at least, am not taking it personally and you have no need to feel bad. Things happen. I will check the import queue etc for my missing books, but since I'm nowhere near finishing tagging what I have imported, let alone importing the whole of my library, a small glitch is not going to dim my appreciation of LibraryThing. :D
Sorry for some startup jitters. It's always hard to start up from nothing, particularly when so many users were just waiting for it to reopen. After a few minutes it has much more of the data in memory (rather than on disk), so it's faster.
Dear Tim,
We realize how you've been struggling. We worked out the three we lost. Not to worry. What a tremendous challenge you've taken on.
Browning said, "A man's reach should exceed his grasp/Or what's a heaven for."
Chava
It's down now, but it's not ME. It's the rackspace people. AAAARRRGH!
This was so bad it's funny. Now it's bad again.
Well for the brief time it was up again it looks like I lost my entire catalog, however: 1) almost all of it is in the queue (though I'll have to re-tag of course); and 2) all of it is also on my local machine in DeliciousLibrary, which I'm now glad I didn't give up quite yet. ;-)
In any case, thanks for handling this graciously with the offer of refunds etc. I only joined a few days ago, but I think I'll donate my free membership to a friend once more of the kinks are ironed out.
*noogie* No worries. Honestly, this is still wayyy less buggy than Bibliophil was, so I'm still happy. I do have a pile of books waiting to be entered, though!
I just wish I would have followed through with the questions about not being able to print out a hard copy list of my library. Now that I'm beginning to grasp how daunting is the task of trying to differentiate the lost 143 individual and manual listings from the remaining 1577. Add to that the extra effort of searching out on the web or scaning in all the cover photos and I guess you might say I am (fill in appropriate word).
Dan
I didn't realize that Library Thing had been done - I've been away.
You are doing an awesome thing and thank you for accepting responsibility - so many people wouldn't.
I don't think I've lost a thing - and, even if I have - I can re-enter.
Thanks again Tim for doing such a great job!
Tim, from one techie to another I hear ya. I have a few projects I have worked on that I never released to the public for fear of the whole thing exploding. My gf got me a lifetime subscription for xmas, and has since got 4 other people to join.
So ya, most people won't mind that they lost some books, and your compensation is very genorous (although if I were in your position I would feel as bad as you, or worse!).
The people that are fuming, are doing so because you are providing a unique service, which is a good thing :)
as an aside, when I was a lowly (4 years ago) co-op student at a big tech company i took a backup of a database before trashing it. 1) it turns out another group was using it for testing. 2) I took an online backup vs an offline, and deleted the logfiles.
Just a tip, if you ever use db2, logfiles are not 'logfiles', they are 'transaction logfiles
. And very much in need, if you do an online backup. Had to rebuilt that s.o.b. from the ground up :/
dan, if you need any help let me know, leave a comment on my profile
'retinaburn', if you think it would be useful.
Quick notes:
[1] Blog comments were down for an hour. THIS WAS A SCHEDULED DOWNTIME AT BLOGGER.COM, NOT AN ATTEMPT TO SILENCE THE BLOG.
[2] No covers were lost, only the book that pointed to them. I will work on a script to allow people to claim these covers and link them to their books.
The post claims that only books that were entered in the last four days were lost. However, all of my books are gone, and they were all entered within a few days of starting the account on January 17th.
Would this indicate that it is possible to recover my books? Or that the problem was more widespread than previously thought?
The post claims that only books that were entered in the last four days were lost. However, all of my books are gone, and they were all entered within a few days of starting the account on January 17th.
Would this indicate that it is possible to recover my books? Or that the problem was more widespread than previously thought?
My librarything username is "ethernight" if that helps.
ethernight -- Thanks. I'm looking at that now.
One other thing: My profile page still lists the number of books that used to be associated with the tag, even though I now have zero books. This seems like it might be indicative of a bug.
Hey Ethernight. I have 82 of your books in the backup.
Now I need to figure out why your books are missing in the current db.
Different problem, or more of the same? Found you OK last night and looked around but didn't do anything. However this morning , IE stops dead and Safari tells me there are too many redirects. This is the only place I can get to. If you aren't back to normal soon, could you post a little something on the google group? Please?
hailelib
wow i lost 469 books, that is 3 days worth of work. and still the site isn't up. it sure was good while it lasted. hope it doesn't go away for good.
I'm a techie but not a sysop nor a db operator but.
May I suggest only making one upgrade at a time. Update the hardware with the exact same OS and software versions. Then when everything is nice and smooth, update the OS or one bit of software. But only after testing them first on an offline machine with a copy of the real database.
Real sysops and db ops may now feel free to flame me if I'm wrong (-:
Tim -- I lost a lot of books and more importantly, a few mini-essays I'd written about some books. So it's upsetting, but LT is a great thing and if you can resolve the stability/performance problems it'll take off. These are "good" problems to have. Still a fan.
OK it's back, it's fast, I only lost 2 things I added yesterday in the ups-and-downs as far as I can see.
But I get a fatal error when I click on an item to add after it was found in Amazon.
Tim,
I claimed my free membership yesterday (thank you), entered a few new books, but now I see that those books are gone again and so is my free 'paid' membership. Do I really need to claim my free 'paid' membership again? Should I wait?
to expand on what hippietrail said:
i got a fatal error for both my books that i was re-adding (i only lost two), one from LoC and one from Amazon. even though it said "fatal error," when i checked my catalog the book was there, but, and here's the weird thing, the summary field was completely wrong. both times, the book ended up with the summary field of a completely random other book. but everything else was fine, and the summary line is easy to change. however, this does seem like something of a bug.
nperrin
According to numbers I have on my profile I have gone down 11 books. The system has guess 12.
However my recently bought/read pile only has 7 books on it. Strange.
BTW I have the same problem as hippietrail - fatal error (5,2) - when using amazon or amazon.co.uk
Above and beyond the call of duty, in my opinion.
Just wanted offer my support. It might be easy for me to say since I didnt lose any books. But you are doing a great job, and I know that I certainly appreciate it. Keep it up.
I lost a handful of books and a ton of tags, but no major damage. One thing I did figure out is that I seemed to have lost a review that I wrote, but when I went to the social data page, my review was there. I had to go back to edit my book, and the review was sitting there. I just hit submit, and then it was showing up that I reviewed it on all of my pages.
Hopefully that can be of help to someone else who "lost" a review even though it is showing in other places.
Thanks for posting the comment to let me know that about 16 books are missing; they can easily be reposted. Thanks also for running this fantastic resource.
I still have all my books but I lost almost all the "date read" fields...
Luckily that's no great damage for me, but I hope from now on things will finally get better. As much as I appreciate Tim's work and dedication, I wouldn't think that users with a year membership are really happy with this.
Well, I had composed a lengthy comment yesterday, with just the right combination of distress, understanding and cautious optimism about the recent crash of LibraryThing and my loss of entries, but when I tried to post it, Blogger.com was DOWN. So I went home from work to find that something was piddly-poo with my wireless internet connection. TODAY, work server back-up failed for the third time in a row. Are the Cyber-gods angry with us for some reason????
Okay, I'm a little annoyed.
Being an experienced db programmer (whose offer of help a couple of weeks ago went unanswered) I knew enough to back up my own data: so I used your 'export' facility periodically, and because I did this, I figured I only lost about an hour's work.
But I was wrong: the system can't import from its own export files! It ignores everything except the ISBNs, which many of my books don't have, and it ignores all the manual changes I entered into LibraryThing itself!
What were you thinking?
Less important: the 'gift' only changed my status from 'paid' to 'paid' -- what was it supposed to do?
I also had the paid-to-paid behavior, unless it was supposed to bump my paid-for-a-year status up until today.
I've worked with the LAMP stack a good bit as a programmer, and the lack of easily findable good contract sysadministration is a major factor keeping it from being used more. Getting these programs and setting them up without knowing them well does not result in a system that can handle real load, unfortunately. (If anyone experienced reading this would like to do some paid work on a per-hour basis or something, please send me Email at rpuchalsky@att.net).
I would like to echo the last anonymous person about reloading the export files. You absolutely should make it so that people can reload their export files, complete with tags and modified fields and so on. I hadn't realized that you couldn't do this, but without, the export files are not really that helpful.
As a veteran of many such breakdowns, my best advice is to take a moment to relax.
for those of us who are NOT "experienced db programmers"....how do we back up our own data????? I didn't know that was an option. I have wondered about that since I signed up.....and while we're on that subject, how about a printer-friendly version of the catalog, so I could have something to hold onto when the site is down ....
Howdy,
I'm certainly not going to ask for a refund of my lifetime membership, since it still amortises out to a pretty darn good value (and I got it while it was cheaper). I also have very little to complain about: the books I lost in my reading-log account were only a few and luckily I hadn't reviewed them yet, plus I can fudge the started/finished dates since they're so recent.
That said, I received an (automated?) comment estimating that I lost eight (8) books in my main account ("angharad"), but I only count five (5) in my recently-bought heap next to the couch here. Better than the other way around, I suppose.
you can export your collection in the Extras tab.
Ken and others: Listen, I can recover your reviews. The trick is this: I have the book review data (also comments), but I don't have the data to link it to a book or user. My thought is, why not just put up a page with all the revies that were lost. People can find their review there and add it to the book?
Moloch: Yuck. Sorry about the date fields getting blown. Unfortunately, they are in the same db as the book titles, etc.
Technical difficulties. Not only was blogger down, but my colocation facility had a power outage.
Liz: Sorry about that. Not many people managed to redeem a gift before it went down. What's your userid? I'll upgrade it.
Anonymous with the import/export problem: The free account doesn't upgrade you to life, but I would be glad to do so.
I'll look into the proposed CSV export/import feature. I can understand why there might be confusion there, although the functionality is stated rather explicity. The trick is getting an export that's not full of junk and in one file—LibraryThing is a relational db and any export would be full of keys and time stamps and status codes that would be quite useless and even confusing.
I wonder if I could deliver a real database dump for a given user, but encrypt it somehow that it didn't lay bare the whole schema.
Rich: Send me your ID too.
More later. Incidentally, I'm responding to email faster than I am profile comments or blog posts. Email me at timspalding@librarything.com.
Sorry if I'm asking something obvious, but how do I get to see any comments on my profile? When I looked at my profile after reading about the various problems it had a highlit note at the top saying something to the effect 'You have a new comment' but then there's nothing on the page below the 'Member since ...' line. I had wondered in the past whether this was because I am not yet a paid member. On looking at my profile now this highlit line has also disappeared but I presume it was about the book entries I have lost.
My user name is fancett.
I've also discovered one of the hazards of using LT for producing a list of books I want to read (with comments and reviews from elsewhere noted) as I chucked out the pieces of paper reviewing them after I entered them in LT. Please, Tim, can you make sure all user added data such as this can be exported and reimported in user backups to give our data added security - though hopefully your colleague is right and you've had your lightning hit.
I've also discovered one of the hazards of using LT for producing a list of books I want to read (with comments and reviews from elsewhere noted) as I chucked out the pieces of paper reviewing them after I entered them in LT. Please, Tim, can you make sure all user added data such as this can be exported and reimported in user backups to give our data added security - though hopefully your colleague is right and you've had your lightning hit.
I've had the same problem. I can reenter the lost books that I own, but the "books I don't own, but want to read" are lost to the ether. If there's any way of fixing the printout problem that I and a lot of other people are having (when we load the printable view, only the first page will print-the rest of the table scrolls off and won't print (both in Firefox and IE)), that would be greatly appreciated. A printed record of our catalog is better than nothing. Thanks!
I am the one who redeemed my free gift and then lost it again. My librarything user id is librarytavern.
Tim - thanks for all your hard work. It's a relief to get all my books back, but I would still be grateful for your efforts on this even if I didn't.
In regards to the import / export thing, may I make a suggestion? You said,
"The trick is getting an export that's not full of junk and in one file—LibraryThing is a relational db and any export would be full of keys and time stamps and status codes that would be quite useless and even confusing."
It seems as though you are thinking that the export would need to be more or less a database dump. However, couldn't the import act as a batch add? In other words, treat the import as a way to go through and add the books again, rather than to upload a backup. This would result in historical data, such as the date the book was originally added being lost, however I suspect this would still be the best solution.
Thanks again.
--ethernight
I'm sitting home in Anaheim Hills, CA roughly 24 hours after a mandatory evacuation order due to the Sierra Wildfire. Yesterday there was a real danger than my small book collection, along with other valuables, would soon be ash. The danger seems to have passed, but we are still supposed to be 'evacuated'. This is the first time I've thought to check on LT since my last blog post looking for a 'downtime update'!
No apologies or free offers are necessary for me. I will re-enter the handfull of books I lost. I have a scanner on the way, so if some of the books have barcodes, it will be easier this time. I feel for those who have lost reviews, extensive comments, or tags. I'm a minimalist there. I just want a record of the books I have.
LT is a bargain for those who have a use for it. I'm still amazed at its low cost and real functionality. Thanks again, Tim.
I haven't checked the site for a week so I missed all the excitement. In any case, I was missing 11 books which have all been added back in. Seems like an amazing confluence of events that caused this problem. NOT A BIG DEAL. The speed improvement should make the site much more palatable. Thank you and keep up the good work . . .
WIth regard to the printing issue, if you go to the "Change Fields" tab, and lower the number of books per page to 10, I think it will print OK.
> Please let me know if there is anything I can do.
well, the new lamp for the kitchen should be put up.. ;-)
Tim, I hope the server will run smoothly now so you can get back to enjoying LibraryThing - it's such a cool site.
The gift membership will make a nice gift :-)
thank you
sunny
I'm not sure if this is universal, but it might help other people who lost a lot of tags in the last few days - I am finding that searching on tags is bringing up books that I know I tagged just before the crash - even though they are showing no tags in the catalogue view. I can ultimately, albeit slowly, use power edit to recover tags on most of these, therefore. [Now is when I'm wishing I'd implemented a rather simpler tagging system but that's my own fault.]
If you're not sure what I mean, take a look at 'living room' in my tags. I've stopped updating that tag for the moment, so some of those results will still show no tags or no 'living room' tag.
Hi there,
I definitely lost all of the books that I entered 238 - the start of my account. I would like my membership payment back and would like to keep my account. I suppose I'll just have to enter them again, unless of course you have other solutions/suggestions.
As far as it being Beta and things crashing - I understand and am glad to see an appropriate reaction (more backups.) This is a free and/or nominally priced service - you pay for what you get, you know? It's very nice to see you admit full responsibility. I applaud you for that.
username: bookishwendy
speaking of backups, for people writing long reviews, etc., never, EVER write such a thing in a web form, online, on librarything, etc. always write and save it on your local machine, then paste it into the online text box, or wherever
otherwise, you are guaranteed to be burned by losing something at some stage
dogzballz
Hey, it'll give me a chance to go through my books again. And I'll enjoy the chance to get someone hooked with the account gift.
Hey, Tim,
I've been waiting for the pressure to die down before piping in.
This is no biggie to me; after all my books didn't disappear, just some electrons!
I think there are 67 books missing off the end of my account (sorted by date of entry).
If you have them anywhere and can restore them, great. If not, ce la vie.
My account said "you have a new comment" but there was no comment and no e-mail (I have the e-mail box checked).
Thanks! Cheers!
My book loss is minimal, but I did do a fair amount of tag editing recently and all that was lost - according to the Tags list, at least.
However, as bopeep mentions, some of the tags appear still to be there, but "invisible". For example, my tag kanji book brings up all the texts that I assigned it to, although the tag itself is nowhere to be seen.
Tim, will these tags be restored? Or would it be best to take bopeep's suggestion and attempt to power-edit them as best we can before these ephemeral connections vanish for good?
Hey Tim, you let me know that about 80 of my books are gone and that looks about right. I guess there is absolutely no way of recovering them?
Keziah. Yes, unfortunately that's true.
I should mention on hail-mary possibility. If you subscribe to an RSS feeder, like Bloglines, you might have an archive of your recently-added books. Unfortuantely, this can't be imported directly--it has no ISBN. But it might jog the mind.
Tim I'm so thrilled I lost 24 books so no free anything but I don't care because I remembered them all so even though I forget where I parked my car and why I've gone into a room I'm not such a dippy old dear on the short term memory after all.
Tim--Just wanted to let you know that you've done an amazing job with The Thing, and I hope you haven't lost sleep over the data loss! I was lucky and just lost a few things, but even if I'd lost the lot, I would have considered it 1/2 my fault for not exporting the data when I *knew* export exists and I *knew* I was using a Beta product. Keep up the great work!!!
How do exports work? I tried but Excel didn't seem to work. What am I doing wrong?
Keziah Hill said...
How do exports work? I tried but Excel didn't seem to work. What am I doing wrong?
If you are asking how to take the exported data file and then import it into Excel, here's how.
First, you go to the Extras page and click on the "Export all records as text" link. The resulting file, bookdata.csv (CSV = comma-separated values), is saved to your hard drive.
Next, open Excel first. THEN use File > Open to open bookdata.csv. (To see it, you will probably have to change the "Files of type" field to "All Files".)
Excel now opens the Text Import Wizard window, to guide you through importing the CSV file so that it is Excel-formatted. The first screen defaults to Delimited, so you can leave that one as is. On the second screen, under Delimiters, select the Comma checkbox and deselect the Tab checkbox. The remaining defaults can usually be left as they are.
Lastly, do a File > Save As and ensure that you're saving your new file as a Microsoft Excel Workbook. Excel will default to calling the file "bookdata.csv" and saving it as a Unicode file, so you will have to manually change the file extension to .xls, too.
Is that what you needed?
chamekke
(writing anonymously from work because I can't remember what my password is :-)
Great thanks!
Hi,
oh NOOOOOOOO, so THAT´s what happened :(...I am new to the librarything and just entered my entire booklist during those four days...I think it was over 40 or something :(...already thought I did something wrong. So I´ll do it again :(...I saw I got a COMMENT (somewhere in the Profile it said "1 new comment" - but now it´s gone) - where can I read this? Just wanna know if I am one of the lucky unlucky who get a memebschip...Was planning on getting one anyway, but now that all is gone...well, but with the new server it should work smoothly I hope :)!
Nice greetings from Germany
Eliwagar
Post a Comment
<< Home