Thursday, February 23, 2006

LibraryThing leaps forward: Everyone a librarian

UPDATE: I changed the way I load covers. Is anyone still getting a "stack" error?

In the last three days I've added a slew of new features, and a new structure to support future improvements:

"Works" : user-controlled book combination and separation

LibraryThing would be pretty useless if connections, ratings, comments and recommendations considered every edition of something like Pride and Prejudice a separate thing, as other online cataloging applications do. In the past LibraryThing made a guess, like Amazon does, and failed about as often. Library technologists are starting to do somewhat better, but there was no perfect answer. It was time to try something new.

Starting three days ago, I announced a trial project to let users determine what books belonged together, the first time anything like this has been attempted. Using simple check boxes, users could go through a favorite author's works, combining and separating editions as necessary.

The response has been startling to say the least: In three days, users have combined 17,000 times, mashing together 42,000 works! Users have spent hours at the task, and debated the nuances in a blog post that now sports 182 comments. Although only a few of these Christmas elves are actual librarians, but most are experts on the authors they labor over. As one wrote on the blog, Isaac Asimov's Nightfall the short story collection, is distinct from Nightfall the novel and from Nightfall One. Do libraries know that? Does Amazon?

As with tagging, reviews, ratings, uploading scanned covers and other user content, LibraryThing users are taking book information into their own hands. And they're doing it because it's fun and they see the benefit right away—filling out the catalog with covers and cataloging data they wouldn't otherwise have, and connecting them with like-minded reasons, even if the person who's read every Asimov book they have did it in Finnish.

Book pages revised

Book information pages (the card-catalog, people and pencil icons) have been enhanced in a variety of ways.
  • Social pages show all the work's covers, by popularity, including user-contributed ones. (Note: Some Windows IE machines fail to load all the images. If the book has very many covers, this can show an error. I am working to solve this.)
  • Each edition links to Amazon, Abebooks, Alibris and other online merchants.
  • Editions also link to the OCLC's "Find in a Library" project.
  • Users can now swap covers easily, and for the first time snap up covers uploaded by other users.

New Recommendation Options

The new "works" system has opened things wide up for improvements. One of the first is an enhanced "recommendations" engine, now based on the "full work," not just some of its editions.

The new engine shows both "People who own X also own Y" and "Similarly tagged" books. The "raw" option shows "People who own X also own Y" without any weighting applied (J. K. Rowling rules!). The "exclude author" option is useful when a book triggers ten or fifteen suggestions by the same author, as happens with authors like Agatha Christie or Stephen King.

Search

LibraryThing now allows library-wide title and author searches based on the new work system.


Deweys and LC Call Numbers for everyone!

Until now, members who found books through the Amazon search (most of us), had no access to Library of Congress Call Numbers and Dewey Decimals, which only came through library searches. The new "work" system leverages everyone's cataloging, bringing true library data to 175,000 works, including most popular ones. These numbers now appear in your catalog, in green to distinguish them from your own numbers.

In addition to Deweys and LC Call Numbers, card catalogue pages now allow you to browse works' MARC records, a Matrix-like stream of data librarians are said to be able to decipher, and even enjoy looking at.

Forward

The new features have gotten a workout since starting testing three days ago, and users have been very helpful with bug reports and suggestions. Some systems are still transitioning, and some problems remain, but issues are being knocked down one-by-one. In my defense, developing LibraryThing is usually like working on a train while it's in motion. The recent changes were like turning a train into a monorail while it was in motion...

Some anticipated improvements include:
  • Fixing all occurrences of that MSIE6 "stack overflow" bug, and the pesky double-frames issue
  • A definitive statement on when translations should be combined (I'm working on it!)
  • Work disambiguation—there, I said it—available through the search system
  • Improved author disambiguation
  • Deweys and LC Call Numbers for all books, not just books in the system
  • Browsable LC Subject headings for all and sundry
  • A space-based laser to smite people who write in library books
Comments, suggestions, criticisms, complaints and bug reports always welcome.

47 Comments:

Blogger Ken said...

The new features are wonderful, but LT has been "down for 10 minutes" for half an hour. Please say there's no data problem going on.

2/23/2006 4:05 PM  
Anonymous Anonymous said...

So good you had to post it twice.

Just kiddin ya.

Great job

2/23/2006 4:32 PM  
Blogger Ed said...

Anonymous (that guy is everywhere and has a lot to say!) beat me to it. The news is so good we get to read it twice!

2/23/2006 4:32 PM  
Anonymous Anonymous said...

A problem with similar libraries: if someone else owns two copies of the same work (say, two different translations of Dante's Inferno) that I have then it shows up that I share two books with them. But I don't; I only share one with them. Maybe it needs to be changed to works shared, so that it'd show up accurately (although even if it's books I still don't share two books with them.) Also, when I click on the number next to the username on the similar libraries list on my profile, I can only see the first page of our shared books; when I click "2" or "next" it goes to the first page of their complete catalog.

--kelsey

2/23/2006 4:34 PM  
Anonymous Anonymous said...

The look and feel of the new card-catalog pages is terrific -- very professional.

2/23/2006 4:36 PM  
Anonymous Anonymous said...

Oh, and "who's helped" is great--now I know (or at least can assume) if edition combinations have been done manually or automatically. Also, I'd be interested in seeing a feature that prompts you to combine a newly-entered book with a pre-existing work (maybe only if the new book is the only book in that work or something)...would save a lot of future work!

--kelsey

2/23/2006 4:39 PM  
Anonymous Anonymous said...

Yes, kelsey (who posted as anonymous----that other guy will be mad you're using his name!), I noticed the double sharing issue too. I own two copies of To Kill a Mockingbird, a paperback and the 35th Anniversary Edition. It shows me sharing two copies of it with my daughter, who owns one other paperback edition of the book. Furthermore, it lists my username twice in the list of users who own the book! As they say in PA Dutch country, that needs FIXED.

Another bug: I've been getting multiple headings (the LibraryThing logo and tabs) at the top of the page lately while using the site. Is that what Tim means by the pesky dobule-frames issue?

2/23/2006 4:46 PM  
Blogger Tim said...

Well, I had to post it again this time with pictures. They add such flair!

Sorry about the "down for 10 minutes" message. I figured I'd optimize the database too, since the work-disambiguation has been so heavy. I forgot to change the default time.

Kelsey. Yes, the thing that prevents multiple copies from matching doesn't update when works are combined. The works are changing so fast that lots of "cached" features have gotten out of snych. I'll run some synching things tonight when the load is less.

That said, I think you SHOULD be listed twice if you gave the books different ratings (maybe you rate by edition qualities, not generally). And a case could be made that someone who has ten editions of Ovid's Ars Amatoria (um, me) should be matched up with other such hoarders. (And maybe all of them should be committed.)

2/23/2006 5:10 PM  
Blogger Tim said...

Incidentally, is anyone having that stack error thing still? I think it only happens now when there are dozens or hundreds of ISBNs for a book. The system tries to load them all from Amazon, "hiding" any that Amazon denies or, more commonly, returns as a one-pixel image. Those one-pixel images are my bane—they totally screw up a design! Anyway, when there are dozens of images to load, the browser can't handle the input at the same time as it's doing all the dynamic hiding of one-pixelers, and chokes.

I'm going to change how images are loaded (preloading them). If that doesn't work, I'll have to start storing which covers Amazon has and which they don't. There's a tricky Amazon terms-of-service issue there, but I think I can skate over it if I just store whether Amazon has it, not the image itself.

2/23/2006 5:15 PM  
Anonymous Anonymous said...

That said, I think you SHOULD be listed twice if you gave the books different ratings (maybe you rate by edition qualities, not generally). And a case could be made that someone who has ten editions of Ovid's Ars Amatoria (um, me) should be matched up with other such hoarders. (And maybe all of them should be committed.)

I think we should be told how many "works" we share, but when browsing another library in "books you share mode" (currently broken - see the Google group for bug report) it should only have multiple entries for multiple editions they have. Right now it has multiple entries no matter whether the other person or yourself has the duplicates. For instance when I look at what I share with bloomsday I like to their 2 editions of Animal Farm with different covers but I don't need to see three copies of their single Baudolino with the same cover just because I own 3 editions (all with different covers).

Hope you understand, because even I almost don't (-:

2/23/2006 6:09 PM  
Anonymous Anonymous said...

can I beg, plead, reason with y'all to move long discussions over to google groups? It's so hard to follow them here, with `180+ comments.

2/23/2006 6:15 PM  
Anonymous Anonymous said...

Tim, could you please, pretty please look at the social information page for the novel Homestead by Rosina Lippi and tell me how to fix the error that has the author as Sara Donati?

I'm sure there are other books with this problem, where an author has more than one name they publish under and the result is confusion in the book and social pages.

2/23/2006 6:18 PM  
Anonymous Anonymous said...

Yes, please do try Google Groups if you haven't. This seems pretty inefficient, and the Group over there is quite active.

2/23/2006 6:20 PM  
Anonymous Anonymous said...

The combine works is graet. What's now is missing is a way to combine authors. For example some people spell the russian author Anton Tschechow this way, others spell it Cechov and I guess native Russian speakers will use Cyrillic. There should be a way to mark them as the same person.

2/23/2006 6:30 PM  
Anonymous Anonymous said...

Thank you, thank you, thank you! For the LC and Dewey Call numbers. My books are organized by LC, so having the actual number instead of having to guess after fruitlessly looking them up on-line is a major help. Thanks again for this great service!

2/23/2006 6:31 PM  
Anonymous Anonymous said...

The link to the LibraryThing Google group:
http://groups.google.com/group/Librarything
;-)

2/23/2006 6:38 PM  
Anonymous Anonymous said...

LibraryThing Google group

2/23/2006 6:43 PM  
Anonymous Anonymous said...

I'm sorry I can't share all the excitement of these days... I still wait for the "definitive word" on translations before starting combining :-|

2/23/2006 7:16 PM  
Anonymous Anonymous said...

andi -- there is a way to combine authors, but you have to go over to Google Groups to find it.

2/23/2006 7:33 PM  
Anonymous Anonymous said...

Space-based laser?

Curses, I was in the "targeted lightning strike" camp.

2/23/2006 10:13 PM  
Blogger Tim said...

No it's not a corner case. The corner case is that the year (and modifier) are supposed to be stripped off it. I'll get the code to do that.

If you haven't used a library, the Deweys and LC Call Numbers come in from the works system. They are, as with other works data, the "winning" information. If libraries differ about a Dewey or LC Call Number, as they sometimes do, the majority opinion wins. The full run of possible numbers is shown on the card-catalog page, with the number of library MARC records supporting it in descending order. If you want to dig deeper, LibraryThing provides the MARC records themselves, also in descending order of frequency.

The alternative would be for everyone to use only libraries. Book requests would take five-fifteen seconds each. Alternately, LibraryThing could, in theory, decide on a single, ruling library, and fetch records from there during off hours. Some of that will probably be done, but I'm not keen on making in the hub of the system. For one thing, no library—even the LC—has all the records. Users who care about library data are also generally savvy about which libraries to search for it. Using their data is the right move.

I already flag when a Dewey or LC Call Number is coming from the works system. I can provide an option to disable Deweys and LC Call Numbers not expressly provided by the user.

2/23/2006 10:31 PM  
Anonymous Anonymous said...

Your last planned upgrade is excellent. ;)
I just wanted to say keep up the good work. I thought this was a fantastic site the moment I heard about it, and you just keep making it better. I know you get lots of comments about problems and what not, so I thought you should also get some praise (though I also love the way you handle these comments, and I think that it improves the site immensely that there's this great feedback between the users and the programmer/creator/grand poobah).

2/24/2006 12:36 AM  
Blogger Tim said...

Thanks for the kudos. I don't know if "grand poobah" fits the situation. I think there's an Aesop's fable about trying to satisfy everyone that's more on target.

2/24/2006 12:45 AM  
Blogger Tim said...

Saralaughs. That's an interesting problem. Once you separate authors, the way to get it to "rethink" a book's author is to "separate" one of the editions. (Whenever a separation occurs, the "democratic process" happens for both books.) So I separated it. See http://www.librarything.com/author_combine.php?author=lippirosina for the books.

In an ideal world, LibraryThing would have both AKAs and pen-names. They're not quite the same, I think.

2/24/2006 1:05 AM  
Anonymous Anonymous said...

In an ideal world, LibraryThing would have both AKAs and pen-names. They're not quite the same, I think.

They're not. Libraries get around this by using author authority records, choosing one definitive name (Samuel Clemens, for example) to link all works by that author (even if it says Mark Twain on the title page).

I will also note that translations are typically considered to be distinct from original works, to a degree. I forget exactly what degree it is, whether it's at the exemplar or manifestation level according to FRBR. However, this can be enough to assign different call numbers, just as different editions in the same language will receive them.

Finally, I will say that LCC order is still kinda wonky. It still seems to be going digit-by-digit, which it shouldn't be doing. Also, a few of mine are manifestly out of order (a PS in the middle of the Bs, for instance) and I'm wondering if this could be a consequence of work combination?

Your best source for call number data would be OCLC WorldCat, but since it's a subscription-only database you would need to access it through a library.

2/24/2006 1:20 AM  
Anonymous Anonymous said...

Finally, I will say that LCC order is still kinda wonky. It still seems to be going digit-by-digit, which it shouldn't be doing. Also, a few of mine are manifestly out of order (a PS in the middle of the Bs, for instance) and I'm wondering if this could be a consequence of work combination?

I just noticed this also and poked at it a bit. It is indeed a product of the new "works" system: if you check the works page for that PS volume, you'll find that other people have it classed as a B, and majority rule is determining the sorting within your own catalog. (Somehow the sorting doesn't "see" your copy's call number; it only sees the majority-rule call number for the "work" as a whole.) Obviously this isn't right, but it's probably an unexpected consequence of the new system that will be fixed sometime soon.

2/24/2006 2:51 AM  
Anonymous Anonymous said...

It's beautiful. Thank you.

LT user Chavala

2/24/2006 3:12 AM  
Blogger Tim said...

Can you give me an example of the PS/B problem in a real catalog. It's not happening in mine.

T

2/24/2006 3:17 AM  
Anonymous Anonymous said...

Can you give me an example of the PS/B problem in a real catalog. It's not happening in mine.

Well, I swear it was there an hour or so ago, but it's cleared up in mine also and everything is as it was before. (You waved the dead chicken just right.)

I even had one book that was appearing just before "Z" where "Y" would be -- which I knew was odd because Y isn't a defined LC class. I looked at the social data, and in the call number field another user had entered "yes" -- so sure enough, it was sorted in my catalog under Y.

But it seems to be fine now. Never mind...

2/24/2006 3:55 AM  
Blogger Tim said...

No, I made a change. It was a silly change, but a change. I *thought* it might make the difference, but I had no way to test it. Thanks!

2/24/2006 4:02 AM  
Blogger Tim said...

What's your username, RJO? I tried to leave my message there.

2/24/2006 4:03 AM  
Anonymous Anonymous said...

Sorry -- RJO = rjohara

And whatever you did, it worked!

2/24/2006 4:07 AM  
Anonymous Anonymous said...

Hi Tim,

My quick review: LibraryThing is great and getting better. Thank you.

Now for the challenge I've created for you.

I collect editions of Clement Clarke Moore's 'A Visit From St. Nicholas' (aka 'The Night Before Christmas'). I have more than 100 copies. I wanted to list them online because friends and family often spot editions and wonder whether or not I have that copy yet. In the past, they've gone ahead and bought a particular edition only to discover I already have it. LibraryThing should limit that from happening.

Unfortunately, as Kelsey noted, multiple copies can really clog up the social page! Additionally, because of my multiple copies, anyone with a copy of 'The Night Before Christmas' is getting referred to my library multiple times. Any suggestions how to handle this?

You'll find the books under the tag NB4X in my library if you want to see what this is doing...

_Celeste_

2/24/2006 4:34 AM  
Blogger Tim said...

Well, that's a POWERFUL argument for screening out "duplicate" books. Doing that is not hard at all. I run the process regularly. But I might want to have the process happen every time you add a book. I'll give it some thought.

2/24/2006 4:47 AM  
Anonymous Anonymous said...

I'm done entering my 'The Night Before Christmas' editions now, Tim, until next Christmas when I get a handful more. If you screen out duplicate books at this point, it should STAY screened for a while.

If I later update information on the books already there (add reviews, comments, tags, etc) would that trigger duplicates?

_Celeste_

2/24/2006 12:50 PM  
Anonymous Anonymous said...

Combining is great, but separating is just as good, because finally my copies of books with a general title and a different subtitles can be separated. I've found I've had to separate a lot before I could combine properly because of that pesky colon! (and in some cases, was delighted to be be able to do it)

Also, would love to be able to combine books without an author, or with multiple authors with a different one as the first author, or where the user has put the "wrong" author in!

2/24/2006 5:39 PM  
Anonymous Anonymous said...

I'm still getting the "stack overflow at line 0" error...

2/24/2006 9:01 PM  
Anonymous Anonymous said...

I still got the "stack" bug here:

Title Principles of Biochemistry: Advance Chapters from the 2000 Edition
Author Albert L. Lehninger

@10:30pm CDT 2/24/06

2/24/2006 11:30 PM  
Anonymous Anonymous said...

First, thank you, thank you, Tim, for doing such good work and letting us play with your site. It's amazing to watch it all evolve.

Two problems I've noticed this week:

1) A book I entered the other day now appears in my catalog (bibliofile) in a different edition than what I entered.

Example: I own the Plume (1985) paperback of Doris Lessing's _Martha Quest_. Currently my catalog lists the 2001 Perennial edition. It's possible that it's my error, but this has happened with several books.

Going to the shared page for that book, I can see the ISBN for the Plume edition, which leads me to the other problem:

2) At the shared info page for a book, clicking on another edition's ISBN pops up a little yellow menu window. However, none of the links work in that menu. I go to the destination site (e.g., Amazon, OCLC) but see only an error message.

Problem #1 is more important, I think.

2/25/2006 2:13 AM  
Blogger Tim said...

Bibliofile. LibraryThing certainly has problems, but changing the publisher between copies of a book is not one of them. There just isn't a mechanism for that to happen, so I'd bet money that was in there originally. If your book was entered recently (post crash) I could tell you the exact information as it entered the database (I now make an instant backup). But not for an older book.

Note that, as stated on the page, "Changing your cover to an Amazon cover also changes your book's ISBN." But doesn't re-fetch the publisher.

The second problem, with the links, I mostly fixed. The links were getting eaten whenever the most popular ISBN didn't have an Amazon cover (or LibraryThing hadn't discovered it yet). Anyway, the text links will never always work. Many older ISBNs aren't stocked by Amazon. It would be nice if they said "we don't have this ISBN in stock" instead of bitching about a broken link...

2/25/2006 4:44 AM  
Anonymous Anonymous said...

Thanks for adding the 'search' link to the homepage :-)

2/25/2006 7:25 AM  
Anonymous Anonymous said...

why not link the title in my catalog to the book/work information? that would be more intuitive.

also, a title tag in the link in the last column would pop up in firefox (it doesn't display the alt tag, which really is a bug in IE)

great site, man!

2/25/2006 9:20 AM  
Anonymous Anonymous said...

Howdy, I was the one with the call number issue. As RJO mentioned, it seems to be resolved. Thanks very much!

2/25/2006 6:55 PM  
Anonymous Anonymous said...

Tim, thanks for responding so quickly.

With my first problem, it's not just the publisher changing--it's the ISBN, everything. Alas, I entered the book awhile ago. That's why it's so weird--the book switched editions or something. I'll send email with more info, in case that will help.

Thanks for explaining about the links not working when a source doesn't. I'd tried one of the used-book sites for one title; it must have been a scarce book.

2/25/2006 7:34 PM  
Anonymous Anonymous said...

This is such a great idea. Maybe we can also start groups with similar interests and reading habits. I am glad that I did not spend money and buy a clunky software program to catalogue my books. This system is so much more intuitive. I think I am going to have to go for the lifetime membership.

3/15/2006 2:09 PM  
Blogger Tim said...

Hey, what's the nature of the sorting error with the rating system?

Thanks,
Tim

3/19/2006 10:54 PM  
Blogger Tim said...

It does when it can--when it has a library record to go from, either yours or a from a book *linked* to yours via the works system.

10/10/2006 11:34 PM  

Post a Comment

<< Home