A small plug for Wordie

Fellow Portland Mainer John McGrath, of Squirl, hacked together Wordie, social cataloging and social networking for words. Basically, you "catalog" words and arrange them in lists (eg., my products named after their place of origin). If two users share a word, that connects them. It's a deeply silly idea, but I love that he did it.

PS: Wordie is now included among LibraryThing's "also on" list, available from your profile.

Small features and bug-fixes

I hope everyone enjoyed their Thanksgiving as much as we did. With all the tryptophan in our systems, we didn't make any major progress over the holiday. But we chipped away at some lesser features and bugs, in between courses:

1. Groups now sport RSS feeds for recent messages. We plan to add feeds for your groups, your posts, etc.
   
2. Tag pages now offer feeds for the most recent books tagged X in your library. They are also available when you look at a tag in your catalog too.
   
3. LibraryThing widgets are now available in the Latin-1 character set (UTF-8 remains the default). If you have a blog in Latin-1, and a lot of non-English books, widgets now work.
4. LibraryThing's universal import feature now accepts raw, encrypted CueCat data, so you can scan your books away from an internet connection.
   
5. The "all books" links on tag pages (eg., biography) is much faster now. (It's not always fast, but it won't take five minutes.)
   
6. The same goes for "recently tagged X" RSS feed; it's faster on high-frequency tags. I don't think many of you were watching the most-recent "fiction" tags, but Google and Technorati were, and all the "database churning" was slowing the site down.
7. Chris may have solved a major forum bug—the "Bermuda triangle" bug, where one message in 50 or 100 gets inexplicably lost. Cross your fingers and hope he's right.
   

LibraryThing integrates with Shaman Drum

LibraryThing now integrates with Shaman Drum, the legendary independent bookshop in Ann Arbor, MI. Edit your profile, check a checkbox (down at the bottom) and your work page will sport availability and pricing information from Shaman Drum and a link to their site.

Right now, it's just Shaman Drum. But the program is open to any bookstore. So long as you have a decent inventory system, it should be a snap. We've published participation details on our other blog. I'm going to approach a few, but feel free to let your local bookstore know about it.

If you've spent time in Ann Arbor, you know that Shaman Drum is the best bookstore in town, and one of the best independents in the country. It doesn't exactly lack for competition, with the flagship Borders store across the street and the Dawn Treader one street farther. I went to grad school in Ann Arbor, and Shaman Drum was practically a second home. I'm so glad the fine folks who run the place were receptive to my idea.

Germans: Swap English books

An eighth swap site has set themselves up to integrate with LibraryThing—this time with a twist: Bookswapper.de is for Germans—you need to have a German address—to swap books in English.

I think it's a great idea. For someone like me, living near a good library and two or three good bookstores, swapping doesn't necessarily make that much sense. When I weigh the mailing and bother costs against the costs of buying or borrowing a book, I usually choose the latter. But it would make a lot of sense for, say, French books, which can't be found near me.* By catering to residents of Germany (Germans and a lot of expats) who read English, Bookswapper makes easy the satisfying of wants that otherwise would take a lot of effort to satisfy.

About the site, the creators, Kata and Resi write:

"The makers of bookswapper.de are two real people, not some corporation. And bookswapper.de wasn't born in an office building but in our home, on a sofa, beneath overflowing bookshelves."

Amen to that.

Note: The original rules didn't anticipate a site like this, so I've decided to list Bookswapper.de second in LibraryThing.de and last in all other LibraryThings.

PS: I fixed a bug that was preventing LibraryThing from seeing everything our swap-site partners had. Numbers have jumped!

8,388,608 books—sort of

Last night LibraryThing hit 8,388,608 books. That's not books in LibraryThing—which stands at 7,268,540—but books ever in LibraryThing, including ones later deleted and some shadows. You might not think it, but 8,388,608 is a significant number. It's half of 2²⁴, the largest number you can store in three bytes. It's also the limit for MySQL's "signed medium integer." It's 111111111111111111111111. The drawers are full of ones and there ain't no twos.

Anyway, we hit the brick wall last night. I had previously expanded the book number field, but I forgot to change the databases that store some related metadata and reviews. So, last night, you couldn't add a book, and this morning you couldn't review one.

I'm really sorry about this. We're good to go now. We won't hit another wall until 8.4 billion books.

Interestingly, the same thing happend to Slashdot last week. Even Homer nods.

PS: I also fixed a bad problem with "search all fields." Some queries ran quickly but some took ten or twenty minutes, by which time the user has generally gone on to better things (after re-running the query a dozen times which, let me tell you, doesn't help much). It turns out MySQL was making periodic mis-guesses about which index to use. Somehow the index with eight million integers looked better than the one with a few hundred strings.

Bonjour SUDOC

LibraryThing now connects to SUDOC, the Système universitaire de documentation, a French union catalog of university libraries (see Wikipedia). SUDOC sports some seven million records—a huge boost to LibraryThing.fr and Thingamabrarians with French books generally.

Three cheers to Nicomo for helping me on this. He found it, fiddled with YAZ and sent me the exact connection info. SUDOC is actually the first time I've managed to parse Unimarc. Solving that (mostly) opens the door to many other libraries. Nicomo, you're a star!

Speaking of French, my family has a non-English-speaking Frenchman over to Thanksgiving. My French is extremely rusty, but I'm thinking I can jog my memory by listening to some simple conversations, news reports and especially vocabulary lists while I work. Any suggestions?

UnSuggester fallout

Our new anti-recommendation engine UnSuggester (blogged about below), has taken off in a weird way. Today was covered in everything from top-shelf library bloggers Tom Roper, Steven Abram and Karen Schneider (via ALA TechSource), to the bouncy, zippy, web-culture vlog MobuzzTV.

Here Mobuzzer Karina Stenquist explains the disconnect between Middlesex and The Passion of Jesus Christ: Fifty reasons he came to die. Karina did something on us before and—I'm sorry—she's great.



I love everything people are saying about it, with big discussions here and on various blogs, mosty people strive to find the oddest, funniest unsuggestions. My favorite blog discussion came off a short note by "Fontana Labs" (a philosophy professor?) on Unfogged, and is now past 120 comments. Much of it came from the proposal:

"There's got to be some ultimately evil book out that that will generate the ideal library. Not The Bridges of Madison County, but like that."

Speculation ensues, with the unsuggestions for Who moved my Cheese? much praised. I enjoyed this exchange:

"Sadly for Unfogged (but perhaps good for Western civilization) Who Moved My Cheese? For Kids, as perfectly soul-deadening a title as I've ever encountered, isn't owned by enough LibraryThing users to produce results. ..."

"Who Moved My Cheese? For Kids. Oh my God, my soul just broke into three thousand tiny wretched pieces."

Hey, I didn't say it. De gustibus...

BookSuggester and UnSuggester

People do not generally like BOTH Shopaholic and Critique of Pure Reason.

The "real" news today is the debut of BookSuggester, a new feature designed to expose LibraryThing's excellent and varied recommendations to members and non-members alike. We put them alongside Amazon's, which are also quite good. We are proud of our recommendations, but haven't perfected the perfect algorithm yet. When we've made things as good as we can, we're going to start offering recommended book data to libraries.**

But to heck with that! Let's talk about bad recommendations. Today we introduce UnSuggester, "the worst recommendation system ever devised™."

UnSuggester is a brand new idea in recommender technology. Recommender systems usually work by similarities. Amazon's "Customers who bought this item also bought" and LibraryThing's "People with this book also have" are typical of the type—What books do people buy together? What books occur often in the same member libraries?

UnSuggester flips this logic: What books DON'T occur in the same libraries? We took our "similars" algorithm and changed "sort ascending" to "sort descending" and—hey presto!—instead of similar books, we get opposite ones. You bet we're going to patent it!

How does it work?

UnSuggester starts by finding every copy of the book in question and all of its owners. So, taking Thucydides' History of the Peloponnesian War as an example, LibraryThing finds the 600-odd people who have entered this ancient classic in their account. Then it makes a big pile of all their other books, a pile of some 623,000 books in all. Then it does a little math. If LibraryThing has seven million books, then a pool of 623,000 book is about 8% of the total. If this pool were average, it would also contain 8% of the Harry Potters, 8% of the Derridas and 8% of the Danielle Steels. But this isn't so. People who own Thucydides aren't a random cross-section of the book-loving public. For example that 8% also contains almost half the Caesar and Plutarch in LibraryThing. At the other end of the scale, Thucydides-fanciers are particularly immune to the novels of Marian Keyes and Dean Koontz. The greatest disconnect occurs with Louise Rennison's popular, teeny-bopper chick-lit novel Angus, thongs and full-frontal snogging : confessions of Georgia Nicolson—the top UnSuggestion.

What patterns emerge?

The Mists of Avalon and Desiring God are very uncommon shelf-mates.

Play with it a few minutes, and patterns emerge. Philosophy and postmodern literary criticism oppose chick lit, popular thrillers and the young adult section. Programming does not truck with classic literature. Memoirs of depression, like Prozac Nation, meet their match in the cheery The Night Before Christmas. Ann Coulter and David Sedaris do not see eye-to-eye. There is a strong disconnect between readers of much recent Protestant, mostly evangelical, non-fiction, and large swaths of contemporary literary fiction. For example, LibraryThing includes 2,300 readers who've logged Jeffrey Eugenides' epic gender-bender novel Middlesex, and 222 readers of John Piper's The Passion of the Christ: 50 Reasons He Came to Die. But the groups don't overlap. No reader has both. Similar instances occur again and again.

These disconnects sadden me. Of course readers have tastes, and nearly everyone has books they'd never read. But, as serious readers, books make our world. A shared book is a sort of shared space between two people. As far as I'm concerned, the more of these the better.

So, in the spirit of unity and understanding, why not enter your favorite book, then read its opposite?

By the way, how about putting this up on Del.icio.us or Digg? Wait, we have a Digg now.*

*LibraryThing never been Dugg. But recently a pale immitation of LibraryThing was lofted to the heavens as the first social network for books. For one day Digg gave them twice our traffic. Fortunately, they fell like a stone after that and, this morning, Alexa has them at "too low to measure."
**My friend Ben correctly points out to me that he suggested "find your book nemesis" almost a year ago.

Seven million books

With spooky synchronicity we hit three major milestones today:

• Seven million books cataloged
• 100,000 registered members
• 1,000 groups

The seven-millionth book was added by Liftedviolets, a new member with just over 150 books cataloged. If the profile picture (right) is anything to go by, there's a lot more to come! The book was Fried Green Tomatoes at the Whistle Stop Cafe by Fannie Flagg. Liftedviolets wins a free paid account—congratulations.

The other two milestones are also big. The 100,000 members is, of course, a little inflated, including members who signed up and didn't do much of anything. It even has a few sign-up do-overs and test accounts. Still the number has been creeping up, so it's worth noticing a milestone. We've been running around 10% non-English sign-ups, to LibraryThing.de, LibraryThing.fr and the like.

1,000 groups is also impressive. Groups were only introduced in July. Use has increased of late. The 1,000th group is Ringers of Handbells, "A group for anyone who enjoys the art of English Handbell Ringing." At present group has just one member, and is private. And that's all I know!

An election-day tag cloud

For our US members:

abortion(423) bush(211) bush administration(22) campaigning(8) character(384) congress(140) conservatism(832) corruption(176) culture war(12) current affairs(2,138) democracy(956) democratic party(35) democrats(34) domestic issues(2) economy(483) election(70) elections(320) environment(2,968) foreign policy(536) gay marriage(10) george w. bush(100) governors(5) House of Representatives(2) immigration(1,045) iraq(880) katrina(55) liberalism(865) minimum wage(26) moral issues(31) politics(30,807) red states(2) republican party(19) Republicans(48) senate(28) terrorism(1,398) unemployment(43) voting(49) war on terrorism(14)

LibraryThing is strictly non-partisan; I'm not trying to make any points with my selections! I tried to think of issues and topics that were in play this year, or that commonly motivate Americans' votes. I left out education because there are so many books tagged education that it made all the other tags look like ants! And what this? Nine million tags and not one gubernatorial?.

I do have one semi-political comment to make. I went down to our new voting place, a freshly built school on Munjoy Hill in Portland. The school also houses the local branch of the Portland Public Library, and indeed the school's library and the branch library are intermixed.

Well THAT lasted a few weeks! Now the branch library is closed to adults during the day so that patrons and school children can't come into contact. How sad. How very sad.

TitleTrader joins the crew

Today we're adding our seventh swap site.* The site is TitleTrader, and it's a BIG addition. We can't give site-by-site holdings statistics, but TitleTrader adds a lot. Finding a book to swap through LibraryThing is getting easier and easier. Check out these numbers:

• Of the top 1,000 works on LibraryThing, 92% have a least one copy available for swapping.
• Of the top 10,000 works, 73% have a least one copy available for swapping.

Here's what the creator, Dan Abercrombie, has to say about making the TitleTrader and what sets it apart:

"The site was started in Sept of 2004. A few months earlier my sister came to me with the idea for a new website ... She had a lot of books and a small house and thought it would be a good idea to get rid of them. I put it together in a few weeks and have been working on it since. Since then there have been a lot of entrants into the space, each with their own take on the idea. ... From what I can tell, we're the largest site that has not only books, but CDs, DVDs, VHS, etc.** It provides a nice flexibility being able to trade a CD for a DVD for example. We're also one of the few sites that supports international trading, users can select which countries they will ship to. Our users also seem to like the ability to choose which item is sent to them and from whom, based on condition or rating."

So give a warm welcome to TitleTrader, and to its members.

To the remaining sites that haven't joined: What's keeping you? Choice is good. Letting your users integrate their favorite websites is good. Aren't you getting lonely in your walled gardens?

*Actually, we have eight in the system. I'm waiting on a technical tweak by number eight before I show the logo and blog about it. Does anyone know of any sites I've missed?
**You'll notice that TitleTrader comes in fouth in the list. This is according to the original invitation, that gives book-only sites priority over non-book sites, and then sorts within those categories by Alexa rating.

PS: Minor technical note. The "Give yours" link for TitleTrader currently only works if you're signed in. Dan tells me he can fix that.

CueCats, meet Abby cats

Back in stock and ready to go! They tried to deliver the latest shipment to me while I was away in Wisconsin, but this afternoon the UPS guy and I finally managed to be in the same place at the same time. It didn't come, of course, in time for me to make it to the post office before it closed-but I have three paper grocery bags filled up with the envelopes, ready for an early morning trip. Now I understand why Tim was saying his hand hurt from writing out so many addresses...

So if you ordered a CueCat in the past few days, I'm sorry about the delay, but it'll be en route to you as soon as I wake up in the morning. For everyone else, we still have some left, and even more coming by the end of the week - get your order in now and we promise a quick turn around. Only fifteen dollars, including shipping anywhere in the US.

(neither animals nor CueCats were harmed during this photoshoot.)

Review pages

Bloggers rejoice! I've added dedicated pages for every review, so you can point directly to your review, not to the work page generally.

Basically, the link appears wherever you see a review. For example, you'll see it on work pages and on your review page (here's mine). I'm using the "permanent link" icon. I hope it's not confusing.

Let your reviews go!

Before now, it was never really clear what LibraryThing could or couldn't do with your reviews. Unlike other such venues we've never had a "do anything" clause, although cynical members probably assumed we did. Now that we have some opportunities, we'd love to show some reviews outside of the site.

But they're your reviews, so you decide where they can go. We're giving you three options:

• Keep reviews on LibraryThing only
• Allow LibraryThing to give your reviews to non-commercial entities (libraries mostly)
• Allow LibraryThing to give reviews to commercial entities (booksellers, publishers, authors, street gangs)

Change your setting by going to edit your profile.

If you don't decide by December 1, we will default you to unrestricted use. In a day or two—once I do the code—all members who have posted a review will receive a message about this. New people will get a message when they post their first review. Whatever you choose, if your library itself is private, your reviews will be private too.

I'm in the process of writing protections into the TOS, eg., that reviews not be changed in any way. (I think I'm going to allow obscuring of swear words, like f**k.) I'd be interested to hear if anyone has any other concerns. LibraryThing won't be DOING anything for some time, and we want to do this carefully, and listen to what people think, as we usualy do.

Why are we doing this? LibraryThing users have built up quite an impressive corpus of book reviews (almost 100,000). Our focus on books as books—not on selling books—together with the lack of any "review rules" has given rein to a lot of excellent writing.

In the last few months, Abby and I have been going to library conferences and meeting a lot of librarians. We've learned that libraries are eager to show more information in their catalog—particularly reviews and recommendations. The few already doing so are paying through the nose for this data. Often this means snippets from "professional reviews" assembled by the publisher for their ONIX feed and repackaged and resold to libraries by data companies. (Library patrons are, I think, generally unaware when reviews come from the book's publishers!) Amazon of course provides user reviews, but only if a library is willing to make itself a sales channel in return. Commercial concerns, such as Abebooks, are also interested in showing reviews on their sites, and obviously can't use Amazon's.

So, we'd love to open up LibraryThing's reviews—to sell them, probably, although we will be underselling the data companies by a mile. By allowing us to do it, you'll be helping LibraryThing financially, and giving your opinions wider currency to boot.

In case it's said, LibraryThing is not turning evil. Letting your review go is your choice, not ours. If you ever reviewed a book on an online bookseller you agreed to unlimited distribution and modification. One of LibraryThing's competitors has a TOS that asserts COPYRIGHT over user reviews. (Prima noctis too, probably.) LibraryThing wants to give you control over your reviews. If you want to help us out, great. If not, and we're very grateful you want them on LibraryThing. We'd kiss you if we could.

Instead, post here or come talk on Talk.

MyPictr.com

A quick plug for MyPictr.com, which takes a pictr—I mean picture*—and scales and crops it for various social-networking sites. LibraryThing actually doesn't have set sizes. (Many sites require small squares, which appear all over the site in lieu of names, which, as we all know, involve those tricky letters things.**) Even so an easy-to-use scale-and-crop service is welcome. (Hat tip: Steve Cohen)

*Creative site; lemming name.
**The chosen picture, smiling college student with beer about to spill is telling.

Recommendations refreshed, improved

I just completed a rather extensive regeneration of the book recommendations—the "people who own X also own Y" recommendations and the "similarly tagged" ones. (The "Special Sauce" is next.) Recommendations, in turn, affect the "Pssst!" system (recommendations based on your whole library) and involved recreating all the work-to-tag clouds. Quality has, I think, risen significantly. We've improved the algorithms and the non-stop growth of the LibraryThing data set—now at 6.8 million books—continually improve the results anyway.

Scope has also improved. The system only makes recommendations for works with more than ten copies. That comes to juse 93,000 out of 1.3 million works. But these works account for over 60% of the books in LibraryThing. Before the current re-do, only 50,000 works were covered.

The recommendations are better, but hardly perfect! We've made progress toward better algorithms, and have big plans for future improvements. On the UI side, we're going to introduce users-feedback on recommendations, including both thumbs-up and thumbs-down buttons, and an "obscurity knob" for Pssst! (People love that on Last.fm.)

The system works best in the "middle," on books with 25-500 copies, non-fiction, genre fiction, books with a well-defined readership, and books that are "about" something—books like Touching the Void, Prozac Nation, The Historical Figure of Jesus and An encyclopedia of fairies. It has the most trouble with bestsellers, "obligatory books" (think high-school classics), and literary fiction—books like the Da Vinci Code and Great Expectations. To some extent, the problem is almost philosophical. What would be good recommendations for the Da Vinci Code? Statistically-speaking not much stands out. At the high-obscurity end, there are of course books on the book and its themes—Cracking Da Vinci's code or Baigen's Holy blood, holy grail.—but most Da Vinci Code readers aren't interested in that stuff. Ideally, we'd like a better mix of suggestions—some obscure stuff, and some of the quick, high-popularity reads it correlates with.

Burying the lead? We figure we're a month or so away from getting the algorithms where we want them. Once they are, we're going to start making them available to libraires, to spice up their online catalogs with top-notch readers' advisory—and for much less than they're currently paying for inferior services. We think it will make quite a splash!

NEWS: Tag-combining is back!