Tuesday, May 20, 2008

Import upgrade

Mike—welcome Mike!*—has completed a major upgrade to the import system. The improvements are:
  • Better user interface.
  • Import now reads files from LibraryThing competitors, so you can move to us or synch your accounts.
  • Depending on the site, we pick up tags, reviews, ratings and comments. If you already have the books in your library, you avoiding adding the books again, but synch your user data.
  • The sites include Anobii, Shelfari and Goodreads. If you use someone else—there are more than 35 of them!—let us know. If the offer export—not all sites do—we can work it out.
  • If your file is formatted properly—formatted like the LibraryThing export or any of our competitors'—we now import non-ISBN books.
Import is still based on the idea that—when possible—LibraryThing re-fetches the bibliographic data. This adds another step, an "import queue." But it also allows members to import full records, which no other site exports, and to get high-quality library data, if they want it.

Tell us what you think on Talk. It's probably going to take a while to spell out what it does and doesn't do and to update the old Adding and Importing FAQs.

*Mike (member: notmydadslibrary) is a new intern up here in Portland. This was a doozy of a first project!

Labels: ,

15 Comments:

Blogger Michał Paluchowski said...

Darn... I just did a large import last weekend :D Nevertheless, damn good upgrade. I'm dying to see the new "works" system deployed right now.

Since I can't check now, does the system support importing from Collectorz.com database files?

5/21/2008 3:27 AM  
Anonymous Anonymous said...

"We're rolling out a much-improved "works" system" on the Down notice - this sounds somewhat different from playing with the import queue? Is it another new feature or a re-used down notice or ?

Can you put a timestamp on the Update "another two hours should do it" 2 hours from when?

5/21/2008 5:53 AM  
Anonymous Anonymous said...

reading_fox:

As far as I remember from the warning note last night/yesterday, the new "works" system should avoid hash collisions, and allow combining AND separating by Title/Author/ISBN, rather than just Author/Title. Combining will be down for a while though, until things settle down.

5/21/2008 6:43 AM  
Blogger amandaellis said...

Definitely a great first project!

5/21/2008 9:53 AM  
Blogger Unknown said...

How about importing Fictionwise bookshelf exported CSV files? No ISBNs in them though...

5/21/2008 12:45 PM  
Blogger Johan A said...

I can't get it to work. I try to import a tab-delimited file with ISBN-less books but it can't find any books in it. To understand what you mean by "properly—formatted like the LibraryThing export" I export my library - 936 books, of which 403 have ISBNs - and try reimporting it, and it finds only 403 books, the ones with ISBNs. What's up?

5/21/2008 3:38 PM  
Blogger Tim said...

Johan, can you email us your file (tim@librarything.com or mike...)?

5/21/2008 3:51 PM  
Blogger Unknown said...

Is there an API that will let me add reviews for books that are already listed in my account? I have a long recommended reading list as an HTML page, and have already imported the titles; would hate to have to import the one-paragraph review of each book by hand. (gvwilson@cs.toronto.edu if you have a solution, and thanks in advance...)

5/21/2008 4:59 PM  
Blogger Johan A said...

Tim; doing so. (Emailing the export file, that is. The import file is at home.)

5/22/2008 3:58 AM  
Blogger Mike said...

This comment has been removed by the author.

5/24/2008 1:42 PM  
Blogger Mike said...

Fixes added today:

1) Large files (over 2K books) were not importing correctly. Should read anything within the 2MB file upload limit now. This accounts for unexpected results reported here and via email.

2) Ugly database error experienced by some users this morning are Fixed.

3) ISBNs that have had their leading 0s stripped by another program were not importing. Fixed.

I haven't been good about posting updates here. Here's a cumulative list of items posted on Talk (copied/pasted might not make perfect sense if you haven't been following along at http://www.librarything.com/talktopic.php?topic=36938 ):

1) Tags and authors were not being imported (Kaethe) if you used field names from the XLS files because the we were only supporting the field names in the CSV files. I've added support wherever possible for the XLS field names so that it will work but we recommend using the CSV export in favor of the XLS format when exporting your books to guarantee all your info is imported. I've added a test for this scenario which produces an alert on the import configuration screen so you'll be made aware of this. So you should be able to reimport and uncheck the fields that have already imported (or not) to get your tags and authors synced. Unfortunately this won't help with ISBN-less books because the import algorithm has no concept of 'duplicates' for these books. We may need to think about a workaround for this so you can easily import the same file over and over again (possibly your making changes with your some offline application or text editor, etc). I propose that if the title matches it's a duplicate and you get another set of checkboxes to configure how your books personal info in synced. Tim, what do you think?

2) Issues reported on the blog by Johan where his XLS file was not imported cleanly are fixed.

3) Translation issue. I missed a couple fields but translation junkies will be happy to know that they can now translate everything new on the import screens.

4) When you import reviews the tally on your profile page is not updated. Fixed.

5) Added mass tagging field. Now you can designate a set of tags that will apply to all books being imported.

6) The reason your list didn't import is because we don't support the XLS format per se. You should be able to 'Save Copy As' in Excel as a CSV file. Once you do this you can continue to edit this new file in Excel but you will have a file that will also import cleanly into LibraryThing as long as your fields are the same as ours. The fields we currently support are:

isbn
rating
review
tags
entry date
author (last, first)
publication info

NOTE: a 'dateread' field will be supported soon.
Also, these fields aren't case sensitive.

7) The problem was that I wasn't handling the date field correctly. That's one that should have been caught. The easiest way to fix your batch is to delete any non-ISBN that were in the file and re-import them. Dates should import cleanly now.

5/29/2008 6:05 PM  
Blogger Mike said...

Update...

Fixed issue when trying to upload files with leading whitespace. I'm not sure why this happens but now it's not a problem. Lines with nothing but whitespace are ignored.

Thanks to vintage_books for pointing this out!

-Mike

5/29/2008 11:55 PM  
Blogger icichat said...

I have an excel file with about 10 fields. The fist two are author and title. Most of these books are very old and do not have ISBN numbers. In the off chance that the book is new enough, I unfortunately did not add them. I'm new at this, obviously, but I do have close to 900 books. Is there a way I can import this? Do I need to copy my file with only some of the fields? Can you give me some steps to do this or point me to some directions? Thanks for any help you can give.

6/12/2008 11:08 AM  
Blogger Mike said...

icichat, If you rename your fields to correspond to the fields in the LT export files you should be able to import all your books without issue. There's now a sample file available on the import page as well (see following post). Feel free to email me at mike at librarthing.com or post here if you have more questions. -Mike

6/22/2008 2:00 PM  
Blogger Mike said...

Update...

Thanks to vintage_books for your help and feedback that led to these changes.

* Made sample CSV file available for download (on tools and import pages). This can be used as a starting point if you want to create a spreadsheet that will be importable to LibraryThing.

* Altered algorithm so it's less picky about what constitutes a properly formatted file for import. In other words less fields have to match for non-ISBN books in the file to be importable. We'll see how this works.

* There was a problem that led to users losing their tags (OOPS, thanks to Tim for magically restoring them with his ninja-like SQL skills). Tags are now always appended (until we add finer grained import control).

More to come...

Mike

6/22/2008 2:01 PM  

Post a Comment

<< Home