Statistically Improbable Phrases
Sometimes I wish I’d stayed at Amazon. I love my life here, but I envy the interesting things my ex-colleagues in Seattle get to work on.
They recently introduced SIPs, “statistically improbable phrases.” They analyze all the text in the millions of books they’ve scanned, and then pull out the unique phrases which most characterize each book. It works simultaneously as a summary of the book, an insight into its feel, and an automated folksonomy.
This is just hours of fun going through some of my books to discover their SIPs. Here are a couple of examples from a few books I highly recommend:
Kitchen Confidential: chiffonaded parsley, broiler man, manger man, feed the bitch
The Eyes of the Dragon: royal napkins, instinct for mischief, watery ale, ash bucket, two guilders
The Tipping Point: social epidemics, transactive memory, mouth epidemics, teenage smoking
Good to Great: sustained great results, unsustained comparisons, red flag mechanisms, profit per customer visit, confronting the brutal facts
John Sumser, whose trade newsletter for online recruiting I read even though I’m not full time with (the award-winning) BostonWorks anymore, brainstorms ideas for using this technology to slice and dice resume information. All sorts of interesting possibilities.
