The Sheep Market by Aaron Koblin is an interesting art project that sheds some light on both Labeler and the Turk:
The Sheep Market is a collection of 10,000 sheep created by workers on Amazon’s Mechanical Turk. Each worker was paid $.02 (US) to “draw a sheep facing left.”
Average time spent on each sheep: 105 seconds
Average wage: $0.69 / hour
Rejected sheep: 662
Collection period: 40 days
Collection rate: about 11 sheep / hour
Unique IP addresses: 7599
The project design notes are quite clever, with references to sheep-like behavior, ‘Dolly’, and other cultural significations of sheep. But most captivating of all is the section on the book Le Petit Prince:
“‘If you please–draw me a sheep…’
When a mystery is too overpowering, one dare not disobey. Absurd as it might seem to me, a thousand miles from any human habitation and in danger of death, I took out of my pocket a sheet of paper and my fountain-pen. But then I remembered how my studies had been concentrated on geography, history, arithmetic and grammar, and I told the little chap (a little crossly, too) that I did not know how to draw. He answered me:
‘That doesn’t matter. Draw me a sheep . . .’”
- Le Petit Prince
Le Petit Prince, by Antoine de Saint-Exupery originally published in 1943, is one piece of fiction which subtly incorporates images of sheep as a pivotal point in the narrative. A tale about life and the truths that are often more apparent to children than adults, Le Petit Prince discusses perception, creativity, and purpose. In the story, the narrator comes upon a prince in the middle of the desert and is asked to draw a sheep. The narrator makes some attempts, each of which is dismissed by the Prince.
Ultimately, the narrator draws a box with holes and explains that the sheep is inside the box, finally the prince is satisfied.
The novel subtly references the sheep in a commentary about the scientific approach to explanation. As the narrator struggles to use relatively formal techniques to render the visible image of a sheep, the prince seems to suggest that there could be different goals and objectives guiding his process. Ultimately, the image itself contains no literal sheep at all, but simply a view from ‘outside the box’.
The Sheep Market raises issues not just for the Mechanical Turk, but also for Google Image Labeler, and indeed any informatic game that implicitly demands “label this a sheep!”]]>
Google Image Labeler is another recent phenomenon in leveraging collective intelligence - this time presented as a game (like pictionary) with partners. Pairs compete with other teams to label as many photos during a time limit - if both players suggest the same tag, a photo is considered labeled.
It is a great concept to make important data a byproduct of play. Still, the gameplay mechanic means I’m much more likely to label a picture “bird” than “cardinal.” An even bigger concern is that for people intent on winning, their goal is orthogonal to the goal of the system.
When I looked at the ratings board, one of the top rated players was using the account AlwaysSayCool. It reminds me of the Southampton team that won an interated prisoner’s dilemma contest by programming their bots to recognize each other and collude. There are many ways that “are we thinking the same thing?” is a different question from “are we seeing the same picture?” - it would be interesting to find out what kinds of data artifacts get pulled out of the Image Labeler result set.
For an interesting discussion on the research of Luis von Ahn, developer of the ESP Game on which Google Image Labeler is based, see an interesting discussion at O’Reilly Radar of his July 2006 tech talk at Google[video].]]>
Two answers Mark:
First, I’m sure that there will be some value to the “human subjects”-type data that is a byproduct of AMT operating, although I’m not sure yet how valuable that will be to Amazon. What would be fascinating is what (if any) disclosure rules govern the selling any subject data NOT submitted as work-for-hire… selling it back to the job owner, that is.
Example: I’m a market analyst, interested in how quickly people recognize the brand of a car and whether that is influenced by car color. I submit scads of jobs for “identify which picture has a Honda” etc. and pay - but I don’t care about the work data (car IDs) because that is a red herring - I already knew that. What I care about is reaction time - how fast each job was completed correctly - and Amazon sells me the reaction times, which I correlate against car color. The question: should AMT workers know that they are selling their reaction times in that case - even if they normally give that data away for free?
Second answer: By supposedly, I was actually just referring cynically to the term “Web 2.0,” which is so amorphous that it means just about anything. Social software / collectivizing is one of a number of Web 2.0 buzzwords… AJAX is another, etc….]]>
The result could be described as is a human intelligence brokerage, a marketplace for micro-tasks, or a harnessing of the collective intelligence that supposedly characterizes most developments of the ‘Web 2.0′ era
Do you think that the AMT (not Alan M. Turing, but the Turkish delight) Website collects data about those who participate and use the system? In other words, are users providing service for a payment that acknowledges part of their work (their HITs) but hides the work they do for Amazon product development (it is beta still, right?), sales, and marketting. Does the site, or its developers, use this data for optimization? Amazon already seems to be built on this kind of obvious work (the reviews you write) and hidden work (your contributions to the data on book-buying habits — reader’s of this book also like… in which recommendations are the sign of the data you have contributed?)
But perhaps I should just ask what you mean to say with “supposedly”?]]>