Strict Standards: Non-static method installk2::installer() should not be called statically in /home/writerresponse/writerresponsetheory.org/wordpress/wp-content/themes/3k2w-b2r344/functions.php on line 23

Warning: Cannot modify header information - headers already sent by (output started at /home/writerresponse/writerresponsetheory.org/wordpress/wp-content/themes/3k2w-b2r344/functions.php:23) in /home/writerresponse/writerresponsetheory.org/wordpress/wp-commentsrss2.php on line 15
Comments on: Amazon Mechanical Turk http://writerresponsetheory.org/wordpress/2005/11/16/amazon-mechanical-turk/ a blog and podcast dedicated to discussing text arts forms Thu, 14 Dec 2017 13:14:01 +0000 http://wordpress.org/?v=2.0.2 en Writer Response Theory 2004-2005 wrt@writerresponsetheory.org (Writer Response Theory) wrt@writerresponsetheory.org Talk Radio Comment-cast: Amazon Mechanical Turk Comment-cast: Amazon Mechanical Turk Writer Response Theory Writer Response Theory wrt@writerresponsetheory.org http://writerresponsetheory.org/wordpress/wp-content/uploads/iTWRT.JPG WRT: Writer Response Theory http://writerresponsetheory.org/wordpress 144 144 by: Jeremy Douglass http://writerresponsetheory.org/wordpress/2005/11/16/amazon-mechanical-turk/#comment-14086 Tue, 12 Sep 2006 04:23:38 +0000 http://writerresponsetheory.org/wordpress/2005/11/16/amazon-mechanical-turk/#comment-14086 <a href="http://www.thesheepmarket.com/" rel="nofollow">The Sheep Market</a> by <a href="http://users.design.ucla.edu/~akoblin/bio.html" rel="nofollow">Aaron Koblin</a> is an interesting art project that sheds some light on both Labeler and the Turk: <blockquote> The Sheep Market is a collection of 10,000 sheep created by workers on Amazon's Mechanical Turk. Each worker was paid $.02 (US) to "draw a sheep facing left." Average time spent on each sheep: 105 seconds Average wage: $0.69 / hour Rejected sheep: 662 Collection period: 40 days Collection rate: about 11 sheep / hour Unique IP addresses: 7599 </blockquote> The <a href="http://users.design.ucla.edu/~akoblin/work/thesheepmarket/index.html" rel="nofollow">project design notes</a> are quite clever, with references to sheep-like behavior, 'Dolly', and other cultural significations of sheep. But most captivating of all is the section on the book Le Petit Prince: <blockquote> <em>"'If you please--draw me a sheep...' When a mystery is too overpowering, one dare not disobey. Absurd as it might seem to me, a thousand miles from any human habitation and in danger of death, I took out of my pocket a sheet of paper and my fountain-pen. But then I remembered how my studies had been concentrated on geography, history, arithmetic and grammar, and I told the little chap (a little crossly, too) that I did not know how to draw. He answered me: 'That doesn't matter. Draw me a sheep . . .'" - Le Petit Prince </em> Le Petit Prince, by Antoine de Saint-Exupery originally published in 1943, is one piece of fiction which subtly incorporates images of sheep as a pivotal point in the narrative. A tale about life and the truths that are often more apparent to children than adults, Le Petit Prince discusses perception, creativity, and purpose. In the story, the narrator comes upon a prince in the middle of the desert and is asked to draw a sheep. The narrator makes some attempts, each of which is dismissed by the Prince. Ultimately, the narrator draws a box with holes and explains that the sheep is inside the box, finally the prince is satisfied. The novel subtly references the sheep in a commentary about the scientific approach to explanation. As the narrator struggles to use relatively formal techniques to render the visible image of a sheep, the prince seems to suggest that there could be different goals and objectives guiding his process. Ultimately, the image itself contains no literal sheep at all, but simply a view from 'outside the box'. </blockquote> The Sheep Market raises issues not just for the Mechanical Turk, but also for Google Image Labeler, and indeed any informatic game that implicitly demands "label this a sheep!" Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/writerresponse/writerresponsetheory.org/wordpress/wp-includes/functions-formatting.php on line 76

The Sheep Market by Aaron Koblin is an interesting art project that sheds some light on both Labeler and the Turk:

The Sheep Market is a collection of 10,000 sheep created by workers on Amazon’s Mechanical Turk. Each worker was paid $.02 (US) to “draw a sheep facing left.”

Average time spent on each sheep: 105 seconds
Average wage: $0.69 / hour
Rejected sheep: 662
Collection period: 40 days
Collection rate: about 11 sheep / hour
Unique IP addresses: 7599

The project design notes are quite clever, with references to sheep-like behavior, ‘Dolly’, and other cultural significations of sheep. But most captivating of all is the section on the book Le Petit Prince:

“‘If you please–draw me a sheep…’

When a mystery is too overpowering, one dare not disobey. Absurd as it might seem to me, a thousand miles from any human habitation and in danger of death, I took out of my pocket a sheet of paper and my fountain-pen. But then I remembered how my studies had been concentrated on geography, history, arithmetic and grammar, and I told the little chap (a little crossly, too) that I did not know how to draw. He answered me:

‘That doesn’t matter. Draw me a sheep . . .’”
- Le Petit Prince

Le Petit Prince, by Antoine de Saint-Exupery originally published in 1943, is one piece of fiction which subtly incorporates images of sheep as a pivotal point in the narrative. A tale about life and the truths that are often more apparent to children than adults, Le Petit Prince discusses perception, creativity, and purpose. In the story, the narrator comes upon a prince in the middle of the desert and is asked to draw a sheep. The narrator makes some attempts, each of which is dismissed by the Prince.

Ultimately, the narrator draws a box with holes and explains that the sheep is inside the box, finally the prince is satisfied.

The novel subtly references the sheep in a commentary about the scientific approach to explanation. As the narrator struggles to use relatively formal techniques to render the visible image of a sheep, the prince seems to suggest that there could be different goals and objectives guiding his process. Ultimately, the image itself contains no literal sheep at all, but simply a view from ‘outside the box’.

The Sheep Market raises issues not just for the Mechanical Turk, but also for Google Image Labeler, and indeed any informatic game that implicitly demands “label this a sheep!”

]]>
Jeremy Douglass The Sheep Market by Aaron Koblin is an interesting art project that sheds some light on both Labeler and the ... The Sheep Market by Aaron Koblin is an interesting art project that sheds some light on both Labeler and the ...
by: Jeremy Douglass http://writerresponsetheory.org/wordpress/2005/11/16/amazon-mechanical-turk/#comment-13854 Fri, 08 Sep 2006 03:03:08 +0000 http://writerresponsetheory.org/wordpress/2005/11/16/amazon-mechanical-turk/#comment-13854 <a href="http://images.google.com/imagelabeler/" rel="nofollow">Google Image Labeler</a> is another recent phenomenon in leveraging collective intelligence - this time presented as a game (like pictionary) with partners. Pairs compete with other teams to label as many photos during a time limit - if both players suggest the same tag, a photo is considered labeled. It is a great concept to make important data a byproduct of play. Still, the gameplay mechanic means I'm much more likely to label a picture "bird" than "cardinal." An even bigger concern is that for people intent on winning, their goal is orthogonal to the goal of the system. When I looked at the ratings board, one of the top rated players was using the account AlwaysSayCool. It reminds me of the Southampton team that won an <a href="http://en.wikipedia.org/wiki/Prisoner%27s_dilemma#The_iterated_prisoner.27s_dilemma" rel="nofollow">interated prisoner's dilemma</a> contest by programming their bots to recognize each other and collude. There are many ways that "are we thinking the same thing?" is a different question from "are we seeing the same picture?" - it would be interesting to find out what kinds of data artifacts get pulled out of the Image Labeler result set. [via <a href="http://www.unmediated.org/archives/2006/09/google_image_la.php" rel="nofollow">Unmediated</a>] For an interesting discussion on the research of <a href="http://www.cs.cmu.edu/~biglou/research.html">Luis von Ahn</a>, developer of the <a href="http://www.espgame.org/">ESP Game</a> on which Google Image Labeler is based, see an interesting <a href="http://radar.oreilly.com/archives/2006/09/more_on_google_image_labeler.html">discussion at O'Reilly Radar</a> of his <a href="http://video.google.com/videoplay?docid=-8246463980976635143">July 2006 tech talk at Google</a>[video]. Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/writerresponse/writerresponsetheory.org/wordpress/wp-includes/functions-formatting.php on line 76

Google Image Labeler is another recent phenomenon in leveraging collective intelligence - this time presented as a game (like pictionary) with partners. Pairs compete with other teams to label as many photos during a time limit - if both players suggest the same tag, a photo is considered labeled.

It is a great concept to make important data a byproduct of play. Still, the gameplay mechanic means I’m much more likely to label a picture “bird” than “cardinal.” An even bigger concern is that for people intent on winning, their goal is orthogonal to the goal of the system.

When I looked at the ratings board, one of the top rated players was using the account AlwaysSayCool. It reminds me of the Southampton team that won an interated prisoner’s dilemma contest by programming their bots to recognize each other and collude. There are many ways that “are we thinking the same thing?” is a different question from “are we seeing the same picture?” - it would be interesting to find out what kinds of data artifacts get pulled out of the Image Labeler result set.

[via Unmediated]

For an interesting discussion on the research of Luis von Ahn, developer of the ESP Game on which Google Image Labeler is based, see an interesting discussion at O’Reilly Radar of his July 2006 tech talk at Google[video].

]]>
Jeremy Douglass Google Image Labeler is another recent phenomenon in leveraging collective intelligence - this time presented as a game (like pictionary) ... Google Image Labeler is another recent phenomenon in leveraging collective intelligence - this time presented as a game (like pictionary) ...
by: Jeremy Douglass http://writerresponsetheory.org/wordpress/2005/11/16/amazon-mechanical-turk/#comment-696 Wed, 23 Nov 2005 11:00:35 +0000 http://writerresponsetheory.org/wordpress/2005/11/16/amazon-mechanical-turk/#comment-696 Two answers Mark: First, I'm sure that there will be some value to the "human subjects"-type data that is a byproduct of AMT operating, although I'm not sure yet how valuable that will be to Amazon. What would be fascinating is what (if any) disclosure rules govern the selling any subject data NOT submitted as work-for-hire... selling it back to the job owner, that is. Example: I'm a market analyst, interested in how quickly people recognize the brand of a car and whether that is influenced by car color. I submit scads of jobs for "identify which picture has a Honda" etc. and pay - but I don't care about the work data (car IDs) because that is a red herring - I already knew that. What I care about is reaction time - how fast each job was completed correctly - and Amazon sells me the reaction times, which I correlate against car color. The question: should AMT workers know that they are selling their reaction times in that case - even if they normally give that data away for free? Second answer: By supposedly, I was actually just referring cynically to the term "Web 2.0," which is so amorphous that it means just about anything. Social software / collectivizing is one of a number of Web 2.0 buzzwords... AJAX is another, etc.... Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/writerresponse/writerresponsetheory.org/wordpress/wp-includes/functions-formatting.php on line 76

Two answers Mark:

First, I’m sure that there will be some value to the “human subjects”-type data that is a byproduct of AMT operating, although I’m not sure yet how valuable that will be to Amazon. What would be fascinating is what (if any) disclosure rules govern the selling any subject data NOT submitted as work-for-hire… selling it back to the job owner, that is.

Example: I’m a market analyst, interested in how quickly people recognize the brand of a car and whether that is influenced by car color. I submit scads of jobs for “identify which picture has a Honda” etc. and pay - but I don’t care about the work data (car IDs) because that is a red herring - I already knew that. What I care about is reaction time - how fast each job was completed correctly - and Amazon sells me the reaction times, which I correlate against car color. The question: should AMT workers know that they are selling their reaction times in that case - even if they normally give that data away for free?

Second answer: By supposedly, I was actually just referring cynically to the term “Web 2.0,” which is so amorphous that it means just about anything. Social software / collectivizing is one of a number of Web 2.0 buzzwords… AJAX is another, etc….

]]>
Jeremy Douglass Two answers Mark: First, I'm sure that there will be some value to the "human subjects"-type data that is a byproduct ... Two answers Mark: First, I'm sure that there will be some value to the "human subjects"-type data that is a byproduct ...
by: Mark Marino http://writerresponsetheory.org/wordpress/2005/11/16/amazon-mechanical-turk/#comment-695 Wed, 23 Nov 2005 07:05:35 +0000 http://writerresponsetheory.org/wordpress/2005/11/16/amazon-mechanical-turk/#comment-695 re: <blockquote>The result could be described as is a human intelligence brokerage, a marketplace for micro-tasks, or a harnessing of the collective intelligence that supposedly characterizes most developments of the 'Web 2.0' era </blockquote> Do you think that the AMT (not Alan M. Turing, but the Turkish delight) Website collects data about those who participate and use the system? In other words, are users providing service for a payment that acknowledges part of their work (their HITs) but hides the work they do for Amazon product development (it is beta still, right?), sales, and marketting. Does the site, or its developers, use this data for optimization? Amazon already seems to be built on this kind of obvious work (the reviews you write) and hidden work (your contributions to the data on book-buying habits -- reader's of this book also like... in which recommendations are the sign of the data you have contributed?) But perhaps I should just ask what you mean to say with "supposedly"? Deprecated: preg_replace(): The /e modifier is deprecated, use preg_replace_callback instead in /home/writerresponse/writerresponsetheory.org/wordpress/wp-includes/functions-formatting.php on line 76

re:

The result could be described as is a human intelligence brokerage, a marketplace for micro-tasks, or a harnessing of the collective intelligence that supposedly characterizes most developments of the ‘Web 2.0′ era 

Do you think that the AMT (not Alan M. Turing, but the Turkish delight) Website collects data about those who participate and use the system? In other words, are users providing service for a payment that acknowledges part of their work (their HITs) but hides the work they do for Amazon product development (it is beta still, right?), sales, and marketting. Does the site, or its developers, use this data for optimization? Amazon already seems to be built on this kind of obvious work (the reviews you write) and hidden work (your contributions to the data on book-buying habits — reader’s of this book also like… in which recommendations are the sign of the data you have contributed?)

But perhaps I should just ask what you mean to say with “supposedly”?

]]>
Mark Marino re: The result could be described as is a human intelligence brokerage, a marketplace for micro-tasks, or a harnessing of the ... re: The result could be described as is a human intelligence brokerage, a marketplace for micro-tasks, or a harnessing of the ...