The Pistos Test: Bot Preaching, not Passing
Published by Jeremy Douglass May 26th, 2005 in Uncategorized, Software.
Pistos, author of chatterbot Mathetes (ne GeoBot) has proposed the eponymous Pistos Test - a challenge to chatterbot developers worldwide to develop bots capable of prostheletizing.
- The Pistos Test
- If a robot converts or aids in the conversion of a human being from one religion, faith or belief system to another, it has passed the Pistos Test.
The Pistos Test is an interesting idea, but it may not be a Test at all. To be fair, Pistos adds the caveat “I’ve deliberately left it fairly open ended, as my intention is not to be legalistic” - but the idea of passing a “Test” (in the style of the Turing Test) is pretty compelling, so I’m going to try to think through the idea in detail.
Like the Turing Test, the Pistos Test specifies a result (testimonial of the human subject to a conversion experience). Unlike the Turing Test, however, it doesn’t describe any experimental conditions leading up to that result. Would “testing” involve a brief conversation, or would it be longitudinal? Would the conversation be initiated by the bot? Would the human tester be prompted as to the topic in some way? Would the tester know it was a bot, or would the bot be simultaneously be playing Turing’s “imitation game” and trying to pass as human?
The Turing Test is anchored in machine intelligence discourse by a sketch of experimental conditions and a brief argument about why the results are significant. The Pistos Test lacks these things. Still, it is thought-provoking, and I would love to see it developed further. The idea of a machine evangelist does seem to share with Turing a certain spirit of respect, a premise that machines might have their own value and perspectives to bring to a conversational table. Here is a bot not just to be interrogated, but to be attended to! In my limited interactions with the Pistos chatterbot Mathetes and particularly during my reading of conversation transcripts, I’ve been impressed with the appeals for respect - to be treated like a human and engaged in civil discourse. This attempts to head off in some ways the bad behavior and limits-testing which are the common strategies of humans who know they are interacting with bots.
As a goal rather than a test, a bot with an agenda feels good. Let us put bots behind podiums, not stick them in under a bare bulb in an interrogation cell! On the other hand, making a bot a vehicle of higher truth (especially in the form of a canonical holy text) may reduce that bot to an immutable object, like a book or a stone tablet. People can change in conversation, but can (or should) a holy text change? Bibles are distributed by the Gideons in confidence that the text itself can act as evangelist - why not distribute bot-wisdom in the same way? Evangelical bots might be an intelligent topical search engine to a concordance of quotations, but could they be real interlocutors?
I’m tempted to say no, they would not, not unless they had the potential to be converted as well as to convert. This may seem an unfair standard, given that I’ve had conversations with human missionaries who were completely uninterested in being converted themselves - that is generally part of the proposition of a conversion conversation. Yet this information-broadcast style is what leads many people confronted with door-to-door missionaries to feel that they aren’t really participating in a ‘conversation’ at all.
This is not to sell evangelism short - the attempt to teach about one’s own beliefs is a profoundly compelling component of conversation, and should be explored. Perhaps, before testing for conversion, a Pre-Pistos Test should measure ideological transmission. Don’t ask the participant if they converted. Ask them if they feel they understood the bot’s world view!
Still, the value of measuring this understanding in a “Test” may be limited. Is understanding indicative of the bot communicating complex ideas clearly? It might however indicate that humans are incredibly good at gleaning understanding from text - even the non-interactive kind. Perhaps instead we should attempt bots which are modeled, not an evangelists, but a potential converts, with the interlocutor attempting to convert the bot which would respond with questions, objections, etc….
4 Responses to “The Pistos Test: Bot Preaching, not Passing”
- 1 Pingback on Apr 21st, 2006 at 12:44 am
This is a fascinating post that seems in keeping with Christy’s Bot God post from a little while back. It also seems to ride the wave of faith-based scholarship that is cresting through our country.
I’m not sure why this needs to be a “Test” so much as a goal or programming objective. This is not to say the goal is unimportant, but as you point out, Jeremy, making this a Test seems to defeat the possibilities of interplay.
It does seem to relate to a tradition of catechesis in Christianity. Would a dogma bot be a modern-day Baltimore Catechism?
I also very much like your idea that the bot should be convertible. It should be open to learning. As with all electronic narrative objects, our creations tend to follow our world view, especially in the case of chatbots.
It’s interesting that the Pistos Test has completely different views of identity than the Turing Test. In the latter, we do not know who is really on the other side. In the Pistos Test, we seem to take faith at “interface value” as Sherry Turkle calls it. In a medium so imbued with doubt, here is a test that wants to believe you are what you say you are.
I’m particularly interested in how a person will have a conversation about faith with a chatbot which has (presumably) never experienced faith directly. In my experience many conversations about faith are testimonials regarding an unquantifiable, untestable experience of the world. Yet, with bots, the exact quantifiability of their experiences is painfully obvious. That is why I’m presuming that evangelicalbots or dogmabots will testify about the written truth of a trusted texts (i.e. the Bible, the Koran), which then have “experienced,” rather than talking about how “Buddha changed my life” etc.
That is, unless they are playing the imitation game as well. I’ll leave a note in the author’s forum about the article, and see if he would like to comment here.
Do you remember which is the Turkle piece where she coins the prhase “interface value”?
A well-written, thought-provoking article. Thank you for your interest in Mathetes.
Indeed, there is no strict process as per Turing’s Test, wherein I might specify what participants are involved, what they are to know or not know about each other, and how one might definitively measure a pass or a failure.
As I shared in the Robitron mailing list, I proposed the Pistos Test more with the intention of catalysing the creation and development of bots which have significant portions of their KB and engine focused on religious and philosophical topics, especially with an eye towards dissemination of religious beliefs, to the point of conversion, or at least challenging bot visitors to justify their beliefs.
“Let us put bots behind podiums, not stick them in under a bare bulb in an interrogation cell!” Indeed, I think it would be more interesting to interact with a bot for whom people would be interested in what it had to say, rather than how it would react. (Gosh, that English is awkward, heh…)
Technologically speaking, Mathetes is not convertible (except for the times he veers outside of operation parameters and spouts off something heretical), but technically, since I consider myself convertible, he is too, indirectly. But a bot which was really convertible (in terms of KB and/or engine) certainly would be a most fun conversation partner! I can imagine people streaming in droves to try to change its beliefs!
“Still, the value of measuring this understanding in a ???Test??? may be limited. Is understanding indicative of the bot communicating complex ideas clearly? It might however indicate that humans are incredibly good at gleaning understanding from text - even the non-interactive kind.” Yes, it would be more of a test, measure, or examination of the human, rather than the bot or botmaster; and that would be okay by me.
I have tried to make Mathetes speak truth as much as possible, veering only when he resorts to external sources for data to help him respond, and one other exception: I have permitted him to exhibit Jabberwacky-esque behaviour insofar as he occasionally thinks he is human (and stubbornly tries to assert this). Bearing this in mind, I admit that, obviously, Mathetes has no flesh to walk the earth, nor does he autonomously crawl the web to learn new things (well… at least not in the sense that we are discussing in this thread), and so he has no spiritual experiences the same way humans would. But I have molded his KB to have him respond as if he were a human in this respect, so as to make argument and debate with him more… workable, you might say.