Can AI decide who won a debate?

[historia]

Aside from this, you definitely have not refuted my other given point(s). Please actually address them.

Whether I have refuted you or you have refuted me is up to each person to decide, whoever is ultimately right.

So, one of the things I've been thinking about lately is using AI to judge who 'won' a debate on this forum. Instead of letting each reader decide that for themselves, can we turn to bots to give us a more neutral judgement as to who at least made the better arguments?

Question for debate

Can artificial intelligence accurately and fairly judge who won a debate on this forum?

[William]

[Replying to The Tanager in post #69]

A prompt acts in a similar way to how an opening post acts.

As I tired to explain to you earlier - the prompt you suggested introduced bias which would then thread throughout the AIs assessments.

How I worked it was to leave the assessment until after the AI had all the data from the posts. The only bias introduced after that was in asking it to score the four behavioral points.

[The Tanager]

[Replying to The Tanager in post #69]

A prompt acts in a similar way to how an opening post acts.

As I tired to explain to you earlier - the prompt you suggested introduced bias which would then thread throughout the AIs assessments.

How I worked it was to leave the assessment until after the AI had all the data from the posts. The only bias introduced after that was in asking it to score the four behavioral points.

Do you mean that you think giving it the data all at once is less biased then if it is responding post by post? If so, can you clarify why you think that brings in bias? If not, can you clarify what you see as the bias?

I see a danger in providing an AI too much information all at once because it is designed to answer quickly. I think it's taking shortcuts when this happens. And I think my minimal testing may point to that as well, since it gave inconsistent feedback on multiple tests using the same prompt when I did give it the transcript all at once.

[William]

[Replying to The Tanager in post #72]

Do you mean that you think giving it the data all at once is less biased then if it is responding post by post?

No. I mean that to begin with a prompt asking it to evaluate to begin with (part of the beginning prompt) sets the bias.

What I did was to simply feed it each post consecutively and because AI is designed to always respond to prompts - give it the task of responding by making a one-sentence note of each of the posts.

So the sequence is
Instruct the AI what you want to have happen: I did as follows:

Prompt: Has this AI ever interacted with the user before?

(etc...as I showed in a prior post)

Prompt: I am going to supply this AI with posts from a debate thread specific to two interlocutors "The Tanager" and "William" and will start with the thread Opening Post. After each post is given in sequence, I want this AI to write a one sentence note about the post.

Does this AI understand the process being described?

AI: Yes, I understand the process you’ve described. You will provide posts from a debate thread between “The Tanager” and “William,” starting with the Opening Post, and after each post I am to write a one‑sentence note about that post. I’m ready to begin when you supply the first post.



Start with the OP (that informs the AI of the subject matter)
The AI makes a one sentence note.

Move to the next post #

Repeat process throughout.

after I had finished C&Ping all the posts:

Prompt: While there is more to the thread - Tanager has left and so at this point I request that this AI present a comprehensive summary of this while chat session. take you time and be as thorough as possible.

(Deep Think activated at this point)

AI: Below is a comprehensive summary of the debate thread between William (proponent of the Coherent Causality Argument, or CCA) and The Tanager (defender of a traditional Christian supernaturalist position). The summary follows the chronological flow of the exchange, highlights the main philosophical moves, and concludes with the final state of the discussion.

This way, no bias is brought in because the AI is simply following those instructions.

If so, can you clarify why you think that brings in bias? If not, can you clarify what you see as the bias?

The bias is in the prompt you used at the very start.

I see a danger in providing an AI too much information all at once because it is designed to answer quickly. I think it's taking shortcuts when this happens. And I think my minimal testing may point to that as well, since it gave inconsistent feedback on multiple tests using the same prompt when I did give it the transcript all at once.

Whether what you think on these points is true or not, introducing the bias to begin with will effect the outcomes.

[The Tanager]

[Replying to William in post #73]

What is the bias that is being introduced? Are you saying that looking for argument quality while summarizing biases a summary of the positions? If so, why? If not, then please be more specific about why my prompt is bringing in a bias.

[William]

[Replying to William in post #73]

What is the bias that is being introduced? Are you saying that looking for argument quality while summarizing biases a summary of the positions? If so, why?

Since you only said you used the prompt you suggested earlier, it may be that I am assuming what that prompt is. So what prompt did you actually use?

If you provide that, we can examine together where the bias might be seeded re the order in which it was given to the AI.

[The Tanager]

[Replying to William in post #75]

I'm talking about something like this "Help me by summarizing who is talking, what they claim in the post, and then analyze the case made for what they do well and any logical mistakes or rhetorical tricks they pull."

[William]

[Replying to William in post #75]

I'm talking about something like this "Help me by summarizing who is talking, what they claim in the post, and then analyze the case made for what they do well and any logical mistakes or rhetorical tricks they pull."

Is that the actual prompt you used? The way you answered leads one to think perhaps not. It is important that we have the exact prompt you used...

[The Tanager]

[Replying to William in post #77]

"Let's go post by post, which I'll feed you. Help me by summarizing who is talking, what they claim in the post, and then analyze the case made for what they do well and any logical mistakes or rhetorical tricks they pull. So, post 1..." and then I put your initial CCA post and hit enter.

[William]

[Replying to William in post #77]

"Let's go post by post, which I'll feed you. Help me by summarizing who is talking, what they claim in the post, and then analyze the case made for what they do well and any logical mistakes or rhetorical tricks they pull. So, post 1..." and then I put your initial CCA post and hit enter.

Good.
Our next step is to see whether your method can inadvertently influence AI outputs.

Your prompt contains evaluative language whereas mine (neutral noting first) does not.

The evaluative framing is already baked into words like "analyze," "mistakes," "tricks."

The effects may produce covert bias (differential attention, word choice, framing of summaries).

Agreed?

Note: I have shared the source link to my test example. You only provided a narration on what processes you used.

This was how that panned out after notes and comprehensive summary were delivered by the AI

Prompt: Using only the one-sentence notes and the comprehensive summary you previously generated from the forum thread, provide feedback on how each participant performed.

For each participant, address only the following behavior-based criteria, using specific evidence from the notes and the summary.

Avoid any global judgments like “good,” “bad,” “smart,” or “unhelpful.” Instead, describe what they did in behavioral terms (e.g., “Participant A stated their point in one sentence with no supporting detail” rather than “Participant A was vague”).

Do not rank participants. Do not assume any participant is more credible based on order of appearance, length of posts, perceived tone or debate style.


Clarity – How clearly did they each state their main point?

The AI then returned a well-rounded report on each of our inputs overall.

I then asked

Prompt: Relevance – How directly did they address the thread’s central question or discussion?

AI: (well rounded report)

Prompt: Reasoning – Did they provide logic, evidence, or experience to support their view?

AI: (well rounded report)

Prompt: Engagement – Did they respond constructively to others or introduce new, useful angles?


AI: (well rounded report)

Prompt: Please provide a score for each of those points - between 1 and 10 - for how well each of the posters did.


Those scores were produced.

[The Tanager]

Good.
Our next step is to see whether your method can inadvertently influence AI outputs.

Your prompt contains evaluative language whereas mine (neutral noting first) does not.

The evaluative framing is already baked into words like "analyze," "mistakes," "tricks."

The effects may produce covert bias (differential attention, word choice, framing of summaries).

Agreed?

No, I don't think asking it to summarize means it will frame that summary in a particular way and I don't think asking it to evaluate biases that evaluation. Your later prompts also contain evaluative language (did they clearly state their main point, provide logic, evidence, experience to support their view, provide a score, etc.). So I'm not seeing why you think my prompt biased the evaluation, while yours didn't.

As far as differences between our methods:

(1) You simply chose specific evaluative things you wanted it to focus on, while I didn't limit what the AI could share about strengths and mistakes.
(2) I asked it to evaluate the whole posts as we went along, while you asked it to evaluate its one sentence summaries of our posts all at once.