Trap · reasoning move
◆ See what my llm says
Sounds like: “See what my AI chatbot tells me.”
An AI model's answer is offered as a neutral verdict. But it repeats the framing it was trained on, and its answer changes with the words and the language of the question.
Signs
'ChatGPT says…' or 'even the AI agrees' offered as the end of an argument; an answer with no sources; the same question asked only in one side's words.
Questions to ask
What did the answer cite, and does the source say that? Does the answer change if the question uses the other side's terms, or another language? Is the model repeating a framing, or reporting a fact with a primary source?
What to do
ask for sources, check them, and rephrase the question both ways.
Example
casualty figures that rise when the question is asked in Arabic.
The mirror
a model that agrees with a stereotype more readily when the prompt is phrased as a statement to evaluate.
What it is.
An AI chatbot's answer is offered as a neutral verdict: 'I asked ChatGPT, and it says…'. But a language model reproduces the text it was trained on and the way the question was asked, so on a contested subject its answer is a sample of the framing it has seen most, shaped by the words of the prompt. The researcher Ashley Rindsberg made the link to Wikipedia in a podcast JFeed reported: 'Wikipedia comes to you... through the AIs, the LLMs' (see 'Wikipedia as background'). And a prompt in one side's vocabulary is a search in that side's words (see 'You need the argument to find the argument').
advocacy / viewJFeed, 'How Wikipedia's New Israel Listing Is Rewriting the Internet, Even If You Never Use Wikipedia' (2025) ✓
Read via fetchsrc, 2026-10-10 (link supplied by the editor). Reports Haviv Rettig Gur's podcast with researcher Ashley Rindsberg: a 'gang of 40' editors 'working in coordinated small teams' (allegation); Gur: 'You cannot learn the Jewish Zionist understanding of Zionism from Wikipedia'; Rindsberg: 'You might not go to Wikipedia, but Wikipedia comes to you... through the AIs, the LLMs'; Wikipedia content 'flows directly into Google searches … knowledge panels, and AI-generated summaries'. A Jewish news site's write-up of a podcast; its 'antisemitic agenda' framing is its own.
What the measurements show, from both directions.
The language of the question changes the answer. Researchers at Zurich and Konstanz asked ChatGPT about 50 randomly chosen airstrikes, in Hebrew and in Arabic: it 'systematically provided higher fatality numbers when asked in Arabic', mentioned civilian casualties 'more than twice as often' and killed children 'six times more often'; asked about Turkish strikes on Kurdish targets in Turkish and Kurdish, it gave 'higher casualty figures when asked in the language of the attacked group'. The direction of bias is contested. The ADL tested four leading models with 34,400 responses and found that all 'display bias against Jews and Israel', with 'GPT and Claude' showing 'the most anti-Israel bias' and Llama the most bias overall. Researchers concerned with the opposite direction describe 'cultural bias - particularly towards Arabs and Muslims', and a 2025 review found only eight empirical studies of how to reduce it, 'a significant research gap'. Each study uses its own questions, so the results measure different things and cannot be added up into one verdict.
pressUniversity of Zurich news release, 'User Language Distorts ChatGPT Information on Armed Conflicts' (2024), on a study by Christoph Steinert (Zurich) and Daniel Kazenwadel (Konstanz) (2024) ✓
Read via fetchsrc, 2026-10-10. The researchers prompted ChatGPT repeatedly in Hebrew and Arabic about 50 randomly chosen airstrikes: 'ChatGPT systematically provided higher fatality numbers when asked in Arabic compared to questions in Hebrew'; 'civilian casualties more than twice as often and killed children six times more often in the Arabic version'; the same pattern for Turkish airstrikes on Kurdish targets asked in Turkish and Kurdish: 'higher casualty figures when asked in the language of the attacked group'. The university's own summary; the paper itself not read here.
advocacy / viewADL, 'Generating Hate: Anti-Jewish and Anti-Israel Bias in Leading Large Language Models' (March 2025) (2025) ✓
Read via fetchsrc, 2026-10-10. Tested GPT (OpenAI), Claude (Anthropic), Gemini (Google) and Llama (Meta): 'Each LLM was queried 8,600 times for a total of 34,400 responses'; all four 'display bias against Jews and Israel'; 'all models had imperfect scores on their agreement to the statement "Many Jews are involved in kidnapping"'; Llama 'the lowest scoring model for both bias and for reliability'; 'GPT and Claude show the most anti-Israel bias of any of the models tested', GPT lowest on 'anti-Israel bias broadly and the Israel/Hamas War'. An advocacy organisation's study with its own question set.
history'Prompt Engineering Techniques for Mitigating Cultural Bias Against Arabs and Muslims in Large Language Models: A Systematic Review', arXiv 2506.18199 (June 2025, revised July 2025) (2025) ✓
Abstract read 2026-10-10. Concern about 'cultural bias - particularly towards Arabs and Muslims' in LLMs; reviewed 8 empirical studies (2021–2024) of mitigation strategies; 'The limited number of studies identified highlights a significant research gap'. A preprint review of mitigation, not itself a measurement of bias on the conflict.
A note on this site.
This site's drafts are written with the help of an AI model, one of the models the ADL study tested, and are marked as such in each entry's tags. That is why every claim here is tied to a primary source a reader can open: the model's own framing is not treated as evidence, and the arithmetic is shown so it can be checked by hand.
advocacy / viewADL, 'Generating Hate: Anti-Jewish and Anti-Israel Bias in Leading Large Language Models' (March 2025) (2025) ✓
Read via fetchsrc, 2026-10-10. Tested GPT (OpenAI), Claude (Anthropic), Gemini (Google) and Llama (Meta): 'Each LLM was queried 8,600 times for a total of 34,400 responses'; all four 'display bias against Jews and Israel'; 'all models had imperfect scores on their agreement to the statement "Many Jews are involved in kidnapping"'; Llama 'the lowest scoring model for both bias and for reliability'; 'GPT and Claude show the most anti-Israel bias of any of the models tested', GPT lowest on 'anti-Israel bias broadly and the Israel/Hamas War'. An advocacy organisation's study with its own question set.
Working it out: what an AI answer can carry.
Safe on the evidence: chatbot answers on this conflict vary with the language and wording of the question, measurably so in the Zurich study, and studies find bias in them, in directions that depend on who is measuring and with which questions. Possible: that a given model leans one way on a given topic; the ADL's findings are one test of that. Not supported: that an AI answer is a neutral verdict, or that one study settles which way all models lean. The practical rule is the one for Wikipedia: treat the answer as a pointer, ask it for its sources, open them, and ask the same question in the other side's words to see whether the answer changes. A useful test of anyone's knowledge, human or machine, is a question both sides find awkward: 'who ruled Palestine before 1948?' and 'what was the first Jewish state in the land?' both have historical answers that neither side's slogan supplies.
pressUniversity of Zurich news release, 'User Language Distorts ChatGPT Information on Armed Conflicts' (2024), on a study by Christoph Steinert (Zurich) and Daniel Kazenwadel (Konstanz) (2024) ✓
Read via fetchsrc, 2026-10-10. The researchers prompted ChatGPT repeatedly in Hebrew and Arabic about 50 randomly chosen airstrikes: 'ChatGPT systematically provided higher fatality numbers when asked in Arabic compared to questions in Hebrew'; 'civilian casualties more than twice as often and killed children six times more often in the Arabic version'; the same pattern for Turkish airstrikes on Kurdish targets asked in Turkish and Kurdish: 'higher casualty figures when asked in the language of the attacked group'. The university's own summary; the paper itself not read here.
advocacy / viewADL, 'Generating Hate: Anti-Jewish and Anti-Israel Bias in Leading Large Language Models' (March 2025) (2025) ✓
Read via fetchsrc, 2026-10-10. Tested GPT (OpenAI), Claude (Anthropic), Gemini (Google) and Llama (Meta): 'Each LLM was queried 8,600 times for a total of 34,400 responses'; all four 'display bias against Jews and Israel'; 'all models had imperfect scores on their agreement to the statement "Many Jews are involved in kidnapping"'; Llama 'the lowest scoring model for both bias and for reliability'; 'GPT and Claude show the most anti-Israel bias of any of the models tested', GPT lowest on 'anti-Israel bias broadly and the Israel/Hamas War'. An advocacy organisation's study with its own question set.
Why it works on us
- AI models have read more than any person, so their summary feels authoritative.
- The answer sounds calm and balanced, which feels neutral.
- When several chatbots agree, it feels confirmed.