How to Use LLMs to Analyse User Feedback Without Flattening Minority Opinions

The Tyranny of the Average Sentiment

You have just closed a successful beta test, and your database is overflowing with hundreds of raw user survey responses, Discord bug logs, and support transcripts. You want to understand what makes your users tick. Naturally, you head over to Gemini or Claude, paste the raw spreadsheet, and type: “Summarize the top five pain points our users are experiencing.”

Within seconds, the LLM tells you that 80% of your users loved the speed of the app, but found the onboarding flow slightly confusing. It recommends you focus on cleaning up the sign-up page.

It feels like a massive win for productivity. But beneath that clean, executive-ready summary lies a quiet ethical trap.

By design, Large Language Models are probability engines built to identify common patterns and generalise. When you ask them to synthesise qualitative data into a neat list of top themes, they act like a filter that prioritises consensus. In doing so, they systematically flatten minority opinions, quiet frustrations, and crucial accessibility outliers.

If only one user out of five hundred mentioned that the new UI layout is completely unusable for screen readers, that vital feedback will be deleted from the summary because it is statistically insignificant to the model.

Why Consensus-Driven AI Silences Your Best Users

In product development, innovation rarely comes from the average user's average request. It comes from the edge cases, the power users pushing your system to its limits, and the vulnerable groups pointing out fundamental barriers to access.

When we hand over the job of synthesis to AI without a deliberate ethical strategy, we make our products intentionally bland and potentially exclusionary. We miss the quiet, highly specific bug report that points to a systemic security flaw because ninety-nine other users were busy commenting on the primary button colour.

As product builders, our ethical duty is not just to build for the majority; it is to ensure we do not become deaf to those who speak quietly.

How to Prompt for the Outliers

To prevent your LLM from acting as a homogenising filter, you must change how you ask it to read your data. Instead of asking for summaries, you need to prompt for variance and dissent.

When writing your prompts, avoid generic terms like "summarise." Use precise, analytical instructions. You can learn how to build complex prompt structures in our /prompts workspace.

Here is an ethical prompting framework to help expose the edge cases in your user data:

`text System Instructions: - You are an objective, highly critical UX researcher analyzing qualitative feedback. - Your goal is to identify both dominant trends AND distinct minority viewpoints. - Do not aggregate feedback into broad categories if doing so erases specific, unique technical critiques.

Output Structure Required: 1. Dominant Sentiments: (The feedback shared by the vast majority of users). 2. Critical Outliers: Identify individual, non-duplicate feedback points that represent severe usability, accessibility, or technical issues, even if they were only mentioned by a single user. 3. Dissenting Opinions: Highlight areas where users actively disagreed with the majority consensus. `

By explicitly creating a bucket for "Critical Outliers," you instruct the model to look past the high frequency noise and capture the lone voices that matter. If you are handling large files and hit token limits or encounter API timeouts while running these detailed prompts, check out the troubleshooting resources on the Google Gemini Support Hub to optimise your ingestion pipeline.

The Two-Pass Workflow: Categorise, Then Critique

For larger datasets, a single prompt won't cut it. To prevent the model from dropping critical data points due to context dilution, adopt a two-pass workflow:

Pass 1: The Tagging Phase Instead of asking the LLM to write a summary, ask it to act as a tagger. Input your feedback rows and instruct the AI to attach semantic tags to every single entry (e.g., `#accessibility`, `#latency`, `#billing`, `#ui-papercut`). Because this is a low-level categorization task, the model is far less likely to ignore outlier voices.

Pass 2: The Deep-Dive Analysis Filter your dataset by those tags and read the smaller clusters. If you see only three entries under `#accessibility`, do not let an AI summarise them. Read those three entries yourself. Your human brain is still the ultimate tool for empathy; do not outsource the actual reading of human frustration to a server farm.

To understand more about semantic tagging and how vectors parse similarity in qualitative text, explore our /glossary.

When to Put the LLM Away

AI is an exceptional sorting assistant, but it is a terrible moral compass. There are certain qualitative analysis tasks where you should skip the LLM entirely:

• Reviewing sensitive or emotional feedback: If users are discussing difficult personal experiences, mental health, or high-stakes financial stress, using an LLM to summarise their pain into a bullet point is cold and often misses the emotional reality.
• Evaluating complex accessibility studies: Automated tools and LLMs often lack the physical context required to understand how a user with motor or visual impairments navigates your layout.

Keeping Product Ethics Alive in the Data Dump

Efficiency is addictive. It feels incredible to clean up a messy user research project in an afternoon with the help of generative AI. But if that efficiency comes at the cost of ignoring the exact people who need your product's care and attention the most, it is a false economy.

By building an intentional, outlier-aware feedback pipeline, you get the best of both worlds: the speed of automated organisation, and the human decency of making sure everyone is heard. Let the AI do the sorting—but keep your own eyes on the edges.