Using LLMs to Synthesize Survey Data

Survey research is never easy. In today’s rapidly changing world, it is getting increasingly difficult and costly to obtain representative samples when collecting survey data. Fragmented and ever-multiplying methods of communication mean that people no longer reliably respond to calls, texts, emails, or other modes of communication traditionally used to collect public opinion. Complicating this are AI bots that flood online forums and coordinated campaigns by adversaries to sway opinions. As a result, surveys often fail to reach their targeted populations. 

In addition, the ongoing tribalization of different groups that coalesce around their specific gender, generation, political identification, or other identity compound the difficulty public opinion researchers have in reaching a representative sample. Today’s companies often collect massive amounts of data, but many have no way to consistently obtain a comprehensive view of their customers. 

A third approach, large language models (LLMs), may be more effective. Using LLM intelligence can transform messy, disjointed, and organic data into a standardized format across artifacts for analysis.

LLMs as the Third Way

Using LLMs as a means of surveying has five steps: ingest, extract, decompose, classify, and output. Using this method, the source data (irrespective of format) is ingested into an electronic format first. An intelligent extractor (text-based or multi-modal) then decomposes the data into pre-defined classifications to ensure usefulness and consistency based on its use case. Those standardized outputs are then combined and analyzed.

The strength of this approach is to leverage the pattern-recognition capability of the LLMs to understand contextual nuances. The latest models are exceptionally helpful, as users can focus on content engineering (i.e., telling the LLMs their intended results) rather than prompt engineering (i.e., telling the LLMs what to do). This is crucially important to use cases such as researching “fake news” or misinformation, because it is difficult to pinpoint the specifics beforehand.

For example, suppose someone has spread the fake news that a city is going to ban all red cars. The first step of the LLM approach is to have the LLM search from pre-selected creditable sources to check if such a policy exists and summarize its findings. Having a pre-selected source (“white list”) is an important set-up to minimize hallucinating or sourcing from unscrupulous sites built by adversaries. 

The second step is to perform automated content extraction and decomposition. This transforms the unstructured data into structured records more suitable for LLM-based classification and interpretation. Further analysis may reveal that many online comments come from recently opened accounts with similar wording or arguments, and that many public reactions are replies from a few emotionally charged postings.

In an unpublished proof-of-concept analysis, 417 cases from the Taiwanese government’s Public Policy Online Discussion data were analyzed to assess the influence of online public opinion on policymaking.1 Using a baseline open-weight LLM (Gemma 3 1B parameters model) with zero-shot prompt and no manual tuning, the LLM automatically categorized all messages into 17 distinct policy domains (e.g., civic participation, culture, etc.) and identified their sentiments on a 5-point Likert scale, from very positive to very negative. 

Using this method, the classification result was validated via spot-check. It is important to note that such data typically contains a mix of genuine citizen feedback, issue-specific mobilization, and potentially organized messaging. Signals may include sustained discussion across diverse user accounts, while noises may arise from short-lived campaigns or coordinated repetition. Manually replicating this classification would require roughly one week of work by an analyst (assuming five minutes per case).

Who Watches the Watchmen?

The LLM plays a key role in this process because it must correctly extract, decompose, and classify. The LLM decision engine can be a generic LLM (e.g., ChatGPT), a specialized LLM (e.g., MedGemma, a clinically trained LLM that understands clinical terminologies), or a specifically tuned LLM (e.g., to identify fake news).

There are two phases of validation. The first is to validate the LLM’s outputs against expert human judgment on a rolling sample. This will ensure that the LLM is “fit for purpose.” The second step is to use an ensemble of other higher-power LLMs from different families as judges to provide ongoing quality assurance (AI peer review). It is important to continuously monitor the performance degradation (model drift) of the LLM decision engine. Of course, judges can also share blind spots, so regular human adjudication is still necessary.

Noise Cancelling

Since the LLM decision engine is fully configurable, it can be customized based on a specific use case. For example, if the user comments are likely to be from a coordinated promotion or a misinformation propaganda campaign, you can train (or direct) your LLM to identify that specific piece of information and then handle them differently. This is similar to an email spam filter. It will greatly improve the quality of the data and analysis (that is, provide a higher signal-to-noise ratio).

Next Era for Social Analytics

Using an LLM represents an emerging way to complement public opinion or market research. Rather than only asking people what they think, you can complement their answers with observations about what they do and say. This combined data across sources will paint a more complete picture of an increasingly elusive public opinion. Given the noisy nature of online data, 
a better trained and tuned LLM decision engine could provide better intelligence to remove noise 
or amplify signals in a sea of data.

Acknowledgement

This article is based on a presentation to the Research Center for Humanities and Social Science of the Academia Sinica on November 27, 2025. I would like to express my appreciation for the help and guidance provided by Dr. MJ Cho in shaping and refining this article.

Reference

Taiwan Public Policy Online Discussion (2017 Executive Yuan) https://data.gov.tw/en/datasets/58297