Large Language Models in Market Research: Opportunities, Limitations, and Guidelines

Large language models are transforming market research, but knowing when and how to use them responsibly is essential. This session distinguishes between using LLMs for discovery (generating new insights about human behavior) versus measurement (coding and annotating data at scale), explaining why measurement tasks are often better suited for AI tools while discovery faces a fundamental "validation paradox." Through a detailed, hands-on example coding the emotional valence of thousands of words, you will learn a step-by-step validation framework: splitting data into development and test sets, designing prompts that mirror human survey instruments, and benchmarking LLM outputs against human ratings using correlation and error metrics. The session also addresses two critical challenges researchers face when adopting these tools: ensuring replicability across probabilistic models and maintaining data privacy when working with sensitive information. Topics include best practices for using open-source models, setting appropriate model parameters, and building research workflows that keep pace with rapidly evolving AI capabilities.