Feasibility of Locally Deployable LLMs for Open Coding in Grounded Theory: A Pilot Comparison with Human Consensus

Qualitative analysis using grounded theory is essential for interpreting complex textual data in the social sciences and humanities but remains labour-intensive, particularly during open coding. Recent large language models (LLMs) offer potential to accelerate this process, yet reproducible pipelines and fair comparisons with human coding are scarce. This pilot study evaluates a fully automated, locally deployable LLM pipeline (Gemma-3-27B) against a multi-expert human consensus gold standard derived from a corpus of Duolingo technical blog posts (12,665 words, 2013-2025) as an illustrative case. Human coding followed a retrieval-style approach, selectively labelling salient units, while the LLM performed sentence-level classification. Evaluation framed coding as evidence retrieval, using strict exact-match and semantic fuzzy-match metrics (sentence embeddings, cosine threshold 0.6). Results showed modest strict performance (macro Jaccard 0.071, F1 0.128) but substantial improvement with fuzzy matching (macro Jaccard 0.368, F1 0.539). The LLM aligned well with human experts on explicit gamification features (e.g., Milestone Celebration fuzzy F1 0.609) but underperformed on nuanced pedagogical constructs (e.g., Personalized Practice Paths 0.462). These findings highlight LLM strengths in broad evidence gathering for recurrent themes, while underscoring limitations in interpretive depth. The study contributes a reproducible Python-based framework with documented prompts and deterministic decoding, supporting hybrid workflows in qualitative research where artificial intelligence (AI) accelerates initial retrieval and humans refine higher-order interpretation in grounded theory.