Key research papers on large language models (LLMs)

How the literature is organised

The technical papers here read as one sequence: models were scaled up, found to learn tasks from a few examples, tuned to follow instructions, and released for others to build on.

Brown et al. (2020) trained GPT-3, an autoregressive model with 175 billion parameters, and showed it could perform new tasks from a few demonstrations in the prompt, without gradient updates — while also identifying datasets where few-shot learning still struggles. Kaplan et al. (2020) gave the empirical scaling laws: loss falls as a power law in model size, dataset size and compute, across more than seven orders of magnitude, and larger models are more sample-efficient. Ouyang et al. (2022) fine-tuned GPT-3 on labeler demonstrations and then on rankings of model outputs, and found that the 1.3-billion-parameter InstructGPT was preferred to the 175-billion-parameter GPT-3. Touvron et al. (2023) trained LLaMA using publicly available datasets only and released the models to the research community. Chen et al. (2021) introduced Codex and the HumanEval set for measuring functional correctness of generated programs.

Application papers make up the rest of the most-cited list, led by medicine and education. Singhal et al. (2023) built the MultiMedQA benchmark and found that instruction prompt tuning improved their model's answers but left it inferior to clinicians on human evaluation. Kung et al. (2023) reported ChatGPT performing at or near the passing threshold on all three USMLE steps, and Gilson et al. (2023) measured 44% to 64.4% accuracy across four question sets, with performance falling as questions got harder. Kasneci et al. (2023) set out opportunities and challenges for education.

Main debates

The debates are about reliability, evaluation and security. Models generate plausible but nonfactual content, and Huang et al. (2025) give a taxonomy of hallucination with contributing factors, detection benchmarks and mitigation methods, including the limits of retrieval augmentation as a remedy. Chang et al. (2024) argue that evaluation should be treated as a discipline in its own right, organised around what to evaluate, where and how. Yao et al. (2024) sort the security literature into beneficial applications, offensive ones and vulnerabilities, and note that work on model and parameter extraction attacks is limited and largely theoretical. Zhao et al. name theoretical foundations, efficient scaling, alignment and agentic capability as the open problems.

Where recent work is heading

Current directions visible in the most-cited surveys are autonomous agents built on language models (Wang et al., 2024, who propose a unified framework and survey applications across the social sciences, natural sciences and engineering) and software engineering (Hou et al., 2024, who analysed 395 articles published between January 2017 and January 2024). If you are working outside computer science, note that both of those surveys organise their field by task and identify gaps: an evaluation in your own domain, on your own language or data, is a realistic contribution.