Text-to-SQL and Query Generation

Translating natural-language questions into SQL is one of the more mature applied-LLM use cases, and one of the harder ones once the schema gets large. The resources below cover benchmarks, production approaches, and the research behind current methods.

Benchmarks

Spider 2.0: a text-to-SQL benchmark built from real enterprise-scale databases (over 1,000 columns each, on systems like BigQuery and Snowflake). Models need to search long database metadata and dialect documentation, reason over long contexts, and generate multi-hundred-line SQL. Current state-of-the-art models score far below human performance on it. See also the Spider project page.
Bird-SQL: a large-scale, cross-domain benchmark testing whether an LLM can serve as a real database interface.

Production approaches

defog-ai/sql-eval: an open evaluation harness for text-to-SQL systems
Uber's Query GPT: a production write-up of natural-language querying at scale
Combining Text-to-SQL with Semantic Search for RAG and its full LlamaIndex guide: combining structured SQL queries with vector search
MAC-SQL (code): a multi-agent collaboration approach to text-to-SQL

Surveys

A Survey on Employing Large Language Models for Text-to-SQL Tasks (Peking University, July 2024): a full overview of benchmark datasets, prompt engineering, and fine-tuning methods for this task
Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation

Papers on method

TAG: Unifying AI and Databases for Text2SQL: argues text-to-SQL alone isn't enough and proposes unifying it with broader database-AI integration
Decomposition for Enhancing Attention (Workflow Paradigm): improving LLM-based text-to-SQL through a decomposed workflow
ACT-SQL: In-Context Learning with Automatically-Generated Chain-of-Thought
Adapt and Decompose: Least-to-Most Prompting for Text-to-SQL
Few-Shot Text-to-SQL Translation Using Structure and Content
RUSH4SQL: Collective Retrieval Using Schema Hallucination
Prompt-Enhanced Two-Stage Text-to-SQL with Cross-Consistency
Semantic Enhanced Text-to-SQL via Iteratively Learning Schema Linking Graph
DIN-SQL: Decomposed In-Context Learning with Self-Correction
SQL-PaLM: Improved LLM Adaptation for Text-to-SQL
Exploring Chain-of-Thought Prompting for Text-to-SQL
Enhancing Text-to-SQL