🦜 PARROT

Practical And Realistic BenchmaRk for crOss-system SQL Translation

The first comprehensive benchmark for evaluating cross-system SQL translation systems

Leaderboard • Documentation • Submit Results • Paper

---

• 09/2025: Our paper "PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation" has been accepted by NeurIPS 2025! 🎉 🎉 🎉
• 05/2025: We have released PARROT-1.0 (28,003 translation pairs from 38 open-source benchmarks for extensive syntax testing) and published the leaderboard.

• ✅ 598 Translation Pairs from 38+ public benchmarks and production-derived workloads
• 🧠 Broad Dialect Coverage: PostgreSQL, MySQL, SQLite, Oracle, SQL Server, Db2, DuckDB, Trino, Hive, Snowflake, and more
• 🧪 Built-in Validators: Comprehensive parsers and executability checks for multiple engines
• 🛠️ Complete Toolkit: Preprocessing utilities and baseline translation tools included
• 📊 Rigorous Evaluation: Multi-dimensional scoring (syntax and execution)
• 🏆 Live Leaderboard: Track your progress and compete with the community

1. 📋 Prepare Outputs

   • Follow the example in Submission_Example/20250928_LLMTranslator_ExampleTeam.zip
   • Ensure proper folder structure and file formats

2. 📖 Read Guidelines

   • Review Submission_Example/PARROT Submission Guidelines.md
   • Check format requirements and naming conventions

3. 📝 Include System Description

   • Approach and methodology
   • Models and versions used
   • Rules and heuristics applied
   • Training data sources
   • Compute resources

4. 🚀 Submit

   • Upload via the leaderboard site
   • Wait for evaluation results

• Consistent model versions and random seeds
• Clear indication of supported dialect pairs
• Valid UTF-8 text file outputs
• Exact versions of LLM prompts/rule files included
• System description document included
• Reproducibility instructions provided

⚠️ Important: Include exact versions of all dependencies, prompts, and rule files for reproducibility.

We recommend to refer to an LLM-based baseline CrackSQL.

CrackSQL is a powerful SQL dialect translation tool that integrates rule-based strategies with LLMs for high accuracy. It enables seamless conversion between dialects (e.g., PostgreSQL → MySQL) with flexible access through Python API, command line, and web interface.

Goal: Translate SQL from one database dialect to another while preserving semantic equivalence.

Input:  (source_dialect, target_dialect, source_sql)
Output: target_sql

-- Source (PostgreSQL)
SELECT EXTRACT(YEAR FROM created_at) AS year, COUNT(*) 
FROM users 
WHERE age > 25 
GROUP BY EXTRACT(YEAR FROM created_at);
-- Target (MySQL)
SELECT YEAR(created_at) AS year, COUNT(*) 
FROM users 
WHERE age > 25 
GROUP BY YEAR(created_at);

PARROT/
├── 📁 benchmark/          # Source datasets from 38+ benchmarks
│   ├── Spider/           # Cross-domain SQL queries
│   ├── SParC/            # Multi-turn conversations
│   ├── BIRD/             # Complex real-world queries
│   ├── TPC-H FROID/      # UDF-heavy workloads
│   └── ...               # 34+ more benchmarks
├── 🔍 validator/         # Grammar parsers & validators
│   ├── pg_parser/        # PostgreSQL parser
│   ├── mysql_parser/     # MySQL parser
│   ├── oracle_parser/    # Oracle parser
│   └── ...               # 10+ more dialect parsers
├── ⚙️ processor/         # Preprocessing utilities
├── 🔄 translator/        # Baseline translation tools
└── 📤 Submission_Example/ # Submission templates

PARROT evaluates systems across four key dimensions:

If you use PARROT in your research, please cite:

@inproceedings{zhou2025parrot,
  author       = {Wei Zhou and Guoliang Li and Haoyu Wang and Yuxing Han and Xufei Wu and Fan Wu and Xuanhe Zhou},
  title        = {PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation},
  booktitle    = {Advances in Neural Information Processing Systems (NeurIPS)},
  year         = {2025}
}
@article{zhou2025cracksql,
  author       = {Wei Zhou and Yuyang Gao and Xuanhe Zhou and Guoliang Li},
  title        = {Cracking SQL Barriers: An LLM-based Dialect Translation System},
  journal      = {Proceedings of the ACM on Management of Data},
  volume       = {3},
  number       = {3 (SIGMOD)},
  year         = {2025}
}
@article{zhou2025cracksqldemo,
  author       = {Wei Zhou and Yuyang Gao and Xuanhe Zhou and Guoliang Li},
  title        = {CrackSQL: A Hybrid SQL Dialect Translation System Powered by Large Language Models},
  journal      = {arXiv Preprint},
  url          = {https://arxiv.org/abs/2504.00882},
  year         = {2025}
}

This project is released under the MIT License. See LICENSE file for details.