The Vault: A Comprehensive Multilingual Dataset for Advancing Code Understanding and Generation

Table of content

• The Vault Dataset
  • Data Summary
  • Data Structure
    • Data Instance
    • Data Fields
    • Data Near Deduplication
    • Splitting Train/Eval/Test
    • Splitting Train set
  • Download Dataset
    • Load dataset on Huggingface
    • Download via link
• The Vault toolkit
  • Getting Started
  • Processing Pipeline
    • Extracting raw code
    • Filtering extracted code snippet
    • Processing Custom Dataset
• Citing The Vault
• Contact Us
• License

---

The Vault Dataset

Data Summary

The Vault dataset is a comprehensive, large-scale, multilingual parallel dataset that features high-quality code-text pairs derived from The Stack, the largest permissively-licensed source code dataset.

We provide The Vault which contains code snippets from 10 popular programming languages such as Java, JavaScript, Python, Ruby, Rust, Golang, C#, C++, C, and PHP. This dataset provides multiple code-snippet levels, metadata, and 11 docstring styles for enhanced usability and versatility.

Data Structure

Data Instances