Community OSCAR: A Community Effort for Multilingual Web Data

Data is a key ingredient and differentiator for deep learning models such as large language models. Despite this fact being well known in the industry, the data topic is still pretty much under-explored by the academic and open research commmunity. Only recently, the effect of data on large-scale models getting investigated with more attention. Notable examples are Scaling Data-Constrained Language Models (Muennighoff et al., 2023), OLMo 1.7–7B: A 24 point improvement on MMLU, or HuggingFace’s FineWeb. However, most data work is focussed on the English language. To change this and make more multilingual data available to the community, we are releasing Community-OSCAR as a large-scale multilingual dataset.

Community-OSCAR is an unofficial version of the OSCAR Corpus created by community members. The annotation schema follows the OSCAR 23.01 release but is based on 40 monthly dumps of Common Crawl ranging from 2024-22 to 2014-42. With these forty dumps, Community-OSCAR is the largest release of the OSCAR Corpus so far and available for download on Huggingface.

About OSCAR

The OSCAR project (Open Super-large Crawled Aggregated coRpus) is an Open Source project aiming to provide web-based multilingual resources and datasets for Machine Learning (ML) and Artificial Intelligence (AI) applications. The project focuses specifically in providing large quantities of unannotated raw data that is commonly used in the pre-training of large deep learning models. The OSCAR project has developed high-performance data pipelines specifically conceived to classify and filter large amounts of web data. The project has also put special attention in improving the data quality of web-based corpora as well as providing data for low-resource languages, so that these new ML/AI technologies are accessible to as many communities as possible.

Community Effort

Community-OSCAR is a release created by members of the OSCAR community and part of an ongoing effort in close colaboration with the Occiglot research collective. We are working on extending this release to all publicly available CommonCrawl dumps and have plenty of ideas on further improvements. If you want to support our activities and collaborate with us, please join the Discord server from the OSCAR project or Occiglot research collective.

Next steps

Building the Community-OSCAR dataset is a critical step in advancing core projects at Occiglot. We realized that larger amounts of raw text data were needed to achieve our goal of training dedicated LLMs for underrepresented European languages. In the release of our Occiglot-fineweb data curation pipeline, we discussed that many languages needed more (clean) data for meaningful continual pre-training. Community-OSCAR closes that gap for multiple languages.

Consequently, we are actively working on cleaning all of this new data and deduplicating it against other data sources from our multilingual corpus. We expect to release an updated version of Occiglot-fineweb soon.

Initial experiments on continual-pretraining of LLama-3.1 using this new Occiglot-fineweb data delivered promising results. Stay tuned for future model releases.

Data Usage

OSCAR is mainly intended to pre-train language models and word representations.

NOTE: Community-OSCAR contains the raw unfiltered Common Crawl text data but with quality annotations. For language model training, we highly recommend filtering the data first with these annotations. A prefiltered version of the dataset will be released in the near future (following the approach from Occiglot-FineWeb).

Data Annotations

Each sample comes with a series of annotations that allow the removal of low quality data.

• identification: Language identification based on fastText.
• harmful_pp: This perplexity comes from a KenLM model trained on harmful content, previously gathered by using the adult annotation in OSCAR 22.01. In other terms, the lower it is, the more likely a given document contains harmful/adult content.
• tlsh: We use TLSH to compute a hash for each document. Locality sensitive hashing is a hashing method that computes similar hashes for similar documents.
• quality_warnings: Computed through heuristics (see below).
• categories: Content categories arom a URL-based blocklist

The annotation schema is the same as in the OSCAR 23.01 release.

Quality Warnings

• tiny: The document has a low (<5) number of lines.
• short_sentences: The document has a high number (>50%) of short lines (<400 bytes)
• header: The document has a high number of short lines at its head, suggesting the presence of low quality content.
• footer: The document has a high number of short lines at its tail, suggesting the presence of low quality content.
• noisy: The document has a high percentage of punctuation (>50%)
• adult: The document contains adult content. This annotation uses a blocklist and labels a tiny part of the corpus: It does not catch most of the adult content.

More information about the thresholds and annotators are present in the OSCAR paper.

Data Format

The data is stored as ZSTD-compressed JSON line files. Each individual data sample has the following JSON schema:

{
   "content":"English sentence\nphrase en français\n????????????", // (1)
   "warc_headers":{ // (2)
      "warc-identified-content-language":"fra,eng",
      "warc-target-uri":"https://fr.wikipedia.org/wiki/...",
      "warc-record-id":"<urn:uuid:29eaa920-d299-4b1d-b687-c72bd8d68116>",
      "warc-type":"conversion",
      "content-length":"35298", // (3)
      "warc-refers-to":"<urn:uuid:39e42055-0d94-4e45-9c6c-9e7056635d64>",
      "warc-block-digest":"sha1:WFH2A5WHCS2H365GIAFYQPI7UOAMFGHB", // (3)
      "warc-date":"2022-11-26T09:45:47Z",
      "content-type":"text/plain"
   },
   "metadata":{
      "identification":{ // (4)
         "label":"fr",
         "prob":0.8938327
      },
      "harmful_pp":4063.1814, // (5)
      "tlsh":"tlsh:T125315FF2B6088901EEA097015DB39B4600B...", // (6)
      "quality_warnings":[ // (7)
         "short_sentences",
         "header",
         "footer"
      ],
      "categories":[ // (8)
         "examen_pix",
         "liste_bu"
      ],
      "sentence_identifications":[ // (9)
         {
            "label":"fr",
            "prob":0.99837273
         },
         {
            "label":"en",
            "prob":0.9992377
         },
         null
      ]
   }
}

Language statistics

All the data is distributed by language and release name. Up to 151 different languages are available. The table below provides the language code as well as the number of documents and data sizes as number of bytes. The statistics are computed based on uncompressed data and on estimates calculated on a subset of 10 releases and extrapolated to all 40 releases (snapshots).

Community-OSCAR was put together by community members in close collaboration with the Occiglot research collective. The main contributors are Manuel Brack, Pedro Ortiz Suarez, Malte Ostendorff, Patrick Schramowski, Georg Rehm, Kristian Kersting, Jose Javier Saiz, Iñaki Lacunza Castilla, Alexander Shvets, Jorge Palomar-Giner, and Marta Villegas. Moreover, this release is supported by and was enabled by contributions from the OSCAR team at Inria (project-team ALMAnaCH), specially by Julien Abadji, Rua Ismail and Benoit Sagot, the Common Crawl Foundation, the SLT and SAINT teams at DFKI, TU Darmstadt, the LangTech unit at the Barcelona Supercomputing Center, the 42 supercomputer and Hessian AI, the OpenGPT-X project, Fraunhofer, Jülich Supercomputing Centre, TU Dresden, Deutsche Telekom, as well as by members of the OSCAR community, in particular Sotaro Takeshita, Sebastian Nagel.

More Information

More information can be found on the paper and the dataset card on Huggingface.