AI2 Releases Dolma: The Largest Open Text Dataset for Language Models
The Allen Institute for AI (AI2) is shaking up the world of language models with the release of Dolma, the largest open text dataset to date. Unlike other language models like GPT-4 and Claude, whose training data is closely guarded, Dolma is free to use and open for inspection. AI2 aims to promote transparency and foster innovation in the AI research community by providing unrestricted access to the dataset.
Dolma serves as the foundation for AI2’s forthcoming open language model, OLMo (Data to feed OLMo’s Appetite). The organization believes that if the model itself is intended to be freely used and modified by researchers, the dataset should follow suit. This marks the first data artifact AI2 is making available in relation to OLMo, with a comprehensive paper in the works to provide further insight.
While companies like OpenAI and Meta publish some information about their language model datasets, a significant portion is treated as proprietary. This closed approach not only hinders scrutiny and improvement but also raises questions about the ethical and legal acquisition of the data. Speculation suggests that pirated copies of authors’ books may have been ingested without consent.
To address these concerns, AI2 has taken a new approach with Dolma. The organization has publicly documented all sources and processes involved in curating the dataset, including the rationale behind its content selection and the steps taken to ensure high-quality text. Unlike other datasets, Dolma is intended to be transparent, allowing researchers to understand what information was removed, why it was removed, and how personal details were appropriately excised.
While AI2 acknowledges that companies have the right to protect their models’ training processes in a competitive landscape, this secrecy makes it difficult for external researchers to study or replicate their work. With Dolma, AI2 aims to break down these barriers by offering a dataset that is not only the largest but also the most accessible and straightforward to use.
Dolma boasts an impressive 3 billion tokens, providing a vast volume of content for AI research. The dataset is made available under the ImpACT license for medium-risk artifacts, ensuring responsible and ethical usage. Prospective users are required to adhere to specific permissions outlined in the license.
For individuals concerned about their personal data being included in Dolma, AI2 has introduced a removal request form. This allows individuals to request the removal of their specific data from the dataset, providing a solution for those who wish to maintain their privacy.
To access Dolma, interested parties can do so via Hugging Face, a popular platform in the AI community. Dolma’s release marks a significant step towards transparency and collaboration in the field of language models, enabling researchers to explore, analyze, and build upon the dataset freely.
In conclusion, AI2’s release of Dolma marks a milestone in the open text dataset landscape. By providing the largest and most accessible dataset for language models, AI2 aims to inspire innovation, promote ethical practices, and foster collaboration among AI researchers worldwide. Dolma’s transparency and user-friendly nature will undoubtedly contribute to advancements in the field, enabling breakthroughs in natural language processing and AI applications.
