A 5-trillion token permissively licensed dataset for training language models, composed of filtered and cross-deduplicated open-source datasets.

A series of small, efficient language models optimized for edge and consumer devices, employing parameter sharing to reduce memory and compute requirements.