2.5 Years in Class:

Compared to image-text pair data, interleaved corpora enable Vision-Language Models (VLMs) to understand the world more naturally like humans. However, such existing datasets are crawled from webpage, facing challenges like low knowledge density, loose image-text relations, and poor logical coherence between images. On the other hand, the internet hosts vast instructional videos (e.g., online geometry courses) that are widely used by humans to learn foundational subjects, yet these valuable resources remain underexplored in VLM training. In this paper, we introduce a high-quality multimodal textbook corpus with richer foundational knowledge for VLM pretraining. It collects over 2.5 years of instructional videos, totaling 22,000 class hours. We first use an LLM-proposed taxonomy to systematically gather instructional videos. Then we progressively extract and refine visual (keyframes), audio (ASR), and textual knowledge (OCR) from the videos, and organize as an image-text interleaved corpus based on temporal order. Compared to its counterparts, our video-centric textbook offers more coherent context, richer knowledge, and better image-text alignment. Experiments demonstrate its superb pretraining performance, particularly in knowledge- and reasoning-intensive tasks like ScienceQA and MathVista. Moreover, VLMs pre-trained on our textbook exhibit outstanding interleaved context awareness, leveraging visual and textual cues in their few-shot context for task solving.

In this paper, we introduce a multimodal Textbook: a high-quality pre-training corpus that encompasses a wealth of foundational knowledge. Our textbook is constructed from 2.5 years of instructional videos, amounting to 22,000 class hours, covering six fundamental subjects, including mathematics, physics, and others. The whole corpus is presented in an image-text interleaved format, where the text and images are more closely aligned, and the logical relations between images are also more coherent.

• An LLM-powered Pipeline for Automatically Collecting Instructional Videos: We first prompt LLMs to construct a knowledge taxonomy covering six subjects and 3900 knowledge points. Then based on this, we gather relevant instructional videos.
• A Video-to-Textbook Pipeline: we design a multi-level, coarse-to-fine knowledge extraction and data filtering pipeline for these collected videos:
  • From a visual perspective, we extract keyframes and recognition text, symbols, and formulas (OCR).
  • From an auditory perspective, we perform automatic speech recognition (ASR) on the instructor's verbal explanations and refine their quality.
  • The keyframes and tutorial text are organized into an interleaved format, sequenced chronologically.

Our textbook is an openly accessible pre-training dataset with high-quality 6.5 million images interleaving with 0.75 billion texts. It drawn from 75,000 extensive instructional videos, totoaling over 22000 class hours, covering multiple core subjects such as mathematics, physics, chemistry. Our textbook (the first example) presents three keyframes interleaved with four tutorial texts to dynamically illustrate the geometric concept of complementary angles. These more coherent interleaved context and better-aligned image-text sequences enable VLMs to better grasp foundational knowledge during the pretraining.

We first employ LLaVA-1.5-7B as base models to study the pretraining performance on our dataset and reference datasets (MMC4, OBELICS). For LLaVA-1.5-7B, we apply continual pretraining on its pre-trained model (aligned using 558K paired data). To investigate our dataset more comprehensively, we also pre-train Idefics2-8B model on our dataset, which is an advanced VLM that already supports multi-image and interleaved format input. For the Idefics2-8B, we design two pretraining settings: 1. Training from scratch using the architecture of Idefics2-8B (i.e., Idefics2-8B with randomly initialized projector) and 2. Continual pretraining from the Idefics2-8B-base which is already pre-trained on OBELICS. For a fair comparison, we sample an equivalent number of samples (610K) from MMC4 and OBELICS and apply the same training parameters across all datasets.

We synthesize knowledge taxonomy with 3915 knowledge points across 6 subjects, which enabled us to automatically collect 159k English instructional videos based on this taxonomy. Following our video-to-textbook pipeline, we filter 53% low-quality or repetitive videos and retain 75k videos (22,697 class hours) with an average duration of 18 minutes. Then we extract 6.5M keyframes and 0.75B text (ASR+OCR) tokens from these videos. We produce a total of 610K interleaved samples. Each sample contains an average of 10.7 keyframes and 1,297 text tokens. The detailed statistics for each subject is as follows: