dataset construction issue

[Victoriaheiheihei]

Your work is very innovative, but I still have some questions.

1. According to the project, gen_qa uses Wiki data, and cot_construct uses Google Search. Will there be any impact if I change the data source in cot_construct to Wiki as well? Is it necessary to use Google Search?

2. The paper mentions two sizes regarding the amount of training data: one is "During the SFT phase, we utilize the agent-based method to synthesize and filter 58K correct CoT trajectories for training data", and the other is "We propose a hybrid approach for constructing CoT data that combines the two approaches, and construct a 10M CoT dataset (14B tokens) to validate the scalability of MASKSEARCH as a pre-training framework." Could you elaborate on the relationship between these two datasets and the filtering rules?

3. In the comparative experiments, were all the external databases for RAG based on Wiki, or were some based on Google Search?

[clarenceguan]

Thank you for your interest in our work.

1. On Changing the Data Source for cot_construct: It is possible to change the search source. We chose Google Search for the cot_construct phase to align with the open-domain setting of our final evaluation. Using Google Search ensures the model is adapted to a broad range of real-world information. Switching the data source to Wikipedia might lead to a decrease in performance on open-domain tasks. However, if your specific application or domain is primarily focused on Wikipedia-based information, then using Wikipedia as the search source would be a perfectly reasonable and effective choice.

2. The Relationship Between the 58K and 10M Datasets: The two datasets represent different stages of our data construction process for SFT.

   • The initial 58K dataset consists of high-quality CoT trajectories. This set was generated using a powerful Large Language Model as our agent (specifically, Qwen-Max) to ensure high accuracy and reasoning quality.
   • To rapidly and efficiently scale up our training data, we then employed a method we refer to as Self-Evolve Distillation. We used the initial 58K data to train a smaller, more efficient model (Qwen-2.5-7B) to act as a substitute for the larger agent. Through an iterative process of training and data generation with this smaller model, we expanded the SFT dataset to the 10M scale.

3. External Databases in Comparative RAG Experiments: For the comparative experiments involving RAG, all external databases were based on Google Search.

Thank you again for your attention !