We fine-tuned the LLaMA-7b model with a search-augmented instruction training set.
We fine-tune a LLaMA-7b model using the 52k instructions designed by the Alpaca Team with the response generated by GPT-4. In addition, we collect 10 search results (titles + previews only) for each instruction with DuckDuckGO.com and a BM25-based Wikipedia retriever implemented by Pyserini, but feed the top 0 to 5 sampled search results to LLaMA for fine-tuning and evaluation. The training data can be downloaded from our Github repository.
We trained the model on 4 NVIDIA RTX A6000 GPUs (4x48GB). The training takes ~24 hours (4x24GPU hours). The details of training parameters can be found in our Github repository.
We trained our model using the FastChat library