SAIL - Search Augmented Instruction Learning

Authors

Hongyin Luo and Yung-Sung Chuang and Yuan Gong and Tianhua Zhang and Yoon Kim and Danny Fox and Xixin Wu and Helen Meng and James Glass

Affliations

MIT CSAIL and MIT Linguistics and CUHK & CPII

Towards transparent & robust Chatbot

We develop a search engine-grounded large language model that generates language grounding on noisy search results, improving both the transparency and robustness of LLMs with distracting information.

[PREPRINT] | [DEMO] | [GITHUB]

Introducing SAIL-7B Language Model

The Research Question and Our Solution

Can search engines always improve language models?
- No. We found that the improvement by applying search engines of LLMs is minimal on several tasks. While search engines retrieve a vast range of up-to-date information, the retrieval results can be disputing or distracting. Such grounding information is not necessarily helpful to language models.

How to improve language models with search engines?
- We fine-tune a large language model (LLaMA-7B) grounded on real search engine outputs. The fine-tuned model can automatically distill the informative search results and flag distracting items. With the search-augmented fine-tuning, our model can be significantly boosted by a search engine, outperforming state-of-the-art chatbots including ChatGPT and Vicuna-13B with much fewer parameters.

How to evaluate search-augmented large language models?
- Automatic scoring for instruction following with GPT-4. SAIL-7b achieves higher scores than search-augmented ChatGPT and Vicuna models.
- We test the search-grounded answering performance on open-ended QA benchmarks.
- Fact and fairness checking. One of our goals is to fight against misinformation, hate, and stereotype with large language models. We test several LLMs on the UniLC benchmark.

More Details

A subset of our code and data are already publicly available and will be updated to a complete version before June 24. We utilize the pretrained LLaMA model, Alpaca 52k instructions, and GPT4-generated responses. Please consider the term-of-use of these projects.

github repository

Implementation

Backbone Model

We fine-tuned the LLaMA-7b model with a search-augmented instruction training set.

Training Data

We fine-tune a LLaMA-7b model using the 52k instructions designed by the Alpaca Team with the response generated by GPT-4. In addition, we collect 10 search results (titles + previews only) for each instruction with DuckDuckGO.com and a BM25-based Wikipedia retriever implemented by Pyserini, but feed the top 0 to 5 sampled search results to LLaMA for fine-tuning and evaluation. The training data can be downloaded from our Github repository.

Training Details

We trained the model on 4 NVIDIA RTX A6000 GPUs (4x48GB). The training takes ~24 hours (4x24GPU hours). The details of training parameters can be found in our Github repository.

Training Code

We trained our model using the FastChat library

Open Language
Safety Research

SAIL - Search Augmented Instruction Learning

Authors

Affliations

Towards transparent & robust Chatbot

Introducing SAIL-7B Language Model

The Research Question and Our Solution

More Details

SAIL-7B Performance

Implementation

Backbone Model

Training Data

Training Details

Training Code

Interactive Demo with SAIL-7B

SAIL - Search Augmented Instruction Learning

Authors

Affliations

Towards transparent & robust Chatbot

Introducing SAIL-7B Language Model

The Research Question and Our Solution

More Details

SAIL-7B Performance

Implementation

Backbone Model

Training Data

Training Details

Training Code

Interactive Demo with SAIL-7B

This website uses cookies.