question-answering-nlu — What's That AI

Question Answering NLU

Question Answering NLU (QANLU) is an approach that maps the NLU task into question answering,
leveraging pre-trained question-answering models to perform well on few-shot settings. Instead of
training an intent classifier or a slot tagger, for example, we can ask the model intent- and
slot-related questions in natural language:

Context : I'm looking for a cheap flight to Boston.

Question: Is the user looking to book a flight?
Answer  : Yes

Question: Is the user asking about departure time?
Answer  : No

Question: What price is the user looking for?
Answer  : cheap

Question: Where is the user flying from?
Answer  : (empty)

Thus, by asking questions for each intent and slot in natural language, we can effectively construct an NLU hypothesis. For more details,
please read the paper:
Language model is all you need: Natural language understanding as question answering.

This repository contains code to transform MATIS++
NLU data (e.g. utterances and intent / slot annotations) into SQuAD 2.0 format
question-answering data that can be used by QANLU. MATIS++ includes
the original English version of ATIS and a translation into eight languages: German, Spanish, French,
Japanese, Hindi, Portuguese, Turkish, and Chinese.

To create a SQuAD-style dataset, we first need to create a list of questions for
each intent and a list of questions for each slot. Questions in English are saved in the MATIS_questions.json file.
In order to parse data in languages other than English, you need to provide questions in that language (or translate the English
questions we provide in this repository).

While we can have a number of questions for each intent and slot, sometimes QANLU will perform better if it sees
one question per intent and slot. We control this with the optional --single_q argument. If you call the
atis.py script using that argument, only the first question in the list will be chosen for each intent and slot.
In the opposite case, all questions for each intent and slot will be used.

Run the following to parse MATIS NLU data into SQuAD:

python atis.py \
       --data_path <path to the data> \
       --languages <de,en,es,fr,ja,hi,pt,tr,zh> \
       --qas_file <path to intent and slot questions json file> \
       --output_dir <path to where output files are stored> \
       [--single_q]

The output of this process will be in the exact format of SQuAD and can be used
to train question answering models. The next step would be to train a question answering model,
see here
for a guide. Alternatively, you can download a QA model trained on SQuAD-v2 directly from
huggingface here, and fine-tune it with
the MATIS++ NLU data parsed into SQuAD format. Please note that we need a model trained on
SQuAD-v2 in order to support negative examples.

A QANLU model trained using SQuAD-v2 and MATIS++ (English) is also available from huggingface here.

In order to calculate precision, recall, and F1 for predictions done on QANLU test sets by the
fine-tuned question answering model, you need to call:

python calculate_pr.py \
    --pred_file <full path to the predictions file created by transformers> \
    --test_file <full path to the test file that the predictions are for>

Example

In this example, we show how to train QANLU on English MATIS (i.e. the original ATIS). We assume that
MATIS has been downloaded at a folder called MATIS in the root directory of this repository.

The first step is to convert the data into SQuAD format:

mkdir data

python atis.py \
       --data_path MATIS/data/train_dev_test \
       --languages en \
       --qas_file MATIS_questions.json \
       --output_dir data

The next step is to fine-tune a SQuAD-trained QA model on the data we just created. For this
example, we will use the deepset/roberta-base-squad2 model from huggingface.
To do the fine-tuning, we will use the run_squad.py script from here
(assuming 8 GPUs present):

mkdir models

python -m torch.distributed.launch --nproc_per_node=8 run_squad.py \
    --model_type roberta \
    --model_name_or_path deepset/roberta-base-squad2 \
    --do_train \
    --do_eval \
    --do_lower_case \
    --train_file data/matis_en_train_squad.json \
    --predict_file data/matis_en_test_squad.json \
    --learning_rate 3e-5 \
    --num_train_epochs 2 \
    --max_seq_length 384 \
    --doc_stride 64 \
    --output_dir models/qanlu/ \
    --per_gpu_train_batch_size 8 \
    --overwrite_output_dir \
    --version_2_with_negative \
    --save_steps 100000 \
    --gradient_accumulation_steps 8 \
    --seed $RANDOM

Once our model is fine-tuned with MATIS++ data, the model will be saved in the models/qanlu/.
The final step is to calculate performance metrics:

python calculate_pr.py 
    --pred_file models/qanlu/predictions_.json 
    --test_file data/matis_en_test.json >> results_matis_en.txt

The output should look like this:

atis_en.txt 
Precision:  0.9613439306358381
Recall:  0.9582283039250991
F1:  0.9597835888187556
Results:  {'slot': {'Precision': 0.9613439306358381, 'Recall': 0.9582283039250991, 'F1': 0.9597835888187556}}

Citation

If you use this work, please cite:

@inproceedings{namazifar2021language,
  title={Language model is all you need: Natural language understanding as question answering},
  author={Namazifar, Mahdi and Papangelis, Alexandros and Tur, Gokhan and Hakkani-T{\"u}r, Dilek},
  booktitle={ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
  pages={7803--7807},
  year={2021},
  organization={IEEE}
}

Security

See CONTRIBUTING for more information.

License

This library is licensed under the CC BY NC License.

README