An Overview of Question and Answering Challenge (QAC) of the next
An Overview of Question and Answering Challenge(QAC)of the next
NTCIR Workshop
Jun’ichi FUKUMOTO
Department of Computer Science,Ritsumeikan University
1-1-1Noji-higashi,Kusatsu-shi,Shiga525-8577,Japan
fukumoto@cs.ritsumei.ac.jp
Tsuneaki KA TO
Language and Information Science,University of Tokyo
3-8-1Komaba,Meguro-ku,Tokyo153-8902Japan
kato@boz.c.u-tokyo.ac.jp
Abstract
In this paper we will propose the question and an-swering task,called Question and Answering Chal-lenge(QAC),and its?rst evaluation.This will be car-ried out as a sub-task of NTCIR3Workshop scheduled in October2002.In QAC,we aimed to encourage the development of practical Q&A systems in a general domain and focus on research of user interaction and information https://www.wendangku.net/doc/2012332919.html,er interaction technology leads to actual interaction between computer and per-son.In actual Q&A between people,there will typi-cally be several interactions in order to con?rm the in-tention of the questions and so https://www.wendangku.net/doc/2012332919.html,rmation extrac-tion that works in general domain is also an important technology in order to realize real Q&A system. Keywords:Q&A,Information Extraction,user inter-action.
1Introduction
We propose the question and answering task,called Question and Answering Challenge(QAC),and its ?rst evaluation QAC-1.This will be carried out as a sub-task of NTCIR3Workshop scheduled in October 2002.
Question and answering is a task to obtain appropri-ate answers for given domain independent questions written in natural language from a large document col-lection.Question and answering has been considered as an interesting research topic for many years and there has been many results published up to now.Most of Q&A systems can get answers for given questions as a result of interaction between the system and uses although these interactions are limited in some speci?c domains.In the current Q&A research,the goal of sys-tems is to retrieve an answer to a given question from very large amount of information source in general do-main.
Q&A technology is related to information extrac-tion,information retrieval,natural language interac-tion and other NLP research.We will focus on user interaction among several research topics in QAC be-cause it will lead actual interaction between computer and person in general domain.We are also interested in how to extract answer expressions from information source to a given https://www.wendangku.net/doc/2012332919.html,rmation extraction that works in general domain will be an important technol-ogy in order to get such answer expressions.
Question and answering is also related to text sum-marization technology.If a question is why or how type question,answer of the question may consists of several sentences or some sort of summary of the orig-inal text.Moreover,in evaluation of question and an-swering,system is required to provide support infor-mation that is important to know why answer is ob-tained or generated.In order to generate support infor-mation,summarization technology can be sometimes used.
2Issues of QAC
We will describe several technical points to conduct evaluation of Q&A task.Currently we are planning to run QAC evaluation exercises over?ve years.In QAC we aim to encourage the development of practi-cal Q&A systems in a general domain by providing an evaluation bases such as a question answering task set and evaluation criterion.
2.1Goal of QAC evaluation
As we have already described in the previous sec-tion,our main goal of QAC is to encourage user inter-action technology that works in a real domain.When user wants to know something from database,s/he makes a question to the database.If s/he can get insuf-?cient information after the?rst question,s/he makes more questions according to the response of questions until the requiring answer is obtained.Moreover,in a real situation of user interaction user generally uses minimum expression of answer to given question.The user also uses a series of questions related to the?rst question,if they want to know more about the topic of the question.
In Q&A task,it is important to evaluate how cor-rect and valid the answer is and how appropriate the support information for each answer is.Support infor-mation means some part of a text that will be evidence why the answer is correct.It will sometimes be a part of text that includes correct answer elements.It is also important to generate answers for no answer question and multiple answer questions.If there is no answer to a question,the system has to respond,for example, there is no answer.
In the Q&A task of TREC[2][3],systems are re-quired to retrieve sentences or a part of text.How-ever,in actual interaction between persons,several in-teractions such as question answering dialogues fre-quently occur.When one person can not understand what questioner intends in his/her question,the per-son make another question to get the questioner’s in-tention to answer the question.There is a case that a person may formulate the second question according to the answer to the?rst question.Moreover,these an-swers by persons consist of simple words or sentences, which express contents of answer and do not include meaningless expressions.
2.2How to de?ne Q&A task
There have been already some evaluation of ques-tion and answering[4][5].Several points of Q&A task de?nition have to be considered as follows:
1.number of question
How many questions are suf?cient for evaluation of Q&A system?
2.no answer question
If there is no answer in database,how a system reply to user?
3.multiple answer question
How to evaluate multiple answers:all the an-swers should be correct or some of them should be correct
https://www.wendangku.net/doc/2012332919.html,rmation level of answer
A system has to reply an answer in detail level or
general level such as speci?c town name or city name.
5.an amount of answer information
Generated answer express is word,phrase,sen-tence,generated passage and so on.
6.question type
There are several question types such as5W1H type question and yes/no question.
2.3Target information sources
What kinds of information sources are suitable for question and answering task?Newspaper articles are suitable for answering general question and are also good database because newspaper articles are widely used for NLP communities.Question and answering on WWW is very real situation and is very useful but is very dif?cult to evaluate generated answers at the same evaluation criterion.On the other hand,for evaluation using the same WWW environment,it is necessary to gather a number of copyright-free WWW pages.
2.4Multilingualism
At the?rst stage of QAC evaluation,question and answering is conducted only in Japanese.However, Japanese Q&A for English text database or English Q&A for Japanese is very https://www.wendangku.net/doc/2012332919.html,e of other lan-guages is also useful in actual simulations of WWW search.
3Task de?nition of QAC-1
There are many evaluation points in Q&A evalua-tion.In the?rst evaluation of question and answering, QAC subtask of NTCIR3Workshop,we tentatively chose two kinds of evaluation points according to our research focus of QAC.In the?rst QAC evaluation, we will emphases two kinds of technical points in the ?rst QAC evaluation at NTCIR3Workshop,that is, information extraction and user interaction technolo-gies.
1.IE technology
Information Extraction like MUC evaluation[6]
aimed to extract information elements from doc-uments using template.For the extraction of tem-plate elements,detailed de?nitions of Named En-tity,Template Element,Template Relation,and Scenario Template are required,that is,IE tech-nology is domain speci?c one.In Q&A,every element and contents can be extracted from doc-uments as an answer of a given question.One
of our aims is to make IE technology available in general purpose.In NE,the extraction ele-ments are person name,organization name,loca-tion name and so on[7],however,answering ele-ments are not limited to these types.
https://www.wendangku.net/doc/2012332919.html,er interaction
Our main purpose in Q&A research is to develop user interaction technology in real domain.At the?rst stage of QAC,we will propose a se-ries of questions for Q&A evaluation.In these questions,it is necessary to resolve reference be-tween questions and there will be ellipsis in the
following questions,which frequently occurs in
Japanese.
We will describe a tentative QAC task de?nition of the NTCIR3Workshop as follows:
Question type
Questions are short answer questions.Answer will be a noun or noun phrase which indicates person names,organization names,name of vari-ous artifact,money,size,date and so on.
no answer or multiple answer
There will be no answer questions or multiple an-swer questions.That is,there is a case that there is no answer object in documents to a given ques-tion or there are many answer objects.In this case,it is necessary to de?ne how to describe these answers as system responses,for example,a list of top?ve answers or all the answers in some order.
a series of questions
There will be one or more follow-up questions to the?rst question.When enough information
is not obtained in the?rst question or more in-
formation related to the?rst question is needed, follow-up questions are generally given.For ex-ample,if the?rst question is a question of per-son name and the second question is a question of his/her age,In Japanese,there will be ellipsis in the follow-up question.
support information
When a system give an answer,the system is required to provide evidential information why
this answer is obtained from document collection.
Such information is support information which is
taken from original document and includes the system’s answer.
Target documents
We are now planning to use one year Japanese newspaper articles to get answers to given ques-tions.
We are planning to prepare two tools to encourage beginners of this research area:NE tagger and IR sys-tem.It will also be helpful for NLP researchers to share dictionaries,extraction rules and some other re-sources from the use of these tools.
Finally,we describe the Current tentative schedule as follows:
Call for Participation June,2001
dry run winter of2001
formal run spring of2001
NTCIR3Workshop October,2002
4Conclusions
We have described our aim and the current evalua-tion plan as above.Although the details of evaluation have not yet been determined,we will discuss task def-inition and evaluation criterion in the later meetings. We hope this QAC evaluation will attract a lot of re-search groups that are interested in question and an-swering research.
References
[1]Burger,J.,Cardie,C.et al.Issues,Tasks and Pro-
gram Structures to Roadmap Research in Ques-tion&Answering(Q&A)NIST DUC Vision and Roadmap Documents https://www.wendangku.net/doc/2012332919.html,
/projects/duc/roadmapping.html,2001.
[2]V oorhees,E.M.and Harman,D.K.(eds.)Pro-
ceedings of the Eighth Text REtrieval Conference (TREC-8)https://www.wendangku.net/doc/2012332919.html,/pubs/trec8
/t8