[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[eu_members at aclweb dot org] Maryland: Text Analysis Conference (TAC 2009) -- Call for Participation



TAC 2009: Call for Participation

                  Text Analysis Conference (TAC)
                     http://www.nist.gov/tac/

               Track Evaluations: February - October 2009
                    Workshop: November 16-17, 2009


Conducted by:
National Institute of Standards and Technology (NIST)


INTRODUCTION

The Text Analysis Conference (TAC) is a series of evaluation workshops
organized to encourage research in Natural Language Processing and
related applications, by providing a large test collection, common
evaluation procedures, and a forum for organizations to share their
results. TAC comprises sets of tasks known as "tracks," each of which
focuses on a particular subproblem of NLP. TAC tracks focus on
end-user tasks, but also include component evaluations situated within
the context of end-user tasks. In the first Text Analysis Conference
(TAC 2008), 65 teams participated in one or more tracks, representing
20 different countries and six continents.

You are invited to participate in TAC 2009. NIST will provide test
data for each track, and track participants will run their NLP systems
on the data and return their results to NIST for evaluation.
Organizations may choose to participate in any or all of the tracks.
The annual conference culminates in a November workshop at NIST in
Gaithersburg, Maryland, USA. All results submitted to NIST are
archived on the TAC web site, and all evaluations of submitted results
are included in the conference proceedings. Dissemination of TAC work
and results other than in the conference proceedings is welcomed, but
the conditions of participation specifically preclude any advertising
claims based on TAC results.


TRACK DESCRIPTION

TAC 2009 has three tracks. A new Knowledge Base Population track (KBP)
joins the Recognizing Textual Entailment track (RTE-5) and the
Summarization track, which are both returning from TAC 2008.

What's NEW in 2009:

    * The goal of the new Knowledge Base Population track is to
      augment an existing knowledge representation with information
      about entities that is discovered from a collection of
      documents. A snapshot of Wikipedia infoboxes will be used as the
      original knowledge source, and participants will be expected to
      fill in empty slots for entities that do exist, add missing
      entities and their learnable attributes, and provide links
      between entities and references to text supporting extracted
      information. The KBP task lies at the intersection of Question
      Answering and Information Extraction and is expected to be of
      particular interest to groups that have participated in ACE or
      TREC QA.

    * RTE-5 will extend the task of recognizing whether a Text entails
      a given Hypothesis. The Texts will be longer (to promote
      discourse analysis) and will not be edited from their source
      documents; thus, systems will be asked to handle real text that
      may include typographical errors and ungrammatical sentences.

    * The Summarization track brings back the update summarization
      task and adds the task of Automatically Evaluating Summaries Of
      Peers (AESOP) for a given metric. AESOP complements the basic
      summarization task by building a collection of automatic
      evaluation tools that support development of summarization
      systems.

The exact definition of the tasks to be performed in each TAC 2009
track is formulated and discussed on the track mailing list. To be
added to a track mailing list, follow the instructions given in the
track web page for contacting the mailing list. For questions about
the track, send mail to the track coordinator (or post the question to
the track mailing list once you join).

    * Knowledge Base Population (KBP)
      Track Coordinator: Paul McNamee (paul dot mcnamee at jhuapl dot edu)
      Web page: http://apl.jhu.edu/~paulmac/kbp.html
      Mailing list: tac-kbp at nist dot gov

    * Recognizing Textual Entailment (RTE)
      Track Coordinators: Danilo Giampiccolo (giampiccolo at celct dot it)
                          and Luisa Bentivogli (bentivo at fbk dot eu)
      Web page: http://www.nist.gov/tac/2009/RTE/
      Mailing list: rte at nist dot gov

    * Summarization
      Track Coordinator: Hoa Trang Dang (hoa dot dang at nist dot gov)
      Web page: http://www.nist.gov/tac/2009/Summarization/
      Mailing list: duc_list at nist dot gov


TRACK REGISTRATION

Organizations wishing to participate in any of the TAC 2009 tracks are
invited to register online by March 3, 2009 at:

http://www.nist.gov/tac/2009/track-app.html

Registration for a track does not commit you to participating in the
track, but is helpful to know for planning. Late registration will be
permitted only if resources allow. Any questions about conference
participation may be sent to the TAC project manager:
tac-manager at nist dot gov.


WORKSHOP

The TAC 2009 workshop will be held November 16-17, 2009, in
Gaithersburg, Maryland, USA, and is co-located with the meeting of the
Eighteenth Text REtrieval Conference (TREC 2009). The TAC workshop is
a forum both for presentation of results (including failure analyses
and system comparisons), and for more lengthy system presentations
describing techniques used, experiments run on the data, and other
issues of interest to researchers in NLP. Track participants who want
to give a presentation during the workshop will submit a 500-word
abstract in September describing the experiments they performed. As
there is a limited amount of time for oral presentations, the TAC
advisory committee will use the abstracts to determine which
participants are asked to speak and which will present in a poster
session.


PRELIMINARY SCHEDULE

March 3: Deadline for track registration
Beginning March 3: Submit signed User Agreements to NIST
July - early September: Deadlines for results submission
September 16 (estimated): Deadline for workshop presentation proposals
September 16 (estimated): Deadline for proposals for TAC 2010 tracks/tasks
By early October: Release of individual evaluated results to participants
mid October: Deadline for participants' notebook papers
November 16-17: TAC 2009 workshop in Gaithersburg, Maryland, USA
early February, 2010: Deadline for participants' final proceedings papers


ADVISORY COMMITTEE

John Conroy, IDA/CCS
Ido Dagan, Bar Ilan University
Hoa Trang Dang, NIST (chair)
Maarten de Rijke, University of Amsterdam
Bill Dolan, Microsoft Research
Bonnie Dorr, University of Maryland
Donna Harman, NIST
Andy Hickl, Language Computer Corporation
Ed Hovy, ISI/USC
Bernardo Magnini, FBK
Ani Nenkova, University of Pennsylvania
Drago Radev, University of Michigan
Lucy Vanderwende, Microsoft Research
Ellen Voorhees, NIST
Ralph Weischedel, BBN