W-NUT 2020 Shared Task: Extracting COVID-19 Events from Twitter

Extracting COVID-19 Events from Twitter

For this task, participants are asked to develop systems that automatically extract COVID-19 related events from Twitter.

Check our semantic search system built on our annotated corpus.

Results for our shared task can be found here.

Introduction

People usually share a wide variety of information related to COVID-19 publicly on social media. For example, Twitter users often indicate when they might be at increased risk of COVID-19 due to a coworker or other close contact testing positive for the virus, or when they have symptoms but were denied access to testing. They also discuss their opinions on potential cure methods or treatments for the coronavirus. In this shared task, system entries are invited to build systems for automatically extracting COVID-19 related events from Twitter using our newly built corpus.

Our event extraction is formulated as a slot-filling task: to extract text spans from a given tweet for filling pre-designed slots. Specifically, each slot-filling task is set up as a binary classification problem: given a tweet \(t\) with a candidate choice \(s\), the classification model \(f(t, s) \rightarrow \{0, 1\}\) predicts whether \(s\) answers its designated question.

Our annotated corpus currently covers the following 5 events: (1) tested positive, (2) tested negative, (3) can not test, (4) death and (5) cure and prevention. Below are some examples of our annotated tweets.

Task Organizers

Shi Zong (Ohio State University)
Wei Xu (Georgia Tech)
Alan Ritter (Georgia Tech)

Contact Us: If you have any questions, please feel free to contact us at wnut.sharedtask.covid19extract@gmail.com.

Important Dates

Data available: June 22, 2020
Evaluation window: Sept. 7, 2020 - Sept. 11, 2020 (AoE)
System description papers submitted: Sept. 22, 2020
Papers reviewed: Sept. 30, 2020
Papers camera ready: ~~Oct. 8, 2020~~ Oct. 12, 2020 (AoE)
Workshop day: Nov. 19, 2020

Dataset Release and Format

We are currently releasing 7,500 annotated tweets for the following 5 events: (1) tested positive, (2) tested negative, (3) can not test, (4) death and (5) cure and prevention. The dataset could be accessed at: https://github.com/viczong/extract_COVID19_events_from_Twitter.

Get dataset

Due to Twitter's data sharing policy and people's privacy concerns, we are only able to release the tweet IDs. Participants could recover our annotated data by using official Twitter API, for example in the following way.


            import tweepy

            auth = tweepy.OAuthHandler(your_API_key, your_API_secret_key)
            auth.set_access_token(your_Access_token, your_Access_token_secret)

            api = tweepy.API(auth, parser=tweepy.parsers.JSONParser(), wait_on_rate_limit=True)

            a_single_tweet = api.get_status(id=id_for_tweet, tweet_mode='extended')
            tweet_text_we_use = a_single_tweet['full_text']

Based on our own trail for re-collecting tweets through official Twitter API, 7,115 out of 7,500 tweets could be obtained by using above code snippet (as of June 5, 2020).

Annotation format

For each tweet, we provide character offsets for candidate chunks and annotations. Character offsets are calculated by using the 'full_text' field (as shown above in the code).

Our provided annotations are divided into two parts. For more details, please refer to Section 2 of our paper.

Part 1 - specific event identification: The annotation is a binary "yes" or "no" answer, with "yes" meaning the given tweet describes an event for an individual or a small group of people (or describes a cure or prevention method for cure category).
Part 2 - slot filling: For each designed question (slot), annotators are asked to choose from a set of candidate choices. We summarize all questions with corresponding candidate choices here. Please note that there might be more than one choice that could answer a specific question.

Baseline

We will provide a logistic regression baseline for this task.

Submit Results

All submitted systems will be evaluated on a separate test set (we plan to prepare around 400-500 tweets as final test set for each category). The predictions will be compared with the golden data and we will maintain a leaderboard for all participants. We will be releasing an evaluation script for comparing the system generated outputs with golden annotations.

Please follow our submission requirements to format your prediction results. Please submit your prediction results here.

Technical System Papers

We strongly encourage every participant team to submit a paper on system description. Your papers will be reviewed through a single-blind peer-reviewing process. The accepted papers will be published in the EMNLP workshop proceedings, and the authors will be invited to present the paper at the virtual WNUT workshop (though not required).

You may submit a paper up to 8 pages (plus extra pages for references). The submission does not need to be anonymized; you shall include your name and affiliation in the submitted paper. The title of the paper must follow the format: "YOUR_TEAM_NAME at W-NUT 2020 Shared Task-3: YOUR_PAPER_TITLE". For example, "OSU-NLP at W-NUT 2020 Shared Task-3: A BERT-based Model XXX" if the team name is "OSU-NLP".

The submission should conform to EMNLP 2020 style guidelines and needs to be submitted at the SoftConf link.

The deadline to submit the system paper is Sep 22, 2020 (AoE).

You may find examples of shared-task system description papers in the previous WNUT proceedings at https://www.aclweb.org/anthology/volumes/W15-43

Sponsored by

Anti-harassment Policy