Extracting COVID-19 Events from Twitter

For this task, participants are asked to develop systems that automatically extract COVID-19 related events from Twitter.

Data is released on June 22, 2020!

Official evaluation will be between Sept. 7, 2020 - Sept. 11, 2020 (register here to participate).

There is a mailing list for future announcements.


Introduction

People usually share a wide variety of information related to COVID-19 publicly on social media. For example, Twitter users often indicate when they might be at increased risk of COVID-19 due to a coworker or other close contact testing positive for the virus, or when they have symptoms but were denied access to testing. They also discuss their opinions on potential cure methods or treatments for the coronavirus. In this shared task, system entries are invited to build systems for automatically extracting COVID-19 related events from Twitter using our newly built corpus.

Our event extraction is formulated as a slot-filling task: to extract text spans from a given tweet for filling pre-designed slots. Specifically, each slot-filling task is set up as a binary classification problem: given a tweet \(t\) with a candidate choice \(s\), the classification model \(f(t, s) \rightarrow \{0, 1\}\) predicts whether \(s\) answers its designated question.

Our annotated corpus currently covers the following 5 events: (1) tested positive, (2) tested negative, (3) can not test, (4) death and (5) cure and prevention. Below are some examples of our annotated tweets.


Task Organizers

Contact Us: If you have any questions, please feel free to contact us at wnut.sharedtask.covid19extract@gmail.com.


Important Dates


Dataset Release and Format

We are currently releasing 7,500 annotated tweets for the following 5 events: (1) tested positive, (2) tested negative, (3) can not test, (4) death and (5) cure and prevention. The dataset could be accessed at: https://github.com/viczong/extract_COVID19_events_from_Twitter.

Get dataset

Due to Twitter's data sharing policy and people's privacy concerns, we are only able to release the tweet IDs. Participants could recover our annotated data by using official Twitter API, for example in the following way.


            import tweepy

            auth = tweepy.OAuthHandler(your_API_key, your_API_secret_key)
            auth.set_access_token(your_Access_token, your_Access_token_secret)

            api = tweepy.API(auth, parser=tweepy.parsers.JSONParser(), wait_on_rate_limit=True)

            a_single_tweet = api.get_status(id=id_for_tweet, tweet_mode='extended')
            tweet_text_we_use = a_single_tweet['full_text']
        

Based on our own trail for re-collecting tweets through official Twitter API, 7,115 out of 7,500 tweets could be obtained by using above code snippet (as of June 5, 2020).

Annotation format

For each tweet, we provide character offsets for candidate chunks and annotations. Character offsets are calculated by using the 'full_text' field (as shown above in the code).

Our provided annotations are divided into two parts. For more details, please refer to Section 2 of our paper.


Baseline

We will provide a logistic regression baseline for this task.


Submit Results

All submitted systems will be evaluated on a separate test set (we plan to prepare around 400-500 tweets as final test set for each category). The predictions will be compared with the golden data and we will maintain a leaderboard for all participants. We will be releasing an evaluation script for comparing the system generated outputs with golden annotations.

Participants should use the following format (similar to our provided annotations in .jsonl format, but replace character offsets with actual text) to submit their results. We will provide a link for uploading the system results, soon to be published.


        [{'id': '1238504197319995397',
         'predicted_annotation': {'part2-age.Response': ['Not Specified'],
                                  'part2-close_contact.Response': ['Not Specified'],
                                  'part2-employer.Response': ['Not Specified'],
                                  'part2-gender.Response': ['Not Specified'],
                                  'part2-name.Response': ['Rita Wilson', 'Tom Hanks'],
                                  'part2-recent_travel.Response': ['Not Specified'],
                                  'part2-relation.Response': ['Not Specified'],
                                  'part2-when.Response': ['Friday'],
                                  'part2-where.Response': ['Australia']}}]
        


Technical System Papers

We seek submissions of systems and system descriptions. Creators of systems with valid results that are submitted to this shared task are invited to sent a short paper (4 pages plus references) to W-NUT 2020 that describes the system. There is no need to give a detailed description of the shared task in a system submission.

All submissions should conform to EMNLP 2020 style guidelines. Submissions must be anonymized. Abstract submissions should include author information (and where the work was published in a footnote on the front page, if applicable). Please submit your papers at the SoftConf link.

Double Submission Policy: Papers that have been or will be submitted to other meetings or publications must indicate at submission time. Authors of a paper accepted for presentation must notify the workshop organizers by the camera-ready deadline as to whether the paper will be presented or withdrawn.

Sponsored by Twitter logo

Anti-harassment Policy