MultiLexNorm2026: Multilingual Lexical Normalization

Timeline

Date	Event
21-Jul	Test data (Final phase)
01-Aug	Final Evaluation
20-Aug	Paper deadline
05-Sep	Paper reviewed
15-Sep	Camera ready
TBA	Workshop

Lexical Normalization

Social media provides a rich source of information. It is notoriously noisy, but also interesting because of its fast pace, informal nature and the vast amount of available data. The creative language use found on social media introduces many difficulties for existing natural language processing tools. One way to overcome these issues is to `normalize' this data to a more canonical register before processing it. In this task we focus on lexical normalization, which means that replacements are done on the word level.

social  ppl     r    troublesome
Social  people  are  troublesome

Previous work on normalization has mostly been mono-lingual, where a wide variety of approaches, datasets, and evaluation metrics where used. For this shared task we combined existing datasets with new ones, and converted them to the same format.

Data Format

We use a tab-separated format, with pre-tokenized data. Each word is on one line, and sentence boundaries are indicated with an empty line. The normalization is displayed in the first column:

social          Social
ppl             people
r               are
troublesome     troublesome

Some of the languages include annotation for word splits and merges. When a word is split, the normalization column includes a white-space character, and with a merge the normalization is only included for the first word:

if              If
i               i
have            have
a               a
head            headache
ache
tomorro         tomorrow
ima             i'm going to
be              be
pissed          pissed

We encourage all the participants to notify the organizers of any disagreements with the annotation. We will take these into account, and improve the dataset. Please forward any such cases to the communication channel or to an organizer.

Evaluation Metric

We use Error Reduction Rate (ERR), which is word-level accuracy normalized for the number of replacements in the dataset (van der Goot, 2019). The formula for ERR is:

ERR = (TP - FP) / (TP + FN)

Where TP, FP, TN, FN are defined as follows:

TN = Annotators did not normalize, system did not normalize
FP = Annotators did not normalize, system normalized
FN = Annotators normalized, but system did not find the correct normalization.
     This could be because it kept the original word, or proposed a wrong candidate.
TP = Annotators normalized, systems normalized correctly

Note that a word which should be normalized, but is normalized to the wrong candidate, is only a FN. Every input token represents exactly one point in TP, FP, TN, FN. It should also be noted that our evaluation script is case-sensitive, even though some of the datasets do not include capitalization corrections. For a more in depth discussion about evaluation of normalization and ERR in particular we refer to Chapter 5 of Normalization and Parsing Algorithms for Uncertain Input.

Final Ranking

Final rankings are computed using macro-average ERR, weighted at 50% for new languages and 50% for original languages, resulting in two winners:

• One winner averaged across the newly added languages
• One overall winner averaged across all languages

Note: The official winner will be the highest-ranking open-source system.

Baselines

We provide two simple baselines:

• Leave-As-Is (LAI): Simply uses the input as output. Word level accuracy would be equal to the % of words that are not normalized, and ERR is 0.0.
• Most-frequent-Replacement (MFR): Uses the most frequent replacement based on the training data. If the input word is not present in the training data, it returns the input word.

Languages

Our dataset consists of the following languages (two of them are language-pairs, these include code-switched data). Please refer to the Multinorm++ paper for more details.

The 1-n/n-1 column indicates whether words are split and or merged in the annotation, and the caps column indicates whether capitalization is corrected. It should be noted that there were annotation guidelines differences as well as filtering criteria during creation of these datasets, which might hinder cross-lingual learning. This is already quite visible in the "1-n/n-1" and "Caps" columns — there are still some differences. We attempted to converge some of these annotation differences automatically, and some of them manually, but did not have the resources to do a full re-annotation. In general we follow the guidelines from the 2021 shared task.

Some known peculiarities in the data include: 1) Japanese, insertion is annotated (which is not included in the other languages); 2) for Korean, the data is sampled so that each word is unique, this leads to an MFR baseline score of 0.0.

You can use any pre-trained models for the shared task. Additional data is allowed, except for lexicon-normalized datasets in the target languages. If you want to use additional target language lexicon-normalization data, you must share the data with other teams. We can help by adding it to the shared task.

Organizers

Contact

weerayut.b_s20@vistec.ac.th