MultiLexNorm2: Multilingual Lexical Normalization

In our MultiLexNorm2 task, we emphasize non-Indo-European languages, such as Thai, Vietnamese, Korean, Janpanese and Indonesian. Participants are asked to develop a system that performs lexical normalization: the conversion of non-canonical texts to their canonical equivalent form. In particular, this task includes data from 17 languages.

Release Notification!

Feb 24, 2026

Lexical Normalization

Social media provides a rich source of information. It is notoriously noisy, but also interesting because of its fast pace, informal nature and the vast amount of available data. The creative language use found on social media introduces many difficulties for existing natural language processing tools. One way to overcome these issues is to `normalize' this data to a more canonical register before processing it. In this task we focus on lexical normalization, which means that replacements are done on the word level.

  social  ppl     r    troublesome
  Social  people  are  troublesome

Previous work on normalization has mostly been mono-lingual, where a wide variety of approaches, datasets, and evaluation metrics where used. For this shared task we combined existing datasets with new ones, and converted them to the same format.

Data format

We use a tab-seperated format, with pre-tokenized data. Each word is on one line, and sentence boundaries are indicated with an empty line. The normalization is displayed in the first column:

  social          Social
  ppl             people
  r               are
  troublesome     troublesome

Some of the languages include annotation for word splits and merges. When a word is split, the normalization column include a white-space character, and with a merge the normalization is only included for the first word:

  if              If
  i               i
  have            have
  a               a
  head            headache
  ache            
  tomorro         tomorrow
  ima             i'm going to
  be              be
  pissed          pissed

We encourage all the participants to notify the organizers of any disagreements with the annotation. We will take these into account, and improve the dataset. Please forward any such cases to the communication channel or to an organizer.

Evaluation metric

We use Error Reduction Rate (ERR), which is word-level accuracy normalized for the number of replacements in the dataset (van der Goot, 2019). The formula for ERR is: 

  ERR = (TP − FP)/(TP + FN)

Where TP, FP, TN, FN are defined as follows:

  TN = Annotators did not normalize, system did not normalize
  FP = Annotators did not normalize, system normalized
  FN = Annotators normalized, but system did not find the correct normalization. This could be because it kept the original word, or proposed a wrong candidate.
  TP = Annotators normalized, systems normalized correctly

Note that a word which should be normalized, but is normalized to the wrong candidate, is only a FN. Every input token represents exactly one point in TP, FP, TN, FN. It should also be noted that our evaluation script is case-sensitive, even though some of the dataset do not include capitalization corrections. For a more in depth discussion about evaluation of normalization and ERR in particular we refer to Chapter 5 of Normalization and Parsing Algorithms for Uncertain Input.

Final Ranking

Final rankings are computed using macro-average ERR, weighted at 50% for new languages and 50% for original languages, resulting in two winners:

Note: The official winner will be the highest-ranking open-source system.

Baselines

We provide two simple baselines:

Languages

Our dataset consists of the following languages (two of them are language-pairs, these include code-switched data). Please refer to the Multinorm++ paper for more details.

Language Data from Original Source Size (#words) 1-n/n-1 Caps %normed MFR-ERR
Added new languages
Indonesian Instagram Kurnia & Yulianti, 2020 [bib] 48,716 - + 47.47 59.75
Japanese Twitter Tomoyuki, Risa, and Naoki 95,416 + - 7.03 6.32
Korean dcinside Yumin Kim, Jimin Lee, and Hwanhee Lee 16,577 + - 7.54 6.35
Thai Twitter Limkonchotiwat et al. [bib] 200,915 + - 3.99 42.77
Vietnamese Facebook/TikTok Nguyen, Le, & Nguyen, 2024 [bib] 128,685 + - 15.98 75.77
Original languages
Croatian Twitter Ljubešić et al, 2017 [bib] 89,052 - + 8.16 41.53
Danish Twitter/Arto Plank et al, 2020 [bib] 20,206 + + 9.09 49.68
Dutch Twitter/sms/forum Schuur, 2020 [bib] 21,657 + + 28.84 39.39
English Twitter Baldwin et al, 2015 [bib] 73,806 + - 7.62 66.57
German Twitter Sidarenka et al, 2013 [bib] 24,948 + + 17.39 34.35
Indonesian-English Twitter Barik et al, 2019 [bib] 23,124 + - 13.93 61.51
Italian Twitter van der Goot et al, 2020 [bib] 14,641 + + 7.01 16.83
Serbian Twitter Ljubešić et al, 2017 [bib] 91,738 - + 7.88 45.19
Slovenian Twitter Erjavec et al, 2017 [bib] 75,276 - + 14.93 58.70
Spanish Twitter Alegria et al, 2013 [bib] 13,824 - - 7.48 25.57
Turkish Twitter Çolakoğlu et al, 2019 [bib] 8,082 - + 36.83 14.53
Turkish-German Twitter van der Goot & Çetinoglu [bib] 16,508 + + 25.59 22.09

The 1-n/n-1 column indicates whether words are split and or merged in the annotation, and the caps column indicates whether capitalization is corrected. It should be noted that there were annotation guidelines differences as well as filtering criteria during creation of these datasets, which might hinder cross-lingual learning. This is already quite visible in the the "1-n/n-1" and "Caps" collumns there are still some differences. We attempted to converge some of these annotation differences automatically, and some of them manually, but did not have the resources to do a full re-annotation. In general we follow the guidelines from the 2021 shared task. Some known pecularities in the data include: 1) Japanese, insertion is annotated (which is not included in the other languages) 2) for Korean, the data is samples so that each word is unique, this leads to an MFR baseline score of 0.0.

You can use any pre-trained models for the shared task. Additional data is allowed, except for lexicon-normalized datasets in the target languages. If you want to use additional target language lexicon-normalization data, you must share the data with other teams. We can help by adding it to the shared task.

Timeline

Organizers

Less public matters can be communicated to weerayut.b_s20@vistec.ac.th


Anti-harassment Policy