LylaBot Help

Appearance

The "Unicode Normalize" Node ​

Performs Unicode normalization. Unicode is the standard that translates (encodes) text to bytes (and back) for most text on the internet. For some characters, there are multiple ways to en-/decode them, resulting in the same character, but using different bytes. Because of this, you might want to normalize the bytes first, i.e. bring them into a predefined way of encoding. This is useful for operations like comparing texts, or find-and-replace operations.

For a technical description see https://unicode.org/reports/tr15/

And here https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/normalize

Unicode Normalize

  • Normalized
  • Form
  • Text

Inputs ​

  • Form − The normalization form to use.
    • Canonical Decomposition
      • Separates characters with marks and diacritics into the base character and mark/diacritic character Ç → C◌̧  |  ä → a◌̈
      • Orders combining marks q◌̇◌̣ → q◌̣◌̇
      • Separates hangul syllables into jamo 가 → ᄀ​ᅡ
      • Converts equivalent characters into their "normalized" equivalent Ω (Ohm) → Ω (Capital Omega)
    • Canonical Composition − Performs a canonical decomposition, then recomposes
    • Compatibility Decomposition − Performs the operations of a canonical decomposition, and additional compatibility decompositions:
      • Converts font variants into their standard variant ℌ → H  |  ℍ → H
      • Converts characters that control line breaks into their "normalized" variant NBSP → SPACE
      • Converts positional variants into their base form ‌ﻋ‌ → ‌ع‌  |  ‌ﻌ‌ → ‌ع‌  |  ‌ﻊ‌ → ‌ع‌
      • Converts circled variants into their standard variant ① → 1
      • Converts width variants into their standard variant ｶ → カ
      • Converts rotated variants into their standard variant  ︷  → {  |   ︸  → }
      • Converts super- and subscripts into their standard variant ⁹ → 9  |  ₉ → 9
      • Separates square katakana words and abbreviations into their individual characters ㌀ → アパート
      • Separates fraction characters into individual numbers ¼ → 1⁄4
      • Separates other conjoined characters into their base characters (extended Latin, ligatures, roman numerals, other)  dž → dž  |  ffi → f‌f‌i  |  Ⅸ → IX
    • Compatibility Composition − Performs a compatibility decomposition, then recomposes
  • Text − The text to normalize.

Outputs ​

  • Normalized − The normalized text.