Nodejs iconv-lite convert wrong SHIFT JIS Character

Question

I'm having trouble converting UTF8 Japanese characters to SJIS

Library used for conversion: iconv-lite

Code:

const iconv = require('iconv-lite')
const japanText = 'でんぱ組 出会いの歌26 カミソヤマ　ユニ';
const buffer = iconv.encode(japanText, 'Shift_JIS');
fs.writeFileSync('convertedFile.txt', buffer)

Result after conversion:

'て?んは?組 出会いの歌26 カミソヤマ　ユニ'

Does anyone have the same problem or have a solution please help me!

Answer 1

Your .js file seems to saved in UTF-8 with NFD (Normalization Form D). Text editors on macOS often use this type of encoding.

iconv-lite cannot handle NFD, so you have to change japanText to NFC. There are two workarounds:

A) Save .js file with NFC

Check the feature of your text editor. For example, if you use Emacs, you can use ucs-normalize-NFC-region. If your text editor doesn't have such a feature, you can use a service to convert NFC/NFD like: https://www.gesource.jp/weblog/?p=7635

B) Canonicalize to NFC in your code

Convert the string with normalize() method before passing the text to iconv-lite like this:

const fs = require('fs')
const iconv = require('iconv-lite')
const japanText = 'でんぱ組 出会いの歌26 カミソヤマ　ユニ';
const buffer = iconv.encode(japanText.normalize("NFC"), 'Shift_JIS');
fs.writeFileSync('convertedFile.txt', buffer)