Hindi, Tamil, Arabic or Russian song titles showing as ???? or garbage: the encoding fix

A song called नमस्ते दुनिया turns into ?????? ?????? on the car stereo. A Russian album becomes Ïðèâåò. A Tamil title is fine on the laptop and garbage on the phone. These are three different faults with the same root: text stored in a form the reader can't interpret. Each one has a clean fix.

One byte decides everything

In an ID3v2 tag, every text field starts with a single byte that says how the text is encoded. In version 2.3 there are two choices: 00 for Latin-1, an old eight-bit table that covers Western European letters and nothing else, and 01 for UTF-16, which covers every script there is. Version 2.4 adds 03 for UTF-8, which also covers everything.

So a Devanagari, Tamil, Arabic, Cyrillic or Japanese title is fine as long as the tagger wrote 01 or 03. If it wrote 00, the characters had nowhere to go, and what happened next depends on the tool.

What I found by trying it

I wrote six titles into a test MP3 with each encoding, using the mutagen library, and looked at what came back.

A careful library refuses to write what it can't encode. A careless one, and there were plenty in the 2000s, substitutes a question mark for every character it can't fit and writes the result. That's fault number one: the question marks are in the file. No player can recover the title, because the title is gone. The only fix is to type it again with a tool that writes UTF-16.

Notice the cost of doing it right: nine extra bytes for Añjali, a few dozen for the others. There's never been a reason to use Latin-1.

Fault two: the ID3v1 tag at the end of the file

ID3v1, the 128-byte block some players read from the end of the file, has no encoding byte at all. It was designed for ASCII. Here is what my tagging library produced when I asked it for an ID3v1 block with the title नमस्ते दुनिया and the artist Añjali:

TAG?????? ??????                A\xf1jali

The Hindi became question marks, because there's no way to store it. The ñ became the single byte f1, which is its Latin-1 code. A player that reads that block as Latin-1 shows Añjali; a player that assumes UTF-8 shows A�jali. So if your title is right on the computer but question marks in the car, the car is reading ID3v1, and ID3v1 can't hold your title. Nothing you do to the tag will change that. Your options are a car that reads ID3v2, or a Latin transliteration in the ID3v1 copy.

The editor on this site keeps the two apart: the ID3v2.3 tag is always UTF-16 and keeps everything; the optional ID3v1 copy drops any character outside ASCII rather than writing question marks. Untick the box if you'd rather the file had no ID3v1 block at all.

Fault three: the right letters in the wrong table

This is the Ïðèâåò case. Before Unicode was everywhere, Windows programs stored text in the local codepage: Windows-1251 for Russian, Windows-1256 for Arabic, Shift-JIS for Japanese, and so on. Old taggers wrote those bytes into the tag with the Latin-1 encoding byte in front, because that's the only thing they knew. On the same Windows machine it looked perfect. Everywhere else, each byte is looked up in the wrong table and you get Ïðèâåò for Привет.

The letters are still there, just mislabelled. Some desktop taggers (Mp3tag has a codepage setting, foobar2000 has a "fix" action) can reinterpret them in bulk. For one file it's faster to retype the title and save, which writes it correctly.

The fix, in practice

1. Drop the file into the editor. If the Title field shows question marks, the title is gone from the file (fault one); if it shows gibberish, it's fault three. Either way, type the correct title.
2. Save. The new ID3v2.3 tag stores the text as UTF-16, which every player made in the last twenty years reads.
3. If the file is for an old car stereo, accept that the ID3v1 copy will be ASCII only, or type a transliterated title (Namaste Duniya) and live with it.

Then replace the file in your player's library rather than overwriting it in place, or the player may keep showing what it cached the first time.

Filenames are a separate problem

A filename in Hindi can also turn to question marks when the file crosses from one system to another, because file systems have their own encoding rules, and a USB stick formatted as FAT32 has its own opinions. That's nothing to do with the tag; the tag inside can be perfect while the filename is ruined. Players show the tag when there is one, so a good tag is the more durable place for the title.