Bulk Replace
Reference

UTF-8 vs ANSI vs UTF-16: Which Text Encoding Should Your Files Use?

UTF-8, UTF-8 with BOM, ANSI and UTF-16 explained in plain English: how to tell them apart on Windows, when each is used, and when to convert.

Updated 5 minute read The Bulk Replace team

UTF-8, ANSI and UTF-16 are three ways of storing text as bytes. UTF-8 can store every character in every language, uses one byte for plain English letters, and is the standard for web pages, code and data files, so it is the right choice for almost everything new. ANSI is Windows' name for the older single-byte code page of your system locale (usually Windows-1252 in English-speaking countries), which holds only 256 characters. UTF-16 stores every character in two or four bytes; Windows uses it internally, and some Windows files such as registry exports are saved in it. Keep existing files in the encoding their programs expect, and convert to UTF-8 when you have the choice.

Below: what each encoding actually is, how to tell which one a file uses, where each one is still expected, and what goes wrong when a file is read in the wrong one.

The three encodings at a glance

UTF-8ANSI (Windows-1252)UTF-16 LE
Characters availableAll of Unicode256, set by the system localeAll of Unicode
Bytes for A112
Bytes for é212
Bytes for the euro sign312
Bytes for a Chinese character3Not available2
Bytes for an emoji4Not available4
Byte order mark (BOM)Optional: EF BB BFNoneUsually: FF FE
Typical useWeb, code, data, modern appsOlder Windows programs and filesWindows internals, .reg exports, some logs

For English text, UTF-8 and ANSI files are the same size and, as long as only plain ASCII letters are used, byte-for-byte identical. UTF-16 is about twice the size for the same English text, because every letter takes two bytes.

What "ANSI" really means

"ANSI" is a historical misnomer. It refers to the Windows code page set by your system locale, not to a single standard. On a US or Western European system it is Windows-1252. On a Polish system it is Windows-1250, on a Russian system Windows-1251, and on a Japanese system code page 932. The same bytes therefore show different characters on different PCs. A file containing the byte 0xE9 shows é on a Western system and a different letter on a Russian one.

This is the core weakness of ANSI: the file does not say which code page it uses, so the reader has to guess. UTF-8 and UTF-16 avoid the guess, because each byte sequence means the same character everywhere.

UTF-8 with or without BOM

A byte order mark is a short invisible sequence at the very start of a file. For UTF-8 it is the three bytes EF BB BF. It tells programs "this is UTF-8", which helps some older Windows software, but it confuses others:

  • PHP files with a BOM send it as output before anything else, which can trigger "headers already sent" errors.
  • Batch files with a BOM fail on the first line, because cmd.exe reads the BOM as part of the first command.
  • Shell scripts with a BOM before #! will not run on Linux.
  • Excel, on the other hand, opens a UTF-8 CSV correctly when it has a BOM, and may garble accented characters when it does not.

Rule: no BOM for web files, code and scripts; a BOM for CSV files meant for Excel; otherwise match what the reading program expects.

How to tell which encoding a file uses

  1. Notepad. Open the file and look at the right end of the status bar (turn it on under View → Status bar if hidden). It shows UTF-8, UTF-8 with BOM, UTF-16 LE, UTF-16 BE or ANSI.
  2. Notepad++. The Encoding menu shows a dot next to the current encoding, and the status bar shows it too.
  3. VS Code. The encoding appears in the status bar; click it to reopen or save with a different encoding.
  4. PowerShell. Format-Hex .\file.txt | Select-Object -First 1 shows the first 16 bytes. EF BB BF means UTF-8 with BOM, FF FE means UTF-16 LE. No BOM means UTF-8 without BOM or ANSI, and only the non-ASCII characters can tell them apart.

A file that contains only plain English letters, digits and punctuation is valid in both UTF-8 and ANSI. Tools will often report it as one or the other at random. That is harmless until someone adds an accented character.

What goes wrong: the classic symptoms

You seeWhat happened
café instead of caféA UTF-8 file was read as ANSI
A black diamond with a question markAn ANSI file was read as UTF-8
Spaces between every letter, or Chinese-looking textA UTF-16 file was read as ANSI or UTF-8
 at the start of a page or fileA UTF-8 BOM was read as ANSI text
Question marks replacing characters, permanentlyText was saved as ANSI, which could not store those characters

The first three are usually reversible: open the file with the right encoding and the text comes back. The last one is not, because the original characters were thrown away when the file was saved. This is the risk with any tool that rewrites files, including a find and replace. If it reads a UTF-8 file as ANSI and writes it back, accented characters can be damaged in every file it touches. Our guide to why special characters break after a find and replace covers recovery.

Where each encoding is still expected

  • UTF-8 (no BOM): HTML, CSS, JavaScript, JSON, XML, Markdown, Python, PHP, most config files, anything for the web or for Linux.
  • UTF-8 with BOM: CSV files for Excel, some older Windows tools, PowerShell 5.1 scripts that contain non-ASCII text.
  • ANSI: legacy Windows programs, older INI files, batch files that print accented text, some older subtitle files.
  • UTF-16 LE: files exported from Registry Editor (.reg), some Windows logs, and output from Windows PowerShell 5.1's Out-File and > redirection.

That last point surprises many people. In Windows PowerShell 5.1, > writes UTF-16, Set-Content writes ANSI, and -Encoding UTF8 writes UTF-8 with a BOM. PowerShell 7 changed the default to UTF-8 without a BOM. The PowerShell replace guide shows how to set the encoding explicitly.

Converting files

For one file, open it in Notepad, choose File → Save as, pick the encoding in the drop-down next to the Save button, and save. Notepad++ has Encoding → Convert to UTF-8. For many files, use a batch tool. Bulk Replace detects each file's encoding (UTF-8, ANSI or UTF-16) and, by default, writes every file back in the same encoding after a replace, so a mixed folder comes out as it went in. If you want to standardize, Bulk Replace can optionally convert all the files in a run to a single encoding, and every run is backed up so the conversion can be undone.

Before converting a whole folder, check what reads the files. Converting a batch file or a legacy program's config from ANSI to UTF-8 can break the program, even though the text looks the same in an editor.

Frequently asked questions

What does ANSI mean in Notepad?

It means the Windows legacy code page for your system locale, usually Windows-1252 on English systems. It can hold only 256 characters.

Should I use UTF-8 with or without BOM?

Use UTF-8 without BOM for web files and most code. Some older Windows programs need the BOM to recognize UTF-8, so match what your tools expect.

Why are UTF-16 files about twice the size?

UTF-16 uses at least two bytes for every character, while UTF-8 uses one byte for plain English letters and more only for other characters.

Get three guides like this every month, plus software update notices.Subscribe →