Short answer

Accents break because the shapefile's attribute table — the .dbf — stores text as raw bytes with no dependable record of which character encoding produced them. A byte like 0xE9 is é in ISO-8859-1 (Latin-1) but is the first byte of a two-byte sequence in UTF-8. If the writer used UTF-8 and the reader assumes Latin-1 (or vice versa), the bytes are decoded against the wrong table and you get mojibake: Genève becomes Genève, Köln becomes Köln, peña becomes peña. The proper fix is the .cpg sidecar, which records the encoding so every reader uses the right one.

Why the .dbf has this weakness

The .dbf is a dBASE IV table, a format from the 1980s. dBASE has a "language driver ID" byte in its header that was supposed to indicate a code page, but it only covers a fixed list of legacy DOS/Windows code pages and cannot express UTF-8. In practice that byte is often unset (zero) or ignored. So a reader confronted with a .dbf has three unreliable signals:

  1. The header language-driver byte (frequently empty or wrong).
  2. A .cpg sidecar (present only if the writer made one).
  3. Its own default, which depends on the operating system and locale.

That third fallback is why the bug is so slippery. On a French Windows machine the default code page (Windows-1252) decodes Latin-1 accents fine, so the author sees correct text. The recipient on a UTF-8 Linux box, or vice versa, sees garbage from the identical file. Nothing about the file changed — only the assumed encoding did.

What a .cpg file actually contains

The .cpg is a tiny plain-text file next to the .dbf with the same base name, containing just an encoding label on one line. Typical contents:

UTF-8

Other valid values include ISO-8859-1, ISO-8859-15 (Latin-9, adds €), windows-1252, windows-1251 (Cyrillic), or a numeric code page like 65001 (which is UTF-8 on Windows). When GDAL or QGIS opens the .dbf and finds a .cpg, it decodes the bytes with that encoding and the guessing stops. The single most effective thing you can do is always ship a .cpg, and standardize on UTF-8 so that any script worldwide is representable.

Worked example — diagnosing mojibake

You receive communes.shp and the field nom shows Mâcon instead of Mâcon. The pattern à followed by another high character is the classic fingerprint of UTF-8 bytes being read as Latin-1. That tells you: the file is genuinely UTF-8, but your reader assumed a single-byte code page. The reverse failure — Mâcon becoming M?con or a box glyph — usually means Latin-1 bytes being read as UTF-8, where the invalid byte sequence cannot be decoded at all.

So the symptom tells you the direction of the mismatch, which tells you how to fix it.

Worked example — fixing on read with GDAL

If the file is UTF-8 but lacks a .cpg, tell the reader explicitly. The shapefile driver honors a config option and an open option:

# Force the reader to treat the .dbf as UTF-8
ogrinfo --config SHAPE_ENCODING UTF-8 communes.shp communes

# Or as an open option
ogrinfo -oo ENCODING=UTF-8 communes.shp communes

If the file is actually Latin-1, set SHAPE_ENCODING ISO-8859-1 instead. A useful trick: setting SHAPE_ENCODING to an empty string tells GDAL not to recode at all and hand back raw bytes, which can help when you are unsure and want to inspect.

Worked example — re-encoding cleanly to UTF-8

The durable fix is to rewrite the data once with a correct, declared encoding. Read with the true source encoding and write UTF-8:

# Source is Latin-1; produce a clean UTF-8 shapefile with a .cpg
ogr2ogr -f "ESRI Shapefile" communes_utf8.shp communes.shp \
  --config SHAPE_ENCODING ISO-8859-1 \
  -lco ENCODING=UTF-8

The -lco ENCODING=UTF-8 layer creation option makes GDAL write a .cpg of UTF-8 alongside the new file, so the encoding now travels with the data and the next reader cannot get it wrong. Better still, convert to a format that has no encoding ambiguity at all:

ogr2ogr -f GPKG communes.gpkg communes.shp --config SHAPE_ENCODING ISO-8859-1

GeoPackage stores text as UTF-8 by definition in its SQLite tables, so the whole class of code-page bugs disappears.

Worked example — QGIS

QGIS lets you set the encoding per layer. When adding a shapefile, the data source dialog has an Encoding dropdown; if the layer is already loaded, open Layer Properties → Source and change the encoding there, then check the attribute table. QGIS respects a .cpg if present, so the dropdown is mainly for files that lack one. When you finally export, choose UTF-8 and let QGIS write the .cpg.

Validation

  • Open the attribute table and inspect a field you know contains accents or non-Latin script against ground truth, not just any field.
  • Confirm a .cpg exists and reads UTF-8 after export.
  • Spot-check the fingerprint patterns: Ã, Â, é, ’ all mean UTF-8 was read as a single-byte code page somewhere upstream.
  • For non-Latin scripts (Cyrillic, Greek, CJK), Latin-1 cannot represent them at all, so any such data must be UTF-8 — a Latin-1 .cpg is automatically wrong.

Common pitfalls and why they happen

  • No .cpg at all. The most common cause; the file relies on the reader's locale default, which differs between machines.
  • "It works for me." The author's OS default matches the file, masking the bug until handoff. Always test the round trip on a differently-configured machine or by forcing an encoding.
  • Double recoding. Reading Latin-1 as UTF-8, then "fixing" by recoding again, can corrupt data irreversibly. Identify the true source encoding once and convert once.
  • Trusting the dBASE header byte. It cannot express UTF-8 and is often blank, so it is not a reliable signal.

Bathyl perspective

We standardize on UTF-8 everywhere and write an explicit .cpg on any shapefile we cannot avoid producing, but the real fix is to retire the shapefile from handoffs in favor of GeoPackage, where text encoding is not negotiable. An attribute table that reads correctly only on the author's laptop is not a finished deliverable; we verify accents survive on a machine with a different locale before anything ships.

Related reading

Sources