URL encoding explained: when you need it and how it works

URL encoding is one of those things that works ninety percent of the time without thought, until the day a Cyrillic filename, a plus sign, or a stray ampersand breaks production. This article covers the rules so you stop guessing.

What percent-encoding is

Percent-encoding replaces a byte with three ASCII characters: % followed by two hex digits. So a space (byte 0x20) becomes %20. The mechanism exists because URLs are constrained to a limited ASCII subset, but the data they carry — filenames, search queries, JSON payloads — is not. Percent-encoding is the escape hatch.

Two things to understand up front:

• Encoding operates on bytes, not characters. To encode a non-ASCII character, you first encode it to bytes using a character encoding (almost always UTF-8), then percent-encode each byte.
• Whether a given byte needs encoding depends on where in the URL it appears. The same byte may be safe in the path and unsafe in the query.

RFC 3986 reserved characters

RFC 3986 splits ASCII into three buckets:

The key insight: reserved characters are not encoded by default. They are encoded when they appear in a position where they would otherwise be misinterpreted as a delimiter. A literal ? in a path is fine in some specs and ambiguous in others; encode it as %3F to be safe.

Encoding rules per URL component

The rules differ by component. Walking through https://host:port/path/segments?query#fragment:

• Scheme and host: ASCII only. Internationalized domain names (IDN) are encoded with Punycode (xn--), not percent-encoding.
• Path: encode bytes outside the unreserved set. / stays literal as a segment separator; if a path segment contains / as data, encode it as %2F.
• Query: encode each key and value separately, then join with & and =. &, =, #, and + in data must be encoded.
• Fragment: the most permissive component. Most characters are allowed literally, but # itself must be encoded if it appears in fragment data — otherwise it terminates the URL.

Our URL decoder and encoder applies the per-component rules so you do not have to memorize which characters are safe where.

Why space becomes %20 sometimes and + other times

Two encodings of space exist in practice:

• %20 — defined by RFC 3986. Works in every URL component.
• + — defined by application/x-www-form-urlencoded, the format HTML forms use. Valid only in the query component, never in the path.

If you encode a path with + for space, the path will not match the file on disk. If you encode a query string with %20, every browser still decodes it correctly — %20 is the safer choice. Use + only when you are emitting a form-encoded query that a strict form parser will read, and even then, %20 almost always works.

UTF-8, Latin-1, and Windows-1251

The character encoding matters because non-ASCII characters are encoded byte-by-byte. The Cyrillic letter “ж” encodes to different byte sequences depending on the character encoding:

• UTF-8: 0xD0 0xB6 → %D0%B6
• Windows-1251: 0xE6 → %E6
• Latin-1: not representable

Modern specifications mandate UTF-8 for new URLs. WHATWG’s URL Standard explicitly requires it for query strings and paths. Legacy systems that produce Windows-1251 URLs (some older Russian and Bulgarian sites) survive only because servers run permissive decoders.

Three rules:

1. Always encode and decode as UTF-8 unless you have a documented reason otherwise.
2. When you receive a URL from an unknown system, validate that all percent-encoded bytes decode to valid UTF-8. Bytes that do not (lone 0xC0, etc.) are a signal that the source used a different encoding.
3. Never mix encodings inside a single URL. A URL where the path is UTF-8 and the query is Windows-1251 will round-trip through some servers and not others.

Double-encoding bugs

The classic bug: encode once, then encode the result again. hello world becomes hello%20world, then hello%2520world — because the second pass encodes the % in %20 as %25. The user sees a literal %20 in the page, the server cannot find the file, and the bug bounces between front-end and back-end teams until somebody traces the exact handoff.

The fix is to draw a clear line: at exactly one layer, you encode raw user input. Every other layer carries the encoded form through unchanged. If a function is called buildURL, decide whether it takes encoded or unencoded inputs, document the choice, and stick to it. Mixed-mode helpers (maybeEncode) are an anti-pattern; they generate exactly this class of bug.

Symptoms of double-encoding to recognize at a glance: %2520 (space encoded twice), %253A (colon encoded twice), %2526 (ampersand encoded twice). If you see these in production URLs, you have a double-encoding bug somewhere upstream.

encodeURIComponent vs. encodeURI in JavaScript

JavaScript ships two encoders that differ in which characters they leave alone:

• encodeURI(str) — assumes str is a full URI and preserves the delimiters that structure it: : / ? # [ ] @ ! $ & ' ( ) * + , ; =. Use it when you have a complete URL with non-ASCII content in the path or query and you want to fix it up without breaking the structure.
• encodeURIComponent(str) — assumes str is a single piece of data destined to be inserted into a URL. Encodes nearly everything reserved. This is the correct function for encoding a single query value or a single path segment.

The rule of thumb: if you are building a URL piece by piece, use encodeURIComponent on each piece. If you are accepting a URL that you trust to be mostly valid but might contain a stray space or accented character, use encodeURI. Mixing them up is a frequent source of bugs — encodeURI on a query value leaves & alone, and a value that contains & will look like two parameters.

A modern alternative is the URL and URLSearchParams APIs:

const u = new URL("https://example.com/search");
u.searchParams.set("q", "café & chocolate");
u.toString();
// => "https://example.com/search?q=caf%C3%A9+%26+chocolate"

URLSearchParams emits + for spaces (form-encoded style) rather than %20. For most servers that does not matter; for some pedantic servers, it does. Test with a real consumer before assuming either is fine.

Python’s urllib.parse

Python’s urllib.parse module has four functions worth knowing:

• quote(s, safe="/") — percent-encodes s, leaving / alone by default. Equivalent to JavaScript’s encodeURI for paths.
• quote_plus(s) — like quote but encodes space as + rather than %20. For query values.
• unquote(s) — decodes percent-encoding. Treats + as literal.
• unquote_plus(s) — decodes percent-encoding and converts + to space. For form-encoded values.

For building queries from a dict, urllib.parse.urlencode(d) handles encoding for you and emits a valid application/x-www-form-urlencoded string.

How analytics URL parameters break

UTM-style tracking parameters and ad-tech click trackers are unusually fragile because they are concatenated late, often by string substitution rather than proper URL construction. Five patterns we see repeatedly:

1. Click macros that emit encoded URLs into already-encoded query strings. The result is double-encoded and the destination server returns 404.
2. Ampersands in campaign names. utm_campaign=fall%20%26%20winter works; utm_campaign=fall & winter ends the parameter at the unencoded &.
3. Anchor markers. A URL like ?id=foo#section — if you concatenate further query parameters after #section, they are part of the fragment, not the query. The server never sees them.
4. Cyrillic or CJK campaign names stored as Windows-1251 or Shift-JIS in legacy analytics tools. Reports show mojibake (é instead of é) and analysts blame the dashboard.
5. Truncation at unexpected places. Some servers limit URL length to 2048 bytes. A long encoded value silently truncates and the last parameter is corrupt.

The defensive move: at the boundary, decode the URL into a structured object (URL in JavaScript, urlparse in Python), inspect it, then re-emit it with a known-good encoder. Never += a query parameter onto a URL string. That single rule eliminates the majority of analytics breakage.

Closing

URL encoding is well-specified, the libraries are competent, and the rules are not complicated. The bugs almost always come from people building URLs through string concatenation rather than through a proper URL builder, and from confusion about whether a given byte should be encoded at a given layer. Pick an encoder per layer, encode exactly once, document where the boundary is, and most of the failure modes go away.

Related reading

• JSON formatting and validation: tools and best practices
• Impression tracking explained: pixels, beacons, and viewability
• Ad-tech glossary: 50 essential terms