Percent-Encoded URL Duplicates: How to Diagnose and Consolidate Them

Encoded and decoded URL paths can look equivalent while behaving differently across browsers, CDNs, servers and applications. Learn how to trace the difference and consolidate safely.

A URL such as /files/annual%20report.pdf may represent the same path a user intended to enter with a space. Sometimes that is a harmless difference in representation. In other cases, an encoded character changes the path structure, selects a different application route or causes one layer to cache and route the request differently from another.

Percent-encoded URL duplication is therefore a representation problem before it is a canonical-tag problem. The practical question is not simply whether two URLs look alike. It is whether they reach the same resource, follow the same route behaviour, return the same response and share the same intended public representation.

This guide sets out a diagnostic method. It separates the behaviour of the browser, crawler-visible links, CDN or reverse proxy, web server, application router and canonical-generation layer. It then shows how to use logs and representative requests to decide whether variants should be left alone, standardised in generated URLs or redirected.

What percent-encoded URL duplication means

Percent-encoding represents an octet using a percent sign followed by two hexadecimal digits. The case of the hexadecimal letters in a triplet is not significant: %7E and %7e represent the same encoded value under URI syntax. The relevant definitions are set out in RFC 3986.

That does not make every encoded character interchangeable with its decoded appearance. URI characters have different roles. Unreserved characters include letters, digits, hyphen, period, underscore and tilde. Under generic URI normalisation, their percent-encoded forms are generally equivalent to their decoded forms.

Reserved characters are different. They can act as delimiters between URI components or path segments. Decoding one too early can change the meaning of the path rather than simply changing its spelling.

For example:

/files/a%2Fb
/files/a/b

The first path may represent a single files route segment whose value is a/b. The second contains two path segments, a and b. The requests must not be assumed to identify the same route. An application may deliberately map both to one resource, but that is an application decision that needs to be tested.

Double decoding creates a related risk. A value such as %252F can become %2F after one decode and a slash after another. If the second operation occurs before route parsing, data intended to remain inside a parameter can become a path delimiter.

The safe principle is to parse URI components and delimiters before decoding where decoding could introduce a delimiter, and to avoid decoding the same value more than once. RFC 3986 provides the generic URI guidance; the exact implementation still depends on the framework and processing stage.

A worked example: one apparent file, two possible routes

Imagine a document application that publishes this link:

https://example.test/files/a%2Fb

A separate component emits:

https://example.test/files/a/b

There are several possible explanations:

• Both strings are aliases for the same document and return identical responses.
• The first is a document key containing slash data, while the second requests a nested route.
• An edge or web-server rule decodes the first request before the application sees it.
• The application returns the same body for both requests but generates different canonical URLs or internal links.

Only the first case is a straightforward representation duplicate. The others are route, generation or signal-consistency problems. A redirect from the encoded path to the decoded path is safe only if the site has established that the two paths are semantically equivalent. With an encoded reserved character such as %2F, visual similarity is not enough evidence.

Trace the representation through every layer

The most useful diagnostic framing is to treat the issue as a representation trace. This is a Plus IQ diagnostic method, not an established industry or academic term. Record what each layer receives, compares, forwards, generates and returns. “The server” is not a single normalisation step.

1. Browser and URL implementation

Browsers do not necessarily preserve every character typed into the address bar. URL implementations apply component-specific parsing, percent-encoding and serialisation rules. The WHATWG URL Standard documents these behaviours.

The address-bar display is therefore not raw server evidence. A browser may serialise a URL before making the request, and redirects may introduce another representation. Test with a controlled client and inspect the request target captured at the edge or origin.

2. HTML, JavaScript and sitemap output

Start with discovery. Search server-rendered HTML, client-generated links, XML sitemaps, structured data, hreflang annotations and redirect targets for both representations.

URL construction is a frequent source of variation. One component may encode a complete path, while another encodes an individual segment. A helper may also encode an already encoded value, producing double encoding. For the worked example, these are not equivalent construction strategies:

encodePath('/files/a/b')
'/files/' + encodeURIComponent('a/b')

The correct function depends on whether a/b is intended to be two path segments or one segment containing slash data. Centralising URL construction can reduce inconsistency, but only after the route contract is understood.

Internal links should use the chosen public representation. Sitemap entries should use it as well when that representation is stable. Google’s guidance on consolidating duplicate URLs treats consistent internal signals as part of the consolidation process, but changing links does not stop direct requests or external links to alternate forms.

3. CDN and reverse-proxy behaviour

An edge layer may normalise a path for cache or behaviour matching while forwarding a different representation to the origin. It may also apply its own redirect, security rule or cache-key policy.

That creates a potentially important distinction between:

• the request target received from the client;
• the path used to select an edge rule or cache key;
• the path forwarded to the origin; and
• the URL returned in a redirect or generated response.

Capture these values where the platform exposes them. If an edge cache treats encoded and decoded paths as one cache key but the application treats them as different routes, the apparent duplicate is also a cache and routing risk. If the edge preserves the raw path and the origin deliberately maps both forms to one resource, the SEO issue may be limited to inconsistent discovery and output.

4. Web-server matching and rewriting

Web-server normalisation is configuration-specific. Apache documents that encoded slashes may be rejected, decoded or preserved depending on AllowEncodedSlashes, and that rewrite processing can involve an unescaped representation before rules are evaluated. See the Apache documentation on technical details of rewrite processing and encoded paths.

Nginx also uses a normalised URI for location matching. Its documentation explains that this includes decoding certain percent-encoded sequences and resolving some path-normalisation details; adjacent slashes may also be merged depending on configuration. The Nginx core module documentation should be checked alongside the deployed configuration.

Do not infer the upstream request from the location rule alone. Compare the raw request line, the server’s selected route, the upstream request and the application’s recorded parameter. Different variables may contain different representations.

5. Application routing and canonical generation

The application is where a representation difference becomes a route decision. Establish whether the router receives:

• one segment containing an encoded delimiter;
• multiple segments after decoding;
• a decoded value that is later re-encoded; or
• a value that has already been decoded by an earlier layer.

Record the route or controller identifier, parameter values and resource identifier for each test request. Two URLs that return the same visible page may still invoke different controllers, permissions, cache policies or database lookups.

Then inspect the response. Compare the canonical element, any Link headers, internal links, structured data, hreflang URLs, redirect targets and cache headers. A canonical tag is useful only if it reflects a verified route choice. It cannot repair unsafe decoding, a route collision or an inconsistent cache key.

Build evidence before changing URLs

A standard access log often records the raw request line, while an application log may record a decoded route parameter. Neither is a complete request history. Apache’s mod_log_config documentation illustrates why the meaning of each logged field needs to be checked rather than assumed.

For each representative pair, collect:

• the exact URL requested, including hexadecimal case and any double encoding;
• the raw request target at the edge and origin, where available;
• the normalised path used for matching or routing;
• the status code and every Location header in the chain;
• the final URL after redirects;
• the response body or a stable content identifier;
• canonical output and other generated URLs;
• route, controller or application resource identifiers;
• cache status and relevant cache headers; and
• the source of discovery, such as HTML, sitemap, redirect or external referral.

A controlled fetch can be as simple as requesting both forms with a client that does not rewrite the URL unexpectedly, then repeating the test through the public CDN and directly against an approved origin environment. Compare headers and bodies, not just the final browser page.

Also capture every redirect hop. An apparent encoding issue may actually be caused by a trailing-slash rule, host or protocol redirect, locale middleware, CMS behaviour or framework rewrite. Attribution should follow the first observed divergence, not the most visible difference in the final URL.

Decide whether the variants are genuinely equivalent

Use the evidence to place the pair into one of three practical categories.

Harmless alternate representation

Leaving a variant unchanged may be reasonable when it differs only by an unreserved-character encoding, is not actively exposed, receives negligible traffic, returns the same response and creates no routing, cache, security or reporting difference. This is a risk-based decision, not a rule that all encoded URLs are harmless.

Equivalent routes with inconsistent public signals

If both paths reliably identify the same resource, return the same intended response and have no delimiter or security ambiguity, standardise the public representation. Update URL generation, internal links, sitemaps and canonical output. A narrow redirect from the alternate form may be appropriate where direct discovery is material and the redirect cannot change route meaning.

Distinct or unproven routes

If the paths invoke different controllers, return different resources, produce different permissions or use an encoded reserved character whose meaning is unresolved, do not redirect them merely to remove visual duplication. Fix the route contract or leave the representations separate until the product and engineering owners decide which behaviour is intended.

In particular, do not apply a broad rule that decodes every percent-encoded path. It may convert an encoded slash, question mark or other reserved character into a delimiter, create redirect loops or expose a double-decoding problem.

Remediate in a controlled sequence

1. Choose the intended route representation. Document whether each relevant value is a path segment, a set of segments or data inside a segment. Include reserved-character cases.
2. Fix generation first. Correct the URL helper, template, application component or feed that emits the alternate form. Update server-rendered and client-rendered links, structured data, hreflang and sitemaps where relevant.
3. Make canonical output deterministic. Generate the chosen representation from the same route data used for internal links. Treat the canonical as supporting evidence, not the mechanism that resolves a route collision.
4. Add only narrow redirects that have been proven safe. Match the known alternate representation and redirect it to the verified equivalent. Avoid decoding more than once or redirecting an encoded reserved delimiter without route evidence.
5. Deploy with logging in place. Preserve enough raw and normalised data to identify whether the released rule behaves as intended at the edge, origin and application.
6. Validate before broadening the rule. Expand the test set only after representative encoded, decoded, malformed and double-encoded cases have passed.

The redirect status and method behaviour should be selected for the site’s request types and compatibility requirements. A GET page URL and an API endpoint may require different treatment.

Post-release validation

Test at least one example from each affected route family. Include:

• an unreserved character encoded and decoded;
• an encoded reserved character such as %2F;
• mixed hexadecimal case;
• a double-encoded value such as %252F;
• an invalid or incomplete percent triplet; and
• any route with trailing-slash, locale or host middleware.

After release, review edge, origin and application logs for new status codes, redirect loops, route mismatches, cache anomalies and unexpected decoded values. Recrawl the affected templates and inspect newly generated links and sitemap entries. Search crawl data for the old representation rather than assuming that changing the source code removed every historical URL.

Finally, compare request volume and response behaviour over an appropriate period. The purpose is to confirm that the chosen representation is being emitted consistently and that route meaning has not changed. It is not to claim an automatic ranking or crawl-budget gain.

When this issue matters, and when it may not

Google’s duplicate-URL guidance indicates that search engines may cluster duplicate URLs and crawl duplicates less frequently, so every encoded and decoded pair does not create index fragmentation or a material crawl problem. Percent-encoding cleanup is not an automatic SEO win.

The issue deserves priority when alternate forms are actively linked or listed in sitemaps, when responses or canonical signals disagree, when request volume is significant, or when edge, cache, routing or security behaviour differs. It is also worth addressing when inconsistent representations make reporting and diagnosis unreliable.

If the difference is limited to an unreserved character, both forms are unexposed, responses are identical and there is no operational risk, documenting the decision to leave them alone may be better than introducing a broad rewrite.

Conclusion

The important distinction is between a harmless alternate spelling and a route that only appears equivalent. Percent-encoding is component-specific, and an encoded reserved character can change the structure of a path when decoded.

Trace the URL from discovery through client serialisation, edge handling, server rewriting, application routing, response generation and logs. Use that evidence to identify the first divergence. Then standardise generated URLs and internal signals, adding a narrow redirect only where equivalence has been proved.

For a wider treatment of URL signal testing, see Canonical Chain QA: How to Find Non-Convergent URL Signals.