Repository navigation
Conversation
decode_utf8 only checked the lead/continuation bit patterns of a multi-byte sequence, never the value it produced, so overlong encodings, UTF-16 surrogates and 4-byte sequences beyond U+10FFFF were accepted. Overlong bytes such as C0 AE C0 AE C0 AF decoded to the ASCII string "../", and surrogates were re-emitted by encode_utf8 as ill-formed UTF-8 in the manifested JSON. Range-check the decoded codepoint for each sequence length and return JSONNET_CODEPOINT_ERROR, as is already done for truncated and malformed sequences, and make encode_utf8 substitute U+FFFD for surrogates the way it already does for values >= U+110000.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
decode_utf8 only checks the lead and continuation bit patterns of a multi-byte sequence and never the value it produces, so overlong encodings, UTF-16 surrogates and 4-byte sequences above U+10FFFF are accepted from importstr content, ext vars, source literals, std.parseYaml strings and std.decodeUTF8 arrays: the bytes C0 AE C0 AE C0 AF decode to "../" and surrogates come back out of encode_utf8 as ill-formed UTF-8 in the manifested JSON, which strict consumers such as Python's json module reject. Reject those sequences with the same U+FFFD substitution already used for truncated and malformed input, and substitute surrogates in encode_utf8 the way values >= U+110000 already are, so the output is always well-formed. The new cases in unicode_test.cpp fail on the current decoder and pass with this change; the full ctest, Makefile and bazel test runs stay green.