Encoding problems #229

Activity

1. miere commented on Apr 25, 2019

   

   miere

   on Apr 25, 2019

   More actions

   Hey @MarioSilv , I've faced the same issue in the past, but I'm not sure if it's the same situation as yours. I'm afraid your issue would be easily reproducible if you give us more context.

   The best case scenario would be if you provide us a (really) small project containing a simple (straightforward) unit test reproducing this error.
2. leocampos commented on Nov 14, 2019

   

   leocampos

   on Nov 14, 2019

   Last edited by leocampos

   More actions

   How to reproduce the problem:
   Set the JVM to a default encoding (such as US-ASCII)
   // Add a VM option: -Dfile.encoding=US-ASCII

     public static void main(String[] args) {
       String jsonWithVeryCommonCharacterInGerman = "{\"name\":\"Thomas Müller\"}";
       Any anyFromThatJson = JsonIterator.deserialize(jsonWithVeryCommonCharacterInGerman);
       String backToText = JsonStream.serialize(anyFromThatJson);
       System.out.println(backToText);
     }
   

   prints {"name":"Thomas M?ller"}

   What happens is this:

   public static final Any deserialize(String input) {
           return deserialize(input.getBytes()); //<- Uses getBytes without the option to provide the encoding
   }
   

   This part is pretty easy to get around by providing the array of bytes already decoded:
   JsonIterator.deserialize(toBeDeserialized.getBytes(StandardCharsets.UTF_8)); //charset here just the example. You have to know the encoding you have your strings in.

   The biggest problem is in the serialize method:
   JsonStream.serialize
   which creates a new String without providing the encoding
   var4 = new String(stream.buf, 0, stream.count);