Base64 JsonMappingException: Unexpected end-of-input

[ViToni]

My prod environment uses version 2.7.4. but I also tested that the bug occurs with the latest version from maven 2.9.7.

My test case include different serializer /deserializer. To ensure compatibility I have some fuzzy which created random data pass it to one serializer and the result to another deserializer. The output is expected to be the same as the original input. One test however which creates a random byte array fails sporadically with jackson.

Here is a minimal test case (the "short" JSON was specifically chosen because the original error message said some about '/'. after simplifying the data structure the error message changed)

To reproduce it easily the other Base64 decoder is from the JRE 8:

import java.util.Arrays;
import java.util.Base64;

import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.databind.ObjectReader;

public class JSONEncodingTest {

    private static final String SHORT_BASE64_ENCODED_STRING = "A/A=";

    private static final String LONG_BASE64_ENCODED_STRING = //
            "eyJ0b3BpYyI6InRlc3QvdG9waWMvZm9yL2V2ZW50cy90b3BpYzAwMDE5L0U1ZjgiLCJwcm9wZXJ0aWVzIjp7ImNvbS5xaXZpY29uLnNlcnZpY2VzLnJlbW90ZS5ldmVudC5zZXF1ZW5jZS5udW1iZXIiOjM2OCwiZXZlbnQudG9waWNzIjoidGVzdC90b3BpYy9mb3IvZXZlbnRzL3RvcGljMDAwMTkvRTVmOCIsInVuaWNvZGUuc3RyaW5nIjoiZ0JBNVBUZjlhZ2d5aEFHdF/DnFpWZ1U3VMOWSi1hUHpow5ZoWlZOUmRMZVNrMUIxejRoX3BNd21WZ05Qw6RaSEVlcnd5NE9CRUJFRGFFRcOEdFVVNHJHM3VUw5xTSjnigqwwaTR6VVB2UnFqdlBZw4TDljNqbsOfaEFqNjc0X3U0ccO2TFFIeEZZSSJ9fQ";

    private static final String JSON_SHORT = "{ \"data\" : \"" + SHORT_BASE64_ENCODED_STRING + "\"}";
    private static final String JSON_LONG = "{ \"data\" : \"" + LONG_BASE64_ENCODED_STRING + "\"}";

    public static class DataWrapper {
        public byte[] data;
    }

    public static void main(String[] args) throws Exception {
        decode(SHORT_BASE64_ENCODED_STRING, JSON_SHORT);
        decode(LONG_BASE64_ENCODED_STRING, JSON_LONG);
    }

    public static void decode(final String base64, final String json) throws Exception {
        final byte[] bytes = Base64.getDecoder().decode(base64);
        System.out.println(Arrays.toString(bytes));

        final ObjectMapper mapper = new ObjectMapper();

        final ObjectReader textReader = mapper.readerFor(DataWrapper.class);

        final DataWrapper jsonResult = textReader.readValue(json);
        System.out.println(Arrays.equals(bytes, jsonResult.data));

    }
}

Expected out would be:

[3, -16]
true
[123, 34, 116, 111, 112, 105, 99, 34, 58, 34, 116, 101, 115, 116, 47, 116, 111, 112, 105, 99, 47, 102, 111, 114, 47, 101, 118, 101, 110, 116, 115, 47, 116, 111, 112, 105, 99, 48, 48, 48, 49, 57, 47, 69, 53, 102, 56, 34, 44, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 34, 58, 123, 34, 99, 111, 109, 46, 113, 105, 118, 105, 99, 111, 110, 46, 115, 101, 114, 118, 105, 99, 101, 115, 46, 114, 101, 109, 111, 116, 101, 46, 101, 118, 101, 110, 116, 46, 115, 101, 113, 117, 101, 110, 99, 101, 46, 110, 117, 109, 98, 101, 114, 34, 58, 51, 54, 56, 44, 34, 101, 118, 101, 110, 116, 46, 116, 111, 112, 105, 99, 115, 34, 58, 34, 116, 101, 115, 116, 47, 116, 111, 112, 105, 99, 47, 102, 111, 114, 47, 101, 118, 101, 110, 116, 115, 47, 116, 111, 112, 105, 99, 48, 48, 48, 49, 57, 47, 69, 53, 102, 56, 34, 44, 34, 117, 110, 105, 99, 111, 100, 101, 46, 115, 116, 114, 105, 110, 103, 34, 58, 34, 103, 66, 65, 53, 80, 84, 102, 57, 97, 103, 103, 121, 104, 65, 71, 116, 95, -61, -100, 90, 86, 103, 85, 55, 84, -61, -106, 74, 45, 97, 80, 122, 104, -61, -106, 104, 90, 86, 78, 82, 100, 76, 101, 83, 107, 49, 66, 49, 122, 52, 104, 95, 112, 77, 119, 109, 86, 103, 78, 80, -61, -92, 90, 72, 69, 101, 114, 119, 121, 52, 79, 66, 69, 66, 69, 68, 97, 69, 69, -61, -124, 116, 85, 85, 52, 114, 71, 51, 117, 84, -61, -100, 83, 74, 57, -30, -126, -84, 48, 105, 52, 122, 85, 80, 118, 82, 113, 106, 118, 80, 89, -61, -124, -61, -106, 51, 106, 110, -61, -97, 104, 65, 106, 54, 55, 52, 95, 117, 52, 113, -61, -74, 76, 81, 72, 120, 70, 89, 73, 34, 125, 125]
true

but is

[3, -16]
true
[123, 34, 116, 111, 112, 105, 99, 34, 58, 34, 116, 101, 115, 116, 47, 116, 111, 112, 105, 99, 47, 102, 111, 114, 47, 101, 118, 101, 110, 116, 115, 47, 116, 111, 112, 105, 99, 48, 48, 48, 49, 57, 47, 69, 53, 102, 56, 34, 44, 34, 112, 114, 111, 112, 101, 114, 116, 105, 101, 115, 34, 58, 123, 34, 99, 111, 109, 46, 113, 105, 118, 105, 99, 111, 110, 46, 115, 101, 114, 118, 105, 99, 101, 115, 46, 114, 101, 109, 111, 116, 101, 46, 101, 118, 101, 110, 116, 46, 115, 101, 113, 117, 101, 110, 99, 101, 46, 110, 117, 109, 98, 101, 114, 34, 58, 51, 54, 56, 44, 34, 101, 118, 101, 110, 116, 46, 116, 111, 112, 105, 99, 115, 34, 58, 34, 116, 101, 115, 116, 47, 116, 111, 112, 105, 99, 47, 102, 111, 114, 47, 101, 118, 101, 110, 116, 115, 47, 116, 111, 112, 105, 99, 48, 48, 48, 49, 57, 47, 69, 53, 102, 56, 34, 44, 34, 117, 110, 105, 99, 111, 100, 101, 46, 115, 116, 114, 105, 110, 103, 34, 58, 34, 103, 66, 65, 53, 80, 84, 102, 57, 97, 103, 103, 121, 104, 65, 71, 116, 95, -61, -100, 90, 86, 103, 85, 55, 84, -61, -106, 74, 45, 97, 80, 122, 104, -61, -106, 104, 90, 86, 78, 82, 100, 76, 101, 83, 107, 49, 66, 49, 122, 52, 104, 95, 112, 77, 119, 109, 86, 103, 78, 80, -61, -92, 90, 72, 69, 101, 114, 119, 121, 52, 79, 66, 69, 66, 69, 68, 97, 69, 69, -61, -124, 116, 85, 85, 52, 114, 71, 51, 117, 84, -61, -100, 83, 74, 57, -30, -126, -84, 48, 105, 52, 122, 85, 80, 118, 82, 113, 106, 118, 80, 89, -61, -124, -61, -106, 51, 106, 110, -61, -97, 104, 65, 106, 54, 55, 52, 95, 117, 52, 113, -61, -74, 76, 81, 72, 120, 70, 89, 73, 34, 125, 125]
Exception in thread "main" com.fasterxml.jackson.databind.JsonMappingException: Unexpected end-of-input: was expecting closing quote for a string value
at [Source: (String)"{ "data" : "eyJ0b3BpYyI6InRlc3QvdG9waWMvZm9yL2V2ZW50cy90b3BpYzAwMDE5L0U1ZjgiLCJwcm9wZXJ0aWVzIjp7ImNvbS5xaXZpY29uLnNlcnZpY2VzLnJlbW90ZS5ldmVudC5zZXF1ZW5jZS5udW1iZXIiOjM2OCwiZXZlbnQudG9waWNzIjoidGVzdC90b3BpYy9mb3IvZXZlbnRzL3RvcGljMDAwMTkvRTVmOCIsInVuaWNvZGUuc3RyaW5nIjoiZ0JBNVBUZjlhZ2d5aEFHdF/DnFpWZ1U3VMOWSi1hUHpow5ZoWlZOUmRMZVNrMUIxejRoX3BNd21WZ05Qw6RaSEVlcnd5NE9CRUJFRGFFRcOEdFVVNHJHM3VUw5xTSjnigqwwaTR6VVB2UnFqdlBZw4TDljNqbsOfaEFqNjc0X3U0ccO2TFFIeEZZSSJ9fQ"}"; line: 1, column: 12] (through reference chain: JSONEncodingTest$DataWrapper["data"])
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:394)
at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:353)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1711)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:290)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:151)
at com.fasterxml.jackson.databind.ObjectReader._bindAndClose(ObjectReader.java:1611)
at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:1219)
at JSONEncodingTest.decode(JSONEncodingTest.java:36)
at JSONEncodingTest.main(JSONEncodingTest.java:25)
Caused by: com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: was expecting closing quote for a string value
at [Source: (String)"{ "data" : "eyJ0b3BpYyI6InRlc3QvdG9waWMvZm9yL2V2ZW50cy90b3BpYzAwMDE5L0U1ZjgiLCJwcm9wZXJ0aWVzIjp7ImNvbS5xaXZpY29uLnNlcnZpY2VzLnJlbW90ZS5ldmVudC5zZXF1ZW5jZS5udW1iZXIiOjM2OCwiZXZlbnQudG9waWNzIjoidGVzdC90b3BpYy9mb3IvZXZlbnRzL3RvcGljMDAwMTkvRTVmOCIsInVuaWNvZGUuc3RyaW5nIjoiZ0JBNVBUZjlhZ2d5aEFHdF/DnFpWZ1U3VMOWSi1hUHpow5ZoWlZOUmRMZVNrMUIxejRoX3BNd21WZ05Qw6RaSEVlcnd5NE9CRUJFRGFFRcOEdFVVNHJHM3VUw5xTSjnigqwwaTR6VVB2UnFqdlBZw4TDljNqbsOfaEFqNjc0X3U0ccO2TFFIeEZZSSJ9fQ"}"; line: 1, column: 921]
at com.fasterxml.jackson.core.base.ParserMinimalBase._reportInvalidEOF(ParserMinimalBase.java:594)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString2(ReaderBasedJsonParser.java:2031)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._finishString(ReaderBasedJsonParser.java:2018)
at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.getText(ReaderBasedJsonParser.java:278)
at com.fasterxml.jackson.databind.deser.std.PrimitiveArrayDeserializers$ByteDeser.deserialize(PrimitiveArrayDeserializers.java:484)
at com.fasterxml.jackson.databind.deser.std.PrimitiveArrayDeserializers$ByteDeser.deserialize(PrimitiveArrayDeserializers.java:446)
at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:136)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:288)
... 5 more

[ViToni]

After further investigation it turns out jackson fails because it expects padding of the Base64 encoded data.

    public static void testRoundtrip() throws Exception {
        roundtrip(SHORT_BASE64_ENCODED_STRING);
        roundtrip(LONG_BASE64_ENCODED_STRING);
    }

    public static void roundtrip(final String base64) throws Exception {
        final DataWrapper dataWrapper = new DataWrapper();
        dataWrapper.data = Base64.getDecoder().decode(base64);

        final ObjectMapper mapper = new ObjectMapper();

        final ObjectReader textReader = mapper.readerFor(DataWrapper.class);
        final ObjectWriter textWriter = mapper.writerFor(DataWrapper.class);

        final String json = textWriter.writeValueAsString(dataWrapper);

        final DataWrapper jsonResult = textReader.readValue(json);

        System.out.println("Byte arrays equal? = " + Arrays.equals(dataWrapper.data, jsonResult.data));

        final JSONObject jsonObject = new JSONObject(json);
        final String jacksonBase64 = jsonObject.getString("data");
        final byte[] bytesFromJacksonBase64 = Base64.getDecoder().decode(jacksonBase64);

        System.out.println("Byte arrays equal? = " + Arrays.equals(dataWrapper.data, bytesFromJacksonBase64));
        if (!base64.equals(jacksonBase64)) {
            System.out.println("Created JSON does not match:");
            System.out.println("JRE:     " + base64);
            System.out.println("Jackson: " + jacksonBase64);
        }
    }

Output:

Byte arrays equal? = true
Byte arrays equal? = true
Byte arrays equal? = true
Byte arrays equal? = true
Created JSON does not match:
JRE: eyJ0b3BpYyI6InRlc3QvdG9waWMvZm9yL2V2ZW50cy90b3BpYzAwMDE5L0U1ZjgiLCJwcm9wZXJ0aWVzIjp7ImNvbS5xaXZpY29uLnNlcnZpY2VzLnJlbW90ZS5ldmVudC5zZXF1ZW5jZS5udW1iZXI
iOjM2OCwiZXZlbnQudG9waWNzIjoidGVzdC90b3BpYy9mb3IvZXZlbnRzL3RvcGljMDAwMTkvRTVmOCIsInVuaWNvZGUuc3RyaW5nIjoiZ0JBNVBUZjlhZ2d5aEFHdF/DnFpWZ1U3VMOWSi1hUHpow5ZoWlZOUmR
MZVNrMUIxejRoX3BNd21WZ05Qw6RaSEVlcnd5NE9CRUJFRGFFRcOEdFVVNHJHM3VUw5xTSjnigqwwaTR6VVB2UnFqdlBZw4TDljNqbsOfaEFqNjc0X3U0ccO2TFFIeEZZSSJ9fQ
Jackson: eyJ0b3BpYyI6InRlc3QvdG9waWMvZm9yL2V2ZW50cy90b3BpYzAwMDE5L0U1ZjgiLCJwcm9wZXJ0aWVzIjp7ImNvbS5xaXZpY29uLnNlcnZpY2VzLnJlbW90ZS5ldmVudC5zZXF1ZW5jZS5udW1iZXI
iOjM2OCwiZXZlbnQudG9waWNzIjoidGVzdC90b3BpYy9mb3IvZXZlbnRzL3RvcGljMDAwMTkvRTVmOCIsInVuaWNvZGUuc3RyaW5nIjoiZ0JBNVBUZjlhZ2d5aEFHdF/DnFpWZ1U3VMOWSi1hUHpow5ZoWlZOUmR
MZVNrMUIxejRoX3BNd21WZ05Qw6RaSEVlcnd5NE9CRUJFRGFFRcOEdFVVNHJHM3VUw5xTSjnigqwwaTR6VVB2UnFqdlBZw4TDljNqbsOfaEFqNjc0X3U0ccO2TFFIeEZZSSJ9fQ==

In contrast to the JRE the trailing == seems to be required by Jackson.

[ViToni]

The current solution / workaround is to use a custom serializer using annotations:
@JsonSerialize(using = CustomSerializer.class)
@JsonDeserialize(using = CustomDeserializer.class)

[cowtowncoder]

Actually this may be a question of proper error messaging. Jackson default Base64 variant is MIME_NO_LINEFEEDS, which does expect padding, so failure makes sense. But it should be indicated with proper message: current one does not explain problem appropriately.

Now: as to differences; as you probably know, Base64 encoding is rather loosely/ill-defined, f.ex:

https://en.wikipedia.org/wiki/Base64

including whether padding is required or not. There are valid opinions for either way: requiring may be safer against invalid encoding or truncated data; but on the other hand it is theoretically optional if length of encoded String is known.

In your case, since you prefer not requiring padding, you probably want to change the default Base64Variant that ObjectMapper uses. A few choices are defined in Base64Variants, but you can also create differently configured instances, either based on predefined ones, or directly.

[ViToni]

Thanks for the swift response 👍
It's certainly a weakness of Base64 that there are so many variants possible.

My main issue is the misleading wording of the error message and that none of the existing Base64Variants supports non-padding Base64.
Maybe there could be some lenient mode for jackson? The JRE can produce padded/non-padded Base64 (by configuration) and can read both using its default decoder.

[cowtowncoder]

I think we are in agreement here! I actually thought one of the variants already allowed that, but apparently not. And although it is possible to create variant, that is not obvious thing to do and is just something that'd be better to avoid.

So: I hope to resolve this when I get more time to work again on Jackson. Or if anyone with an itch could help, would be happy to get PR merged and such too.

[cowtowncoder]

Hmmmh. Very interesting.

I have been working a bit on improving error message, but now found out that part of the problem is that exception seen is actually side-effect of error handler that tries to get JSON String value, but that parser itself is in inconsistent state due to failure in base64 codec.
So I need to figure out how to untangle that part, in addition to original exception that I have straightened out now in jackson-core.

[ViToni]

You are one of the most responsive persons! Thanks a lot for caring!

[cowtowncoder]

@ViToni you are welcome. I find that having "good users", ones who also care, helps a lot. And discussions that go depth like here, to find good workable solutions.

One remaining thing: I looked at Wikipedia page, and while there are many variants, did not see obvious name for new one to add. Now, of existing variants, MODIFIED_FOR_URL does actually not use padding (i.e. does not add, accepts content without). But that uses different alphabet.

So I was wondering if you had something in mind; I could easily add new variant for 2.10, for convenience. If not that's fine too, but I don't want to forget one aspect that could further help usage.

[ViToni]

@cowtowncoder You might want to take a look at the Enoder class fom the JDK.
It can, so to say, create subvariants of variants:
https://docs.oracle.com/javase/8/docs/api/java/util/Base64.Encoder.html#withoutPadding--

For the decoder part I would go for a lenient default (which can read both: padded / non-padded) with an option to make it strict (but that's just me) if one wants to pin its "other" side to a certain expected format.

Strict might refere to one of both modes:

1. padded-only
2. non-padded-only

But this might be over the top.

[cowtowncoder]

I think the main challenge currently is just that there are named existing variants, applied to both, and I think attempting to change style of API could only be done for 3.x. Adding new methods in Base64Variant for creating alternative configurations is fine (withPadding(char) / withoutPadding()?) for 2.10, as would be adding new pre-configured variant. But I think naming is important here. Existing Base64Variants are:

• MIME
• MIME_NO_LINEFEEDS
• PEM
• MODIFIED_FOR_URL

of which all but last add and require padding; last one neither adds nor requires.

I could add, say, MIME_NO_PADDING, but I haven't seen such usage.

Now... I guess the main remaining challenge here is that there does not exist option of "may but need not" use padding.
So that's what strict/lenien(-padding) would refer to. And if such existed, there'd be less need to have named variants, if one could use method to create "the other" variant.

I'll create a new issue for that work: it would go in 2.10.

[cowtowncoder]

Created follow-up issue in jackson-core: FasterXML/jackson-core#500