MockMvc no longer handles UTF-8 characters

[momega]

I wrote simple controller:

@RestController
public class TestController {

    @GetMapping("/api")
    public Map<String, String> simpleTest() {
        // here is the text with special characters, all in UTF-8
        return Collections.singletonMap("test", "Příliš žluťoučký kůň úpěl ďábelské ódy");
    }
}

Also very simple tutorial-like test:

    @Before
    public void setup() {
        this.mockMvc = MockMvcBuilders.webAppContextSetup(this.wac).build();
    }

    @Test
    public void test() throws Exception {
        this.mockMvc.perform(get("/api"))
                .andDo(print())
                .andExpect(jsonPath("$.test").value("Příliš žluťoučký kůň úpěl ďábelské ódy"));
    }

The test fails in version 5.2.0.M3 now. Test passes in the previous milestone version 5.2.0.M2 and earlier versions.

The issue is somehow related to deprecation of org.springframework.http.MediaType#APPLICATION_JSON_UTF8_VALUE.

Of course it is possible to fix the test by adding accept but I rather do not do that, because APPLICATION_JSON_UTF8_VALUE is deprecated now.

.accept(MediaType.APPLICATION_JSON_UTF8_VALUE)

I have also prepared simple maven project to simulate the problem.
springutf8.zip

It seems to me, that MockMvc client does not handle UTF-8 characters well, like other modern browsers.

[sbrannen]

@sdeleuze, it appears that gh-22788 introduced a regression in MockMvc. Do you have time to look into this?

[sdeleuze]

@sbrannen @rstoyanchev The situation here is the following.

As expected, the JSON data is written as UTF-8 by Jackson converter/encoder regardless of the ISO-8859-1 default Servlet encoding which is only used when the String serialization is performed by the Servlet container itself.

On MockMVC side, by default MockHttpServletResponse#characterEncoding is set to ISO-8859-1 and used for writing and reading String. For writing text, I think that's relevant since it will mimic the Servlet Container behavior. But for reading, I think current behavior can lead to surprising results like the one raised in this issue.

In order to get a more predictable behavior, I think my proposal would be to change:

public String getContentAsString() throws UnsupportedEncodingException {
	return (this.characterEncoding != null ?
			this.content.toString(this.characterEncoding) : this.content.toString());
}

to

public String getContentAsString() throws UnsupportedEncodingException {
	return (this.characterEncoding != null && this.charset ?
			this.content.toString(this.characterEncoding) : this.content.toString());
}

in order to only use characterEncoding in getContentAsString when it has been set explicitly.

Any thoughts?

[rstoyanchev]

What about making a change in JsonPathResultMatchers#getContent instead to check if response.isCharset() is set and if not then default to UTF-8 for extracting the content. That 's a better default for JSON vs the general one for the Servlet API and matches the default assumption of the Jackson message converter. Given this was triggered by a change to the JSON media type it seems a more targeted fix too.

[sdeleuze]

I am not sure because response.getContentAsString() is also pretty commonly used, and I am not sure this default to ISO-8859-1 makes sense on reading the mocked response which matches conceptually with the clients like an http library or the browser which have not this unusual (outside Servlet world) default charset. So I think I am still in favor of my previous proposal.

[rstoyanchev]

@sdeleuze I think MockHttpServletResponse is too level and widely used, and for a very long time. This change is likely to break tests. We should probably discuss with the team. What I like about a change in JsonPathResultMatchers#getContent is that we can afford to make a stronger assumption and it will address very specifically the cause for the regression.

[sgybas]

@sdeleuze Thanks for fixing this bug. However, the same bug exists for .andExpect(content().json(expectedJson)) which still uses the default charset in response.getContentAsString() as of Spring 5.2.0.RC2.

Should I open a new issue and provide a project to reproduce the problem?

[sdeleuze]

yes please

[mobject]

I had to read from getResponse().getContentAsByteArray(); and wrap the byte[] into String.