Publish test execution events to the test ApplicationContext [SPR-13916]

[spring-projects-issues]

Frank Scheffler opened SPR-13916 and commented

We are using Mockito mocks quite often throughout our Spring-based integration tests, which leaves us with the problem of resetting them before/after each test method. While we can use some globally defined custom TestExecutionListener to reset all the mocks found within the ApplicationContext, it would be more practical to be able to define specific event-listeners within the application context itself. This would allow for more fine-grained control over which beans (i.e. mocks) should be reset.

Deliverables

• Implement EventPublishingTestExecutionListener.
• Implement abstract TestContextEvent and all concrete subclasses.
• Implement composed @EventListener annotations for all events.
• Document exception handling and async support in Javadoc.
• Document in the Testing chapter of the reference manual.

---

Issue Links:

• Publish TestStarted and TestCompleted events in the TestContext framework [SPR-8710] #13352 Publish TestStarted and TestCompleted events in the TestContext framework
• Introduce support for test-scoped beans in the TestContext framework [SPR-14034] #18606 Introduce support for test-scoped beans in the TestContext framework

2 votes, 6 watchers

[spring-projects-issues]

Frank Scheffler commented

In fact, one solution would be to provide an additional EventPublishingTestExecutionListener, so users could plug-in the event publishing behavior, if they are in need for it.

TestExecutionListener could look like this, backed by an event class hierarchy reflecting the testcontext lifecycle:

public class EventPublishingTestExecutionListener extends AbstractTestExecutionListener {

  @Override
  public void afterTestMethod(TestContext testContext) throws Exception {
    testContext.getApplicationContext().publishEvent(new AfterMethodTestExecutionEvent(testContext));
  }
  
}

Within the test configuration code you could then, for instance, reset the mocks in question, e.g. by:

@Configuration
public class MyMockConfiguration implements ApplicationListener<AfterMethodTestExecutionEvent> {

   @Bean
   public MyService myService() {
      return Mockito.mock(MyService.class);
   }

   @Override
    public void onApplicationEvent(AfterMethodTestExecutionEvent event) {
      Mockito.reset(myService());
    }
}

[spring-projects-issues]

Frank Scheffler commented

I have created an implementation draft for further discussions at: maverick1601@f07c34c

The tests work fine and I added EventListener-based meta-annotation to simplify integration into configuration classes and beans, e.g. simply annotating methods with @BeforeTestMethod, which is compliant with JUnit life-cycle annotations.

I am, however, currently unsure about the following:

1. What order should the ApplicationEventTestExecutionListener have?
2. Should it be added to the list of default test execution listeners, as i did?
3. Is the eager initialization of the ApplicationContext as part of the listener a problem or not?

Finally I would like to summarize the benefits of this approach and why we need it:

1. Using ApplicationEvents for signaling the test life-cycle enables us to interact with the test framework from inside our Spring configuration classes.
2. We can therefore more concisely prepare or clean up mocks that are beans within the application context. This is inherited behavior for all test cases using such configurations, as compared to copying standard @Before methods around.
3. Event consumption is much more flexible with respect to profiles etc., while TestExecutionListeners have to detect, what needs to be done based on the TestContext only.
4. Moreover, we will be able to reduce the amount of TestExecutionListeners in our case, because the logic can be refactored into event-consuming methods.
5. Being framework agnostic enables us to integrate this with FitNesse and other testing frameworks more easily.

[spring-projects-issues]

Frank Scheffler commented

Any feedback on this or the related question on Google groups (https://groups.google.com/forum/#!topic/spring-framework-contrib/crdJWZk3Wcs). I would like to proceed on finishing the code, if this is considered a useful enhancement to the TCF.

[spring-projects-issues]

Sam Brannen commented

Frank Scheffler,

Thanks for the proposal and draft implementation.

At a quick glance, it looks reasonably sound, though I haven't yet had time to look at it in great detail.

With regard to the automatic registration of ApplicationEventTestExecutionListener, I'm not so sure that's a good idea, especially if the ApplicationContext is always eagerly initialized. Most listeners (except DITEL) tend not to eagerly load a context (or do anything for that matter) if unnecessary. In other words, we wouldn't want to attempt to fire events in a context that will never exist for a given test class.

I'll move this issue to the 4.3 backlog with the hope of finding more time to devote to it as we move toward 4.3 GA.

Regards,

Sam

[spring-projects-issues]

Sam Brannen commented

Please note that this issue is very closely related to and potentially supersedes #13352.

[spring-projects-issues]

Phil Webb commented

You might be interested in spring-projects/spring-boot#5042 which is looking to add better Mockito support to Spring Boot.

[spring-projects-issues]

Frank Scheffler commented

We have implemented this at a customer and enhanced the listener to publish events to the fixture, i.e. the test class, itself as well, if implemented as ApplicationListener or via @EventListener (meta)-annotations. This improves test framework agnosticism, making it easier for us to switch test code from JUnit to TestNG or FitNesse (we have implemented our own Spring-Test integration for that one), later on, because we rely solely on our own meta-annotations instead of JUnit-specific ones.

[sbrannen]

This issue supersedes gh-13352.

[sbrannen]

Related discussion on StackOverflow

[sbrannen]

Hi @maverick1601,

I would like to proceed on finishing the code, if this is considered a useful enhancement to the TCF.

I took a glance at your draft, and it looks very similar to solutions I've implemented for clients.

Are you still interested in submitting a PR?

If so, feel free to simply submit your draft, and I'll take it from there. Otherwise, I'll likely just implement it from scratch.

I'm hoping to get this into Spring 5.2 M1, so I'd be grateful for prompt feedback.

Thanks,

Sam

[maverick1601]

I would do so. Since the issue is about 3 yrs old, I don't know how much I would have to rewrite. Back then we could not fully agree on the fact, that such a TestExecutionListener would be included in the list of default listeners, especially with respect to the fact that such an event publishing TestExecutionListener would trigger context initialization. What is your opinion on that?

[sbrannen]

I would do so.

I like to give credit where credit is due, so that would be great!

Can you provide an estimate of when you think you'll able to do that?

Since the issue is about 3 yrs old, I don't know how much I would have to rewrite.

I don't think you'd have to rewrite much of anything. There are two new methods in the TestExecutionListener API, but I assume you'll make quick work of that. If you don't have time, don't worry: I can add the support for the new methods.

Back then we could not fully agree on the fact, that such a TestExecutionListener would be included in the list of default listeners, especially with respect to the fact that such an event publishing TestExecutionListener would trigger context initialization. What is your opinion on that?

I'm currently inclined not to register this new listener by default. I think it's perhaps best to keep it as an opt-in feature.

In any case, that can be decided later or changed before 5.2 GA is released.

My main objective right now is to get the feature in place in 5.2 M1 so that people can experiment with it and provide feedback.

Cheers,

Sam

p.s. don't worry so much about the documentation at this point unless you have time. If you don't have time, I'll pick up the documentation task after merging in your work.

[maverick1601]

Okay, I took a look at it and it seems the main work is to rebase my previous proposal against master branch and to cover those two new life-cycle methods. I'd say this is done by the end of this week.

[sbrannen]

Sounds good.

Looking forward to your PR!

[maverick1601]

@sbrannen See: #22477

[sbrannen]

@maverick1601, thanks for the PR.

I'll take a look.

[sbrannen]

The work for this issue has been merged into master in 34fee86 and refined in 2d6624d.

Reopening in order to add documentation to the Testing chapter of the reference manual. See new Deliverables section of this issue.

[sbrannen]

Update: commit e272e2e introduced support for conditional event processing in the built-in event listener annotations.

[sbrannen]

Update:

This new feature set is now documented in the Reference Manual.

In addition, the EventPublishingTestExecutionListener is now registered by default for 5.2 M1.

[sbrannen]

Reopening to sort out issues with having the new EventPublishingTestExecutionListener registered by default and having an order of HIGHEST_PRECEDENCE.

The current state of affairs leads to undesirable side effects, such as:

• @DirtiesContext no longer works as expected: a context that is dirtied (and properly closed) after a test method or test class will be magically recreated by the EventPublishingTestExecutionListener, thus effectively breaking @DirtiesContext support.
• The test's ApplicationContext will be eagerly loaded before any other TestExecutionListener and before any before all lifecycle methods in test classes (e.g., JUnit Jupiter's @BeforeAll and JUnit 4's @BeforeClass).

[sbrannen]

Update: the implementation of this feature has been revised. See 353e092 for details.