Fix blocking Akka dispatcher thread on WoT skeleton creation

[thjaeckle]

During creating of Things and Features with a given WoT "ThingModel" definition, an API was used which uses CompletableFuture and which does blocking calls (e.g. in order to retrieve the model(s) via HTTP).

This however was directly called via the Akka "dispatcher" thread - so other processing was completely halted, even the Actor system of the things node could be affected by not being able to schedule any actor related work.

This bugfix enables the internal mechanisms of Ditto to be also able to deal with asynchronously provided events to persist - which will offload the blocking operations to the already provided "wot-dispatcher" instead.

[thjaeckle]

@kalinkostashki @alstanchev this is quite important for me.
Could you squeeze in a review and a system test run?

[alstanchev]

Sure will do my best. Will look in to it later today.

[alstanchev]

I have left few comments in the code other then that looks good.
Tests are running currently had one failure in the Things unit tests which i believe is from the thing i have commented. For the sake of time i tried a local fix and am currently waiting on the builds.

[thjaeckle]

@alstanchev I am still working on this as I encountered some more .join() calls which were done on the Akka main dispatcher thread.
I tried to replace them all with CompletionStage based APIs, I am however stuck when persisting in the AbstractPersistenceActor - there I have an open "TODO" to replace the explicit blocking.

Without that however the event is not persisted correctly (in case of WoT based Thing creations) ..

So still working on it :)

I however fixed the PolicyImportsIT#putPolicyWithTooManyImports() test

[thjaeckle]

@alstanchev I am confident that I solved it now ..

I reduced the potential blocking on the Akka dispatcher thread to a minimum - as I did not manage to get the "persist" to work without it.

Let me sleep one night, maybe I have a better idea tomorrow :)

[thjaeckle]

@alstanchev this morning I got rid of blocking on the Akka dispatcher thread completely 😁
With that I would say it is ready to be reviewed.

Sorry that it is that many changes, I completely made the WoT integration async - I don't know why I did not do that when creating it in the first place.. :/

[thjaeckle]

@alstanchev there is one unit test failing on GitHub action build - locally however this work both when starting in the IDE and when building via maven.

As you are the author of this test, maybe do you have a clue why it would fail?
ThingPersistenceActorTest.policyShouldNotBeDeletedOnThingRetrieveAndActorFail

One hint: I changed something in the "rollback created policy logic when thing creation fails" as this was IMO not done correctly:

ditto/internal/utils/persistent-actors/src/main/java/org/eclipse/ditto/internal/utils/persistentactors/AbstractPersistenceSupervisor.java

Line 786 in 7f96a64

.exceptionallyCompose(error -> handleTargetActorAndEnforcerException(transformed, error))

Here I replaced signal with transformed - as an incoming ModifyThing command could be transformed to a CreateThing by the handleOptionalTransformationException method when the Thing did not yet exist before.
In such a case, the rollback of the policy was not working..

As the unit test tests something in that direction, maybe this is related?

Edit: in the last run, the test was successful

[alstanchev]

LGTM ;)