Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to mock S3's presignGetObject call #1238

Open
hugoncosta opened this issue Feb 29, 2024 · 5 comments
Open

Unable to mock S3's presignGetObject call #1238

hugoncosta opened this issue Feb 29, 2024 · 5 comments
Assignees
Labels
bug This issue is a bug.

Comments

@hugoncosta
Copy link

Describe the bug

I am unable to mock a call to the S3Client's presignGetObject function as part of a unit test.

Expected behavior

The test should run successfully.

Current behavior

The test fails with the following stacktrace

java.lang.IllegalArgumentException:  is not a valid inet host

	at aws.smithy.kotlin.runtime.net.HostKt.hostParseImpl(Host.kt:35)
	at aws.smithy.kotlin.runtime.net.HostKt.access$hostParseImpl(Host.kt:1)
	at aws.smithy.kotlin.runtime.net.Host$Companion.parse(Host.kt:14)
	at aws.smithy.kotlin.runtime.http.operation.OperationEndpointKt$setResolvedEndpoint$1.invoke(OperationEndpoint.kt:70)
	at aws.smithy.kotlin.runtime.http.operation.OperationEndpointKt$setResolvedEndpoint$1.invoke(OperationEndpoint.kt:66)
	at aws.smithy.kotlin.runtime.http.request.HttpRequestBuilderKt.url(HttpRequestBuilder.kt:71)
	at aws.smithy.kotlin.runtime.http.operation.OperationEndpointKt.setResolvedEndpoint(OperationEndpoint.kt:66)
	at aws.smithy.kotlin.runtime.auth.awssigning.PresignerKt.presignRequest(Presigner.kt:35)
	at aws.sdk.kotlin.services.s3.presigners.PresignersKt.presignGetObject(Presigners.kt:52)
	at aws.sdk.kotlin.services.s3.presigners.PresignersKt.presignGetObject$default(Presigners.kt:40)
	at aws.sdk.kotlin.services.s3.presigners.PresignersKt.presignGetObject-exY8QGI(Presigners.kt:30)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1$1.invokeSuspend(S3AccessorTest.kt:74)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1$1.invoke(S3AccessorTest.kt)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1$1.invoke(S3AccessorTest.kt)
	at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2$1.invokeSuspend(RecordedBlockEvaluator.kt:27)
	at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2$1.invoke(RecordedBlockEvaluator.kt)
	at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2$1.invoke(RecordedBlockEvaluator.kt)
	at io.mockk.InternalPlatformDsl$runCoroutine$1.invokeSuspend(InternalPlatformDsl.kt:23)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at io.mockk.InternalPlatformDsl.runCoroutine(InternalPlatformDsl.kt:22)
	at io.mockk.impl.eval.RecordedBlockEvaluator$record$block$2.invoke(RecordedBlockEvaluator.kt:27)
	at io.mockk.impl.eval.RecordedBlockEvaluator$enhanceWithRethrow$1.invoke(RecordedBlockEvaluator.kt:76)
	at io.mockk.impl.recording.JvmAutoHinter.autoHint(JvmAutoHinter.kt:23)
	at io.mockk.impl.eval.RecordedBlockEvaluator.record(RecordedBlockEvaluator.kt:39)
	at io.mockk.impl.eval.EveryBlockEvaluator.every(EveryBlockEvaluator.kt:30)
	at io.mockk.MockKDsl.internalCoEvery(API.kt:100)
	at io.mockk.MockKKt.coEvery(MockK.kt:169)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1.invokeSuspend(S3AccessorTest.kt:74)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1.invoke(S3AccessorTest.kt)
	at com.amazon.networkvalidator.s3.S3AccessorTest$getObjectPresigned$1.invoke(S3AccessorTest.kt)
	at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt$runTest$2$1$1.invokeSuspend(TestBuilders.kt:316)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.test.TestDispatcher.processEvent$kotlinx_coroutines_test(TestDispatcher.kt:24)
	at kotlinx.coroutines.test.TestCoroutineScheduler.tryRunNextTaskUnless$kotlinx_coroutines_test(TestCoroutineScheduler.kt:99)
	at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt$runTest$2$1$workRunner$1.invokeSuspend(TestBuilders.kt:322)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
	at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
	at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
	at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
	at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48)
	at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
	at kotlinx.coroutines.test.TestBuildersJvmKt.createTestResult(TestBuildersJvm.kt:10)
	at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0(TestBuilders.kt:310)
	at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0(Unknown Source)
	at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0(TestBuilders.kt:168)
	at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0(Unknown Source)
	at kotlinx.coroutines.test.TestBuildersKt__TestBuildersKt.runTest-8Mi8wO0$default(TestBuilders.kt:160)
	at kotlinx.coroutines.test.TestBuildersKt.runTest-8Mi8wO0$default(Unknown Source)
	at com.amazon.networkvalidator.s3.S3AccessorTest.getObjectPresigned(S3AccessorTest.kt:73)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
	at org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
	at org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
	at org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
	at org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
	at org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
	at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
	at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
	at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
	at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
	at com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:57)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater$1.execute(IdeaTestRunner.java:38)
	at com.intellij.rt.execution.junit.TestsRepeater.repeat(TestsRepeater.java:11)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:35)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:232)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:55)

Steps to Reproduce

@Disabled
@Test
fun getObjectPresigned_exception() = runTest {
    coEvery { client.presignGetObject(GET_OBJECT_REQUEST, any()) } throws S3Exception("Error")
    assertThrows<IllegalStateException> { s3Accessor.getObjectPresigned(S3BUCKET, KEY) }
}

the function it calls is the following

class S3Accessor(
    private val client: S3Client = S3Client { region = "us-east-1" }
) {

suspend fun getObjectPresigned(
        s3Bucket: String,
        keyName: String
    ): String {
        val request = GetObjectRequest {
            bucket = s3Bucket
            key = keyName
        }

        return try {
            client.presignGetObject(request, 6.hours).url.toString()
        } catch (e: S3Exception) {
            log.error(e)
            error("There was an issue creating the presigned link for $keyName from ${s3Bucket}: ${e.message}")
        }
    }    
}

Possible Solution

No response

Context

I am trying to create a unit test to ensure that if an error is thrown, it is thrown captured as I expect it. I am also facing the issue if I want to mock that the function returns some ByteStream.

AWS Kotlin SDK version used

1.0.67 (?)

Platform (JVM/JS/Native)

JVM (JDK17)

Operating System and version

macOS Sonoma 14.3.1

@hugoncosta hugoncosta added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Feb 29, 2024
@lauzadis
Copy link
Member

lauzadis commented Mar 14, 2024

Thanks for the report! I am able to replicate the failure. The reason it throws this exception is because MockK tries to instantiate an empty Host for mocking which then causes a parse failure in the SDK.

...
2024-03-14 16:24:17 TRACE io.mockk.impl.instantiation.AbstractMockFactory:17 - Building proxy for Host hashcode=dde6f87
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ProxyMaker:17 - Class class aws.smithy.kotlin.runtime.net.Host is sealed, will use its subclass class aws.smithy.kotlin.runtime.net.Host$Domain to build proxy
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.transformation.JvmInlineInstrumentation:17 - Retransforming classes aws.smithy.kotlin.runtime.net.Host$Domain, aws.smithy.kotlin.runtime.net.Host
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ProxyMaker:17 - Taking instance of class aws.smithy.kotlin.runtime.net.Host$Domain itself because it is final.
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ProxyMaker:17 - Instantiating proxy for class aws.smithy.kotlin.runtime.net.Host$Domain via instantiator
2024-03-14 16:24:17 TRACE io.mockk.proxy.jvm.ObjenesisInstantiator:17 - Creating new empty instance of class aws.smithy.kotlin.runtime.net.Host$Domain
hostname: 

Note: hostname is an empty string at this point , so parsing can never succeed.

This seems to be an issue with MockK / your configuration of it rather than the SDK. I'm not too familiar with using MockK, is this enough extra information to help resolve your issue?

Also, do you need to use MockK? If not, you could consider writing an interceptor to throw this exception rather than have MockK do it.

@lauzadis lauzadis added response-requested Waiting on additional info and feedback. Will move to 'closing-soon' in 5 days. and removed needs-triage This issue or PR still needs to be triaged. labels Mar 14, 2024
@hugoncosta
Copy link
Author

I didn't include any configuration, and for mocking purposes, I wouldn't expect any needed. Is it not possible to include a default hostname so that MockK doesn't fail? From all the s3 methods, this is the only one that fails.

As to why use MockK, this function fails as well

@Disabled
@Test
fun getObjectPresigned() = runTest {
    coEvery { client.presignGetObject(GET_OBJECT_REQUEST, 6.hours) } returns HTTP_REQUEST
    val output = s3Accessor.getObjectPresigned(S3BUCKET, KEY)
    assertEquals(Url.toString(), output)
}

@github-actions github-actions bot removed the response-requested Waiting on additional info and feedback. Will move to 'closing-soon' in 5 days. label Mar 20, 2024
@ianbotsf ianbotsf added the p2 This is a standard priority issue label Apr 4, 2024
@ianbotsf ianbotsf assigned ianbotsf and unassigned ianbotsf Apr 11, 2024
@ianbotsf
Copy link
Contributor

@hugoncosta I can't see how you're configuring your mocks but all of the presigner methods (e.g., presignGetObject) are extension methods on the service client interface and MockK requires an additional mock set up for extension methods. I suggest adding the following call to your test code before setting the expectation on the extension method:

mockStatic("aws.sdk.kotlin.services.s3.presigners.PresignersKt")

The SDK does not currently provide explicit support for any specific mocking framework. We strive to generally make our code as mockable as possible but each mocking framework will have its own set up, requirements, and likely some scenarios which cannot be mocked. We're always open to suggested improvements to our code which would improve mockability but we cannot promise 100% mockability via any specific framework.

I'm resolving this issue for now. Feel free to open a new issue if you encounter any SDK-specific bugs in your testing.

Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@ianbotsf ianbotsf self-assigned this Apr 11, 2024
@ianbotsf
Copy link
Contributor

ianbotsf commented Sep 3, 2024

In light of #1396 we should re-examine this request and either provide clearer instructions for how to mock presigning operations and/or improve the mockability of presigning code.

@ianbotsf ianbotsf reopened this Sep 3, 2024
@ianbotsf ianbotsf added needs-triage This issue or PR still needs to be triaged. and removed p2 This is a standard priority issue needs-triage This issue or PR still needs to be triaged. labels Sep 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a bug.
Projects
None yet
Development

No branches or pull requests

3 participants