Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unsure how to assign Baseline when using Arguments #2596

Open
JakenVeina opened this issue Jul 24, 2024 · 3 comments
Open

Unsure how to assign Baseline when using Arguments #2596

JakenVeina opened this issue Jul 24, 2024 · 3 comments

Comments

@JakenVeina
Copy link

I've got the following setup for a benchmark fixture:

public class MyBenchmarks
{
    [Params(1, 2, 3)]
    public int SomeParameter { get; set; }
    
    [Benchmark(Baseline = true)]
    public void ScenarioA();

    [Benchmark]
    [Arguments(4)]
    [Arguments(5)]
    public void ScenarioB(int someArgument);
}

I was taken aback when the benchmarks spit out...

Method SomeParameter someArgument Ratio
ScenarioA 1 ? 1.00
ScenarioB 1 4 ?
ScenarioB 1 5 ?
ScenarioA 2 ? 1.00
ScenarioB 2 4 ?
ScenarioB 2 5 ?
ScenarioA 3 ? 1.00
ScenarioB 3 4 ?
ScenarioB 3 5 ?

What I was going for, and had before I added someArgument was...

Method SomeParameter someArgument Ratio
ScenarioA 1 ? 1.00
ScenarioB 1 4 0.50
ScenarioB 1 5 0.50
ScenarioA 2 ? 1.00
ScenarioB 2 4 0.50
ScenarioB 2 5 0.50
ScenarioA 3 ? 1.00
ScenarioB 3 4 0.50
ScenarioB 3 5 0.50

Makes sense, in retrospect, I suppose. I could achieve this by just splitting a ScenarioC method off of ScenarioB, and getting rid of someParameter. Is there a simpler way to get these to group up?

@timcassell
Copy link
Collaborator

timcassell commented Jul 24, 2024

Benchmarks are not comparable with differing argument/param values. See #2584.

This is definitely an area that can be improved upon.

@JakenVeina
Copy link
Author

What would be the best workaround? Just de-parameterizing that method, and making a second one instead?

@timcassell
Copy link
Collaborator

What would be the best workaround? Just de-parameterizing that method, and making a second one instead?

Yes, that should do the trick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants