Scientist

A library for carefully refactoring critical paths in your elixir appplication.

This is an elixir clone of the ruby gem scientist.

Wait, why be a Scientist?

Suppose you decide to add a new caching layer to your production application, while still being able to make the same guarantees about your data. By processing your new caching strategy through Scientist you'll be able to

• Run both the old and the new code in random order
• Monitor timing for each strategy
• Compare their results and find any mismatches
• Rescue and report any exceptions thrown in your new code
• Publish all of this information in a manner of your choosing.

Externally, a Scientist experiment behaves exactly the same as its control block, returning the value of the control as well as re-raising any of its exceptions.

defmodule MyPhoenixApp.UserController do
  use Scientist
  use MyPhoenixApp.Web, :controller

  alias MyPhoenixApp.User
  alias MyPhoenixApp.Repo

  plug :action

  def get_user(id) do
    # Let's get down to business, with science!
    science "New ETS cache for users" do
      control do: Repo.get(User, id)
      candidate do: MyETSCache.get(User, id)
    end
  end

  # ... other controller logic
end

Rolling your own Experiment

Experiments aren't useful on their own. You need to be able to report their results and control their execution. To define your own custom experiment, you need to use Scientist.Experiment and implement a few callbacks.

defmodule MyCustomExperiment do
  use Scientist.Experiment

  # Required callbacks: publish/1, enabled?/0
  # See "Enabling Experiments" and "Publishing" below
  defdelegate [enabled?(), publish(result)], to: Scientist.Default

  # Optional callbacks
  # Default name
  def default_name, do: "My custom experiment"
  # Default context, see "Need some context?" below
  def default_context, do: %{}
end

Then when using Scientist you can specify your custom experiment to be used instead of Scientist.Default:

defmodule UserController do
  use Scientist, experiment: MyCustomExperiment

  # Now let's get some science done!
end

Now all calls to science will use MyCustomExperiment.new/2 for setup.

Custom Comparison

Out of the box, Scientist will compare observed values with Kernel.==/2 to see if they match. You can override this with a comparison block.

def get_user(id) do
  science "New ETS cache for users" do
    control do: Repo.get(User, id)
    candidate do: MyETSCache.get(User, id)

    # We only care if the user's status is updated.
    compare(%{status: sa}, %{status: sb}) do
      sa == sb
    end
  end
end

Need some context?

Sometimes, you need more information about the environment when checking the results of your experiment. In these cases, you can pass a map of values to your experiment before it's run:

def get_user(id) do
  # Perhaps the cache is filling too quickly
  c = %{cache_size: MyETSCache.size(User)}
  science "New ETS cache for users", context: <