Rexsilient.CircuitBreaker (Rexsilient v1.0.0)

Circuit breakers stop calls to a dependency that keeps failing.

When a dependency fails repeatedly, calling it again usually fails too: it wastes the caller's time and adds load to a service that is already struggling. A circuit breaker watches the calls made to the dependency and, after too many failures, rejects them immediately until the dependency has had time to recover.

A circuit breaker is declared in a module that uses Rexsilient:

defmodule MyApp.Resilience do
  use Rexsilient

  alias Rexsilient.CircuitBreaker.{Recover, Strategy}

  circuit_breaker :payments do
    strategy Strategy.Count, threshold: 5
    recover Recover.BasicTimeout,
      reset_timeout: :timer.seconds(30),
      probing_timeout: :timer.seconds(10)
  end
end

and used with @circuit_breaker :payments from Rexsilient.Decorator:

defmodule MyApp.Payments do
  use Rexsilient.Decorator

  @circuit_breaker :payments
  def charge(order) do
    PaymentsClient.charge(order)
  end
end

or, without a decorator, with run/3:

case Rexsilient.CircuitBreaker.run(:payments, fn -> PaymentsClient.charge(order) end) do
  {:ok, receipt} -> {:ok, receipt}
  {:error, :circuit_open} -> {:error, :payments_unavailable}
  {:error, reason} -> {:error, reason}
end

States

A breaker is always in one of four states, returned by status/1:

:closed - calls go through. Their results are passed to the breaker's strategy, which decides when it opens.
:open - calls are rejected without reaching the dependency. After the time set by its recover module, the breaker becomes half-open.
:half_open - the next call is let through as a probe, to test whether the dependency has recovered.
:probing - the probe is running. Every other call is rejected. If the probe succeeds, the breaker closes and its strategy is reset. If it fails, the breaker opens again. If it runs longer than the time set by the recover module, the breaker becomes half-open again and the slow probe's result is ignored.

Exactly one caller becomes the probe, even when many arrive at once.

Failures

Whether a call counts as a failure is decided by the classify/1 and classify/2 functions of the module that declares the breaker, or by the functions named with classify_with in the breaker's declaration. By default, a call fails when the function returns :error or {:error, _}, or when it raises, throws or exits; every other return value counts as a success. See Rexsilient.Classify to change it.

Starting

Breakers are registered by a single manager process, started by Rexsilient in your supervision tree with the modules that declare them:

children = [
  {Rexsilient, modules: [MyApp.Resilience]}
]

Checking a breaker does not involve the manager: it is a single atomic read in the calling process, so it scales with the number of schedulers. The manager only handles state changes and timers. If it crashes, every breaker restarts closed, so a failure of the breaker itself never blocks traffic.

Summary

Types

name()

The name a breaker is declared with.

state_counter()

Identifies the state a call was allowed in. Returned by acquire/1 and passed unchanged to release/4.

status()

The state of a breaker; see States.

Functions

acquire(name)

Asks the breaker name whether a call may go through.

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

force_close(name)

Forces the breaker name closed, and resets its strategy.

force_open(name)

Forces the breaker name open.

release(name, result, state_counter, outcome \\ nil)

Records the result of a call allowed by acquire/1.

run(name, func, opts \\ [])

Calls func through the breaker name.

start_link(opts)

Starts the breaker manager and registers the declared breakers.

status(name)

Returns the current state of the breaker name.

Types

name()

@type name() :: atom()

The name a breaker is declared with.

state_counter()

@type state_counter() :: non_neg_integer()

Identifies the state a call was allowed in. Returned by acquire/1 and passed unchanged to release/4.

status()

@type status() :: :closed | :open | :half_open | :probing

The state of a breaker; see States.

Functions

acquire(name)

@spec acquire(name()) :: {:ok, state_counter()} | {:error, :circuit_open}

Asks the breaker name whether a call may go through.

Returns {:ok, state_counter} if the call is allowed, or {:error, :circuit_open} if it is rejected. When the breaker is half-open, the caller that gets {:ok, _} becomes the probe.

Together with release/4, it is the building block of run/3. Use them directly when run/3 does not fit, for example when what counts as a failure depends on the call site rather than on the breaker's classify/1. After an allowed call, pass its result, state_counter and outcome to release/4.

They are only the breaker's state machine, without anything run/3 adds around it: no classify, no catching of raises, throws and exits, no :on_open, and no :before, :after or :rejected notifications. Only the state changes caused by release/4 are notified.

Raises ArgumentError if no breaker is registered under name.

Examples

Counting only server errors as failures:

case Rexsilient.CircuitBreaker.acquire(:payments) do
  {:ok, state_counter} ->
    try do
      PaymentsClient.charge(order)
    catch
      kind, reason ->
        Rexsilient.CircuitBreaker.release(:payments, :failure, state_counter, {kind, reason})
        :erlang.raise(kind, reason, __STACKTRACE__)
    else
      %{status: status} = response when status >= 500 ->
        Rexsilient.CircuitBreaker.release(:payments, :failure, state_counter, {:value, response})
        response

      response ->
        Rexsilient.CircuitBreaker.release(:payments, :success, state_counter, {:value, response})
        response
    end

  {:error, :circuit_open} ->
    {:error, :payments_unavailable}
end

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

force_close(name)

@spec force_close(name()) :: :ok

Forces the breaker name closed, and resets its strategy.

Works from any state, including after force_open/1. Timers armed before it are ignored, and so are the results of calls allowed before it.

Raises ArgumentError if no breaker is registered under name.

Examples

Rexsilient.CircuitBreaker.force_close(:payments)
Rexsilient.CircuitBreaker.status(:payments)
#=> :closed

force_open(name)

@spec force_open(name()) :: :ok

Forces the breaker name open.

Calls are rejected until force_close/1 is called: unlike a trip, no timer makes the breaker half-open, and no probe is let through. Meant for operators, for example to stop traffic to a dependency under maintenance.

Calls already allowed, and probes already running, finish normally, but their results are ignored.

Raises ArgumentError if no breaker is registered under name.

Examples

Rexsilient.CircuitBreaker.force_open(:payments)
Rexsilient.CircuitBreaker.status(:payments)
#=> :open

release(name, result, state_counter, outcome \\ nil)

@spec release(
  name(),
  Rexsilient.Classify.result(),
  state_counter(),
  Rexsilient.Classify.outcome() | nil
) :: :ok

Records the result of a call allowed by acquire/1.

result is :success or :failure, as decided by the caller, and state_counter is the value returned by acquire/1; pass it unchanged. Results recorded with a counter from a previous state are ignored, so a slow call cannot affect a breaker that has since changed state.

outcome is what the call produced, {:value, value} or {kind, reason}. When the failure opens the breaker, it is passed to the recover module, so it can decide how long to stay open from the error itself, for example from a Retry-After header. It is optional; without it, the recover module gets nil.

Release every acquire

Call it exactly once for every successful acquire/1, including when the call raises. If the call was a probe and is not released in time, the breaker stays in :probing, rejecting calls, until the probing timeout set by its recover module makes it half-open again. See acquire/1 for an example.

The result is recorded in the calling process, so a call that opens or closes the breaker is reflected by status/1 as soon as release/4 returns. Only the timers that make a breaker half-open run in the manager.

run(name, func, opts \\ [])

@spec run(name(), func :: (-> term()), opts :: [{:on_open, term()}]) :: term()

Calls func through the breaker name.

If the breaker allows the call, func is invoked and its result is returned unchanged. The result is then recorded as a success or a failure (see Failures). If func raises, throws or exits, the outcome is recorded the same way and the error is re-raised unchanged.

If the breaker rejects the call, func is not invoked and {:error, :circuit_open} is returned, unless :on_open says otherwise. To compute another result instead, wrap the call in a Rexsilient.Fallback, which always replaces {:error, :circuit_open}.

Raises ArgumentError if no breaker is registered under name.

Options

:on_open - the value returned when the call is rejected, instead of {:error, :circuit_open}.

Examples

Rexsilient.CircuitBreaker.run(:payments, fn -> Payments.charge(order) end)

Rexsilient.CircuitBreaker.run(:payments, fn -> Payments.charge(order) end,
  on_open: {:error, :unavailable}
)

start_link(opts)

@spec start_link(opts :: [{:modules, [module()]}]) :: GenServer.on_start()

Starts the breaker manager and registers the declared breakers.

Usually started by Rexsilient, as shown in the module documentation. Only one manager runs per node; it is registered under this module's name.

Options

:modules - the modules that declare breakers with Rexsilient. Required. The manager fails to start, with an ArgumentError as the reason, if two breakers share a name or a breaker's options are invalid.

status(name)

@spec status(name()) :: status()

Returns the current state of the breaker name.

Meant for inspection and debugging; see States for what each state means. Raises ArgumentError if no breaker is registered under name.

Examples

Rexsilient.CircuitBreaker.status(:payments)
#=> :closed