Mocking with var redefinition considered harmful

(Disclaimer: “Mocking” is used as an umbrella term for faking/mocking/stubbing in this post.)

Many Clojure developers tend to rely on redefining vars for mocking dependencies during testing. While the approach works for very simple cases, it breaks down as soon as the needs become complex. In this post I want to list several kinds of pitfalls with redefining vars for mocking.

The optional direct-linking feature introduced in Clojure 1.8 may prevent var redefinition. For the purpose of this post we assume direct-linking is not enabled during testing.

What exactly is a var?

For practical purposes, a var is a thread-safe entry in a namespace that may be bound (and rebound) to a value. Since it works like a thread-safe mutable variable, both pros and cons of mutability apply to it. For technical details on vars, refer here.

Consider the following code snippet:

(defn fetch-each-item
  "Return item details, or nil when not found."
  [item-id]
  (db/sql-fetch conn-pool (normalize item-id)))

(defn fetch-items
  "Given item IDs, return a map of item IDs to item details.
  Items not found are omitted from the result."
  [item-ids]
  (->> item-ids
    (map #(vector % (fetch-each-item %)))
    (filter second)
    (apply conj {})))

Let us say, for a test case we want a scenario where certain item IDs are not found. This can be simulated by altering the definition of fetch-items for the scope of the test.

(with-redefs [fetch-items (fn [ids] mock-result-without-certain-IDs)]
  ;; call test code that calls fetch-items
  ...) 

Here, fetch-items is a dependency that the code under test would invoke. This is typically how dependencies are mocked out using var redefinition. This works fine for simple and straightforward mocking needs as in the example above.

Where does it break down?

There are several kinds of issues with redefining vars, mostly associated with the site of redefinition not being isolated together with the site of access. This fundamental problem manifests in the following ways, often combining with one another to form compounded problems.

Diamond dependency

Consider there are four defn’ed functions foo, bar, baz and qux in the following dependency order:

Assuming that a test case requires two different behavior from qux (once called from bar, then later called from baz), how can you mock qux in such a way that bar->qux call succeeds and baz->qux fails? The quick-and-dirty solution to this problem is easy – just add a level of indirection by introducing bar-qux and baz-qux functions to actually call qux:

Instead of mocking qux, now we can mock each of bar-qux and baz-qux. This gets the job done, but introduces phony functions just to satisfy the mocking requirement. This is also cumbersome and difficult to scale for a non-trivial number of such cases.

Diamond dependency example

Let me explain this with a made-up example. Let us say we want to test a function inventory-dashboard that returns inventory information to populate a dashboard display. This function in turn calls main-panel and side-panel functions to find inventory information for configured categories. The panel functions call a function find-inventory-level that connects to several backend systems to fetch the inventory levels of various vendors. The connectivity looks like the the diagram below:

The main-panel is configured for various categories of items whereas the side-panel is configured particularly for consumable items. When over 50% of the consumable stock crosses 80% of the respective shelf life, the side panel is supposed to change structure of the data to reflect the risk in consumable inventory, which leads to some additional changes in the overall dashboard data. To simulate this scenario, how do we mock find-inventory-level function to work normally for main-panel and return at-risk consumable inventory information for the side-panel?

Concurrency

This problem is relatively better known than the diamond dependency problem. Since var redefinition has global visibility, a var redefined by one thread is instantly seen by all other threads in the system. The redefinition cannot be isolated to a thread, or just to a group of threads participating in a test case. Consider the illustration below:

The above diagram illustrates the problem of concurrently running two tests, both of which redefine the same var for the purpose of mocking. Test A mutates the var first, and also restores the old value while Test B is still in progress. The redefinition from two separate threads highlights the issue of non-transactional mutation leading to race condition in the tests and corruption of the var's root binding.

The biggest ill effect of this problem is you cannot concurrently run tests in a predictable manner no matter how powerful your computer is. As a consequence, if your builds include running tests as a step, then the build times will grow proportionally with respect to count and duration of tests.
Some may suggest that dynamic vars can help with the issue. However, since dynamic vars have thread-local visibility, they cannot be used where the execution has to span across more than one thread.

Asynchronous execution

Consider the following code snippet:

(let [items (with-redefs [fetch-items mock-fetch-items]
              (future (home-items user-id)))]
  ;; do something with (deref items)
  ;; test code here
  ...)

This is a variation of the concurrency problem, but with a twist that the var access happens in a different thread than the thread that redefines/restores the var. The future call submits the body of code for execution in a new thread, and immediately returns a future object that may be deref'ed to determine the result. Once the future call returns, there is no guarantee whether with-redefs in the caller thread or the future thread would complete first, which is a race condition. It is quite possible that the var redefinition would be restored before the future call can finish. This is going to lead to unpredictable, inconsistent results.

Laziness

Lazy sequences have the same fundamental problem as asynchronous execution and concurrency issue -- var redefinition and var access may not be isolated together. Consider the following code snippet:

(let [items (with-redefs [fetch-items mock-fetch-items]
              (map order-items orders))]
  ;; "items" lazy/unevaluated here, var redefinition already restored
  ;; test code here
  ...)

In this example, the order in which var access and restoration take place is not correct. By the time the var is accessed (i.e. when the lazy sequence is evaluated in test code) the var's original root binding is already restored. Hence, the purpose of mocking is totally defeated here. This is not a problem with laziness per se, but rather the way it is used in conjunction with var redefinition. However, it is an easy pitfall for an unsuspecting developer.

The Path Forward

Var redefinition is indeed risky for mocking dependencies. However, many projects may choose to just test the happy path and test bad input, hence they may not need mocking at all. Projects that need to simulate conditions are going to need dependencies mocked and that is where var redefinition could be the spoilsport.

Dependency passing style

Fortunately Clojure, being a functional language, already has a way to swap out dependencies using higher-order functions. In fact, higher-order function is just a special case of "dependency passing style", wherein the dependencies are functions the callee can invoke. One can easily use this style to pass in values, configuration data, protocol implementations, functions, Java objects and whatnot. Once the dependency is passed as argument to a function, it is captured in the callee function isolated from other code that may refer to the same dependency – the dependency is no more tied to a var that must be mutated for others to see. In one fell swoop, passing dependencies takes care of all the problems we discussed above about var redefinition.

The Dependency inversion principle states, "Depend upon Abstractions. Do not depend upon concretions." An inside-out design, where a function explicitly accepts the dependencies as arguments, naturally allows dependencies to be swapped out by the caller. The dependencies passed during tests could simply be different from the ones in production. See the code example we discussed at the beginning of this post, reimplemented using higher order functions as follows:

(defn fetch-each-item
  "Return item details, or nil when not found."
  [db-fetcher item-id]
  (db-fetcher (normalize item-id)))

(defn fetch-items
  "Given item IDs, return a map of item IDs to item details.
  Items not found are omitted from the result."
  [item-fetcher item-ids]
  (->> item-ids
    (map #(vector % (item-fetcher %)))
    (filter second)
    (apply conj {})))

The function fetch-each-item must be partially applied with db-fetcher before it is passed to fetch-items as argument. Similarly, fetch-items must be partially applied with item-fetcher before it can be used as arity-1 function elsewhere.

Challenges with Dependency passing style

Dependency passing style poses a different set of problems than directly calling namespace functions, such as:

1. Cascading construction of dependencies to be passed to various higher order functions
2. Lack of editor support to navigate to the dependency source/implementation

It is technically possible to solve the first issue by writing code to create dependencies/partially applied functions. However, there are libraries like DIME (full disclosure: I am the author), Plumatic Graph and Component that to an extent take care of tracking/automating interdependencies without redefining vars. The second issue could be very unsettling for developers who are accustomed to editor/tooling support for navigating to referenced functions, e.g. the Emacs M-. (meta dot) feature provided by CIDER. I hope this can be mitigated with augmented tooling.

Though var redefinition has serious challenges with mocking, we have an alternative in the form of dependency passing style with a different set of tradeoffs than var redefinition.

This blog post is on Hacker News and Reddit. Please let me know your feedback in the comments. You may also like to follow me on Twitter.

(Thanks to Vijay Mathew, Sreenath N and Jerry Jacob for reviewing drafts of this post.)