Tuesday, December 6, 2016

Mocking with var redefinition considered harmful

(Disclaimer: “Mocking” is used as an umbrella term for faking/mocking/stubbing in this post.)

Many Clojure developers tend to rely on redefining vars for mocking dependencies during testing. While the approach works for very simple cases, it breaks down as soon as the needs become complex. In this post I want to list several kinds of pitfalls with redefining vars for mocking.

The optional direct-linking feature introduced in Clojure 1.8 may prevent var redefinition. For the purpose of this post we assume direct-linking is not enabled during testing.

What exactly is a var?

For practical purposes, a var is a thread-safe entry in a namespace that may be bound (and rebound) to a value. Since it works like a thread-safe mutable variable, both pros and cons of mutability apply to it. For technical details on vars, refer here.

Consider the following code snippet:
(defn fetch-each-item
  "Return item details, or nil when not found."
  [item-id]
  (db/sql-fetch conn-pool (normalize item-id)))

(defn fetch-items
  "Given item IDs, return a map of item IDs to item details.
  Items not found are omitted from the result."
  [item-ids]
  (->> item-ids
    (map #(vector % (fetch-each-item %)))
    (filter second)
    (apply conj {})))
Let us say, for a test case we want a scenario where certain item IDs are not found. This can be simulated by altering the definition of fetch-items for the scope of the test.
(with-redefs [fetch-items (fn [ids] mock-result-without-certain-IDs)]
  ;; call test code that calls fetch-items
  ...) 
Here, fetch-items is a dependency that the code under test would invoke. This is typically how dependencies are mocked out using var redefinition. This works fine for simple and straightforward mocking needs as in the example above.

Where does it break down?

There are several kinds of issues with redefining vars, mostly associated with the site of redefinition not being isolated together with the site of access. This fundamental problem manifests in the following ways, often combining with one another to form compounded problems.

Diamond dependency

Consider there are four defn’ed functions foo, bar, baz and qux in the following dependency order:


Assuming that a test case requires two different behavior from qux (once called from bar, then later called from baz), how can you mock qux in such a way that bar->qux call succeeds and baz->qux fails? The quick-and-dirty solution to this problem is easy – just add a level of indirection by introducing bar-qux and baz-qux functions to actually call qux:


Instead of mocking qux, now we can mock each of bar-qux and baz-qux. This gets the job done, but introduces phony functions just to satisfy the mocking requirement. This is also cumbersome and difficult to scale for a non-trivial number of such cases.

Diamond dependency example

Let me explain this with a made-up example. Let us say we want to test a function inventory-dashboard that returns inventory information to populate a dashboard display. This function in turn calls main-panel and side-panel functions to find inventory information for configured categories. The panel functions call a function find-inventory-level that connects to several backend systems to fetch the inventory levels of various vendors. The connectivity looks like the the diagram below:


The main-panel is configured for various categories of items whereas the side-panel is configured particularly for consumable items. When over 50% of the consumable stock crosses 80% of the respective shelf life, the side panel is supposed to change structure of the data to reflect the risk in consumable inventory, which leads to some additional changes in the overall dashboard data. To simulate this scenario, how do we mock find-inventory-level function to work normally for main-panel and return at-risk consumable inventory information for the side-panel?

Concurrency

This problem is relatively better known than the diamond dependency problem. Since var redefinition has global visibility, a var redefined by one thread is instantly seen by all other threads in the system. The redefinition cannot be isolated to a thread, or just to a group of threads participating in a test case. Consider the illustration below:


The above diagram illustrates the problem of concurrently running two tests, both of which redefine the same var for the purpose of mocking. Test A mutates the var first, and also restores the old value while Test B is still in progress. The redefinition from two separate threads highlights the issue of non-transactional mutation leading to race condition in the tests and corruption of the var's root binding.

The biggest ill effect of this problem is you cannot concurrently run tests in a predictable manner no matter how powerful your computer is. As a consequence, if your builds include running tests as a step, then the build times will grow proportionally with respect to count and duration of tests.
Some may suggest that dynamic vars can help with the issue. However, since dynamic vars have thread-local visibility, they cannot be used where the execution has to span across more than one thread.

Asynchronous execution

Consider the following code snippet:
(let [items (with-redefs [fetch-items mock-fetch-items]
              (future (home-items user-id)))]
  ;; do something with (deref items)
  ;; test code here
  ...)
This is a variation of the concurrency problem, but with a twist that the var access happens in a different thread than the thread that redefines/restores the var. The future call submits the body of code for execution in a new thread, and immediately returns a future object that may be deref'ed to determine the result. Once the future call returns, there is no guarantee whether with-redefs in the caller thread or the future thread would complete first, which is a race condition. It is quite possible that the var redefinition would be restored before the future call can finish. This is going to lead to unpredictable, inconsistent results.

Laziness

Lazy sequences have the same fundamental problem as asynchronous execution and concurrency issue -- var redefinition and var access may not be isolated together. Consider the following code snippet:
(let [items (with-redefs [fetch-items mock-fetch-items]
              (map order-items orders))]
  ;; "items" lazy/unevaluated here, var redefinition already restored
  ;; test code here
  ...)
In this example, the order in which var access and restoration take place is not correct. By the time the var is accessed (i.e. when the lazy sequence is evaluated in test code) the var's original root binding is already restored. Hence, the purpose of mocking is totally defeated here. This is not a problem with laziness per se, but rather the way it is used in conjunction with var redefinition. However, it is an easy pitfall for an unsuspecting developer.

The Path Forward

Var redefinition is indeed risky for mocking dependencies. However, many projects may choose to just test the happy path and test bad input, hence they may not need mocking at all. Projects that need to simulate conditions are going to need dependencies mocked and that is where var redefinition could be the spoilsport.

Dependency passing style

Fortunately Clojure, being a functional language, already has a way to swap out dependencies using higher-order functions. In fact, higher-order function is just a special case of "dependency passing style", wherein the dependencies are functions the callee can invoke. One can easily use this style to pass in values, configuration data, protocol implementations, functions, Java objects and whatnot. Once the dependency is passed as argument to a function, it is captured in the callee function isolated from other code that may refer to the same dependency – the dependency is no more tied to a var that must be mutated for others to see. In one fell swoop, passing dependencies takes care of all the problems we discussed above about var redefinition.

The Dependency inversion principle states, "Depend upon Abstractions. Do not depend upon concretions." An inside-out design, where a function explicitly accepts the dependencies as arguments, naturally allows dependencies to be swapped out by the caller. The dependencies passed during tests could simply be different from the ones in production. See the code example we discussed at the beginning of this post, reimplemented using higher order functions as follows:
(defn fetch-each-item
  "Return item details, or nil when not found."
  [db-fetcher item-id]
  (db-fetcher (normalize item-id)))

(defn fetch-items
  "Given item IDs, return a map of item IDs to item details.
  Items not found are omitted from the result."
  [item-fetcher item-ids]
  (->> item-ids
    (map #(vector % (item-fetcher %)))
    (filter second)
    (apply conj {})))
The function fetch-each-item must be partially applied with db-fetcher before it is passed to fetch-items as argument. Similarly, fetch-items must be partially applied with item-fetcher before it can be used as arity-1 function elsewhere.

Challenges with Dependency passing style

Dependency passing style poses a different set of problems than directly calling namespace functions, such as:

  1. Cascading construction of dependencies to be passed to various higher order functions
  2. Lack of editor support to navigate to the dependency source/implementation

It is technically possible to solve the first issue by writing code to create dependencies/partially applied functions. However, there are libraries like DIME (full disclosure: I am the author), Plumatic Graph and Component that to an extent take care of tracking/automating interdependencies without redefining vars. The second issue could be very unsettling for developers who are accustomed to editor/tooling support for navigating to referenced functions, e.g. the Emacs M-. (meta dot) feature provided by CIDER. I hope this can be mitigated with augmented tooling.

Though var redefinition has serious challenges with mocking, we have an alternative in the form of dependency passing style with a different set of tradeoffs than var redefinition.

This blog post is on Hacker News and Reddit. Please let me know your feedback in the comments. You may also like to follow me on Twitter.

(Thanks to Vijay Mathew, Sreenath N and Jerry Jacob for reviewing drafts of this post.)

Wednesday, July 29, 2015

Running ClojureCLR 1.7 on Mac OS X

ClojureCLR 1.7.0 came out few days ago. Trying it out on Mac OS X took some steps that I document in this post.

Step 1: Download latest Mono from the URL below
http://www.mono-project.com/download/#download-mac

Step 2: Download ClojureCLR 1.7.0 using NuGet client
$ nuget install Clojure -Version 1.7.0
This command will create a directory 'Clojure.1.7.0' containing ClojureCLR 1.7 files.

Step 3: Put lib and tools files together
$ cd Clojure.1.7.0
$ mkdir all
$ cp -r lib/net40 all
$ cp -r tools/net40/* all/net40
$ cp -r lib/net35 all/
$ cp -r tools/net35/* all/net35/

Step 4: Verify that files are combined by running the 'tree' command
$ tree
.
├── Clojure.1.7.0.nupkg
├── all
│   ├── net35
│   │   ├── Clojure.Compile.exe
│   │   ├── Clojure.Main.exe
│   │   ├── Clojure.dll
│   │   ├── Microsoft.Dynamic.dll
│   │   ├── Microsoft.Scripting.Core.dll
│   │   └── Microsoft.Scripting.dll
│   └── net40
│       ├── Clojure.Compile.exe
│       ├── Clojure.Main.exe
│       ├── Clojure.dll
│       ├── Microsoft.Dynamic.dll
│       └── Microsoft.Scripting.dll
├── lib
│   ├── net35
│   │   ├── Clojure.dll
│   │   ├── Microsoft.Dynamic.dll
│   │   ├── Microsoft.Scripting.Core.dll
│   │   └── Microsoft.Scripting.dll
│   └── net40
│       ├── Clojure.dll
│       ├── Microsoft.Dynamic.dll
│       └── Microsoft.Scripting.dll
└── tools
    ├── net35
    │   ├── Clojure.Compile.exe
    │   └── Clojure.Main.exe
    └── net40
        ├── Clojure.Compile.exe
        └── Clojure.Main.exe

9 directories, 23 files

Step 5: Create a script called mklinks.sh with the following content
#!/usr/bin/env bash
ln -s Clojure.dll clojure.clr.io.clj.dll
ln -s Clojure.dll clojure.core.clj.dll
ln -s Clojure.dll clojure.core_clr.clj.dll
ln -s Clojure.dll clojure.core_deftype.clj.dll
ln -s Clojure.dll clojure.core_print.clj.dll
ln -s Clojure.dll clojure.core.protocols.clj.dll
ln -s Clojure.dll clojure.core_proxy.clj.dll
ln -s Clojure.dll clojure.genclass.clj.dll
ln -s Clojure.dll clojure.gvec.clj.dll
ln -s Clojure.dll clojure.instant.clj.dll
ln -s Clojure.dll clojure.main.clj.dll
ln -s Clojure.dll clojure.pprint.cl_format.clj.dll
ln -s Clojure.dll clojure.pprint.clj.dll
ln -s Clojure.dll clojure.pprint.column_writer.clj.dll
ln -s Clojure.dll clojure.pprint.dispatch.clj.dll
ln -s Clojure.dll clojure.pprint.pprint_base.clj.dll
ln -s Clojure.dll clojure.pprint.pretty_writer.clj.dll
ln -s Clojure.dll clojure.pprint.print_table.clj.dll
ln -s Clojure.dll clojure.pprint.utilities.clj.dll
ln -s Clojure.dll clojure.repl.clj.dll
ln -s Clojure.dll clojure.set.clj.dll
ln -s Clojure.dll clojure.stacktrace.clj.dll
ln -s Clojure.dll clojure.string.clj.dll
ln -s Clojure.dll clojure.template.clj.dll
ln -s Clojure.dll clojure.test.clj.dll
ln -s Clojure.dll clojure.uuid.clj.dll
ln -s Clojure.dll clojure.walk.clj.dll
Step 6: Create symlinks to Clojure.dll
cd all/net40
bash ../../mklinks.sh
cd ../..
cd all/net35
bash ../../mklinks.sh
cd ../..
Step 7: Run ClojureCLR now

Run the Clojure.Main.exe file with Mono. If everything went fine you will get a REPL.
$ mono all/net40/Clojure.Main.exe
Clojure 1.7.0
user=>
That is all. If you like this post you may like to follow me on Twitter.

Friday, September 19, 2014

Running ClojureCLR 1.6 on Ubuntu 14.04

Recently ClojureCLR 1.6 was released. This post describes how to run ClojureCLR 1.6 on Ubuntu 14.04 - you may be able to run it similarly on other Ubuntu versions.

Installing Mono


The Mono project provides an Open Source implementation of the .NET platform. In order to run ClojureCLR we need to install Mono first. Follow the steps below to install Mono:

1. Create the following directories:
$ mkdir -p ~/app/installed
$ mkdir -p ~/app/src

2. Download the file http://download.mono-project.com/sources/mono/mono-3.8.0.tar.bz2 and untar the file in the src folder:
$ cd ~/app/src
$ tar xvf /path/to/mono-3.8.0.tar.bz2

3. Build Mono binaries from sources:
$ cd ~/app/src/mono-3.8.0
$ mkdir -p ~/app/installed/mono-3.8.0
$ ./configure --prefix ~/app/installed/mono-3.8.0
$ make
$ make install

4. Set the PATH to include the Mono binaries - include following lines in your ~/.bashrc or ~/.zshrc depending upon what shell you use:

export MONO_HOME=~/app/installed/mono-3.8.0
export PATH=$MONO_HOME/bin:$PATH

5. Verify that Mono is configured fine by running the mono command:
$ mono --version
Mono JIT compiler version 3.8.0 (tarball Sat Sep 13 13:21:35 IST 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
 TLS:           __thread
 SIGSEGV:       altstack
 Notifications: epoll
 Architecture:  amd64
 Disabled:      none
 Misc:          softdebug 
 LLVM:          supported, not enabled.
 GC:            sgen

Installing and configuring Nuget


Download the Nuget binary from http://nuget.org/nuget.exe and put it in a directory that is included in PATH. The Nuget binary being in PATH helps when using the lein-clr plugin.

Using Nuget on Linux requires some additional configuration. Run the following commands (note that mozroots and certmgr binaries are in Mono but may not be visible to sudo, so you may have to give full path):
$ sudo $MONO_HOME/bin/mozroots --import --machine --sync
$ sudo $MONO_HOME/bin/certmgr -ssl -m https://go.microsoft.com
$ sudo $MONO_HOME/bin/certmgr -ssl -m https://nugetgallery.blob.core.windows.net
$ sudo $MONO_HOME/bin/certmgr -ssl -m https://nuget.org

Installing and configuring ClojureCLR 1.6


To download ClojureCLR 1.6 you can now use Nuget:

$ cd ~/app/installed
$ mono /path/to/nuget.exe install Clojure -Version 1.6.0.1

This command will download ClojureCLR 1.6 into a folder 'Clojure.1.6.0.1' in the current directory. The 'Clojure.1.6.0.1' directory contains sub-directories 'lib' and 'tools', each further containing 'net35' and 'net40' sub-directories. You will need to store the files from 'lib' and 'tools' in a common directory, so do as follows:

$ cd ~/app/installed/Clojure.1.6.0.1
$ mkdir -p all/net35
$ mkdir -p all/net40
$ cp lib/net35/* all/net35/
$ cp lib/net40/* all/net40/
$ cp tools/net35/* all/net35/
$ cp tools/net40/* all/net40/

Now, in each of 'all/net35' and 'all/net40' sub-directories you need to create symbolic links for the file 'Clojure.dll' - the symbolic link names are listed below:

$ ln -s Clojure.dll clojure.clr.io.clj.dll
$ ln -s Clojure.dll clojure.core.clj.dll
$ ln -s Clojure.dll clojure.core_clr.clj.dll
$ ln -s Clojure.dll clojure.core_deftype.clj.dll
$ ln -s Clojure.dll clojure.core_print.clj.dll
$ ln -s Clojure.dll clojure.core.protocols.clj.dll
$ ln -s Clojure.dll clojure.core_proxy.clj.dll
$ ln -s Clojure.dll clojure.genclass.clj.dll
$ ln -s Clojure.dll clojure.gvec.clj.dll
$ ln -s Clojure.dll clojure.instant.clj.dll
$ ln -s Clojure.dll clojure.main.clj.dll
$ ln -s Clojure.dll clojure.pprint.cl_format.clj.dll
$ ln -s Clojure.dll clojure.pprint.clj.dll
$ ln -s Clojure.dll clojure.pprint.column_writer.clj.dll
$ ln -s Clojure.dll clojure.pprint.dispatch.clj.dll
$ ln -s Clojure.dll clojure.pprint.pprint_base.clj.dll
$ ln -s Clojure.dll clojure.pprint.pretty_writer.clj.dll
$ ln -s Clojure.dll clojure.pprint.print_table.clj.dll
$ ln -s Clojure.dll clojure.pprint.utilities.clj.dll
$ ln -s Clojure.dll clojure.repl.clj.dll
$ ln -s Clojure.dll clojure.set.clj.dll
$ ln -s Clojure.dll clojure.stacktrace.clj.dll
$ ln -s Clojure.dll clojure.string.clj.dll
$ ln -s Clojure.dll clojure.template.clj.dll
$ ln -s Clojure.dll clojure.test.clj.dll
$ ln -s Clojure.dll clojure.uuid.clj.dll
$ ln -s Clojure.dll clojure.walk.clj.dll

Now ClojureCLR 1.6 is ready to run. For example, you can launch a REPL as follows:

$ mono ~/app/installed/Clojure.1.6.0.1/all/net40/Clojure.Main.exe -r

You may notice Ctrl+D key combination does not work in this REPL. Use Ctrl+C to exit.

Using ClojureCLR 1.6 from the Lein-clr plugin


1. To use the Lein-clr plugin you need to have Java and Leiningen installed. To create a new lein-clr project use the following command:

$ lein new lein-clr foo

Make sure the lein-clr plugin version is 0.2.2 (or higher) in the project.clj file.

2. Edit the :clj-exe entry (under :clr => :cmd-templates) to specify an environment variable that points to the 'Clojure.1.6.0.1/all/net40' directory we discussed in the previous section.

Now you should be able to use ClojureCLR 1.6 in your lein-clr app:

$ cd foo
$ lein clr repl
$ lein clr test


Hope you find this useful. For more information on ClojureCLR 1.6 you should join the ClojureCLR Google Group. You may like to follow me on Twitter.

Sunday, April 22, 2012

Scripting Clojure with Leiningen 2

[2012-May-14] Update: Fixed shebang for more Unix variants

What would it take to write scripts in Clojure (and run them on a terminal) the way we write them in Python, Ruby or Groovy? To pipe scripts together to accomplish bigger tasks? What would it take to throw in argument handling and transitive dependency management capabilities to such scripts? This post attempts to answer some of these questions while illustrating how to use the Leiningen plugin lein-exec to write Clojure scripts.

In particular, I assume you have Leiningen 2 and lein-exec 0.2.0 installed. The shebang feature may require that you use a *nix environment.

Command-line eval

Let us begin with evaluating tiny Clojure S-expressions on the command line.

$ lein2 exec -e "(println \"40 and 50 make\" (+ 40 50))"
40 and 50 make 90


Since the plugin can also evaluate the content from STDIN, we can also do the same thing as above by piping the Clojure code to the plugin:

$ echo "(println \"40 and 50 make\" (+ 40 50))" | lein2 exec
40 and 50 make 90


While clojure.core vars are available, vars from other namespaces may need an explicit reference:

$ lein2 exec -e "(use 'clojure.pprint) (pprint (map inc (range 10)))"
(1 2 3 4 5 6 7 8 9 10)


You would need similar quoting when writing the Clojure scripts due to the fact that S-expressions and scripts are simply evaluated by lein-exec.

Scripting

So, let's start with the first script as a file called fib10.clj that simply prints out first 10 numbers in the Fibonacci series:

;; Taken from http://j.mp/IiT8UK
(def fib-seq
  ((fn rfib [a b]
    (lazy-seq (cons a (rfib b (+ a b)))))
    0 1))

(println (take 10 fib-seq))


We can run it using lein-exec:

$ lein2 exec fib10.clj
(0 1 1 2 3 5 8 13 21 34)


OK, that is fine and dandy but how do we run fib10.clj as an executable? Here's how – you edit the file and put a shebang on the first line:

#!/bin/bash lein-exec

;; Taken from http://j.mp/IiT8UK
(def fib-seq
  ((fn rfib [a b]
    (lazy-seq (cons a (rfib b (+ a b)))))
    0 1))

(println (take 10 fib-seq))


Now you can run the fib10.clj as an executable:

$ chmod a+x fib10.clj
$ ./fib10.clj
(0 1 1 2 3 5 8 13 21 34)


Handling arguments

Scripts often need to work with command-line arguments. While using lein-exec arguments are always passed as string via the clojure.core/*command-line-args* dynamic var – first argument is the name of the script followed by the arguments to the script. Below is an example that takes name and scores and prints out name and average:

#!/bin/bash lein-exec

(defn err-println "println for STDERR"
  [& args]
  (binding [*out* *err*]
    (apply println args)))

(defn parse-int "Parse string as an integer. Abort if invalid."
  [n]
  (try (Integer/parseInt n)
    (catch NumberFormatException e
      (err-println (str \' n \')
                   "is not a valid integer")
      (System/exit 1))))

(defn avg "Given a sequence of numbers return their average."
  [nseq]
  (double (/ (apply + nseq) (count nseq))))

(if (>= (count *command-line-args*) 3)
  (println (second *command-line-args*)
           (avg (map parse-int (drop 2 *command-line-args*))))
  (do (err-println "Usage:" (first *command-line-args*)
                   "name score [score2 ..]")
    (System/exit 1)))


Upon running the script we can inspect how it responds to different input arguments:

$ chmod a+x ./avg.clj
$ ./avg.clj
Usage: ./avg.clj name score [score2 ..]
$ ./avg.clj "Nick Foster"
Usage: ./avg.clj name score [score2 ..]
$ ./avg.clj "Nick Foster" 45 78 65
Nick Foster 62.66666666666667
$ ./avg.clj "Nick Foster" notnum 45 78 65
'notnum' is not a valid integer


Pipeline

Pipeline is a very powerful concept that enables us to chain simple tasks using their STDIN and STDOUT to build more sophisticated tasks. As we would see below, it is entirely possible to build pipe together scripts written in Clojure by simply reading input from STDIN and writing result to STDOUT:

#!/bin/bash lein-exec

(loop []
  (let [c (.read *in*)]
    (when (>= c 0)
      (if (and (>= c (int \A)) (<= c (int \Z)))
        (print (Character/toLowerCase (char c)))
        (print (char c)))
      (recur))))


This script lcase.clj converts the input fed via STDIN to lowercase. Let us see it in action:

$ chmod a+x ./lcase.clj
$ echo "5 Quick brown foxes Jumped over 7 lazy dogs" | ./lcase.clj 
5 quick brown foxes jumped over 7 lazy dogs
$ tree | ./lcase.clj
..result omitted..
$ ls -l ~ | ./lcase.clj
..result omitted..


Transitive dependencies

Scripting with Clojure is fun! It would only be nicer to have the power of transitive dependency management baked into it. Well, pomegranate that is part of Leiningen 2 already makes that possible. lein-exec wraps pomegranate API to provide its own simpler API. The example below shows how to create a demo web service using lein-exec's deps function and Ring:

#!/bin/bash lein-exec

(use '[leiningen.exec :only  (deps)])
(deps '[[ring "1.0.1"]])

(defn handler
  [request]
  {:status 200
   :headers {}
   :body "Hello from Ring!"})

(use 'ring.adapter.jetty)
(run-jetty handler {:port 3000})


The deps function above from leiningen.exec namespace, which is in the lein-exec plugin itself. Here we pull the dpendency ring 1.0.1 which depends on a number of components under it. All those dependencies are pulled in as you run this script. Upon running something this is what you see:

$ chmod a+x ring.clj
$ ./ring.clj
2012-04-22 17:55:35.814:INFO::Logging to STDERR via org.mortbay.log.StdErrLog
2012-04-22 17:55:35.815:INFO::jetty-6.1.25
2012-04-22 17:55:35.824:INFO::Started SocketConnector@0.0.0.0:3000


Depending on whether ring is already downloaded to your local Maven repo, you may see the script downloading them first if they don't already exist.

Working in project context

At times you may want to run a script in the context of a project, so that the code being evaluated has access to the project CLASSPATH and other project resources. All of the above usages of lein-exec support an additional switch -p that does the same thing in project scope.

$ lein2 exec -ep "(use 'foo.bar) (baz :qux)"
..result omitted..
$ cat foo.clj | lein2 exec -p
.. result omitted..
$ lein2 exec -p foo.clj
.. result omitted..


The examples above are applicable when you run them in a project. If you use -p outside of a project it will complain about missing project!

Caveats

While the ability to script in Clojure is fascinating with all the shebang operator, pipeline and dependency management, it's important to know its limitations:

  • As widely known, JVM startup time is a pain even though Leiningen works hard to reduce it.
  • Eval'd code runs in the same JVM that runs Leiningen. There's no easy way to customize that.
  • Currently, the deps function pulls in dependencies only from Maven Central and Clojars.
  • When you run lein-exec in project context, the project map is not accessible to the eval'd code.

Few of these limitations may be addressed in future versions.


I think many Clojure beginners may see scripting as a good way to learn the language and explore Clojure libraries. At the same time, some of the Clojure app/tool projects may find it easy to distribute the app via a script instead of a full blown JAR with instructions on how to run it. Whatever is your impression, feedback and ideas, please share it in comments. You may like to follow me on Twitter: @kumarshantanu

Happy scripting with Clojure!

Thursday, November 11, 2010

My notes from "TheDeadline" presentations

Recently TheDeadline[1] presentations were[2] published[3] on Slideshare. I believe they have an incredibly powerful message about a modern style of web development using HTML5 and JavaScript. My notes from the presentations are in this post.

Rules of Modern Web development

1. You should be able to understand what your application does just by reading the code.
  • Do not overuse JavaScript callbacks
  • Don't call other callback functions from within a callback (Spaghetti)
2. Don't try to write Windows-style Desktop apps inside the browser.
  • Begin with UX in mind, proceed with CSS3 and JavaScript
  • De-couple formatting and display from content using CSS3
3. Be prepared to render most of your HTML code on the client-side.
  • Server-side per component content generation doesn't scale, and is limited by the server boundary
  • Consider Closure Templates[4] or Mustache[5] for templates - they work on both JavaScript and server-side
4. Don't write JavaScript in the style of the Java language. Forget everything you learned by writing Java code.
  • No complex class hierarchies. JavaScript was not meant for this
  • Write functional code
  • Consider adopting Google Closure[6] -- Google Clojure Library is to JavaScript what the JDK is to Java
5. Plan ahead for Offline capabilities. But be aware, that users maybe try to sync stale data.
  • Use local storage
  • Use caching
6. You'll need an idea how to cope with concurrent modifications, when it is likely that your users can modify the same data at the same time and this could cause problems.
  • Optimistic Locking
  • While versioning data just store the actions as delta, not the snapshots. Gain? Real-time Analytics!
7. You need push notifications.
  • Event-loop on the client side and a fast, async server-side REST API
8. Key/values != E-R model
  • Use a persistence model that matches the app data model
  • E-R is not it
9. Log client-side exceptions to the server.
  • Wrappers that do this transparently
  • Should gather enough context though

You can catch more of related stuff at their blog[7].

Links:
1. http://the-deadline.appspot.com/
2. http://www.slideshare.net/smartrevolution/using-clojure-nosql-databases-and-functionalstyle-javascript-to-write-gextgeneration-html5-apps
3. http://www.slideshare.net/smartrevolution/writing-html5-apps-with-google-app-engine-google-closure-library-and-clojure
4. http://code.google.com/closure/templates/
5. http://mustache.github.com/
6. http://code.google.com/closure/
7. http://www.hackers-with-attitude.com/

If you have feedback on this post, or any opinion on these topics, please let me know.

Friday, September 10, 2010

Interactive Web Development with Clojure

Interactive web development is a rapid iteration of the following:

1. Create/edit source file. Save.
2. Compile. (not required; the runtime reloads them automatically)
3. Deploy to app server. (not required; the runtime syncs up with embedded web container)
3. Evaluate in REPL or refresh browser.

Languages like PHP, Ruby, Python etc already support this and here we discuss how to achieve the same with Clojure using the Eclipse IDE and CounterClockWise plugin. This example uses Compojure as the web development library.

Make sure you use CounterClockWise plugin version 0.0.62 or later with Eclipse: http://code.google.com/p/counterclockwise/

Create a Clojure project and define your Compojure routes like this:


(ns example.core
(:use compojure.core)
(:require [compojure.route :as route]))

(defroutes handler
(GET "/" [] "Hello World!")
(route/not-found "Page not found"))


Create another file called ipte.clj outside of the regular sources directory. Let's say we create this under a folder called dev, so the folder structure looks like this:


dev
`-- ipte.clj
src
`-- example
|-- core.clj
`-- (other files)


Now make sure both src and dev folders are included as source folders in Eclipse. Edit the file ipte.clj to contain the following:


(ns ipte "In-process Test Environment"
(:use ring.adapter.jetty)
(:use example.core))

(defonce server
(run-jetty (var handler) {:port 8080}))


Note:
1. The file ipte.clj starts Jetty server on compile/reload, so it's put in a separate file (for development environment only).
2. defonce makes sure the Jetty server is not attempted to start again upon namespace reload. Using _ as var name with defonce resulted in non-evaluation of the expresion for me, so I would suggest using a proper name.
3. (var ..) makes sure that updates to the handler data is available across namespace reloads.

Make sure you include all required JAR files in project classpath and we are all set up (leaving out a required JAR may lead to silent failure while running the project in Eclipse). Right-click on the project (root node) in left package/navigation pane and select "Run in JVM". That's all. Now, the changes you make to the source code will reflect in the REPL or browser immediately.

Thursday, July 15, 2010

Setup Emacs for development with Clojure/Leiningen

Edit: [2011 Oct 16] This post works only for Emacs-Starter-Kit 1, and is hence out of date
Edit: [2011 April 8] Works on Mac OS X (Intel)
Edit: [2010 October 10] Updated for Windows 7

This post is to describe how to set up Emacs for Clojure development on Mac, Linux and Windows. I assume you have setup a project structure using Leiningen already. You can follow this to setup Leiningen on Windows.

Swank-Clojure

You need to get Swank-Clojure running first.

1. Mention
:dev-dependencies [[swank-clojure "1.2.1"]]
in project.clj as described here: http://github.com/technomancy/leiningen

2. Get the dependencies:
lein deps

3. Start Swank
lein swank


Run Emacs

1. Download and install Emacs (at least version 22, recommended version 23 or higher). Mac users can get Emacs from here: http://emacsformacosx.com/. Windows users can download Emacs binaries from here and set the PATH: http://ftp.gnu.org/gnu/windows/emacs/

2. Delete any previous .emacs.d directory from home directory.

Mac/Linux:
mv ~/.emacs.d ~/.emacs.d.bak

Windows XP:
C:\Documents and Settings\<username>\Application Data

Windows 7:
C:\Users\<username>\AppData\Roaming

3. Install Git.

Windows users can download Git (MSys Git) from here and set the PATH: http://code.google.com/p/msysgit/downloads/list (version 1.7.3.1 is fine as of October 2010)

4. Clone the Emacs Starter kit in a working directory (this command clones the starter kit as .emacs.d directory):
git clone http://github.com/technomancy/emacs-starter-kit/ .emacs.d

OR
git clone git://github.com/technomancy/emacs-starter-kit.git .emacs.d

5. Copy the cloned .emacs.d directory to your home directory (~ on Mac/Linux, or "C:\Documents and Settings\<username>\Application Data" on Windows XP, or "C:\Users\<username>\AppData\Roaming" on Windows 7).

6. Run Emacs, and follow the steps below ("M" means "Meta", which is the Alt key i.e. M-x means Alt+x. Press Enter after each Emacs command.):

(a) Install necessary Emacs packages.
M-x package-list-packages

You will be presented with a list of modes. Select clojure-mode and swank-clojure packages by navigating to those items in the list and pressing "i" against each. After selection, press "x" to install.

(b) Connect to the Swank server.
M-x slime-connect

It will ask for hostname and port. Press Enter key for both and it will take the defaults. If all goes well upto this point, you will be connected to the Swank-Clojure running via Lein. Emacs would be ready for use.


Just in case you are an Emacs newbie and don't know how to exit, use the following keys to quit Emacs ("C" means "Control", i.e. C-x means Ctrl+x):
C-x C-c

Please let me know your feedback and suggestions.

Disqus for Char Sequence