My notes from "TheDeadline" presentations

Recently TheDeadline[1] presentations were[2] published[3] on Slideshare. I believe they have an incredibly powerful message about a modern style of web development using HTML5 and JavaScript. My notes from the presentations are in this post.

Rules of Modern Web development

1. You should be able to understand what your application does just by reading the code.

• Do not overuse JavaScript callbacks
• Don't call other callback functions from within a callback (Spaghetti)

2. Don't try to write Windows-style Desktop apps inside the browser.

• Begin with UX in mind, proceed with CSS3 and JavaScript
• De-couple formatting and display from content using CSS3

3. Be prepared to render most of your HTML code on the client-side.

• Server-side per component content generation doesn't scale, and is limited by the server boundary
• Consider Closure Templates[4] or Mustache[5] for templates - they work on both JavaScript and server-side

4. Don't write JavaScript in the style of the Java language. Forget everything you learned by writing Java code.

• No complex class hierarchies. JavaScript was not meant for this
• Write functional code
• Consider adopting Google Closure[6] -- Google Clojure Library is to JavaScript what the JDK is to Java

5. Plan ahead for Offline capabilities. But be aware, that users maybe try to sync stale data.

• Use local storage
• Use caching

6. You'll need an idea how to cope with concurrent modifications, when it is likely that your users can modify the same data at the same time and this could cause problems.

• Optimistic Locking
• While versioning data just store the actions as delta, not the snapshots. Gain? Real-time Analytics!

7. You need push notifications.

• Event-loop on the client side and a fast, async server-side REST API

8. Key/values != E-R model

• Use a persistence model that matches the app data model
• E-R is not it

9. Log client-side exceptions to the server.

• Wrappers that do this transparently
• Should gather enough context though

You can catch more of related stuff at their blog[7].

Links:
1. http://the-deadline.appspot.com/
2. http://www.slideshare.net/smartrevolution/using-clojure-nosql-databases-and-functionalstyle-javascript-to-write-gextgeneration-html5-apps
3. http://www.slideshare.net/smartrevolution/writing-html5-apps-with-google-app-engine-google-closure-library-and-clojure
4. http://code.google.com/closure/templates/
5. http://mustache.github.com/
6. http://code.google.com/closure/
7. http://www.hackers-with-attitude.com/

If you have feedback on this post, or any opinion on these topics, please let me know.

Interactive Web Development with Clojure

Interactive web development is a rapid iteration of the following:

1. Create/edit source file. Save.
2. Compile. (not required; the runtime reloads them automatically)
3. Deploy to app server. (not required; the runtime syncs up with embedded web container)
3. Evaluate in REPL or refresh browser.

Languages like PHP, Ruby, Python etc already support this and here we discuss how to achieve the same with Clojure using the Eclipse IDE and CounterClockWise plugin. This example uses Compojure as the web development library.

Make sure you use CounterClockWise plugin version 0.0.62 or later with Eclipse: http://code.google.com/p/counterclockwise/

Create a Clojure project and define your Compojure routes like this:

(ns example.core
  (:use compojure.core)
  (:require [compojure.route :as route]))

(defroutes handler
  (GET "/" [] "Hello World!")
  (route/not-found "Page not found"))

Create another file called ipte.clj outside of the regular sources directory. Let's say we create this under a folder called dev, so the folder structure looks like this:

  dev
   `-- ipte.clj
  src
   `-- example
        |-- core.clj
        `-- (other files)

Now make sure both src and dev folders are included as source folders in Eclipse. Edit the file ipte.clj to contain the following:

(ns ipte "In-process Test Environment"
  (:use ring.adapter.jetty)
  (:use example.core))

(defonce server
  (run-jetty (var handler) {:port 8080}))

Note:
1. The file ipte.clj starts Jetty server on compile/reload, so it's put in a separate file (for development environment only).
2. defonce makes sure the Jetty server is not attempted to start again upon namespace reload. Using _ as var name with defonce resulted in non-evaluation of the expresion for me, so I would suggest using a proper name.
3. (var ..) makes sure that updates to the handler data is available across namespace reloads.

Make sure you include all required JAR files in project classpath and we are all set up (leaving out a required JAR may lead to silent failure while running the project in Eclipse). Right-click on the project (root node) in left package/navigation pane and select "Run in JVM". That's all. Now, the changes you make to the source code will reflect in the REPL or browser immediately.

Setup Emacs for development with Clojure/Leiningen

Edit: [2011 Oct 16] This post works only for Emacs-Starter-Kit 1, and is hence out of date
Edit: [2011 April 8] Works on Mac OS X (Intel)
Edit: [2010 October 10] Updated for Windows 7

This post is to describe how to set up Emacs for Clojure development on Mac, Linux and Windows. I assume you have setup a project structure using Leiningen already. You can follow this to setup Leiningen on Windows.

Swank-Clojure

You need to get Swank-Clojure running first.

1. Mention

:dev-dependencies [[swank-clojure "1.2.1"]]

in project.clj as described here: http://github.com/technomancy/leiningen

2. Get the dependencies:

lein deps

3. Start Swank

lein swank

Run Emacs

1. Download and install Emacs (at least version 22, recommended version 23 or higher). Mac users can get Emacs from here: http://emacsformacosx.com/. Windows users can download Emacs binaries from here and set the PATH: http://ftp.gnu.org/gnu/windows/emacs/

2. Delete any previous .emacs.d directory from home directory.

Mac/Linux:

mv ~/.emacs.d ~/.emacs.d.bak

Windows XP:

C:\Documents and Settings\<username>\Application Data

Windows 7:

C:\Users\<username>\AppData\Roaming

3. Install Git.

Windows users can download Git (MSys Git) from here and set the PATH: http://code.google.com/p/msysgit/downloads/list (version 1.7.3.1 is fine as of October 2010)

4. Clone the Emacs Starter kit in a working directory (this command clones the starter kit as .emacs.d directory):

git clone http://github.com/technomancy/emacs-starter-kit/ .emacs.d

OR

git clone git://github.com/technomancy/emacs-starter-kit.git .emacs.d

5. Copy the cloned .emacs.d directory to your home directory (~ on Mac/Linux, or "C:\Documents and Settings\<username>\Application Data" on Windows XP, or "C:\Users\<username>\AppData\Roaming" on Windows 7).

6. Run Emacs, and follow the steps below ("M" means "Meta", which is the Alt key i.e. M-x means Alt+x. Press Enter after each Emacs command.):

(a) Install necessary Emacs packages.

M-x package-list-packages

You will be presented with a list of modes. Select clojure-mode and swank-clojure packages by navigating to those items in the list and pressing "i" against each. After selection, press "x" to install.

(b) Connect to the Swank server.

M-x slime-connect

It will ask for hostname and port. Press Enter key for both and it will take the defaults. If all goes well upto this point, you will be connected to the Swank-Clojure running via Lein. Emacs would be ready for use.


Just in case you are an Emacs newbie and don't know how to exit, use the following keys to quit Emacs ("C" means "Control", i.e. C-x means Ctrl+x):

C-x C-c

Please let me know your feedback and suggestions.

Setup Leiningen on Windows

Edit: Leiningen 1.3.1 got better Windows support and you don't need to follow these instructions anymore. Get the Windows distribution from here: http://github.com/technomancy/leiningen/downloads, unzip into a folder of choice and include in PATH.

Leiningen is a build tool for Clojure. Using Leiningen has been described here and here.

Leiningen installs and runs well on Linux and Mac. As of June 2010, Leiningen has experimental support for Windows and lacks the self-install feature. This post describes how to setup Leiningen on Windows XP. No prior Ant / Maven / Lancet experience is assumed.

1. Establish a directory where you want to install Leiningen (create if it doesn't exist). Example:

D:\lein

Also, add it to your system PATH environment variable.

2. Download the Leiningen script (right-click and choose "Save as"). Save it as "lein.bat" to the location discussed above.

3. Download the Leiningen JAR (for Leiningen 1.1.0) to the same location discussed above. The JAR filename can be figured out by looking at the comments (REM statements) in the lein.bat file.

4. Set environment variable LEIN_JAR to "D:\lein\leiningen-1.1.0-standalone.jar", or the appropriate path where the file is saved.


Leiningen is setup now. To test the install, try these:

D:\temp>lein new hello
D:\temp>cd hello
D:\temp\hello>lein deps
D:\temp\hello>lein test

Please let me know your comments and feedback.

Mars Rover solution in Clojure

The Mars Rover problem has been solved by other people earlier:

Veera Sundar - Java

Arun Ravindran - Python and Haskell

Baishampayan Ghose - Clojure

A simple and less concise Clojure solution is listed in this post.

Mars Rover Problem


A squad of robotic rovers are to be landed by NASA on a plateau on Mars.

This plateau, which is curiously rectangular, must be navigated by the rovers so that their on-board cameras can get a complete view of the surrounding terrain to send back to Earth.

A rover’s position and location is represented by a combination of x and y co-ordinates and a letter representing one of the four cardinal compass points. The plateau is divided up into a grid to simplify navigation. An example position might be 0, 0, N, which means the rover is in the bottom left corner and facing North.

In order to control a rover , NASA sends a simple string of letters. The possible letters are ‘L’, ‘R’ and ‘M’. ‘L’ and ‘R’ makes the rover spin 90 degrees left or right respectively, without moving from its current spot. ‘M’ means move forward one grid point, and maintain the same heading.

Assume that the square directly North from (x, y) is (x, y+1).

INPUT:

The first line of input is the upper-right coordinates of the plateau, the lower-left coordinates are assumed to be 0,0.

The rest of the input is information pertaining to the rovers that have been deployed. Each rover has two lines of input. The first line gives the rover’s position, and the second line is a series of instructions telling the rover how to explore the plateau.

The position is made up of two integers and a letter separated by spaces, corresponding to the x and y co-ordinates and the rover’s orientation.

Each rover will be finished sequentially, which means that the second rover won’t start to move until the first one has finished moving.

OUTPUT

The output for each rover should be its final co-ordinates and heading.

INPUT AND OUTPUT

Test Input:

5 5
1 2 N
LMLMLMLMM
3 3 E
MMRMMRMRRM

Expected Output:

1 3 N
5 1 E


Clojure Solution

(ns marsrover.main)

(defstruct rover-location :x :y :facing)

(def facing-all [:north :east :west :south])

(def facing-lookup    (zipmap facing-all "NEWS"))

(def direction-lookup (zipmap "NEWS" facing-all))

(def go-left-lookup   (zipmap facing-all [:west :north :south :east]))

(def go-right-lookup  (zipmap facing-all [:east :south :north :west]))

(def move-lookup
  {:north #(struct-map rover-location :x (% :x)       :y (inc (% :y)) :facing :north)
   :east  #(struct-map rover-location :x (inc (% :x)) :y (% :y)       :facing :east)
   :south #(struct-map rover-location :x (% :x)       :y (dec (% :y)) :facing :south)
   :west  #(struct-map rover-location :x (dec (% :x)) :y (% :y)       :facing :west)}
  )


(defn turn-left
  [rloc]
  (struct-map rover-location
    :x (rloc :x)
    :y (rloc :y)
    :facing (go-left-lookup (rloc :facing))))


(defn turn-right
  [rloc]
  (struct-map rover-location
    :x (rloc :x)
    :y (rloc :y)
    :facing (go-right-lookup (rloc :facing))))


(defn move
  [rloc]
  (let [func (move-lookup (rloc :facing))]
    (func rloc)))


(def command-lookup (zipmap "LRM" [turn-left turn-right move]))


(defn process-each-command
  [rloc single-cmd]
  (let [func (command-lookup single-cmd)]
    (func rloc)))


(defn process-commands
  [[x y d] cmds]
  (loop [rloc (struct-map rover-location :x x :y y :facing (direction-lookup d))
         cmd-seq (seq cmds)]
    (if (empty? cmd-seq)
      rloc
      (recur (process-each-command rloc (first cmd-seq)) (rest cmd-seq)))))


(defn print-location
  [rloc]
  (do
      (println (str (rloc :x) (rloc :y) (facing-lookup (rloc :facing))))))


;(defn -main
;  [input]
  (do
      (print-location (process-commands [1 2 \N] "LMLMLMLMM"))
      (print-location (process-commands [3 3 \E] "MMRMMRMRRM")))
;  )

Your comments and feedback are welcome.

Data immutability in Java

Shared mutable state is one of the murkiest areas of concurrent programming in Java. To tackle this issue it is generally advised to prefer data immutability over thread-safety as the latter is hard to manage in large and complex applications. In this post I am noting few measures to approach data immutability in Java.

1. Always declare instance-level (and class-level) data members as "final". Assign only immutable or thread-safe objects to these variables. (See point #6 below on sharing mutable assignments in a thread-safe way.)

While declaring instance variables as final you will have to instantiate them in the constructor. This may lead to constructor-based dependency injection and a declarative design, which is a good thing.

2. Declare local variables as "final". The only exceptions to this are the loop counter primitives for performance reasons:

    for (int i = 0; i < 40; i++) {
        // blah
    }

Edit: This point will not help concurrency but it will make sure you don't accidentally bash the value in place.

3. Use immutable data structures unless you need to modify it immediately.

   java.util.Collections.unmodifiableList(list)
   java.util.Collections.unmodifiableSet(set)
   java.util.Collections.unmodifiableMap(map)

Google Collections is useful for dealing with collections conveniently:

Edit: Consider using persistent collections as it enforces immutability at collection level without giving up efficiency.

4. For concurrent scenarios, use concurrency-optimized data structure implementations.

    java.util.concurrent.CopyOnWriteArrayList
    java.util.concurrent.CopyOnWriteArraySet
    java.util.concurrent.ConcurrentHashMap
    java.util.concurrent.ConcurrentLinkedQueue

Never use the Collections.synchronizedXXX() methods, as they will reduce concurrency to zero. I blogged about this earlier.

5. Use reference copying while constructing new collections from immutable collections.

    public <T>List<T> add(List<T> old, T element) {
        final List<T> newlist = new ArrayList<T>(old);
        newlist.add(element);
        return Collections.unmodifiableList(newlist);
    }

Note: For large data structures this may have a performance penalty.

6. Shared mutable assignments should be atomic. You can use the built-in library in Java5+:

java.util.concurrent.atomic.AtomicBoolean 
java.util.concurrent.atomic.AtomicInteger 
java.util.concurrent.atomic.AtomicIntegerArray 
java.util.concurrent.atomic.AtomicIntegerFieldUpdater 
java.util.concurrent.atomic.AtomicLong 
java.util.concurrent.atomic.AtomicLongArray 
java.util.concurrent.atomic.AtomicLongFieldUpdater 
java.util.concurrent.atomic.AtomicMarkableReference 
java.util.concurrent.atomic.AtomicReference 
java.util.concurrent.atomic.AtomicReferenceArray 
java.util.concurrent.atomic.AtomicReferenceFieldUpdater 
java.util.concurrent.atomic.AtomicStampedReference

7. Never "bash in place" - rather construct a new object upon every action that requires change in state. A desirable approach may look like this:

public class XYPos {
    public final int x;
    public final int y;
    public class XYPos(final int ix, final int iy) {
        this.x = ix;
        this.y = iy;
    }
}


public class Navigator {
    public XYPos moveRight(final XYPos pos) { return new XYPos(pos.x + 1, pos.y); }
    public XYPos moveLeft (final XYPos pos) { return new XYPos(pos.x - 1, pos.y); }
    public XYPos moveUp   (final XYPos pos) { return new XYPos(pos.x, pos.y + 1); }
    public XYPos moveDown (final XYPos pos) { return new XYPos(pos.x, pos.y - 1); }
}

Your opinions, comments and feedback are welcome.

Replacing application properties with Groovy scripts

Application configuration for non-trivial applications are sometimes quite large. Apart from being large, they may also have interdependencies, so much so that you might wish you could use a dynamic script to specify the configuration.

You can use the Groovy scripting language to do the exactly same thing. Groovy has a familiar Java syntax and you can use other powerful features at runtime to specify your application configuration. Here is a 6-part approach:

Pre-condition: You need to have the Groovy JAR file in your classpath that you can get from here: http://groovy.codehaus.org/Download

1. Create a configuration class:

// filename: AppConfigStruct.java
//
public class AppConfigStruct {
    // some properties - JDBC etc
    //
    public volatile String jdbc_driver_class = null;
    public volatile String jdbc_url          = null;
    public volatile String username          = null;
    public volatile String password          = null;
    public volatile String validation_query  = null;
    public volatile String schema_name       = null;
    public volatile String table_name        = null;
}

2. Create an interface for loading application configuration:

// filename: IAppConfigLoader.java
//
public interface IAppConfigLoader {
 
    public abstract AppConfigStruct load();
 
}

3. Have the IAppConfigLoader interface implemented by the groovy config script:

// filename: AppConfigLoaderGroovyImpl.groovy
//
public class AppConfigLoaderGroovyImpl implements IAppConfigLoader {
    public AppConfigStruct load() {
        AppConfigStruct appConfig = new AppConfigStruct();
        setAllProperties(appConfig);
        return appConfig;
    }
    public void setAllProperties(AppConfigStruct appConfig) {
        // set all properties here
        // you may use more functions to divide the work
    }
}    

4. Now comes the meat - write a function that loads the Groovy script and parses it into a config loader:

// filename: AppConfigUtil.java
//
public class AppConfigUtil {
    public static AppConfigStruct getConfigFromGroovyScript(final InputStream in)
    throws IOException, IllegalAccessException, InstantiationException {
        groovy.lang.GroovyClassLoader gcl = new groovy.lang.GroovyClassLoader();
        Class clazz = gcl.parseClass(in);
        Object aScript = clazz.newInstance();
        IAppConfigLoader ifc = (IAppConfigLoader) aScript;
        return ifc.load();
    }
}

5. Use the above-mentioned function by loading the Groovy script from a file (useful for testing or dev mode):

// filename: AppConfigUtil.java
//
    public static AppConfigStruct getConfigFromGroovyScript(final String filename)
    throws
    FileNotFoundException, IOException,
    InstantiationException, IllegalAccessException {
        final InputStream in = new FileInputStream(new File(filename));
        try {
            return getConfigFromGroovyScript(in);
        } finally {
            if (in != null) {
                in.close();
            }
        }
    }

6. You can also use the following mechanism in a servlet to load the groovy script using the servlet context classloader:

// filename: AppConfigUtil.java
//
    public static AppConfigStruct getConfigFromGroovyScript(
            final ServletContext servletContext,
            final String groovyConfigFilename) {
        // filename example -- "/WEB-INF/AppConfigLoaderProductionImpl.groovy"
        final InputStream in = servletContext.getResourceAsStream(groovyConfigFile);
        return getConfigFromGroovyScript(in);
    }

Let me know what you think about this approach.