Clojure, Maven, Emacs, Compojure: Hello Web.

Let's use maven to construct a web server:

Take the setup from the last post (http://learnclojure.blogspot.com/2010/03/clojure-maven-emacs-eternal-golden.html),

Paste the snippet

   
    <dependency>
      <groupId>compojure</groupId>
      <artifactId>compojure</artifactId>
      <version>0.3.2</version>
    </dependency>
    

Into the <dependencies> section of the pom.xml file, and then restart clojure using
mvn clojure:repl or mvn clojure:swank, according to taste.

Maven will again appear to download the internet as it goes and pulls in compojure and all its dependencies.

If you're using emacs, then when the swank server restarts, connect with M-x slime-connect.

At the REPL prompt, try

user> (use 'compojure)
nil
user> (defn yo "yo")
#'user/yo
user> (defroutes my-app (GET "/" yo))
#'user/my-app
user> (run-server {:port 8080} "/*" (servlet my-app))
user.proxy$javax.servlet.http.HttpServlet$0
#<Server Server@a16605>

And you are totally running a web server. Go check it out at http://localhost:8080.

OK, it's not brilliant. Somewhere in the carnage it should say:

Wrong number of args passed to: user$yo

We can fix that:

user> (defn yo [s] "yo")
#'user/yo

Now have another look.

user> (defn yo [s] (html [:h1 "yo"]))
#'user/yo

And finally let's get all standards-compliant:

user> (defn yo [s]
        (html
         [:head
          [:title "hello world"]]
         [:body
          [:h1 "yo"]]))
#'user/yo

So loading a library, itself with many dependencies, and setting up the relevant classpath, are reduced to the finding of one xml snippet.

Once we've done that we can run a process which both serves web pages and runs a swank server that allows us to modify its code as it's running, without restarting anything.

Many clojure library snippets can be found at http://clojars.org.
There are various modified versions of compojure there, which will be found by a search for compojure. The canonical version is at http://clojars.org/compojure.

Clojure, Maven, Emacs: An eternal golden braid

This page still gets lots of hits. 
Several people have said that they refer to it often, even though it's out of date. So I'm leaving it up.

There are more recent instructions on getting clojure working quickly here:
http://www.learningclojure.com/2010/08/clojure-emacs-swank-slime-maven-maven.html

It takes about 30 seconds from a fresh install if your internet connection is fast, and at that point you've got a usable REPL and emacs setup.

It basically goes: install maven, install emacs, make pom, copy cryptic thing into emacs, press magic keys, press i three times, press x and you're done.

I use that page if I need to install from scratch these days.


Here's the original post:
------------------------------------------------------------------------------------------------------------------------------

Maven is a horror from the depths of the Java pit.
But on investigation I find I rather like it:

Install maven (actually maven2) with

$ sudo apt-get install maven2

and then put the xml you'll find below in a file called pom.xml .

Put pom.xml in its own directory, and type

$ mvn clojure:repl

At this point, maven will begin to download the entire internet. This is normal. Go and make a cup of tea. It will only need to do it once.

When you get back, the following prompt should be waiting for you:

Clojure 1.1.0
user=>

Go ahead and play: clojure-contrib is already on the classpath ; Jline is running, so the repl is usable.

But it gets better:

$mvn clojure:swank

leads to:

Connection opened on local port  4005
#<ServerSocket ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=4005]>


So if you've got a clojure-compatible SLIME installed in EMACS then a simple M-x slime-connect will allow you to connect to the running image.

Now at this point, we've already short-circuited a large amount of the unpleasantness which would usually be involved in setting up Clojure.

But in fact this is not what I like about using maven.

From here on in a lot of tedious and annoying tasks to do with versions and classpaths and tests and libraries and setting up interactive development become very easy and automatic.

I have almost stopped missing PLT scheme's excellent PLaneT system.

There will be further posts.


Here is the pom file:

<project>

  <modelVersion>4.0.0</modelVersion>
  <groupId>com.example</groupId>
  <artifactId>hello-maven-clojure-swank</artifactId>
  <version>1.0-SNAPSHOT</version>
  <name>hello-maven</name>
  <description>maven, clojure, emacs: together at last</description>

  <repositories>
    <repository>
      <id>clojars</id>
      <url>http://clojars.org/repo/</url>
    </repository>
    <repository>
      <id>clojure</id>
      <url>http://build.clojure.org/releases</url>
    </repository>
    <repository>
      <id>central</id>
      <url>http://repo1.maven.org/maven2</url>
    </repository>
  </repositories>

  <dependencies>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure</artifactId>
      <version>1.1.0</version>
    </dependency>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>clojure-contrib</artifactId>
      <version>1.1.0</version>
    </dependency>
    <dependency>
      <groupId>jline</groupId>
      <artifactId>jline</artifactId>
      <version>0.9.94</version>
    </dependency>
    <dependency>
      <groupId>swank-clojure</groupId>
      <artifactId>swank-clojure</artifactId>
      <version>1.2.0-break-SNAPSHOT</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>com.theoryinpractise</groupId>
        <artifactId>clojure-maven-plugin</artifactId>
        <version>1.3.2</version>
      </plugin>
    </plugins>
  </build>

</project>

Moving Average of a List

Someone on Stack Overflow asked how to take the moving average of a list.
These were my answers:

;; The first one is elegant, but inefficient for long windows.
(defn average [lst] (/ (reduce + lst) (count lst)))
(defn moving-average [window lst] (map average (partition window 1 lst)))



;; The second is harder to understand, but keeps a running sum.
(defn partialsums [start lst]
  (lazy-seq
    (if-let [lst (seq lst)] 
          (cons start (partialsums (+ start (first lst)) (rest lst)))
          (list start))))

(defn sliding-window-moving-average [window lst]
  (map #(/ % window)
       (let [start   (apply + (take window lst))
             diffseq (map   - (drop window lst) lst)]
         (partialsums start diffseq))))


;; Here is a list
(def integers (iterate inc 0))

;; Here are some averages of it:
(take 10 (moving-average 5 integers))
(take 10 (sliding-window-moving-average 5 integers))

Athena sprang fully-armed from the forehead of Zeus, and I have presented these functions in a similar spirit.

If anyone is interested in the gory details, I have a long file of increasingly pretty attempts, stack blowing versions, tail recursive versions, timings etc, which could be turned into a post if there is demand.

These two are not necessarily the fastest versions, but they are certainly the most elegant of the various versions that I tried, and fairly careful benchmarking has not shown any of my other versions to be consistently faster.

With the algorithms done, the next step would be to nail down the types and shell out to Java.

Scala vs Clojure

At dev8d last week I was keen to go to the Scala dojo. Scala is another modern JVM language, but much more like ML than Lisp.

While there I translated the fractal tree program from Clojure to Scala.

As you can see, they're effectively the same program.

The command lines to run them are:

$ clj fractaltree.clj
$ scalac ScalaFractalTree.scala && scala ScalaFractalTree

It instantly becomes obvious that the Scala version is considerably faster.

I tried using (set! *warn-on-reflection* true), which told me that I needed to type hint the graphics objects in the clojure program, and it also says something about the proxy JPanel. I don't know how to fix that, but I don't think it's called often enough to be important.

The graphics type hints did seem to speed the Clojure version up a bit, but it's still noticeably slower than the Scala.

I'm presuming that the recursive calls to draw-tree are boxing and unboxing the parameters in the Clojure version.

Does anyone know if that's the case, and if so what can be done about it?

(import '(javax.swing JFrame JPanel )
        '(java.awt Color Graphics Graphics2D))

(defn draw-tree [ #^Graphics g2d angle x y length branch-angle depth]
  (if (> depth 0)
    (let [new-x (- x (* length (Math/sin (Math/toRadians angle))))
          new-y (- y (* length (Math/cos (Math/toRadians angle))))
          new-length (fn [] (* length (+ 0.75 (rand 0.1))))
          new-angle  (fn [op] (op angle (* branch-angle (+ 0.75 (rand)))))]
      (. g2d drawLine x y new-x new-y)
      (draw-tree g2d (new-angle +) new-x new-y (new-length) branch-angle (- depth 1))
      (draw-tree g2d (new-angle -) new-x new-y (new-length) branch-angle (- depth 1)))))

(defn render [ #^Graphics g w h ]
  (doto g
    (.setColor (Color/BLACK))
    (.fillRect 0 0 w h)
    (.setColor (Color/GREEN)))
  (let [init-length ( / (min w h) 5),
        branch-angle (* 10 (/ w h)),
        max-depth 12]
    (draw-tree  g 0.0 (/ w 2) h init-length branch-angle max-depth)))

(defn create-panel []
    "Create a panel with a customised render"

  (proxy [JPanel] []
    (paintComponent [g]
                    (proxy-super paintComponent g)
                    (render g (. this getWidth) (. this getHeight)))))


(defn run []
  (let [frame (JFrame. "Clojure Fractal Tree")
        panel (create-panel)]
    (doto frame
      (.add panel)
      (.setSize 640 400)
      (.setVisible true))))

(run)

Here's the scala version

import scala.swing._
import java.awt._
import javax.swing._

object ScalaFractalTree {
       def main(args: Array[String]){
                      val frame=new JFrame("Scala Fractal Tree")
                      val panel=new MyPanel()
                      frame add panel
                      frame setSize (640, 400)
                      frame setVisible true

                }

       }

class MyPanel extends JPanel{
  override def paintComponent(g:Graphics):Unit = {
    super.paintComponent(g)
    render(g, getWidth(), this.getHeight())
   
  }
  def render(g:Graphics, w:Int, h:Int){
      g.setColor (Color.BLACK)
      g.fillRect( 0, 0, w, h)
      g.setColor (Color.GREEN)
      val initlength=if (w<h) w/5 else h/5
      val branchangle=10*w/h
      val maxdepth=12
      drawtree( g, 0.0, w/2.0, h ,initlength, branchangle, maxdepth)
      }
      
  def drawtree(g:Graphics, angle:Double, x:Double, y:Double, length:Double, 
                             branchangle:Double, depth:Double){
     if (depth>0){
        val newx= x-length*(Math.sin(Math.toRadians( angle)))
        val newy= y-length*(Math.cos(Math.toRadians( angle)))
        val newlength1 = length*(0.75+0.1*Math.random)
        val newlength2 = length*(0.75+0.1*Math.random)
        val newangle1  = angle+branchangle*(0.75+Math.random)
        val newangle2  = angle-branchangle*(0.75+Math.random)
        g.drawLine(x.toInt, y.toInt, newx.toInt, newy.toInt)
        drawtree(g, newangle1, newx, newy, newlength1, branchangle, depth-1)
        drawtree(g, newangle2, newx, newy, newlength2, branchangle, depth-1)
        }
  }
}
                             

How the Clojure Dojo worked in London

I ended up giving my clojure dojo http://learnclojure.blogspot.com/2010/01/clojure-dojo-method-of-heron-of.html twice at dev8d in London, after two dry runs in Cambridge. The results were as follows:

I In Cambridge to the BioInformatics Group of the Chemistry Department (8 people present for 1 1/2 hours)

Worked well, got as far as a general square root solver. Everyone stayed involved and I think everyone enjoyed it.

II In Cambridge to a man randomly encountered in a pub ( 1 person for 1 hour )

Worked a treat, got as far as a general square root solver. That will teach people to ask me what I'm doing when I look busy.

III In London at dev8d in an extra session organised in advance of the main event (5 people for 2 1/2 hours)

There was a non-native English speaker there, and the dojo format quickly proved its effectiveness. When he sat down at the keyboard it was obvious that he hadn't understood much of my introductory talk, but when he was forced to try to construct expressions at the REPL he quickly got the hang of it, and turned out to be very good.

We got through the square root finder very quickly, and then the group wanted to know about macros. I thought this was perhaps a bit much for people who'd only learned the language an hour ago, and instead led them through the symbolic differentiator, as an example of how LISP can process its own code. This is after all the basis of macros!

One girl out of the five had only the sketchiest notion of what a derivative was, so we made her do most of the typing in this second half.

Afterwards, when I pointed out that a program which takes code and gives back other code is usually known as a compiler, and that she'd just written one, her face lit up with pride. This was the most rewarding moment of the week for me.


IV In London at dev8d in the planned session (27 people for 2 1/2 hours)


With so many people it was very hard to keep everyone properly involved, and so we broke the rules and had people who already had a clojure setup working on their own laptops separately from the communal effort.

After an hour and a half or so we made our square root finder, and there was a big round of applause.


I should really have canned it there, but people were still keen and wanted to know more, so we pressed on to do the general Newton-Raphson method solver.

At that point it went a bit wrong:


I'd been expecting Newton's method to be common knowledge amongst programmers (I'm sure it was on my maths A-level), and so to be a nice simple example of how powerful the general iterative-improve strategy is.

But in fact the method seemed to confuse people, and they got distracted from the programming by trying to understand the algorithm. There were too many people in the room for me to make sure that everyone stayed on board.


Nevertheless, they got the thing done with not too much help from me. I think by the end I had lost some of my audience, and I was a bit disappointed.

But afterwards lots of people came up and told me how much they'd enjoyed it, and apparently there were a lot of positive comments made afterwards (I had spies planted to find out what people really thought.).

I suspect that some people loved it, and some people lost the plot and were bored. Sorry to those people! As they say, if someone isn't learning, someone isn't teaching....


I think next time I either need to find a less scary example (it wasn't meant to be scary!!) or find a better way of explaining the strategy without using the fear-inducing word "derivative"! That may in fact not be too hard. Something like 'try moving the guess a bit, and see how much difference that makes, ok how much does that mean we should change our guess by?'.

I'll also see if I can keep the numbers down to below ten for the next one. Then you can rescue people who are floundering rather than ignoring the fact so that the group can progress.

There will be a next time. Various friends have expressed interest, and I enjoyed doing it, so we'll organise some more sessions, and for the advanced students, I'll try and think of some suitable examples that will illustrate what's particularly good about Clojure, once people have got the hang of why LISP itself is so cool.

Conditioning the REPL

Clojure's REPL is fairly minimal on startup.

I find that I always want to load clojure.contrib.repl-utils, amongst other libraries, and I have some handy debugging and exploring functions which I like to have loaded when interacting with my programs.

Also, the otherwise excellent 'apropos' function find-doc only looks in namespaces which have been loaded, so it's no good for finding things when you don't know which namespace they're in.

For this reason, it's useful to require every namespace that you can find.

The number of batteries included in clojure is getting quite phenomenal, and find-doc, doc, and source and javadoc are good tools for exploring.

I also often want to know what the classpath and current directory are, and I sometimes find it frustrating to have to remember whether various functions take a string, a symbol or a quoted symbol.

Recently I've found myself putting all these little tweaks in a file which I can load first thing on starting a REPL. On my netbook requiring everything takes about 30 seconds.

 

;; LOAD ME WITH
;; (load-file "/home/john/hobby-code/require-all-snippet.clj")

;; This file conditions a repl in various ways that I do all the time.

;; Firstly we want to 'require' all the namespaces on the classpath
;; This ensures that find-doc and the like will work
(require 'clojure.contrib.find-namespaces)

;; Some namespaces may fail to load, so catch any exceptions thrown
(defn- require-may-fail [ns]
  (try

   (print "Attempting to require " ns ": ")
   (require ns)
   (println "success")
   (catch Exception e (println "couldn't require " ns "\nException\n" e "\n\n"))))


;; Generally we'd want clojure.*, clojure.contrib.*, and any project-specific namespaces
(defn require-all-namespaces-starting-with [strng]
  (doall (map require-may-fail 
              (filter #(. (str %) startsWith strng) 
                      (clojure.contrib.find-namespaces/find-namespaces-on-classpath)))))


;; The functions in these namespaces are so useful at the REPL that I want them 'use'd.
;; I.e. I want to be able to type 'source' rather than 'clojure.contrib.repl-utils/source'
(use 'clojure.contrib.repl-utils)
(use 'clojure.inspector)
(use 'clojure.contrib.pprint)
(use 'clojure.contrib.repl-utils)
(use 'clojure.contrib.trace)

;; It drives me up the wall that it's (doc re-pattern) but (find-doc "re-pattern").
;; Can use macros so that (fd re-pattern) (fd "re-pattern") and (fd 're-pattern) all mean the same thing
(defn- stringify [x]
  (println "stringify given" (str x))
  (let [s  (cond (string? x) x
                 (symbol? x) (str x)
                 (and (list? x) (= (first x) 'quote)) (str (second x))
                 :else (str x)) ]
    (println (str "translating to: \"" s "\""))
    s))



;; Sometimes I like to ask which public functions a namespace provides.
(defn- ns-publics-list [ns] (#(list (ns-name %) (map first (ns-publics %))) ns))
;; And occasionally which functions it pulls in (with refer or use)
(defn- ns-refers-list  [ns] (#(list (ns-name %) (map first (ns-refers %))) ns))


;; Nice pretty-printed versions of these functions, accepting strings, symbols or quoted symbol
(defmacro list-publics     
  ([]   `(pprint (ns-publics-list *ns*)))
  ([symbol-or-string] `(pprint (ns-publics-list (find-ns (symbol (stringify '~symbol-or-string)))))))

(defmacro list-refers
  ([]   `(pprint (ns-refers-list *ns*)))
  ([symbol-or-string] `(pprint (ns-refers-list (find-ns (symbol (stringify '~symbol-or-string)))))))

;; List all the namespaces
(defn list-all-ns [] (pprint (map ns-name (all-ns))))

;; List all public functions in all namespaces!
(defn list-publics-all-ns [] (pprint (map #(list (ns-name %) (map first (ns-publics %))) (all-ns))))

;; With all the namespaces loaded, find-doc can be overwhelming.
;; This is like find-doc, but just gives the associated names.

(defn- find-doc-names
  "Prints the name of any var whose documentation or name contains a match for re-string-or-pattern"
  [re-string-or-pattern]
    (let [re  (re-pattern re-string-or-pattern)]
      (doseq [ns (all-ns)
              v (sort-by (comp :name meta) (vals (ns-interns ns)))
              :when (and (:doc ^v)
                         (or (re-find (re-matcher re (:doc ^v)))
                             (re-find (re-matcher re (str (:name ^v))))))]
               (print v "\n"))))





;;find symbol or string in docs 
(defmacro fd [symbol-or-string] `(find-doc (stringify '~symbol-or-string)))

(defmacro fdn [symbol-or-string] `(find-doc-names (stringify '~symbol-or-string)))



;;debugging macro                                try: (* 2 (dbg (* 3 4)))
(defmacro dbg [x] `(let [x# ~x] (do (println '~x "->" x#) x#))) 

;;and pretty-printing version 
(defmacro ppdbg [x]`(let [x# ~x] (do (println "--")(pprint '~x)(println "->")(pprint x#) (println "--") x#))) 


;;and one for running tests 
(defmacro run-test [fn] `(test (resolve '~fn)))


;; Sometimes it's nice to check the classpath
(defn- get-classpath []
   (sort (map (memfn getPath) 
              (seq (.getURLs (java.lang.ClassLoader/getSystemClassLoader))))))

(defn print-classpath []
  (clojure.contrib.pprint/pprint (get-classpath)))

(defn get-current-directory []
  (. (java.io.File. ".") getCanonicalPath))


;;require everything from clojure and clojure.contrib, so that find-doc can find it
(require-all-namespaces-starting-with "clojure")

;;print the classpath
(println "Classpath:")
(print-classpath)

(println "Current Directory" (get-current-directory))

;;print the public functions in the current namespace
(println "Current Namespace")
(list-publics)


;;hint on how to require project specific namespaces
(println "to require all namespaces starting with example:")
(println "(require-all-namespaces-starting-with \"example\")")