Ari Lerner

Neotoma - Super powerful parsing for erlang

2010-04-21T00:00:00-07:00

Erlang strings are painful

Oh it’s so true. The pain is super apparent, especially when trying to parse configuration files. The traditional way to parse a configuration file that is not in the erlang format can be pretty hard to do. For instance, for beehive, the application configuration template looks like:

# Config file
# For example, a rack app
bundle: echo "Bundle java stuff"
start: /bin/rackstart.sh
# etc. etc.

Originally, this was parsed in lex/yacc and consumed in c++ (shudder). The code for that is available buried deep within the history of babysitter

A traditional parser would look something like this:

  -module (config_parser).
  -export ([file/1]).

  -define (SEPARATOR, $:).

  file(Filename) ->
    {ok, Fd} = file:open(Filename, [read]),
    io:setopts(Fd, [binary]),
    for_each_line(Fd, fun parse/3, 1, []).

  for_each_line(Device, Proc, Count, Accum) ->
    case io:get_line(Device, "") of
      eof  -> file:close(Device), Accum;
      Line -> 
        NewAccum = Proc(Line, Count, Accum), 
        for_each_line(Device, Proc, Count + 1, NewAccum)
    end.

  parse(Line, Count, Acc) ->
    [peg_parse(Line, Count)|Acc].

  % START PEG PARSING
  peg_parse(Line, Count) ->
    {Field, X} = parse_line(Line, [], []),
    case Field of
      comment -> ok;
      X -> io:format("[~p] ~p:~p~n", [Count, Field, X])
    end.

  % Top
  % Strip comments
  parse_line(<<$#, _Rest/binary>>, [], _Acc) -> {comment, []};
  % Is this the field?
  parse_line(<<?SEPARATOR, Rest/binary>>, _Field, Acc) -> parse_line(Rest, lists:reverse(Acc), []);
  parse_line(<<Char, Rest/binary>>, Field, Acc) -> parse_line(Rest, Field, [Char|Acc]);
  parse_line(<<>>, Field, Acc) -> {Field, lists:reverse(Acc)};
  parse_line(<<$\n>>, Field, Acc) -> {Field, lists:reverse(Acc)}.

I’ll only touch on the basics of what that is here (so if you want to skip it, just go to the next section).

Basically we open a file descriptor to the file and tell it to read in the binary format (a little faster and less work on the vm). For every line, we go through character by character and examine based on the position and context that the character is in and store the value in the context where it appears. Later we’ll come back (notice where the io:format is?) and store it in some meaningful way. This is just a demo. If there is enough interest, I can finish it and post it here. Otherwise, I won’t spend more time on it as there is a better way.

Introducing Neotoma

Neotoma, a project by Sean Cribbs that makes PEG parsing in erlang easy. It’s a nifty tool that generates an unambiguous parser that generates a parse tree. Don’t try to use this to create a parser to examine natural languages though, it’s not an CFG (context free grammar) parser.

There aren’t too many resources available yet through google.com, so after some head scratching and pm’ing with Sean Cribbs, the author, I was able to sketch the parser to dig out the parse tree for the grammar in a ridiculously little amount of code.

Before we get into the PEG grammar, there are a few basic pieces of terminology that you need to know.

Terminal symbols

There are two disjoint sets of items called terminal symbols. These are basic pieces (or atoms) of a grammar. There are two types of terminals, the non-terminal and the terminal symbol. They must be, by necessity disjoint sets of symbols for a valid grammar. The reason will become apparent shortly.

non-terminal symbol - Symbol representing a ‘variable’ in a grammar. These are symbols that can be replaced by other elements of the grammar.
terminal symbol - Symbols that cannot be broken down any further, but that can be consumed by non-terminals.

Let’s look at an example. These are two non-terminals that describe a signed integer in the BNF form of a grammar:

  <integer> ::= ['-'] <digit> {<digit>}
  <digit> ::= '0' | '1' | '2' | '3' | '4' | '5' | '6' | '7' | '8' | '9'

The integer and the digit are the non-terminals, while the 0-9 and - are the terminal symbols.

PEG grammars

A “parsing expression grammar”, or PEG is an formal description of an analytic language model that describes a set of rules for recognizing strings and their context in the language. For instance, in the English language a proper declarative sentence requires a subject and a predicate

  My name is Mary.
    <subject> = My name
    <predicate> = is Mary

Although you should never use a PEG to parse natural language (use a Backus-Naur Form), the corresponding PEG would look something like (incomplete, but only used for example):

  % Psst, a Neotoma PEG for the sentence "My name is Mary"
  declarative_sentence <- predicate noun ".";
  predicate <- "My name";
  noun <- "is Mary";

This is a VERY basic PEG parser. What does all this mean? Well…

Introducing Neotoma

Let’s look at the simplest possible grammar for describing the exact same thing, a signed integer.

  rule <- Decimal ~;
  Decimal <- [0-9]+ `list_to_integer(Node)`;

That’s it? Yep. This might look a little confusing, but don’t worry, I’ll explain what’s happening here. First, look at the `Decimal` rule. That says that any digit 0 through 9 will be consumed by this rule. The stuff in the “ will be what happens to the matched number. We’ll get back to that soon. The `rule` non-terminal will ‘take’ the Decimal rule in. This won’t do anything interesting yet, other than say that the rule will contain an integer. Let’s change that:

  additive <- multitative "+" additive / multitative ~;
  multitative <- primary "*" multitative / primary ~;
  primary <- "(" additive ")" / decimal ~;
  decimal <- [0-9]+ `list_to_integer(Node)`;

This shouldn’t be too hard to understand from the rule above, but there are two new things in these rules. First, the stuff that happens in between the “ is the semantic part of the parse tree. Sean calls this the ‘transformation’ of the grammar. What it does is stuff all the matches of the rules into a variable called `Node`. From there, you can do basically what you want with what the transformation returns (so long as the rules that use it understand it). If instead of putting stuff in “ at the end of a rule definition, you put a `~`, this says return the Node variable untouched.

The second new introduction in this set of rules is the `/`. This an `or` statement. It’s a precedence-based ‘or’. So, for our rules above, the primary terminal can either be an additive surrounded by parenthesis OR a decimal. Note, I said that precedence is denoted here. The first item in the list is the first item matched. It’s always a good idea (even the author Sean Cribbs suggests) to try to match the longest rule possible. Precedence is important.

Alright, so remember the very incomplete code example from above that didn’t do anything yet, that looks ugly and is exactly 37 lines long? Well, here is a very complete Neotoma version that can parse the entire file:

  % This is the PEG compiler for babysitter configuration files
  config_element <- ws? elem_list ws? `list_utils:merge_proplists(Node)`;
  elem_list <- head:elem tail:(ws? elem)* `
    case Node of
      [] -> [];
      [""] -> [];
      _ ->
        Head = proplists:get_value(head, Node),
        Tail = [R || [_,R] <- proplists:get_value(tail, Node)],
        [Head|Tail]
    end
  `;
  elem <- hook_elem / action_elem ~;
  hook_elem <- action '.' (bef / aft) ':' [ \t]* string (crlf / !.) `
    {lists:nth(1, Node), 
      {lists:nth(3, Node), lists:flatten(lists:nth(6, Node))}
    }
  `;
  action_elem <- action ':' [ \t]* string (crlf / !.) `
    {lists:nth(1, Node), 
      {command, lists:flatten(lists:nth(4, Node))}
    }
  `;

  action <- "bundle" / "mount" / "run" / "unmount" / "cleanup" `erlang:list_to_atom(lists:flatten(Node))`;
  ws <- (comment / space)* `{}`;
  % Atoms
  bef <- "before" `pre`;
  aft <- "after" `post`;

  string <- bracketed_string / nonbracketed_string ~;
  nonbracketed_string <- (!crlf .)* ~;
  bracketed_string <- '{' str:(!'}' (.))* '}' `proplists:get_value(str, Node)`;

  comment <- '#' (!crlf .)* crlf ~;
  space <- [ \t\n\s\r] ~;
  crlf <- "\r\n" / "\n" ~;

That’s it! Looks like a lot, but it’s not as bad once you start looking at it. I’ll leave this as an exercise to look through it for now. Some hints that I picked up to keep in mind either from Sean Cribbs or from working with the code for a while.

(crlf / !.) at the end of a line means either a newline (note the crlf rule at the bottom) or the EOF character at the end of the file
The ’!’ means NOT, it’s a negative look-ahead character, so for example, the nonbracketed_string can be zero or more of any character except the crlf
The ‘string’ rule can be either a string or the stuff inside of {}s. Notice that the bracketed_string comes first in the string rule, this is why precedence matters. Try it the other way
You can assign ‘variables’ to the matches. For instance, look at the bracketed_string rule. The tag is: ‘str:’. This will put the tuple: {str, Value} into the Node variable, so you can pull it out later.

To actually use this, make sure the neotoma.beam is in your code path (or use the -p(a/z) switch to load it) and type:

  neotoma:file("file.peg").

If all the syntax is correct, neotoma will generate a parser with the two exported functions `parse/1` and `file/1`, which you can compile and use at your leisure.

To get a copy of the code discussed in this tutorial in full, clone it from this repository here: http://github.com/auser/neotoma_template.

Some quick links before I go:

* Neotoma source * Video introduction * Parsing Expression Grammar Wikipedia * Google group

Thanks and I hope this helps you figure out Neotoma. Don’t hesitate to ask.

Snow Leopard Erlang woes (and the fix!)

2009-10-12T00:00:00-07:00

After upgrading to Snow Leopard, I found my os_mon erlang application exploded in a very ugly error message.

=CRASH REPORT==== 12-Oct-2009::23:29:20 ===
  crasher:
    initial call: memsup:init/1
    pid: <0.76.0>
    registered_name: memsup
    exception exit: },
                     [{memsup,get_memory_usage,1},
                      {memsup,'-handle_info/2-fun-0-',2}]}
      in function  gen_server:terminate/6
    ancestors: [os_mon_sup,<0.46.0>]
    messages: []
    links: [<0.47.0>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 233
    stack_size: 24
    reductions: 172
  neighbours:

After some investigation, it turns out that Snow Leopard changed their output on vm_stat, the tool to look at the memory available on the system. The new output added the line:

Pages speculative:                   42219.

Where the erlang module memsup depends upon that not being there. The erlang developers get the ugliness. In any case, the patch looks like:

--- a/lib/os_mon/src/memsup.erl
+++ b/lib/os_mon/src/memsup.erl
@@ -728,8 +728,12 @@ get_memory_usage({unix,darwin}) ->
        io_lib:fread("Pages active:~d.", skip_to_eol(Str2)),
     {ok, [Inactive],Str4} =
        io_lib:fread("Pages inactive:~d.", skip_to_eol(Str3)),
+         {ok, _,Str5} =
+       io_lib:fread("Pages speculative:~d.", skip_to_eol(Str4)),
     {ok, [Wired],_} =
-       io_lib:fread("Pages wired down:~d.", skip_to_eol(Str4)),
+       io_lib:fread("Pages wired down:~d.", skip_to_eol(Str5)),
+  %     {ok, [Wired],_} =
+  % io_lib:fread("Pages wired down:~d.", skip_to_eol(Str4)),
     NMemUsed  = (Wired + Active + Inactive) * 4000,
     NMemTotal = NMemUsed + Free * 4000,
     {NMemUsed,NMemTotal};

Save this to a file, such as /tmp/erlang_patch. Full instructions for upgrading your erlang:

git clone git://github.com/mfoemmel/erlang-otp.git
cd erlang-otp
patch -l -i /tmp/erlang_patch -p1
./configure --prefix=/opt/erlang --enable-hipe
make
make install
export PATH=/opt/erlang/bin:$PATH

After that, you should be able to start os_mon:

erl
1> application:start(sasl).
2> application:start(os_mon).

If it starts, you’re done!

Hope this helps.

Fixed typo, thanks to Craig Krigsman

Getting command-line options into erlang

2009-10-04T00:00:00-07:00

So you have your killer erlang application that possibly could make you millions, but it was written in a test environment. Shoot, how do you change that “on the fly” at the application runtime? There are a many different ways this can be accomplished. This post will go over the basics of this typical issue.

Application variables

Application variables must be declared in the .app file for your application. For instance:

{application, killer_app,
 [{description, "The most killer application ever"},
  {modules, []}, {registered,[]},{applications, [kernel,stdlib,sasl]},
  {env, [
    {key, 'value'} % proplist
        ]
...

From here, the key is settable from the command-line simply by passing it (with a little erlang idiom):

erl -pa ./ebin -killer_app key 'new_value'

From within the application, this can be fetched by looking it up:

Value = case application:get_env(killer_app, key) of
  undefined   -> 'default';
  {ok, V}     -> V
end.

Environment variables

Sometimes it’s just easier and the application runtime environment requires that variables need to be fetched from an environment variable. These are also super easy to lookup, arguably even easier:

EnvParam = string:to_upper(erlang:atom_to_list('key')),
Value = case os:getenv(EnvParam) of
  false -> Default;
  E -> E
end.

This can obviously be set the standard way an environment variable is set:

KEY='awesome_value' erl -pa ./ebin

Configuration file

Other times I just want to set my configuration in a file and be done with it, so that deployment is only dependent upon a change of the configuration file. An application configuration file is a newline separated set of proplists. For instance, it might look like:

{port, 8080}.
{log_path, "logs/killer_app.log"}.

These are pretty easy to look up as well, but it’s important to note that the variables set here must be in the application configuration file as shown above. Fetching these variables might look something like:

Proplists = case file:consult("config/config.cfg") of
  {ok, C} -> C;
  O -> O
end,
Value = proplists:get_value(key, Proplists).

I tend to like more niceties than this, don’t you? When fetching from a configuration file, I tend to use a helper:

-module (config).
-include ("killer_app.hrl").
-compile (export_all).

%%--------------------------------------------------------------------
%% Function: Read the config file () -> {ok, Config} | 
%%                                      {error, Reason}
%% Description: Read the configuration data
%%--------------------------------------------------------------------
read() ->
  case read_1(?CONFIG_FILE) of
    {ok, C} -> {ok, C};
    {error, enoent} -> {error, no_file};
    Err -> Err
  end.
 
read_1(Location) ->
  case file:consult(Location) of
    {ok, C} -> C;
    O -> O
  end.
%%--------------------------------------------------------------------
%% Function: get (Key, Config) -> {error, not_found} |
%%                                {ok, Value}
%% Description: Get the value of a config element
%%--------------------------------------------------------------------
get(Key) -> get(Key, read()).
get(_Key, []) ->
  {error, not_found};
get(Key, [{Key, Value} | _Config]) ->
  {ok, Value};
get(Key, [{_Other, _Value} | Config]) ->
  get(Key, Config).

By using that, I can simply call:

config:get(key).

Finally, I hate to clutter my code with all the funkiness of fetching an application variable, so I tend to use a utility that cleans it up pretty nicely.

-module (apps).

-export ([search_for_application_value/3]).

% Find the application config value
search_for_application_value(Param, Default, App) ->
  case application:get_env(App, Param) of
    undefined         -> search_for_application_value_from_config(Param, Default);
    {ok, undefined}   -> search_for_application_value_from_config(Param, Default);
    {ok, V}    -> V
  end.

search_for_application_value_from_config(Param, Default) ->
	case config:get(Param) of
		{error, _} -> search_for_application_value_from_environment(Param, Default);
		V -> V
	end.

search_for_application_value_from_environment(Param, Default) ->
  EnvParam = string:to_upper(erlang:atom_to_list(Param)),
  case os:getenv(EnvParam) of
    false -> Default;
    E -> E
  end.

Using this, I can simply call and I get built-in defaults for free:

AppDir = apps:search_for_application_value(port, 8080, killer_app),

AMQP cake with Ruby: Using Ruby and RabbitMQ together

2009-08-04T00:00:00-07:00

There are a bunch of different ways to get connected to RabbitMQ. Thankfully the community has somewhat rallied behind the queue messaging system and there are a growing number of clients/libraries to aid in accessing the queue’s functionality.

In this post, we’ll explore the different options for connecting a ruby program to the AMQP-driven queue server. So, with no further ado:

AMQP: the grandfather

Installing

gem sources -a http://gems.github.com
sudo gem install tmm1-amqp
# OR, for the developers in the crowd
git clone git://github.com/tmm1/amqp.git
rake gem
sudo gem install amqp-*.gem

Now that we’ve got it installed, let’s write a quick example to get connected:

require "rubygems"
require 'mq'

EM.run {
  AMQP.start(:host => 'localhost') do
    amq = MQ.new
    EM.add_periodic_timer(1) { amq.queue("noises").publish("moo") }
  end
}

Exactly as it looks, we’ll be using EventMachine to provide us with a periodic timer. When the timer has been exhausted, the block will be run, in our case we’ll push the message “moo” to the queue named “noises” at the RabbitMQ host (in this example, it’s at localhost).

Now, let’s consume a message now. This is nearly straight from the examples/ directory of tmm1-amqp library:

require "rubygems"
require 'mq'

AMQP.start(:host => 'localhost') do
  MQ.queue("noises").publish("moo")
  MQ.queue("noises").publish("quack")
  MQ.queue("noises").publish("bark")
  
  i = 0
  
  MQ.queue('noises').subscribe(:ack => true) do |h,m|
    if (i+=1) == 3
      puts 'Shutting down...'
      AMQP.stop{ EM.stop }
    end

    if AMQP.closing?
      puts "#{m} (ignored, redelivered later)"
    else
      puts m
      h.ack
    end
  end
end  

# moo
# quack
# Shutting down...
# bark (ignored, redelivered later)

In this example, we’ve pushed a few sounds to the noises queue and then later subscribe to the queue (and demanding an acknowledgement receipt).

There are alternative methods to pull a message off the queue. For instance, you can use pop (if you don’t need an ack of the message):

queue.pop{ |msg|
  unless msg
    # queue was empty
    p [Time.now, :queue_empty!]

    # try again in 1 second
    EM.add_timer(1){ queue.pop }
  else
    # process this message
    p [Time.now, msg]

    # get the next message in the queue
    queue.pop
  end
}

Up until now, we’ve only used a queue type called: type. Now, there are N+1 ways to skin a cat, so let’s look at sending a queue message everyone who is subscribed to the fanout. First off, to understand how this works, let’s define a fanout. A fanout in RabbitMQ terminology is like a direct exchange, except that it represents a 1:N message delivery pattern. Now, one more, we need to define a binding as well, you’ll see why in just a minute. First, a binding, from the RabbitMQ documentation:

‘When you publish a message, you send a "routing key" along with it, that's used by the exchange when it decides which queues to forward a copy of the message on to. The links between exchanges and queues are created through binding, with a "binding pattern" that is used by the exchange when comparing against routing keys.

Now that we’ve got that under control, let’s actually write one. Again, this is nearly directly from the examples/mq/clock.rb if you’d like to look at a more complete example

require "rubygems"
require "mq"

AMQP.start(:host => 'localhost') do
  def log *args
    p args
  end
  
  clock = MQ.new.fanout('clock')
  EM.add_periodic_timer(1){
    log :publishing, time = Time.now
    clock.publish(Marshal.dump(time))
  }

  amq = MQ.new
  amq.queue('every second').bind(amq.fanout('clock')).subscribe{ |time|
    log 'every second', :received, Marshal.load(time)
  }
  
  amq = MQ.new
  amq.queue('every 5 seconds').bind(amq.fanout('clock')).subscribe{ |time|
    time = Marshal.load(time)
    log 'every 5 seconds', :received, time if time.strftime('%S').to_i%5 == 0
  }
end

Okay, So first off, we created a new fanout and using MQ and EventMachine again, we’ve created a periodic timer that will publish to the clock fanout (again, a 1:N communication). Then we bind (recall a binding is just a link to the queue’s “router”) to the fanout named “clock” and subscribe to it whn there is a new message in the clock queue. Notice we take two differently named queues (one is called “every second” and the other “every 5 seconds”) and bind them to the same fanout. This works because the fanout provides the 1:N communication again.

Now, notice that we are only logging if the time mod 5 is zero, or essentially every 5 seconds. However, if we remove that if block, then you’ll notice that the queue still receives messages. This is because the fanout is publishing the message to EVERYONE who is subscribed to the fanout. That’s how the fanout works.

Now, the AMQP library provides something really kind of neat. You can send “RPC” calls across the wire to the queue. How? Let’s look at another example from the examples/ directory:

require "rubygems"
require 'mq'

AMQP.start(:host => 'localhost') do

  def log *args
    p args
  end

  class HashTable < Hash
    def get(key)
      self[key]
    end
    
    def set(key, value)
      self[key] = value
    end
  end

  server = MQ.new.rpc('hash table node', HashTable.new)

  client = MQ.new.rpc('hash table node')
  client.set(:now, time = Time.now)
  client.get(:now) do |res|
    log 'client', :now => res, :eql? => res == time
  end

  client.set(:one, 1)
  client.get :one do |res| 
    p res
  end
  client.keys do |res|
    log 'client', :keys => res
    AMQP.stop{ EM.stop }
  end

end

Sweetness. We are taking a HashTable object and storing it’s in a Marshalled format (thanks to the amqp gem) into the queue. When we pop it out, we get a full object back.

A few other notes.

We can create a topic exchange, which is kind of like a fanout, except that instead of 1:N, it’s N:N. To do this, we’ll write a mini chat server (well, we’ll pull it from the very smart blog: http://www.randomhacks.net/articles/2009/05/08/chat-client-ruby-amqp-eventmachine-shoes). Some more quick overhead before we get to the code. When we created the binding before, we set a :key that was equivalent to the name of the queue. We can do better than that and specifically set it to an exact queue channel via the bind method:

queue.bind('chat', :key => $channel)

This sets a binding on the chat queue to match any key with the name of the channel in the chat program. You can set wildcards, and arrays in the key parameter. Now, when we publish a message, we’ll use the routing key to ensure it gets sent to the right bindings:

MQ.queue('chat').publish(" -> #{data}", :routing_key => $channel)

Now, on to the entire program:

require 'rubygems'
gem 'amqp'
require 'mq'

unless ARGV.length == 2
  STDERR.puts "Usage: #{$0} <channel> <nick>"
  exit 1
end
$channel, $nick = ARGV

AMQP.start(:host => 'localhost') do
  $chat = MQ.topic('chat')

  # Print any messages on our channel.
  queue = MQ.queue($nick)
  queue.bind('chat', :key => $channel)
  queue.subscribe do |msg|
    if msg.index("#{$nick}:") != 0
      puts msg
    end
  end

  # Forward console input to our channel.
  module KeyboardInput
    include EM::Protocols::LineText2
    def receive_line data
      $chat.publish("#{$nick}: #{data}",
                    :routing_key => $channel)
    end
  end
  EM.open_keyboard(KeyboardInput)
end

Last example of the day: We can distribute work in our programs as well.

require "rubygems"
require 'mq'

MAX = 1000

# logging
def log *args
  p args
end

# spawn workers
workers = ARGV[0] ? (Integer(ARGV[0]) rescue 1) : 1
AMQP.fork(workers) do

  log MQ.id, :started

  class Fixnum
    def prime?
      ('1' * self) !~ /^1?$|^(11+?)\1+$/
    end
  end

  class PrimeChecker
    def is_prime? number
      log "prime checker #{MQ.id}", :prime?, number
      number.prime?
    end
  end

  MQ.rpc('prime checker', PrimeChecker.new)

end

# use workers to check which numbers are prime
AMQP.start(:host => 'localhost') do
  
  prime_checker = MQ.rpc('prime checker')

  (10_000...(10_000+MAX)).each do |num|
    log :checking, num

    prime_checker.is_prime?(num) { |is_prime|
      log :prime?, num, is_prime
      (@primes||=[]) << num if is_prime
      
      if (@responses = (@responses || 0) + 1) == MAX
        log :primes=, @primes
        EM.stop_event_loop
      end
    }

  end
  
end

By using the fork method in AMQP, we are forking several consumers of the messages in the ‘prime checker’ rpc queue. This shows just how easy it is to multithread a consumer for an amqp queue.

Using mochiweb to create a web framework in erlang

2009-07-29T00:00:00-07:00

Recently, I used Mochiweb for several projects (Alice) I’ve been working on. After some investigation of the current erlang web frameworks, Mochiweb suited our needs well. It’s lightweight, fast, open-source and pretty source code. Throughout this post, we’ll build a little mochiweb application, so note that it will be available in full for download at the end of the article, but we’ll write bits and pieces at a time.

So with no further ado:

Using mochiweb in erlang

Starting mochiweb is pretty straightforward. Just calling the mochiweb_http:start/1 function, we’ll start the mochiweb application. This is, to me the most dynamic way to start a mochiweb application. While it can be done in a different way, this allows more flexibility. Notice that we’ll be passing a loop function. We’ll define that shortly, but for now, just note that it’s the function that will be receiving the requests.

-module (mochiweb_server).
-export ([start_mochiweb/1]).

start_mochiweb(Args) ->
  [Port] = Args,
  io:format("Starting mochiweb_http with ~p~n", [Port]),
  mochiweb_http:start([ {port, Port},
                        {loop, fun dispatch_requests/1}]).

Now, as promised, let’s look at how to handle we’ll handle the requests:

-export ([dispatch_requests/1]).

% ...

dispatch_requests(Req) ->
  Path = Req:get(path),
  Action = clean_path(Path),
  handle(Action, Req).

% Get a clean path
% strips off the query string
clean_path(Path) ->
  case string:str(Path, "?") of
    0 -> Path;
    N -> string:substr(Path, 1, N - 1)
  end.

The Req record that is passed in is a mochiweb_request record, which gives us access to all the methods defined in the mochiweb_request record. We’ll use the Req:get(path) method to pull out the path. Notice that we are also pulling out the Action the path defines by stripping off any query string at the end. Sweet.

Now, for some nifty request handling, we’ll use the handle method to give us the ability to handle requests with Erlang’s pattern matching:

handle("/favicon.ico", Req) -> Req:respond({200, [{"Content-Type", "text/html"}], ""});
handle(Path, Req) -> 
	Req:respond({200, [{"Content-Type", "text/html"}], "<h3>Hello world</h3>"});

Sweet! Digging a little deeper, we can see that any request that is not /favicon.ico is going to respond with Hello World (Also, notice the use of respond on the Req record). The respond method takes a tuple that consists of:

{status, [{proplist_of, headers}], Body}

So obviously we can respond in different ways to our different requests. Let’s dig a little deeper and build out some controllers, an obvious enhancement. First, we’ll modify our handle method:

handle(Path, Req) ->
  BaseController = lists:concat([top_level_request(clean_path(Path)), "_controller"]),
  CAtom = list_to_atom(BaseController),
  ControllerPath = parse_controller_path(clean_path(Path)),
  
  case CAtom of
    home ->
      IndexContents = ?ERROR_HTML("Uh oh"),
      Req:ok({"text/html", IndexContents});
    ControllerAtom -> 
    Meth = clean_method(Req:get(method)),
    case Meth of
      get -> 
        run_controller(Req, ControllerAtom, Meth, [ControllerPath]);
      _ -> 
        run_controller(Req, ControllerAtom, Meth, [ControllerPath, decode_data_from_request(Req)])
    end
  end.

% parse the controller path
parse_controller_path(CleanPath) ->
  case string:tokens(CleanPath, "/") of
    [] -> [];
    [_RootPath|Rest] -> Rest
  end.
    
% Call the controller action here
run_controller(Req, ControllerAtom, Meth, Args) ->
  case (catch erlang:apply(ControllerAtom, Meth, Args)) of
    {'EXIT', {undef, _}} = E ->
      Req:ok({"text/html", "Unimplemented: there is nothing to see here"});
    {'EXIT', E} -> 
      Req:not_found();
    Body -> 
      Req:ok({"text/html", Body})
  end.

% Other methods
% Get the data off the request
decode_data_from_request(Req) ->
  RecvBody = Req:recv_body(),
  Data = case RecvBody of
    <<>> -> erlang:list_to_binary("{}");
    Bin -> Bin
  end,
  {struct, Struct} = mochijson2:decode(Data),
  Struct.

% parse the controller path
parse_controller_path(CleanPath) ->
  case string:tokens(CleanPath, "/") of
    [] -> [];
    [_RootPath|Rest] -> Rest
  end.

% Get a clean path
% strips off the query string
clean_path(Path) ->
  case string:str(Path, "?") of
    0 -> Path;
    N -> string:substr(Path, 1, N - 1)
  end.

top_level_request(Path) ->
  case string:tokens(Path, "/") of
    [CleanPath|_Others] -> CleanPath;
    [] -> "home"
  end.

Now we can use any controller with the path appended to call out to a controller of our choosing that respond to the four http methods, get, put, post and delete! To finish off, let’s add a controller that responds with our hello world message:

-module (home_controller).
-export ([get/1, post/2, put/2, delete/2]).

get(Path) -> "hello world".

post(_Path, _Data) -> "unhandled".
put(_Path, _Data) -> "unhandled".
delete(_Path, _Data) -> "unhandled".

Now we have an application-scalable web framework written in erlang with mochiweb.

This code can be retrieved at: /code/erlang/rest_server.erl.

Thanks to damjan for pointing out the clean_path correction.

Announcing Alice and Wonderland

2009-07-13T00:00:00-07:00

Alice

http://alicetheapp.com.

As a queue server, RabbitMQ is super cool, but my company is hesitant to use it without a nice front-end or access to statistics about the server. So we set out to develop the latest RabbitMQ REST interface, Alice.

Alice is a RESTful interface to the RabbitMQ server that talks directly through erlang’s native interface, epmd. The purely RESTful server responds to the same interface as the RabbitMQ’s command-line interface and presents a native HTTP interface to the data. Alice is written with Mochiweb.

Quickstart

How to get started.

git clone git://github.com/auser/alice.git
cd alice
./start.sh

Currently exposed RESTful routes

  /conn - Current connection information
  /exchanges - Current exchanges information
  /queues - Current queues
  /users - Current users
  /bindings - Current bindings
  /control - Access to the RabbitMQ control
  /permissions - Current permissions
  /vhosts - Current vhosts

These endpoints all are exposed with the four verbs (get, post, put, delete) and respond in the JSON format, (except the root / endpoint which responds with text/html).

Usage

Users

# List users
curl -i http://localhost:9999/users 
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:08:20 GMT
Content-Type: text/json
Content-Length: 19

{"users":["guest"]}

# Viewing a specific user
curl -i http://localhost:9999/users/guest
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:01:01 GMT
Content-Type: text/json
Content-Length: 17

{"users":"guest"}

# If the user is not a user:
curl -i http://localhost:9999/users/bob  
HTTP/1.1 400 Bad Request
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:01:20 GMT
Content-Type: text/json
Content-Length: 20

{"bob":"not a user"}

# Add a user
curl -i -XPOST \
        -d'{"username":"ari", "password":"weak password"}' \
        http://localhost:9999/users
        
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Thu, 16 Jul 2009 00:10:35 GMT
Content-Type: text/json
Content-Length: 25

{"users":["ari","guest"]}

# Deleting a user
curl -i -XDELETE  http://localhost:9999/users/ari
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:19:24 GMT
Content-Type: text/json
Content-Length: 19

{"users":["guest"]}

Notice that when we list the user that doesn’t exist, bob from the second example above, the return is a 400. This is especially useful when you want to access the data programmatically. More on extending Alice below and how to get access to the return value of the requested route.

The same basic usage is applied to all the routes listed, as you can see:

Connections

# List connections
curl -i http://localhost:9999/conn
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:30:52 GMT
Content-Type: text/json
Content-Length: 287

{"conn":[{"pid":"...","ip":"127.0.0.1","port":"5672","peer_address":"127.0.0.1" ...}]}

Exchanges

# List the current exchanges
curl -i http://localhost:9999/exchanges
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:34:14 GMT
Content-Type: text/json
Content-Length: 654

{"exchanges":[{"name":"amq.rabbitmq.log","type":"topic","durable":"true","auto_delete":...}

Queues

# List the current queues
curl -i http://localhost:9999/queues   
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:35:42 GMT
Content-Type: text/json
Content-Length: 60

{"queues":[{"memory":"212988","name":"noises","vhost":"/"}]}

Bindings

# List the current bindings
curl -i http://localhost:9999/bindings
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:36:13 GMT
Content-Type: text/json
Content-Length: 69

{"bindings":[{"queue":"noises","exchange":"","from_queue":"noises"}]}

Permissions

# List permissions
curl -i http://localhost:9999/permissions
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:37:32 GMT
Content-Type: text/json
Content-Length: 42

{"permissions":[["guest",".*",".*",".*"]]}

# You can specify permissions on a vhost
curl -i http://localhost:9999/permissions/vhost/root
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:50:33 GMT
Content-Type: text/json
Content-Length: 42

{"permissions":[["guest",".*",".*",".*"]]}
# Setting permissions
curl -i -XPOST -d '{"vhost":"/", "configure":".*", "read":".*", "write":".*"}' \
  http://localhost:9999/permissions/guest
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:55:33 GMT
Content-Type: text/json
Content-Length: 38

{"permissions":[["/",".*",".*",".*"]]}

Vhosts

# List vhosts
curl -i http://localhost:9999/vhostsHTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:57:10 GMT
Content-Type: text/json
Content-Length: 16

{"vhosts":["/"]}

# Viewing a specific vhost
curl -i http://localhost:9999/vhosts/barneys%20list
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:59:29 GMT
Content-Type: text/json
Content-Length: 25

{"vhosts":"barneys list"}

# If it doesn't exist:
curl -i http://localhost:9999/vhosts/barneys%20listings
HTTP/1.1 400 Bad Request
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:59:59 GMT
Content-Type: text/json
Content-Length: 34

{"barneys listings":"not a vhost"}

# Add a vhost
curl -i http://localhost:9999/vhosts -XPOST -d'{"name":"barneys list"}'
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 07:58:09 GMT
Content-Type: text/json
Content-Length: 31

{"vhosts":["/","barneys list"]}

# Delete a vhost
curl -XDELETE -i http://localhost:9999/vhosts/barneys%20list
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:02:44 GMT
Content-Type: text/json
Content-Length: 16

{"vhosts":["/"]}

Now, there is a module in the Alice called control. There are a lot of routes and a lot of functionality built-in here, so let’s dig in.

Control

# Getting the status of the server
curl -i http://localhost:9999/control 
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:05:19 GMT
Content-Type: text/json
Content-Length: 151

{"status":[{"applications":["rabbit","mnesia","os_mon","sasl","stdlib","kernel"], \
"nodes":["rabbit@YPCMC05591"],"running_nodes":["rabbit@YPCMC05591"]}]}

# Stopping the rabbitmq-server
curl -XPOST -i http://localhost:9999/control/stop  
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:06:02 GMT
Content-Type: text/json
Content-Length: 20

{"status":"stopped"}

# Starting the rabbitmq-server application
curl -XPOST -i http://localhost:9999/control/start_app
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:06:50 GMT
Content-Type: text/json
Content-Length: 20

{"status":"started"}

# Stopping the rabbitmq-server application
curl -XDELETE -i http://localhost:9999/control/stop_app
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:15:56 GMT
Content-Type: text/json
Content-Length: 20

{"status":"stopped"}

# Reset the rabbitmq-server application
curl -XPOST -i http://localhost:9999/control/reset    
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:07:15 GMT
Content-Type: text/json
Content-Length: 18

{"status":"reset"}

# Or force-resetting the server
curl -XPOST -i http://localhost:9999/control/force_reset
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:07:27 GMT
Content-Type: text/json
Content-Length: 18

{"status":"reset"}

# Clustering a set of nodes
curl -XPOST -i http://localhost:9999/control/cluster -d'{"nodes":["bob@otherhost"]}'
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:14:10 GMT
Content-Type: text/json
Content-Length: 20

{"status":"cluster"}

# Rotating rabbit logs
curl -XPOST -i http://localhost:9999/control/rotate_logs -d'{"prefix":"mn_"}'
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:15:12 GMT
Content-Type: text/json
Content-Length: 25

{"status":"rotated_logs"}

Extending

Alice is written with the intention of being highly extensible and makes it easy to do so. The controllers respond only to the four verbs with pattern-matching on the routes.

For instance, a very basic controller looks like this:

-module (say).
-export ([get/1, post/2, put/2, delete/2]).

get([]) -> {"hello", <<"world">>};
get(_Path) -> {"error", <<"unhandled">>}.

post(_Path, _Data) -> {"error", <<"unhandled">>}.
put(_Path, _Data) -> {"error", <<"unhandled">>}.
delete(_Path, _Data) -> {"error", <<"unhandled">>}.

There are the 4 RESTful verbs that the controller responds. Now, if you were to compile this in Alice (in src/rest_server/controllers), then the route http://localhost:9999/say would now be accessible. Cool!

curl -i http://localhost:9999/say
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:20:57 GMT
Content-Type: text/json
Content-Length: 17

{"hello":"world"}

Now let’s add a route to say hello to someone:

-module (say).
-export ([get/1, post/2, put/2, delete/2]).

get([Name]) -> {"hello", erlang:list_to_binary(Name)};
get([]) -> {"hello", <<"world">>};
% ....

curl -i http://localhost:9999/say/ari
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:21:54 GMT
Content-Type: text/json
Content-Length: 15

{"hello":"ari"}

Finally, with every other verb than get, we are given data to extract. Let’s see how to pull some data in a post. The data is given as a proplist with binary keys, we it’s pretty easy to pull them out:

% ...
post([], Data) ->
  Name = erlang:binary_to_list(proplists:get_value(<<"name">>, Data)),
  {"hello back", erlang:list_to_binary(Name)};
post([]) -> 
% ...

Let’s check it:

curl -i http://localhost:9999/say -XPOST -d'{"name":"ari"}'
HTTP/1.1 200 OK
Server: MochiWeb/1.0 (Any of you quaids got a smint?)
Date: Tue, 04 Aug 2009 08:23:54 GMT
Content-Type: text/json
Content-Length: 20

{"hello back":"ari"}

It’s as easy as pie to extend Alice.

Wonderland

Wonderland is the webUI to Alice. It is driven by the javascript framework Sammy and Alice in the backend. Because the framework is client-side and accesses the data through ajax, Wonderland can be deployed nearly anywhere.

Quickstart

cd alice
make wonderland

Screenshots:

Check these two projects out on github at:
http://github.com/auser/alice
http://github.com/auser/wonderland.

Community

Or feel free to ping me on email (arilerner dot mac dot com) if you have any questions.

Alice, the app http://alicetheapp.com

Makefiles making erlang easy

2009-07-10T00:00:00-07:00

Update (01/04/2010): Added a github repos with a project template for starting new projects here: http://github.com/auser/erlproject_template.

The Rakefile in a ruby project is almost just as important as the code itself. Ask any rubyish to show you their project and you can bet your bottom dollar that nine out of every 10 projects of theirs has a Rakefile (most of the time, it’s 10/10). This is one thing that can make starting an erlang project painful… the Makefile (bum bum buuuummmm). Today, I’ll share a Makefile (for my own future reference too!) that works really well for me and my projects.

I’ve attached a sample project directory to get started (for the impatient, you can them here if you’d like to follow along). It includes a sample application file, a sample gen_server and of course all the files in this post. Let’s get started:

First, the Makefile:

# Makefile
LIBDIR		= `erl -eval \
  'io:format("~s~n", [code:lib_dir()])' -s init stop -noshell`
VERSION		= 0.0.1
CC  			= erlc
ERL     	= erl
EBIN			= ebin
CFLAGS  	= -I include -pa $(EBIN)
COMPILE		= $(CC) $(CFLAGS) -o $(EBIN)
EBIN_DIRS = $(wildcard deps/*/ebin)

all: mochi ebin compile
all_boot: all make_boot
start: all start_all

mochi:
  @(cd deps/mochiweb;$(MAKE))

compile:
  @$(ERL) -pa $(EBIN_DIRS) -noinput +B \
  -eval 'case make:all() of up_to_date -> halt(0); \
        error -> halt(1) end.'

edoc:
  @echo Generating $(APP) documentation from srcs
  @erl -noinput -eval 'edoc:application($(APP), "./", \
        [{doc, "doc/"}, {files, "src/"}])' -s erlang halt
  
make_boot:
  (cd ebin; erl -pa ebin -noshell \
    -run make_boot write_scripts rest_app)

start_all:
  (cd ebin; erl -pa ebin -noshell -sname _name_ -boot _name_)

ebin:
  @mkdir ebin

clean:
  rm -rf ebin/*.beam ebin/erl_crash.dump erl_crash.dump
  rm -rf ebin/*.boot ebin/*.rel ebin/*.script
  rm -rf doc/*.html doc/*.css doc/erlang.png doc/edoc-info

This particular project (not yet announced) uses mochiweb (and it’s a good example to show dependencies, so I left it), so we have a task called mochi so that we compile all of the mochiweb sources. Before showing the EMakefile, which is what drives the compile task, it’s important to note that there is also a make_boot task that creates a boot file for the project. This is pretty interesting, so we’ll dive into that real quick:

% make_boot.erl
-module(make_boot).
-export([write_scripts/1]).

write_scripts(Args) -> 
  [Name] = Args,
  io:format("write_scripts for ~p~n", [Name]),
  Erts = erlang:system_info(version),
  application:load(sasl),
  Version = "0.1",
  {value, {kernel, _, Kernel}} = lists:keysearch(kernel, 1,
    application:loaded_applications()),
  {value, {stdlib, _, Stdlib}} = lists:keysearch(stdlib, 1,
    application:loaded_applications()),
  {value, {sasl, _, Sasl}} = lists:keysearch(sasl, 1,
    application:loaded_applications()),

  Rel = "{release, {\"~s\", \"~s\"}, {erts, \"~s\"}, ["
          "{kernel, \"~s\"}, {stdlib, \"~s\"}, 
          {sasl, \"~s\"}, {~s, \"~s\"}]}.",
  
  Lowername = string:to_lower(Name),

  Filename = lists:flatten(Lowername ++ ".rel"),
  io:format("Writing to ~p (as ~s)~n", [Filename, Lowername]),
  {ok, Fs} = file:open(Filename, [write]),

  io:format(Fs, Rel, [Name, 
                      Version, 
                      Erts, 
                      Kernel, 
                      Stdlib, 
                      Sasl, 
                      Lowername, 
                      Version]),
  file:close(Fs),

  systools:make_script(Lowername, [local]),
  halt().

To actually write a bootfile, we need to supply the name of the bootfile to the method call, so we call this like so:

% shell command
erl -pa ebin -noshell -run make_boot write_scripts rest_app

Finally, let’s make sure we have a .app file in the ebin/ directory, sample one below:

% _name_.app
{application, _name_, [
  {description, "_Name_"},
  {vsn, "0.1"},
  {modules, [_modules_]},
  {env, [
    {port, 9999}
  ]},
  {registered, [_name_]},
  {applications, [kernel, stdlib]},
  {mod, {_name_, []}}
]}.

Lastly, break out the EMakefile so that we can actually compile the project:

% EMakefile
% -*- mode: erlang -*-
{["src/*", "src/*/*", "src/*/*/*"],
 [{i, "include"},
  {outdir, "ebin"},
  debug_info]
}.

Hurray, we are almost set. Now let’s make a start.sh file so we can just call that to start the application:

#!/bin/sh
# start.sh
cd `dirname $0`
erl -pa $PWD/ebin -pa $PWD/deps/*/ebin \
    -sname alice -s reloader -boot rest_app $1

Easy as pie.

To show you why this is a great setup, simply navigate to the project directory and type

make && start.sh

And your server should start right up.

A quick-tip I’ve picked up… get rstakeout so that anytime you change a file in the src/ directory, your application will recompile:

rstakeout "make" "src/**/*"

Lemme know if this helps, I love to hear feedback!

Download project files here

Update

I’ve added a generic makefile generator in my erlang snippet project: Skelerl. Get it on github and type:

makefile app_name

New blog engine, same ole' blog

2009-06-23T00:00:00-07:00

Welcome to the new WillCodeForFoo.com blog. It’s gone through plenty of changes and is now going through yet one more! Powered by Jekyll!