Sergi Mansilla

My talk at DynCon 2011

2011-04-04T00:00:00-07:00

The video with my talk about Cloud9 IDE at DynCon 2011 is already out! It’s quite long, but definitely worth it if you are interested in the future of the IDE, since I had the chance to answer many questions about it during the talk. About half of the video contains demo and questions from the audience, as it was quite interactive.

Also, I want to thank Peter Svensson for creating such a wildly interesting conference and being a really great host and organizer. It is a very, very hard job that he pulled off amazingly well.

This talk was especially interesting to me because although the audience included mainly professional developers, most of them weren’t using JavaScript as their main language (as is often the case in our talks). It was fun to show them the power of JavaScript. And as always when you add people with different backgrounds to a discussion, ideas and good questions quickly started popping up. It was great. We’re also working hard at getting more languages incorporated, so hopefully we can have more of these types of discussions in the future, hang in there!

The best part was to see how programmers from all kinds of backgrounds were equally interested in having the advantages of a development platform like Cloud9 on the Cloud. It is incredibly motivating to get approval and excitement from the very audience that we are targeting with Cloud9.

As a side note, it is amazing how much our own perception of past events diverges from the real thing. I thought the talk was quite fluent, but I can (painfully) see that I could have used more practice (LOTS of “ums”, especially at the beginning!). Ah well…it was my first big public talk, so I guess this is to be expected :) I’ve taken notes and my next talk will be much better!

Still, it’s a talk I’m proud of about a product that I’m proud of. Check it out if you want to see a glimpse of the future of development!!

Update: Find the slides for the talk here

Dynamically generating MIDI in JavaScript

2010-11-08T00:00:00-08:00

Last weekend I open-sourced a small side project consisting of a JavaScript library that generates MIDI in a simple manner. As an example, the code to generate a MIDI file that plays 3 notes (C, E and G) would look like the following:

// We pass some notes to |MidiWriter.createNote| to create the MIDI
// events that define the notes that will be in the final MIDI stream. If
// no other parameters are specified to |createNote|, a NoteOff event
// will be inserted automatically, instead of letting the note ring forever.

// Disregard the |push.apply|, it is used here simply to flatten the
// resulting array, since |createNote| returns an array of events.

var noteEvents = [];
["C4", "E4", "G4"].forEach(function(note) {
    Array.prototype.push.apply(noteEvents, MidiEvent.createNote(note));
});

// Create a track that contains the events to play the notes above
var track = new MidiTrack({ events: noteEvents });

// Creates an object that contains the final MIDI track in base64 and some
// useful methods.
var song  = MidiWriter({ tracks: [track] });

// Alert the base64 representation of the MIDI file
alert(song.b64);

// Play the song
song.play();

// Play/save the song (depending of MIDI plugins in the browser). It opens
// a new window and loads the generated MIDI file with the proper MIME type
song.save();

The library is work in progress, but it already outputs standard MIDI files (SMF) encoded in base64, and can play them in any browser that has a plugin that can reproduce MIDI installed. It is usually Quicktime in most machines. And now, please allow me to rant a little bit…

The surprisingly bad state of MIDI in the browser

While developing this small library I was shocked at how inconsistent the state of MIDI reproducibility is in the browser. I might have been very naive, but I expected that a MIDI file would just sound right away inside a browser. Boy was I wrong.

Disclaimer: Any time I talk about reproducing MIDI, I am trying to do so using a data URI encoded in base64. Although all the browsers mentioned should be compatible with it, there are some bugs related to how data URI is supported when using it along with external plugins.

The new and shiny HTML5 <audio> element doesn’t support MIDI in any browser, so I had to revert to the good ol’ <embed> element, which supports it ONLY if the user has a proper plugin installed. All the browsers are using the Quicktime 7 plugin (I was very surprised to see that the mighty VLC can’t reproduce MIDI out of the box).

In some cases, even with the plugin installed, things didn’t work. That is the case for Webkit in Mac, which can’t reproduce the MIDI file generated by the library, while Webkit on Windows and Firefox 4b6 (Mac/Win) have no problems with it. I suspect this is due to bad interaction of the plugins with data URIs.

Of course you can always try to force a file download, but that’s not the point. It would be very nice if something as basic as cross-browser MIDI reproducibility would exist, and it is surprising that it is not possible.

But its inconsistent behavior isn’t actually the worse part. The worse part is that the user has to manually install a plug-in if she wants to be able to reproduce MIDI inside the browser. Of course iTunes and Media Player can play MIDI, but that’s not the experience I am aiming for. MIDI is still a strong standard nowadays (apparently the new iOS 4.2 is betting hard for it), specially in the world of professional musicians. For some reason this format has been ditched from HTML5 audio, or was never considered for it, leaving MIDI reproduction in the browser in the hands of mediocre plugins and careless browser vendors.

Try it!

Go and try it! Find bugs, suggest improvements to the API or features you would like to see there. And if you find solutions or have ideas on how to improve MIDI reproducibility in the browser, please don’t hesitate to tell me!

You can find the library here. Remember that it is not complete and there is heavy work in progress going on, so expect that new versions will break compatibility (certainly) and keep updating it because I will improve it during the following weeks.

Writing a JavaScript interpreter for DBN using PEG.js and canvas (Part II)

2010-08-20T00:00:00-07:00

In this article I build an interpreter for the DBN language from the parsed AST generated by the grammar we defined in the previous article in PEG.js. If you haven’t read the first part, I strongly reccomend to do so, otherwise this will make little sense to you.

It should be quite easy to follow by just looking at the code and its comments. Still, I will be explaining how the interpreter works as I put the code examples in.

Choices and assumptions

Due to little time and given the lack of any reference of the language, I made some assumptions and I left out some advanced features of the language. I hope I can include them soon into the project.

Scope

Since I don’t know how scope works in DBN, I assumed Lexical scoping, given that it is the most common among modern programming languages (JavaScript has lexical scoping as well, for example).

The interpreter

The interpreter takes care of evaluating the arguments for each statement and then execute the statement with its parsed arguments and the given scope, if any:

var Interpreter = function(canvas) {
    this.canvas = canvas
    this.ctx = canvas.getContext('2d')
    this.vars = {}
}

Interpreter.prototype = {
    evalType: function (e, scope) {
        return Interpreter.typeTable[e.type].call(this, e, scope)
    },
    evalExpression: function(ast, scope) {
        _.each(ast, function (c) {
            // Parse the arguments real value
            var args = _.map(c.args, function(a) {
                return this.evalType(a, scope)
            }, this)
            Interpreter.expTable[c.name].call(this, {
                args: args,
                block: c.block,
                scope: scope
            })
        }, this)
    },
    reset: function() {
        var ctx = this.ctx
        // Resets all cnavas state and properties
        this.canvas.width = this.canvas.width
        this.r = { vars: {}, cmds: {} }
    }
}

The global variables will reside in the vars property, and the most important methods here are obviously evalType and evalExpression.

Evaluating types

evalType receives the type object and its scope and calls the proper static method in Interpreter.typeTable with the same parameters. Interpreter.typeTable holds the functions that interpret the different types in DBN:

// In DBN, everything is an integer or a set of integers, and the developer has
// the possibility to put the values directly in the code or store them in
// variables.

Interpreter.typeTable = {
    "command": function(e, scope) {
        var values = _.map(e.args, function(a) {
                return this.evalType(a, scope)
            }, this)
        return Interpreter.cmdTable[e.name].apply(this, values)
    },
    "string": function(e, scope) {
        var vars = this.r.vars
        var v = e.value

        if (scope && scope.hasOwnProperty(v)) return scope[v]
        else if (vars.hasOwnProperty(v))      return vars[v]

        return v
    },
    "integer": function(e, scope) { return e.value },
    "point": function(e, scope) {
        return {
            type: "point",
            x: this.evalType(e.x, scope),
            y: this.evalType(e.y, scope)
        }
    }
}

The types are the following:

Command

A user-defined function in DBN jargon. The user can define custom commands with parameters that execute a block of code when called.

String

Used mainly for variable names. Given a string from a generated AST, I check whether the variable is defined in its local scope (in case there is one) or in the global scope. In case it is in neither of them, I assume that it is the name of a parameter and doesn’t have an associated value.

Integer

The basic type. Its value is returned right away.

Point

A point refers to a coordinate in the canvas. It contains x and y values that can be integers, variables or arithmetic expressions. Returns an object with the evaluated x and y values.

Evaluating expressions

Expressions are DBN functions. In this interpreter, everything that computes is considered a function, including the arithmetic operators.

evalExpression takes an AST (or a portion of one, as it is done for code blocks) and loops through every statement, resolving the types of its arguments using evalType and calling the proper static method in Interpreter.expTable with them. Interpreter.expTable is a dictionary that stores the supported DBN expressions’ equivalents in JavaScript (same as with types).

Inside the Interpreter.expTable we can find the following kinds of expressions:

Arithmetic expressions

"*": function(a, b) { return parseInt(a * b) },
"/": function(a, b) { return parseInt(a / b) },
"+": function(a, b) { return parseInt(a + b) },
"-": function(a, b) { return parseInt(a - b) }

Arithmetic expressions are always used inside other expressions parameters, never by themselves. The functions are called always with 2 arguments that have been already resolved by evalType, so they are integers when they get to the expression.

Drawing expressions

/*
 * paper will fill up the canvas with the specified color (the only argument
 * of the expression)
 */
paper: function(p) {
    var ctx = this.ctx
    var w = ctx.canvas.clientWidth
    var h = ctx.canvas.clientHeight

    ctx.fillStyle = Interpreter.gray2rgb(p.args[0])
    ctx.fillRect(0, 0, w, h)
},
pen: function(p) {
    this.ctx.strokeStyle = Interpreter.gray2rgb(p.args[0])
},
line: function(p) {
    var a = p.args
    var ctx = this.ctx
    var height = ctx.canvas.clientHeight

    ctx.moveTo(a[0], (height-a[1]))
    ctx.lineTo(a[2], (height-a[3]))
    ctx.closePath()
    ctx.stroke()
}

The main difference between the DBN ‘canvas’ and HTML5 canvas is that the Y coordinate is inverted; in DBN the Y coordinate starts at the bottom whereas in HTML5 canvas it starts at the top. The other important difference is that DBN’s pixels color can only be a gray value from 0 (black) to 100 (white), so we have to use the very simple gray2rgb method to translate them to canvas rgb syntax:

/*
 * gray2rgb transforms a 0 to 100 gray value into a rgb
 * string compatible with canvas
 */
Interpreter.gray2rgb = function(gray) {
    var value = Math.ceil(255 - (gray * 2.55))
    return "rgb(" + value + "," + value  +"," + value + ")"
}

With these two changes in place it is quite straightforward to translate drawing functions using the basic HTML5 canvas functions and passing the arguments properly.

Block expressions

A block expression is a statement that contains a block of nested expressions. DBN’s repeat and command statements are the ones implemented here that belong to this kind. The peculiarity of block expressions is that they have a local scope:

repeat: function(p) {
    var args = p.args,
        id = args[0], f = args[1], c = args[2],
        // Cloning context object because otherwise we will change
        // properties of the parent context, messing up everything
        scopeObj = _.clone(p.scope) || {}

    _.each(_.range(f, c), function(n) {
        scopeObj[id] = n // Update the 'counter' variable to reflect the current loop number
        this.evalExpression(p.block, scopeObj)
    }, this)
},
command: function(p) {
    var args = _.rest(p.args)

    Interpreter.expTable[p.args[0]] = function(_p) {
        // Cloning context object because otherwise we will change
        // properties of the parent context, messing up everything
        var scopeObj = _.clone(p.scope) || {}

        // Associate every argument name with the given argument
        // value of the command call
        _.each(args, function(p, i) { scopeObj[p] = _p.args[i] })
        this.evalExpression(p.block, scopeObj)
    }
}

The local scope here is implemented in a naïve (and a bit inefficient) way. Basically and if it exists, the parent scope of the block expression is cloned and then the local variables of the block are copied on it. After that, the block of nested expressions is executed with the cloned context.

repeat is a ‘for’ loop that gets as arguments the ‘counter’ variable, the lower bound of the range and the higher bound of the range. From that it creates a loop that assigns the counter variable to the local scope and executes the block with this scope.

command builds a function from the given name and arguments and the block expression that forms the body of the function. In that case I create a function and store it in Interpreter.expTable so it can be called anywhere in the code. I do the same trick with the scope as in repeat, with the difference that in this case I asociate the name of the command arguments with the value passed to the call of the function.

Demo

That’s it, you can try a quick demo here. In the demo page you can find some examples and you can play around to better understand how DBN works and what its powers and limitations are.

Conclusion

As you can see this is a very simple implmentation, but it is enough to run most of the DBN programs around. Some things like arrays, timers and mouse position and events were left out and will be implemented at some point when I have time, but the actual ‘meat’ of the intepreter won’t change much anyway.

You can find the complete code for this interpreter here and the complete repository here. If I missed something or yo have some suggestions of how to better implement DBN drop me a line, or even better: fork it!

Writing a JavaScript interpreter for DBN using PEG.js and canvas (Part I)

2010-08-03T00:00:00-07:00

In this first part of the article, I will define a grammar for DBN and generate a parser for it that outputs an AST, so I can interpret the syntax tree it later on with JavaScript and draw it into an HTML5 Canvas.

John Maeda created the DBN language as a tool to teach programming to non-developers. It was quite an influential language; in fact it was the precursor of the popular processing language, developed by Maeda’s students Casey Reas and Ben Fry, taking many ideas from DBN.

Unfortunately, DBN hasn’t been open-sourced and the documentation of it is almost nonexisting, let alone any official specification of the language. There is only one book written about it (although a blurry copy exists in google books, censored enough to miss very important parts), and the DBN software amounts to a Java applet in the official website and a Java program that you can run on your own computer.

All this lack of proper information was a challenge enough to get me interested on building a JavaScript interpreter for it (this, and my deep hatred for Java applets) and a good excuse to play with some cool JavaScript technologies.

Crash course in PEG.js grammar definitions

PEG.js is a parser generator for JavaScript. Given a proper grammar definition of the language, it will generate a parser for it that you can use wherever.

If you want to quickly test the grammar output as you go, use PEG.js online version, it is immensely useful on the early stages of grammar definition.

Also, keep in mind that what I explain here is a 10.000 foot overview of PEG.js and it is by no means a tutorial; if you want to learn real language grammar definition with PEG.js you must read the real documentation for it, and have some idea about parsers, lexers, tokens and their friends.

1. Defining the basics

Before we start, one thing worth oting is that PEG.js doesn’t have a separate stage for “lexing”. The lexer is integrated seamlessly in the generated parser, so we don’t have to worry about it.

Ok, so the first thing to do is to define the building bocks for the language. Note that the parser doesn’t take anything for granted, so we will have to define how our language handles whitespace as well.

variable
 = v:([a-zA-Z_][a-zA-Z0-9_]*) { // The first character must not be a number
     return { 
       type: "string",
       value: v.join("")
     } 
   }

integer
  = digits:[0-9]+ { 
      return { 
        type: "integer", 
        value: parseInt(digits.join(""), 10) 
      }
    }

point // Matches expressions of the form [value1 value2]
= "[" left:value ws right:value "]" {
    return {
        type: "point",
        x:left,
        y:right
      }
    }

special // Matches expressions of the form <command value1 value2 ...>
= "<" _ left:variable args:(ws value)+ ">" {
    return {
      type: "special",
      args: [left, args.map(function(a) { return a[1] })]
    }
  }

additive
  = left:muldiv _ sign:[+-] _ right:additive { 
      return { 
        type: "command", 
        name:sign, 
        args:[left,right] 
      }
    }
  / muldiv

muldiv
  = left:primary _ sign:[*/] _ right:muldiv {
      return { 
        type: "command", 
        name: sign, 
        args:[left, right] 
      }
    }
  / primary

primary
 = (variable / integer / special)
 / "(" _ additive:additive _ ")" { return additive }

value
= variable / integer / additive / point / special

comment // Matches single-line comments
 = "//" (!lb .)*

ws
 = [ \t]

_ // Matches any number of whitespace/comments in a row
 = (ws / comment)*

As you can see, the language is very clear and takes inspiration from regular expressions. The above peg grammar defines strings, integers (DBN has no decimals), points in a 2D canvas, and basic arithmetic (the latter adapted from the arithmetic grammar given at the PEG.js site as example).

A rule is composed of several parts: its name, the matching expression and the optional parser action, which is enclosed between brackets and transforms the output of the rule.

For example, for the point rule:

point
 = "[" left:value ws right:value "]" // <-- This is the matching expression
   // The following code is the parser action
   {
     return {
       type: "point",
       x:left,
       y:right
     }
   }

This rule checks for a “[” character followed by a value rule (which is defined separately), followed by a whitespace rule (also defined separately), followed by another value rule and a closing “]”.

In most non-trivial rules we include a parser action, which is the JavaScript code inside the brackets. The parser action can access the matched values of the rule and modify them using JavaScript before the output is returned. Here I return an object that contains the two values inside a point, which were labeled left and right in the matching expression so they can be accessed from the parser action. Without a parser action the output would be a JavaScript array with all the matches in the matching expression, so with the input [4 2] we would get the output ["[", "a", " ", "b", "]"].

The point rule references two other rules, value and ws, which look like this:

value // matches any of the standard values in DBN
= variable / integer / additive / point / special

ws // Matches space and tab characters
= [ \t]

value will try to sequentially match any of the expressions separated by “/”, and it will return the first sucessful match. The whitespace ws rule uses a RegExp construct to match any single space or tab character.

2. DBN Commands

Now we will define the “meat” of our grammar. Fortunately enough, DBN is a language with a rather simple syntax. A basic program could look like this:

paper 30

repeat A 0 300
{
    pen 50
    line 0 55 100 55
    line 0 20 100 20

    pen 100
    line (A*2) 0 (A*2) 20
    line (A/3)  55 (A/3) 100
    paper 30
}

This program runs the instructions inside the repeat loop, which is equivalent to a classical for loop. It loops from 0 to 300 while assigning the current iteration value to the variable A and drawing different lines on the canvas.

As we see above, a basic command in DBN takes the following form:

command_name value_1 value_2 value_n

And a rule command to parse it:

command
 = _ cmd:[A-Za-z0-9?]+ args:((ws+ value)+)? _ lb+ {
     return {
       type: "command",
       name: cmd.join("").toLowerCase(),
       args: args ? args.map(function(a) { return a[1] }) : null
     }
   }

Let’s break the matching expresion down:

_: Match 0 or more whitespace characters.
cmd:[A-Za-z0-9?]+: Match an alphanumeric word that might include the ‘?’ char, and label it as “cmd”.
args:((ws+ value)+)?: Match 0 or more DBN value expressions preceded by a whitespace and label them as “args”.
lb+: Match at least one linebreak.

In case an expression command is matched, we will pass the cmd and the args results to the parsing expression, who will return an object with some particular properties that manipulate the values in the way we want. In that case we just do some basic filtering to the arrays resulting from the matches.

An important thing to note down is that any match gotten using the operators [] * or + will return an array with as many coincidences as there are in the matching expression. That means that we will have to take care of filtering these arrays so we obtain the results we want. A clear example is the cmd variable, which returns an array with all the characters of the command name matched, so in the parsing expression we have to join it to form a real string.

3. DBN Blocks

Our grammar can already process DBN commands, but this is pretty limited functionality. Any interesting program will at least have loops and block commands, and we still can’t parse these.

In order to process block commands we will create the new rules block and block_command:

block
 = "{" lb* e:(block_command / command)+ lb* _ "}" { return e }

block_command
 = e:command _ b:block lb* {
     e.block = b;
     return e;
}

In block we match any group of commands or nested block_commands that are inside braces, taking into account the possible whitespaces that we might find.

In block_command the matching expression is very straightforward, but there is an interesting bit in the parsing expression, where we attach b to c as a property, and we return c after that, so the result is a JavaScript object with the properties type: “command”, array of parameters args and that block property we just added, which contains the commands inside the block.

Conclusion

And we’re finished! Our grammar can now parse DBN programs and generate its JSON AST. Parsing the basic program that I wrote above as an example generates this tree. You can find the complete grammar file here.

As you can see, using PEG.js is a very clean and concise way to define language grammars. And best of all, we can use JavaScript for doing it which is pretty awesome. Moreover, the generated parser for it is pure JavaScript and thus independent from the PEG.js library itself. In fact, using the online version of it, you can just download the generated parser without even having to download the library itself.

I am sure very soon we will hear of apps that use PEG.js in very creative ways, surely involving node (as it seems to be the trend now) or some other cool technologies. JavaScript is definitely ready for prime time.

In the second part of this article I will create the interpreter that turns the AST into JavaScript code that paints directly into canvas. Stay tuned!

GitHub in Catalan

2010-07-28T00:00:00-07:00

This is awesome. From now on, Catalan users of GitHub can use the website in their native language. Thanks to the Github team initiative and the brilliant approach of crowd-sourcing the translations (along with some very convenient tooling to do so), I could translate almost all the strings of the website in a record time two weeks ago.

What I found out today is that the website is already making use of it and available to everyone! Just point your browser to http://github.com/blog?locale=ca and the page will be offered to you in Catalan already. The translation is currently 87% complete, so there is still some work to be done, which I expect to finish between today and tomorrow.

Anyway, if Catalan is your native language and the lack of it in Github was stopping you from using it, now you have the chance!

GitHub en Català

Des d’ara mateix, els usuaris Catalans de Github poden utilitzar la web en el seu propi llenguatge. Gràcies a la iniciativa de l’equip GitHub i a la manera brillant d’utilitzar crowd-sourcing per a traduïr la web a multitud de llenguatges (i a unes eines de traducció molt útils), he pogut traduïr quasi totes les cadenes de la web fa dues setmanes en un temps récord.

El que no m’esperava i he descobert avui és que GitHub ja està disponible online en Català! Només cal que introduïu la següent direcció al vostre navegador: http://github.com/blog?locale=ca i la pàgina us preguntarà si voleu establir Català com a llenguatge del lloc per defecte. La traducció està 87% complerta, així que encara hi ha una mica de feina per fer, que espero finalitzar entre avui i demà.

Així doncs, si la teva llengua és el Català i el fet que GitHub estigués només en anglès t’impedia utilitzar-lo, ara ja no tens excuses!

Blog running on Jekyll now

2010-04-13T00:00:00-07:00

That’s right, I couldn’t resist switching to Jekyll. I feel so free now!

For those who don’t know, Jekyll is a static site generator. It takes a template directory, runs it through a textile or markdown parser, and outputs a complete website ready to be served with any web server, since it is simple HTML. That means no more server queries or content stored in databases. You own your content, and it is stored in plain text files. Of course, that also means that it is perfect to be maintained with a VCS like Github, where by the way, you can host your Jekyll sites for free!

Ah, simplicity…

You can see the source code for my blog and get a feeling of how it works.

ABN Amro to Wesabe script

2009-02-25T00:00:00-08:00

Wesabe is a great service. And a very big part of its greatness resides in the fact that it retrieves your bank accounts data automatically from their sources whenever you log in into the page.

That is, if you don’t have a Dutch bank account.

The reason for this is that the security of Internet banking in The Netherlands is quite strong, probably much more than the ones in the US (where most Wesabe users are from). Every bank customer owns a little device in which you have to put your bank card in, type your PIN number, type a number given to you by the bank website, and type back the code calculated by the device. Only then you can access your personal bank website. Once in the site, you will have to repeat this process for every transaction or order that you make.

I much rather prefer this increased security when it comes to Internet banking as it adds an extra layer of security that makes it much more difficult to hack illegally into people’s bank accounts, but this also means that Wesabe (obviously) won’t be able to access the data in my bank website automatically.
Thus, the process for the suffered dutch-account-owners/Wesabe users (category in which I happen to fall in), is the following:

Download ABN Amro’s own QIF format for transactions
Convert it to QIF
Log into Wesabe
Upload the QIF file
Enter my account current balance manually, for it is not stored in the QIF file

No need to say that this is a very annoying and time consuming process…

So, after having to do that for the nth time, I decided to write a python script to automatize the process as much as I could. The process now becomes:

Download ABN Amro’s own CSV format for transactions
Execute script

Which is slightly a less annoying process to me.

Its usage is dead simple: abn2wesabe.py STATEMENTFILE.TAB

You can look at the script source code here or download it directly from here.

Many thanks to Johan Kool, who gave me permission to me use part of his ABN Amro 2 QIF Converter" script to convert ABN bank statements to QIF files.

Disclaimer: This script does not check the validity of the certificate issued by Wesabe, so if any virus, malware, or plain user stupidity causes it to do damage of ANY kind to a computer or a human being, I will accept no responsibility for it.
In other words: this software could post your financial data to undesired sites, delete all the files in your computer, insult you, burn your house and cause all kinds of innumerable misfortunes to whomever uses it, and I don’t want to be the one to blame. If you are afraid of using it, don’t. You are warned.

YQL mashup with GWT

2008-11-16T00:00:00-08:00

Yahoo came up with YQL a few weeks ago, and I think it is a very interesting idea. It is basically a SQL-like language that allows developers to retrieve structured data from multiple Yahoo web services without having to deal with each service particularities; the developer can query any service in a generic way, just like he would do retrieving rows and columns from a SQL database.

An example of YQL query to retrieve the 5 first results returned by the search engine with the keyword “pizza” would be:

select title, abstract from search.web where query="pizza"

which returns a response in the format that you specify (either XML or JSON) that includes some metadata about the query itself and the actual requested results in a format like this (JSON):

"results": {
    "result": [
    {
     "abstract": "Official site for the international <strong>pizza</strong> chain, with online ordering, etc.",
     "title": "<strong>Pizza</strong> Hut"
    },
    {
     "abstract": "<strong>...</strong> site for the delivery chain offering thin crust, deep dish, etc.",
     "title": "Domino's <strong>Pizza</strong>"
    },
    ...
    ]
}

As you can see it is a very intuitive way to interact with web services (if only they were not all from Yahoo) for everybody that knows a bit of SQL.

If you want to try other queries or see examples of its use with other services you can use the YQL Console.

Querying YQL with Google Web Toolkit

So how would we query YQL with GWT? As it turns out, very easily. I am going to create a GWT component that queries flickr through YQL and retrieves the first 9 occurrences related to the keyword entered in a textbox and presents them in a Grid, like the one on the left.

Authentication

YQL uses 2-legged OAuth authentication like most OpenSocial containers, so we will have to have this into account when making the request. Fortunately there are good OAuth libraries available for every language. In my example I will use the ones made by Google, available at http://oauth.googlecode.com. I copied them to the /public folder and I referenced them from the module xml file instead of loading them directly in the html, which is more GWT-ish.

When I browsed the web for OAuth libraries I stumbled upon Paul Donnelly’s blog, who made a simple Javascrifunction for processing the parameters returned by OAUth libraries in the fashion that YQL wants, so instead of doing it with pure Java I took advantage of the toolkit’s JSNI classes and I just made a dumb OAuth class with a single static container method makeSignedRequest that wraPaul’s very slightly modified function so that it references the OAuth object properly.

Calling back

Note: This section is explained in much more detail at Dan Morrill’s JSON tutorial.

Now that we have an OAuth class that will deal with the authentication, we only have to worry about the actual communication. Before getting our hands dirty with it, I will explain a bit some particularities of YQL and GWT that are necessary to get the mashup working properly.

GWT doesn’t directly allow fetching JSON data from an external domain for security reasons (any JSON data is a Javascriobject after all, and although it should be used only for data transport, it can contain executable code as well), so I will use a bridge method setup to fetch it from Yahoo’s domain.

When we send a request to YQL through OAuth, we send some important parameters with it: our personal YQL key and YQL secret (generated when we request a new application account), and the URL in which we specify the encoded query, the response format we want and the callback function name that YQL will use in its response. The request method looks like this:

public void executeQuery(String keyword)
{
    String callback = this.reserveCallback();

    // Generate a function in the page with the known name we just created
    flyqlr.setup(this, callback);

    // Encode the URL using GWT's |URL.encode| and replace remaining unencoded characters
    String query = URL.encode("select * from flickr.photos.search where text = '" + keyword + "' limit 9")
                  .replaceAll("'", "%22").replaceAll("=", "%3D");

    // Makes the request using OAuth, which calls our just created callback
    // and adds the response as a scrito the page
    this.addScript(OAuth.makeSignedRequest(
        flyqlr.Y_KEY, flyqlr.Y_SECRET, flyqlr.YQL_URL 
            + "q=" + query + "&amp;format=json" + "&amp;callback=" + callback));
}

The first thing that the method does is to generate a unique name for our future function and pass it as a parameter to the setup static method, which is the JSNI “bridge” method mentioned above, and that looks like this:

public native static void setup(flyqlr f, String callback) /*-{
  $wnd[callback] = function(data) {
    f.@com.sergi.client.flyqlr::process(Lcom/google/gwt/core/client/JavaScriptObject;)(data);
  }
}-*/;

This method creates a new function inside the window object ($wnd in JSNI slang) named after the callback parameter. This function calls the Java method process with a parameter data which contains the JSON string returned by YQL. When we know that our callback exists in the window we can safely compose the query and make the OAuth request. Notice that the latest parameter of the request string is callback, which tells YQL services our callback function’s name.

The addScrimethod just creates a scritag and injects its argument as its content and appends the tag to the page which executes its contents straight away, thus executing our callback.

Processing the response

At this point we have the JSON data returned by the YQL server, passed as a parameter to our process method, which looks like the following:

public void process(JavaScriptObject jso)
{
    JSONObject json = new JSONObject(jso);
    JSONArray ary = json.get("query").isObject().get("results").isObject().get("photo").isArray();

    this.updateGrid(ary);
}

This method is pretty straightforward, it simply converts the JavaScriptObject to a JSON one so we can extract from it the specific information that we want in form of JSONArray and pass it to updateGrid, which renders the retrieved photos:

public void updateGrid(JSONArray ary)
{
    for (int i = 0; i &lt; ary.size(); ++i)
    {
    JSONObject photo = ary.get(i).isObject();

    Image img = new Image
        (   // Constructs the flickr image URL from JSON data
        "http://farm"+photo.get("farm").isString().stringValue() + ".static.flickr.com" +
        "/" + photo.get("server").isString().stringValue() +
        "/" + photo.get("id").isString().stringValue() +
        "_" + photo.get("secret").isString().stringValue() +
        "_s.jpg"
        );
    img.setWidth("75");
    this.theGrid.setWidget((int)Math.ceil(i/3), ((i+1) % 3), img);
    }
}

updateGrid constructs the 9 element grid and composes the flickr pictures URLs out of the data that we extracted from YQL. We are done!!

Although I am not sure about using GWT to build a complete website, I love its potential for building independent small components that hook on any web page. Even with an example like the one above in which the request is not a direct Rcall from Java, AJAX is ridiculously easy to use from GWT, and it provides a whole set of widgets ready to use. Besides, there are very interesting and impressive addons to GWT like GWT-Ext which boosts its possibilities on the UI side. This, and the goodness of developing the whole thing in one single robust language (most of the time) makes it a very attractive option for building web front-ends.

flyqlr.java
OAuth.java

Solitaire Encryption Algorithm implementation in F#

2008-10-13T00:00:00-07:00

I first read about the Solitaire algorithm in the novel Cryptonomicon, by Neal Stephenson. Being a geeky kid fascinated by computers I thought that it was a brilliant idea, but my curiosity didn’t go so far as to make me write an implementation of it (you can find a perl version in the book, which by the way I consider to be another great way to encrypt information).

Solitaire is a stream cipher created by Bruce Schneier and designed to be used with only a deck of cards. No electronic devices are needed and its security is intended to be of military-strength; that means that even if the way to generate/decipher messages is manual and might look a bit rudimentary, ideally not even the most powerful government could ever decipher a message generated with Solitaire (although some potential flaws have been reported).

So after playing with F# lately, I decided to write some non-trivial code in order to see how it suits my coding style and habits, and yet another naive implementation of this curious crypto-algorithm was born.

Notes about the implementation

After the initial struggle learning the particularities of F# (which are basically the same as ocaml) I quickly felt comfortable with it. It’s a very versatile language, given that it allows the programmer to use functional and imperative programming at his own will, so the learning curve is not very steep. And although the functional approach should be used most of the time, sometimes imperative code is more convenient and can make some programmers with a C or Java background feel more in control. For an example, consider this snippet of code:

member s.KeyStream (c: char list) f =
    match c with
    |x :: xs  -> let mutable o = s.getOutput
                 while o = jokerA or o = jokerB do o <- s.getOutput;
                 (Convert.ToChar (((f x o) % 26) + 65) :: s.KeyStream xs f)
    |[] ->  [];

Here I am using a functional approach to go through a list of characters and apply a given function and some calculations to each one of them. The whole transformation is based on the value of the variable o, but I only want to use o if it is not equal to a joker value (you can read about the algorithm internals on the wikipedia article), so I use a very convenient while block that does the job. Sure, it can be done with recursivity as well, but F# lets you choose what suits you better.
In this snippet I am also using one of the most powerful ideas of the functional world: pattern matching. As the name says, a pattern match takes an expression between the match ... with block and compares it against a set of rules which test the expression given and returns a result. The concept is close to the good old switch/case statement, but much more powerful, because it allows the programmer to use conditions, name binding and object decomposing in the rules. The net is full of examples and articles about it where you can learn how to use them effectively.

The code also makes a basic use of some of the different paradigms that F# allows. For example, it uses OO for the Solitaire class to take advantage of mutable variables (a clear deviation from functional pureness and as I said, one of the strong points of the language) so the same variable can be used when keying the deck and by the encryption/decryption methods, but I use global functions for card manipulation operations. Sure, there could be a Deck class that includes card manipulation methods, but as I said before, this code is just for my own experimentation purposes.

I borrowed some utility functions from Haskell Prelude (span, splitAt) to help me with string manipulation, although I am sure that there are dozens of Ocaml libraries that already have a version of them. I am lazy and I just implemented the ones I needed and knew from Haskell.

This implementation is not intended to be efficient and of course there might be better ways to write it, this code was only written for fun and to teach myself some F# discipline, so it is not written in the most optimal way, but in the clearest one. I would gladly accept suggestions of how to improve it. I feel like I just started scratching the surface of what can be done with F# and I can’t wait to unleash its real potential.

The code below is the working algorithm, and you can find the complete source code including tests here.

#light
#r "FSharp.PowerPack.dll"

open System

(* Utility functions *)

let getCharValue c =
    let alphabet = Seq.to_list "ABCDEFGHIJKLMNOPQRSTUVWXYZ"
    (List.find_index (fun x -> c = x) alphabet)

(* Adds 'X' characters until the list has a length multiple of 5, as stated in
   Schneier's page *)
let rec mod5 (msg: char list) =
    if (List.length msg) % 5 <> 0 then mod5 (msg @ ['X']) else msg

let sanitize _string =
    _string
    |> Seq.to_list
    |> List.filter (fun c -> Char.IsLetter c)
    |> List.map Char.ToUpper

(* Functions borrowed from Haskell *)
let span p xs = (Seq.to_list (Seq.take_while p xs), Seq.to_list (Seq.skip_while p xs))

(* Splits a list into two lists at the position corresponding to the given
   integer *)
let rec splitAt n xs =
    match (n, xs) with
    |(0, xs) -> ([], xs)
    |(_, []) -> ([], [])
    |(n, x :: xs) ->  let t = splitAt (n-1) xs;
                      (x :: fst t, snd t)

(* Deck Manipulation functions *)
let jokerA = 53
let jokerB = 54

(* Moves down a card in the deck *)
let md (c:int) (d) =
    match span (fun x -> x <> c) d with
    | (y :: ys, [x])     -> y :: x :: ys
    | (ys, x :: z :: zs) -> ys @ [z;x] @ zs
    | (_,_) -> []

let md2 c d = md c (md c d)

let tripleCut deck =
    let tripleCut2 c acc =
        match acc with
        | []      -> [c] :: acc
        | h :: t  -> if c = jokerA or c = jokerB then [] :: [c] :: acc
                     else (c :: h) :: t
    let s = List.fold_right tripleCut2 deck [ [] ]
    [s.[4]; s.[1]; s.[2]; s.[3]; s.[0]] |> List.concat

let countCut index deck =
    match splitAt index deck with
    | (xs, [x]) ->  deck
    | (xs, ys)  ->  let split = splitAt (ys.Length-1) ys;
                    fst split @ xs @ snd split

type Solitaire() =
    let mutable Cards = [1 .. jokerB]
    member s.Deck with get() = Cards

    member s.getOutput =
        Cards <- Cards |> md jokerA |> md2 jokerB |> tripleCut
        Cards <- countCut (if Cards.[jokerA] = jokerB
                           then jokerA
                           else Cards.[jokerA]) Cards

        if Cards.Head = jokerB then Cards.[jokerA] else Cards.[Cards.Head]

    member s.keyDeck keys =
        Cards <- [1 .. jokerB]
        let keyArray = keys |> sanitize |> Seq.to_list
        List.iter (fun (c:char) -> s.getOutput |> ignore;
                                   Cards <- countCut (Convert.ToInt32 c - 64) Cards) keyArray

    member s.KeyStream (c: char list) f =
        match c with
        |x :: xs  -> let mutable o = s.getOutput
                     while o = jokerA or o = jokerB do o <- s.getOutput;
                     (Convert.ToChar (((f x o) % 26) + 65) :: s.KeyStream xs f)
        |[] ->  [];

    member s.Encrypt msg = s.KeyStream (msg |> sanitize |> mod5)
                                       (fun x o -> (getCharValue x) + (o % 26))

    member s.Decrypt msg = s.KeyStream (msg |> sanitize |> mod5)
                                       (fun x o -> let k = (getCharValue x) - (o % 26)
                                                   if k < 65 then k + 26 else k)
(* Unit testing *)
let vectors =
   [["AAAAAAAAAAAAAAA";"";"EXKYIZSGEHUNTIQ"];
    ["AAAAAAAAAAAAAAA";"f";"XYIUQBMHKKJBEGY"];
    ["AAAAAAAAAAAAAAA";"fo";"TUJYMBERLGXNDIW"];
    ["AAAAAAAAAAAAAAA";"foo";"ITHZUJIWGRFARMW"];
    ["AAAAAAAAAAAAAAA";"aa";"OHGWMXXCAIMCIQP"];
    ["AAAAAAAAAAAAAAA";"aaa";"DCSQYHBQZNGDRUT"];
    ["AAAAAAAAAAAAAAA";"b";"XQEEMOITLZVDSQS"];
    ["AAAAAAAAAAAAAAA";"bc";"QNGRKQIHCLGWSCE"];
    ["AAAAAAAAAAAAAAA";"bcd";"FMUBYBMAXHNQXCJ"];
    ["AAAAAAAAAAAAAAAAAAAAAAAAA";"cryptonomicon";"SUGSRSXSWQRMXOHIPBFPXARYQ"];
    ["SOLITAIRE";"cryptonomicon";"KIRAKSFJAN"]]

let rec test (arr:string list list) =
    match arr with
    |x::xs    ->    let key = x.Tail.Head
                    let correct_enc = List.hd (List.tl x.Tail)
                    print_endline("Testing vector: " ^ x.Head ^
                                      "\nwith key: " ^ key ^
                                      "\nshould output:\t" ^ correct_enc)

                    let cipher = new Solitaire()
                    cipher.keyDeck key
                    let enc = new String (List.to_array (cipher.Encrypt x.Head))
                    cipher.keyDeck key
                    let dec = String.sub (new String (List.to_array (cipher.Decrypt enc))) 0 x.Head.Length
                    print_endline
                        ("and it outputs:\t" ^ enc ^ "\ndecryption:\t" ^ dec ^
                        "\n[Encryption test: " ^
                            if enc = correct_enc then "CORRECT!]" else "FAIL!]")
                    print_endline
                        ("[Decryption test: " ^
                            if dec = x.Head then "CORRECT!]\n\n" else "FAIL!]\n\n\n")
                    test xs
    |[]       ->    print_endline "All tests Finished."

(* Command line parser *)

let _ = match Sys.argv with
        |[|_;"-test"|]      ->  test vectors
        |[|_;"-enc";_;_|]   ->  let cipher = new Solitaire()
                                cipher.keyDeck (Sys.argv.[3])
                                print_endline
                                    (new String (List.to_array
                                        (cipher.Encrypt (Sys.argv.[2]))))
        |[|_;"-dec";_;_|]   ->  let cipher = new Solitaire()
                                cipher.keyDeck (Sys.argv.[3])
                                print_endline
                                    (new String (List.to_array
                                        (cipher.Decrypt (Sys.argv.[2]))))
        |_  ->  print_endline("Usage:\tSolitaire -test\n\t" ^
                                  "Solitaire -enc text key\n\t" ^
                                  "Solitaire -dec ciphertext key\n\n")
                print_endline("Example: Solitaire -enc SECRETMESSAGE foo")

The great thing about F# is that is a member of the .NET family of languages, so it can interact with the .NET framework just as C# would. In the code above I am not using much of explicit .NET related code, but I definitely want to explore it…just imagine the potential uses of combining ideas like pattern matching with existing .NET objects. Cool.

Nth attempt

2008-10-08T00:00:00-07:00

Yes, I am blogging again. But this time I have a plan.

My last blog tried to cover too many things, from my life abroad, to ramblings on software engineering and even some poor attempts to deep thinking every other post. Well, you really can’t bite off more than you can chew, and that was too broad and disorganized to maintain it properly, so this time the theme of this blog will be quite specific: programming and sotware engineering (mainly), focusing on languages, techniques and my day-to-day development.

I already have some few articles prepared, and many ideas for future ones. Stay tuned!