Christian Heilmann

Posts Tagged ‘fronteers2009’

Of Hamsters, Feature Creatures and Missed Opportunities – my talk at Fronteers 2009

Thursday, November 5th, 2009

The outline

In this session Chris Heilmann will show how some of the traits we have as developers keep us from evolving and our market from maturing. Instead of picking and choosing from already amazing products we tend to want to build and maintain everything ourselves – and fail at keeping up the quality leaving both security holes and carwrecks on the information superhighway behind. There is another way and Chris will show just how much more fun we can have with the web if we use systems that are available for us for free, remix the web with simple meta-mixing systems and liberate our developer minds from the barriers technology throws at us. If you always thought that there has to be more to our jobs than fixing bugs, this session is for you.


The audio recording

You can download the MP3 of the talk at – 42MB.

Following are the notes which will resemble what I am talking about…

Remember, remember the 5th of November…

First of all, how cool is it to have a conference on Guy Fawkes? Whoever can recite the “V” alliteration later on flawlessly I will buy a beer for.


Here’s a thing that happened in my garden in London the other day. My friend keeps putting bird feed in the garden to, well, feed the birds. She also wonders why she keeps having weeds in her garden. Well, the reason is this: a squirrel keeps coming and takes the bird feed and buries it a meter away from the bird feeder (video of the theft). This is stupid as we keep replenishing the bird feed anyways. However in the mind of the squirrel it is a safe idea to keep a cache. Luckily we humans are cleverer, right?

No. One very human trait we have is that we want to collect things – especially when they are free. If you followed my exploits this year you’d have seen that I travelled – a lot. What I found myself doing is collecting both inflight sock and toothpaste packs and little soap/shampoo/body lotion bottles. I am amazed just how many of these things I accumulated without ever needing them.

This is exactly what we do with knowledge and code examples. I have about 20GB of e-books on all kind of technical subjects. All of these I haven’t looked at for years as most of the topics are outdated. However, we accumulate knowledge and think we do a great job once we found the solution to end all solutions. That doesn’t work any longer. We should not fill up our head with useless information about how some browser fails to do one thing or another. This secured our jobs during the first .com crash but it is time to let go.

Amazing resources

Nowadays we have amazing resources for beginners to learn what we are doing. The times of copy and paste and then try to understand someone else’s code is over. We should not put random stuff on the web. Both the Opera Web Standards Curriculum and the WaSP interact curriculum needs your input.

Evolving the web

Our job as developers is to evolve the web to something more than it is now. However, we are not really concentrating on that. There are a few things that are in our way.

Specialising without caring about the consequences

One thing is that we tend to become specialists in a certain area and forget about the others. Web development is a task of working together with other specialists to build amazing interfaces that make a change in people’s lives. If you are not passionate about a certain aspect of it, don’t do that job – instead partner with somebody who is passionate about it.


The amount of energy wasted every day fighting each other on forums, mailing lists and other resources about which technology is best and what library to use pains and bores me. It is not helpful and it doesn’t make any sense to try to make the world fit your tool of choice.

Be a librarian

Good developers should be like librarians. You don’t need to know everything, but you need to know where to find the right information at the right time.

Use libraries

And for that, use libraries like Dojo, jQuery, YUI, mootools or whatever to work around the browser problems so that you can concentrate on what you want to build rather than how it fails.

Libraries fix browsers

Our main enemy to build cool products are browsers. The technologies and standards that drive the web are dead easy – we just don’t have time using them as we spend most of our time testing and wondering why browsers don’t implement them in a predictable manner. This is where libraries come in and my plea here is to use them. Using libraries you can write predictable applications built on a base that can be altered for all the browsers to come – if you build towards the browsers in use today you build for the past. We’ve done that before and paid dearly for it.

Use don’t abuse libraries

Libraries give you an amazing amount of power to do quick solutions. Be aware that when you use them though, you use them wisely. Do not mix HTML, CSS and JavaScript in the same script. This creates unmaintainable solutions and we had years and years of those.

The feature creature

The first time I heard about the feature creature was in 2004 here in Amsterdam at the PHP conference. I loved the image of every developer having a little creature on their shoulder that keeps telling us what we should be doing. Most of the time the feature creature tells us that whatever is out there is not nearly as good as the thing we can come up with. It also keeps telling us that whatever we do needs more features. This leads to something I call the feature loop.

The feature loop

As developers we are driven to writing simple, elegant and clever solutions for complex issues. Then we do the right thing and release those to the whole world. Then we do the wrong thing by listening to everybody who wants to have an extra feature and support for some very esoteric edge case. We add all of these extensions and build something complex again that another developer might find and simplify again.

Breaking the loop

The only way to break this vicious circle is by broadening our horizons and being cleverly lazy. I talked about this before in detail at the web expo in Prague. In essence what I am saying is that a lot of what we think is our job to do has been done for us already. And now I am going to show you some of these things.

Missed opportunities

The thing is that by building on tested and working solutions we can build faster, more secure and working solutions. While we think that we need to squeeze the last out of everything we do ourselves very successful web sites are being built in other countries where developers are very pragmatic of what has to be done. Questioning an out-of-the box solution is a good safety measure but you also need to draw the line at trying to replace them. Instead, why not add to the already existing system – all of them are open source!

The other thing we keep forgetting as web developers is that the technologies we are experts in right now reach much further than the browser world.

Our technologies are hot right now

Mobile development, W3C mobile widgets, iPhone development, TV widgets and applications running inside Facebook, Yahoo or iGoogle are all based on CSS, HTML and JavaScript and run in much more defined and easier to use environments. We can take our knowledge and finally work without the sword of damocles of bad browser support hanging over our head.

Changing the idea of what the web is

The web is not web sites, it is not even code. Deep down all of it is about data and information. The Yahoo homepage for example moved from a directory of bookmarks to a portal full of third party content to a starting point for users where they can mix and match content of the web to their liking. This goes as far as now allowing people to update Facebook from the Yahoo homepage or you to write an application that is shown on it.

Building with components

Interfaces like that are only possible when you move away from the old idea of having a single page but you think of small components in the interface that could hold whatever the end user pleases. This is were we run into issues with the web standards, but that is another talk.

Based on APIs

All the data to remix and show is available for us on the web already – offered as APIs and listed on sites like Programmable Web.

Kobayashi Maru

Looking at the sheer amount of APIs – over 1500 right now – and knowing full well that all of them were built by developers with the feature creature on their shoulder it seems impossible to use a lot of them at the same time as you need to read up on all the documentation, get developer keys, learn the authentication and so on. We have that same problem internally at Yahoo, which is why we tackled it.

YQL is there for you

YQL is a meta language for the web that over a single, simple REST service turns the whole web into a giant database for you to explore and take part in. Let’s see some examples:

Photos from Flickr?

select * from where text = "cat"

Headlines of the New York Times?

select * from where query="spontaneous combustion"

Latest commits on a GitHub repo?

select * from github.repo.commits where id='yql' and repo='yql-tables'

Latest headlines of the Volkskrant?

select title from html where url="" and xpath="//h2[@title]"

Latest “English” headlines of the Volkskrant?

select * from google.translate where q in (select title from html where url="" and xpath="//h2[@title]") and target='en'

Updating Twitter?

use '';
insert into twitter.status 
values (
  "In your fronteers, blowing ur mindz",          

Updating WordPress?

insert into 
(title, description, blogurl, username, 
("Test Title", "This is a test body", 
"", "codepo8", "iedoesitright")

Updating the internet?

delete from internet where user_agent="MSIE" and version < 8;

– Not yet, sorry

Search one term in Google, Bing and Yahoo?

select * from query.multi where queries='
select * from where query="Jimmy Hoffa" and source in ("web");
select * from where q = "Jimmy Hoffa";
select * from search.web where query = "Jimmy Hoffa"

You can do all of this in the REST API{uri-encoded-query}&

Things you can do in YQL

  • You can mix and match whatever you want.
  • You can sort and limit.
  • You can extend with your own data tables by writing a schema and putting it on GitHub.
  • You can store information on the YQL servers (the cloud, the cloud)

Where’s the catch?

  • 10k hits an hour (20k with oAuth).
  • 100k hits a day.
  • Data is cached for 15 minutes (soon customisable).
  • Execution limit on the server is 30 seconds (for now).

That’s all!

An example – loading photos for a location from Flickr

Find Amsterdam

select * from geo.places where text="amsterdam"

Define Amsterdam and get photos taken there

select * from where woe_id in (
select woeid from geo.places where text="amsterdam"

Get all the information about these photos

select * from where photo_id in (
  select id from where woe_id in (
    select woeid from geo.places where text="amsterdam"

Get only what we need

from where photo_id in (
  select id from where woe_id in (
    select woeid from geo.places where text="amsterdam"

Select format JSON, define a callback and copy and paste the URL.*

Copy into a script src and write some DOM scripting to display

<script type="text/javascript">
function flickr(o){
  var f = document.getElementById('flickr');
  var out = '';
  var photos =;
  for(var i=0,j=photos.length;i<j;i++){
    var cur = photos[i];
    out += '<li><a href="' + cur.urls.url + '"><img src="http://farm' +
   + '' + cur.server + '/' + 
   + '_' + cur.secret + '_s.jpg" alt="' + cur.title + 
  f.innerHTML = out;
<script type="text/javascript" src="*"></script>

And you have a mashup!

Well, to a degree. You can see that it pulls photos from Flickr and shows them on a page.

You also have FAIL

As much as we love it, JavaScript is not reliably available in browsers and cannot be trusted for providing very important content. Or can it?

Moving JavaScript solutions server-side with YQL execute

Open data tables providing YQL with data can contain JavaScript that will be executed on the server to convert information. That way you can take the script just shown and make it a server-side conversion. The benefit of this is that you are not bound by client security flaws and you have full language support, including E4X for XML. The earlier script will thus become the following:

var amt = amount || 10;
var query = 'select farm,id,secret,owner.realname,server,title,'+
            'urls.url.content from where '+
            'photo_id in (select id from'+
             amount + ') where ';
  query += 'woe_id in (select woeid from geo.places where text="' + 
            location+'") and ';
query += ' text="' + text + '" and license=4)'
var x = y.query(query);
var out = <ul/>;
for each(var cur in{
  var li = <li/>;
  var a = <a/>;
  a.@["href"] = cur.urls.url;
  var img = <img/>;
  var url = 'http://farm' + cur.@farm + '' +
            cur.@server + '/'+cur.@id + '_' + cur.@secret +
  img.@["src"] = url;
  img.@["alt"] = cur.title;
  a.img = img;
  li.a = a; += li;
response.object = out;

This, embedded in an open table means you can retrieve photos from Flickr as a UL now using the following YQL statement:

use "" as flickr;
select * from flickr where text="me" and location="uk" and amount=20

You can then display the photos returned either with PHP:

$url = ';%20select%20*%20from%20flickr%20where%20text=%22me%22%20and%20location=%22uk%22%20and%20amount=20&format=xml&diagnostics=false';
$ch = curl_init(); 
curl_setopt($ch, CURLOPT_URL, $url); 
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); 
$output = curl_exec($ch); 
$output = preg_replace('/.*<ul>/','<ul>',$output);
$output = preg_replace('/<\/ul>.*/','</ul>',$output);
$output = preg_replace('/<\?.*\?>/','',$output);
$output = preg_replace('/<!--.*-->/','',$output);
echo $output;

Or you can of course use JavaScript:

<div id="jsflickr"></div>
<script type="text/javascript" charset="utf-8">
function flickrlist(o){
  var ct = document.getElementById('jsflickr');
  ct.innerHTML = o.results;
<script type="text/javascript" src=";%20select%20*%20from%20flickr%20where%20text=%22me%22%20and%20location=%22uk%22%20and%20amount=20&format=xml&diagnostics=false&callback=flickrlist">

Be part of the movement

If you have other practical solutions like that, please write them and add them to the GitHub repository so they can become part of the interface.

Read the YQL documentation

Other than that you can read the YQL docs, there’s a YQL Blog and a Forum available. See you there.

Be clever, not busy

I think it is time we find solutions like libraries and YQL for our problems rather than doing everything by hand. Spend your time on making the web fun for humans, not on pleasing machines. If you are clever, you are allowed to be lazy. Replacing the web would be tough, so use it well.