Scraping Data from websites is pretty cool. But what if we could not only grab and parse the markup from external websites, but take full-blown screenshots and even modify that markup before we take that screenshot? We can do all that with an excellent package called PhantomJS.

A Browser without the Browser.

PhantomJS describes itself as a "headless WebKit with JavaScript API". For those that are not familiar, WebKit is the open-source web browsing engine that powers popular browsers including Chrome and Safari. "Headless" refers to fact that the program can be run from the command line without a window system, making it perfect for the server environment. The JavaScript API means that we can easily write scripts that interact with PhantomJS in the language of the web, allowing us to modify the browser output on the fly, just like you can do with Firebug or Developer Tools locally!


The hardest part of working with PhantomJS is getting it installed. Because of the requirements you are not going to be able to run this on a standard shared host, but an average vps will handle it fine. The following instructions are for getting things up and running with Ubuntu 11.10 on Linode.

Step1: Install PhantomJS Requirements

This will install everything we need to compile PhantomJS, along with the ability to run it virtually without a window system.

Step2: Install Browser Goodies

Installs flash plugin and windows fonts so sites appear more accurate.

Step3: Compile!

Clone the repo and build from source.
After everything finishes we should now have the program installed. Test it at the command-line by typing "phantomjs"

Taking Screenshots

Since PhantomJS scripts are simply javascript with some extra api calls baked in, the barrier to entry for web developers is zero! Create the following script and save it somewhere on your server as "shotty.js". Check out the comments for an explanation of what is happening.
To run this script, navigate to where the script lives on your server and run the following command. We run "Xvfb -screen 0 1024x768x24&"  first to set the parameters of our Xvfb screen buffer. This is basically a virtual screen that allows us to emulate a window environment. Then we call the script with "DISPLAY=:0 phantomjs --load-plugins=yes shotty.js" to ensure Phantom runs in the buffer. PhantomJS will execute our script and save our screenshot in the same folder the script is in. If everything went well our screenshot of should look just like the real thing. How cool is that?

Screenshot of ESPN homepage

Muck'n with Markup

Because PhantomJS scripts are JavaScript, that means we can easily perform actions on the result of a page, and then take a screenshot of the results! Below we have created a script that grabs a page, embeds jQuery, and then performs some DOM manipulation before finally taking a screenshot. Priceless. The following code will allow us to edit the headline on to say whatever we want.

Screenshot of ESPN Homepage 2

I'm sure by now your gears are already turning on fun uses of this technology. In Part II of this series I will go over how to integrate with Node.js to create a Phantom powered web app.