In most cases, content retrieved by Ajax after the normal page is being loaded, isn’t a problem. If you really need the content, you can usually see how the Ajax call is made and use the same url for getting the data directly. In Most Cases. Sometimes the Ajax calls are not so obvious; the javascript might be obfuscated or you can’t really tell where the data is being loaded from. Or sometimes you just need to simulate the actual page loading. Test automation could be one of such situation. None the less, the problem is loading the actual data. Before Ajax-times, you could just use wget, or maybe a library provided by your programming language of choice (Python – urllib2) – to fetch the HTML source. Javascript changed this, as almost nothing, except the actual web browers, can execute it while reading the source.

Few days ago I faced such a problem. I spent most of my day figuring out how to read Javascript-rich web 2.0 page content using only tools available in shell environment. I preferred something I could use with Python, but nothing seemed to work. As I had almost lost all hope of achieving my goal, I decided to take one last look of PyQt4. I knew QT had a webkit component but I didn’t know if it can be used in a non graphical environemnt or if PyQt4 supported it. It can but it cannot and it does but it does not…

When I first tried PyQt4 it seemed to include WebKit module. But it didn’t. After a long session of banging my head against the wall I realized that even though the module QtWebKit was there, it didn’t actually do anything. It just failed silently when I used it. Then I came across this: http://www.insecure.ws/2008/09/16/xserver-less-webpage-screenshot#comment-239. It seems version 4.4.2 of PyQt4 is sometimes build without support of WebKit. Guess which version I was using? I updated my PyQt and got immediately some results.  I also found out that QWebPage can be used in a widgetless situation. I also learnt that PyQt crashes when you create such object without first starting QtGui app by: QtGui.QApplication(sys.argv). It doesn’t work if you pass the second argument as False, for starting a non-Gui QT application. This seemed like a deal breaker to me, even though I was able to render the complex html with ajax calls just correctly with all the data included. It bugged the hell out of me. I drew no windows, no dialogs, nothing graphical. And yes, you cannot run a PyQt GUI application in shell, even though it produces no graphics. There was no physical QWebPage to be seen, everything happened on the background, why the hell it didn’t work as non Gui Qt Application? I never found an answer for this, but I found a way around it: XVFB, thanks to the post I referenced earlier.

Xvfb is a Virtual Framebuffer for X. What a life saver. I don’t know how it does it what it does, but basically it fakes a Display in linux environment where X is not available. I guess it directs everything to /dev/null or something, which in my case was just perfect, as I didn’t even want to produce anything graphical. My only interest was the HTML content, in textual format. So now, with the help of Xvfb, I can run “graphical programs” in shell and render my complex web pages programmatically, and parse the results as I please. Maybe my solution is not perfect and maybe there is some cleaner way of doing it, but I couldn’t find it. If you know, please share! Until that, I’m quite happy running my PyQt application (even though it seems to be quite heavy for such a basic task).

After I gave PyQt a shot, I did some googling and found out that others have used the same approach as me. Here’s one link:

http://aezell.wordpress.com/2009/02/13/screenshot-a-url-with-python-and-qt-and-webkit/

Running stuff on a shared host can be tricky. Installing stuff can be even trickier. My first task was to install Django. Even though my host/webhotel provided Django, I wanted to use a newer version. After digging around I found a wonderful little script for creating a virtual environemnt for Python. It basically creates bin & lib (& site-packages) directories to your home directory and links them to the system-wide python. Only now when you use your “local” python you can safely install stuff to the site-packages directory, using basic “setup.py install” method. It works great and you need no additional priviledges as you install everything under your home directory. Pretty cool. Check out more information about virtual-python here.

After installing Django I was ready to try to set up a live site using it. I didn’t have any real needs for my Django site, so I decided to just hack something up, that demonstrates that I can use Django on my shared host. Displayofpatience.net runs on Apache, so I figured fcgi was the way to go. I created a .fcgi script according Django documentation and it worked out of the box. I, of course, used my own “local” python for running the fcgi script. Then I just configured  .htaccess file to handle .fcgi files (all in the Django docs). Django takes care from that on, according to the project urls.py.

Here is the outcome: The awesome Lottery machine!

To spice things up a little, I also added some JQuery ajax and animations to my program.  I hadn’t tried JQuery before, so that too was also a nice learning experience. I’m really happy about the outcome. Of course, the actual program is pretty useless, but now I can run my wordpress site as usually and Django applications along with it.