Reading ajax content programmatically

October 28th, 2009 by bodhi 4 comments »
1
Digg me

Javascript and dynamic content loading with Ajax has changed the way web is being used. User interfaces have advanced rapidly and the “web content” is becoming more and more complex. This, however, presents whole new kinds of problems for parties other than the expected end user. Web content is intended to be read with a graphical browser, such as Firefox or Google Chrome. Javascript capabilities are taken for granded, as they should, as we live in the modern ages :) However, sometimes there still is a need for reading the content programmatically. Ajax content loading can become quite a pain in the ass in such situations, as I now have learnt.

In most cases, content retrieved by Ajax after the normal page is being loaded, isn’t a problem. If you really need the content, you can usually see how the Ajax call is made and use the same url for getting the data directly. In Most Cases. Sometimes the Ajax calls are not so obvious; the javascript might be obfuscated or you can’t really tell where the data is being loaded from. Or sometimes you just need to simulate the actual page loading. Test automation could be one of such situation. None the less, the problem is loading the actual data. Before Ajax-times, you could just use wget, or maybe a library provided by your programming language of choice (Python – urllib2) – to fetch the HTML source. Javascript changed this, as almost nothing, except the actual web browers, can execute it while reading the source.

Few days ago I faced such a problem. I spent most of my day figuring out how to read Javascript-rich web 2.0 page content using only tools available in shell environment. I preferred something I could use with Python, but nothing seemed to work. As I had almost lost all hope of achieving my goal, I decided to take one last look of PyQt4. I knew QT had a webkit component but I didn’t know if it can be used in a non graphical environemnt or if PyQt4 supported it. It can but it cannot and it does but it does not…

When I first tried PyQt4 it seemed to include WebKit module. But it didn’t. After a long session of banging my head against the wall I realized that even though the module QtWebKit was there, it didn’t actually do anything. It just failed silently when I used it. Then I came across this: http://www.insecure.ws/2008/09/16/xserver-less-webpage-screenshot#comment-239. It seems version 4.4.2 of PyQt4 is sometimes build without support of WebKit. Guess which version I was using? I updated my PyQt and got immediately some results.  I also found out that QWebPage can be used in a widgetless situation. I also learnt that PyQt crashes when you create such object without first starting QtGui app by: QtGui.QApplication(sys.argv). It doesn’t work if you pass the second argument as False, for starting a non-Gui QT application. This seemed like a deal breaker to me, even though I was able to render the complex html with ajax calls just correctly with all the data included. It bugged the hell out of me. I drew no windows, no dialogs, nothing graphical. And yes, you cannot run a PyQt GUI application in shell, even though it produces no graphics. There was no physical QWebPage to be seen, everything happened on the background, why the hell it didn’t work as non Gui Qt Application? I never found an answer for this, but I found a way around it: XVFB, thanks to the post I referenced earlier.

Xvfb is a Virtual Framebuffer for X. What a life saver. I don’t know how it does it what it does, but basically it fakes a Display in linux environment where X is not available. I guess it directs everything to /dev/null or something, which in my case was just perfect, as I didn’t even want to produce anything graphical. My only interest was the HTML content, in textual format. So now, with the help of Xvfb, I can run “graphical programs” in shell and render my complex web pages programmatically, and parse the results as I please. Maybe my solution is not perfect and maybe there is some cleaner way of doing it, but I couldn’t find it. If you know, please share! Until that, I’m quite happy running my PyQt application (even though it seems to be quite heavy for such a basic task).

After I gave PyQt a shot, I did some googling and found out that others have used the same approach as me. Here’s one link:

http://aezell.wordpress.com/2009/02/13/screenshot-a-url-with-python-and-qt-and-webkit/

Simple example of Django on shared host

October 12th, 2009 by bodhi 2 comments »
2
Digg me

Displayofpatience.net runs on a shared host. It’s basically your avarage wordpress site, and the whole domain is dedicated to it. In addition to wordpress, I wanted to try if I could run a django site along with it. The first thing was to install Django.

Running stuff on a shared host can be tricky. Installing stuff can be even trickier. My first task was to install Django. Even though my host/webhotel provided Django, I wanted to use a newer version. After digging around I found a wonderful little script for creating a virtual environemnt for Python. It basically creates bin & lib (& site-packages) directories to your home directory and links them to the system-wide python. Only now when you use your “local” python you can safely install stuff to the site-packages directory, using basic “setup.py install” method. It works great and you need no additional priviledges as you install everything under your home directory. Pretty cool. Check out more information about virtual-python here.

After installing Django I was ready to try to set up a live site using it. I didn’t have any real needs for my Django site, so I decided to just hack something up, that demonstrates that I can use Django on my shared host. Displayofpatience.net runs on Apache, so I figured fcgi was the way to go. I created a .fcgi script according Django documentation and it worked out of the box. I, of course, used my own “local” python for running the fcgi script. Then I just configured  .htaccess file to handle .fcgi files (all in the Django docs). Django takes care from that on, according to the project urls.py.

Here is the outcome: The awesome Lottery machine!

To spice things up a little, I also added some JQuery ajax and animations to my program.  I hadn’t tried JQuery before, so that too was also a nice learning experience. I’m really happy about the outcome. Of course, the actual program is pretty useless, but now I can run my wordpress site as usually and Django applications along with it.

Scaler frontend

August 2nd, 2009 by bodhi No comments »
0
Digg me

In my previous ps3 post I tinkered with spe-scaler, which is a very nice tool to help you (for example) play emulator games in fullscreen. Unfortunately, it’s not very userfriendly, and I decided to write a little frontend program for it, to act as a gui. I first thought about writing it in QT/C++, but as I’m not that familiar with QT, I decided to do it in Python due time constraints. I chose Tkinter as a framework, which also was new to me, but as the program doesn’t do much I got it working pretty fast. Anyway, let me know what you think about it: scalergui.

Playing emulator games with my ps3 with linux pt. 3 – scaler

July 30th, 2009 by bodhi 5 comments »
1
Digg me

This is a follow up on my earlier posts of playing emulator games on ps3 with linux… (pt. 1, pt. 2)

As I said earlier, I wanted to do a clean install with the brand new Yellow Dog Linux 6.2 and that’s what I did. The installation wen’t fine, same as usual, and it took forever. Suprisingly the display was set correctly to 720p without any manual tweaking. At the time I thought it would be the ideal resolution for playing emulator games (from my earlier experiences).  There were narrow black borders around the screen, but I can live with that. So the next step, once again, was to install the actual emulators.

I added the PS3Bodega repos as usual and installed following emulators: xe, vice and e-uae. At the time I did this, the YDL 6.2 repos were still missing content, so I had to manually add the dvd as a repository source (there were instructions on the YDL forums for that), but it should only be a temporary problem and not needed in the future. Also, the bluetooth is not currently working in YDL 6.2, but I hope that too will be fixed soon. With the bluetooth not working, you have to use a usb cord for plugging in your controller, but luckily that works out of the box.

All the emulators I tried worked just about the same as before. I knew I couldn’t run most of the them in full screen, because the fps would drop dramatically. Instead I wanted to try the scaling method. With scaling, you run the emulators at their native resolution, which is usually very low. This way the fps should be fine. The scaler is an external program, which is developed for the CELL processor, and it uses the additional SPE’s to do all the work. The main CPU (PPE), which is running the emulator and the operating system, is not stressed. In other words the scaler program magnifies part of the screen, so you can get a bigger picture without using the emulators full screen or double screen actions, which would stress the main cpu and make the fps drop. The scaling method works quite well, but unfortunately it’s not very user friendly.

Scaler

To install the scaler, as root, do:

yum install scaler

Scaler is the original scaler program, but there are some modifications available, like f-scaler, which provides some additional features. Scaler is a command-line program which takes two arguments: -s and -t. “s” is for the scale, it has to be an interger, usually something between 2 and 4. The “t” is for timeout, I usually set it far (like 100000) and kill the process manually. This is how I use the scaler:

  1. Start the emulator just as usual and load up the game you want to play
  2. Open terminal and start the scaler: “scaler -t 10000 -s 2″
  3. -> The display should now be scaled in the middle of the screen
  4. Press alt-tab to switch to the emulator game
  5. Use alt-mousedrag to center the window so you can see the whole screen.
  6. Play
  7. When finished, alt-tab change back to terminal and ctrl-c to kill the scaler process

Not very userfriendly as you can see…

You probably might want to try the different scale options of the scaler for each emulator. Unfortunately you have to use integers, which makes it a somewhat limited tool. You cannot give the scaler an option like -s 1.5. I also figured that 720p is not very ideal for screen resolution, when using scaler, because anything over scale option 2 usually gets too big. So I changed my resolution to 1080p to get me more leverage when using the scaler. As you’re not using the emulator to get full screen, it runs perfectly ok, even if the reolution is so big. With vice for example, I use the scale option 4.

You can also start the scaler along with the emulator or create a script for it. You do it like this: “scaler -t 10000 -s 2 & x64 && killall scaler”. So that would first run the scaler, then the emulator x64 (vice) and afterwards when you quit the emulator it would also kill the scaler. I found this to be quite unusable, as you need menus to load up the games etc, but if you can or want to load the game using command line options of your emulator, it might be good way to go. I think people use that when playing mame games for example.

Afterthoughts

So using the scaler in my opinion is currently the best way to play emulator games in (almost) fullscreen. It’s not as easy and userfriendly as I’d hope it to be, but considering the alternatives, well, it’s basically the only way. Scaler is very helpful as it is, but I would like to see following features in the future:

  • A Graphical User Interface. Would make it a lot easier to use for newbies
  • More scale options, not only integers. At least “half modes” like 2.5 needed.
  • The scaler could automatically get the target window size and calculate the optimal scaling to reach fullscreen (dynamic scaling)
  • Interactive scaling through hot-keys (like Ctrl+ would increase the scale and Ctrl- decrease)

I know those features will most probably never be implemented but one can always wish right?

EDIT: Implemented Scalergui. Check it out!

And here’s a little tip for those who want to use the xe multisystem emulator. It’s from YDL forum. I had some trouble when I wanted to use the controller to play games, but replacing these options to the configuration file(s) helped (~/.xe/rc/xerc):

# Joystick bindings

Joy     1       up              4
Joy     1       down            6
Joy     1       left            7
Joy     1       right           5
Joy     1       start           3
Joy     1       select          0
Joy     1       button1         15
Joy     1       button2         12
Joy     1       button3         14
Joy     1       button4         13

Going to try out ps3 YDL 6.2 soon

July 29th, 2009 by bodhi 2 comments »
0
Digg me

It’s been a while since I last played around with my ps3 and linux and I thought I’d try out the latest version of Yellow Dog Linux (6.2). The goal is the same as in my previous posts: emulator games (megadrive, c64, amiga). What I have learned from past experiences is that I should probably set the display straight to way to 720p, and forget about the fullscreen options found within the emulator software. Instead I should concentrate on how to use scalers specifically developed to take advantage of ps3 features (additional spu’s). Scalers are external programs that will magnify a part of the screen and they might be the way to play games in fullscreen. The trick is that they don’t use the main cpu, but the additional spe’s so the framerate shouldn’t drop. Last time I had some problems using them (they worked, but it was a pain to set them up for different emulators each time you wanted a play a game). It will be fun to see if there has been any development on that area.

I’m still going for YDL, since they have made some additional tweaks concerning ps3, that I’d have to do manually if using other distributions such as Ubuntu. There should be some VRAM swap tricks etc. I’m planning to do a clean install and I’m actually downloading the iso as I’m writing this. I hope I find the time to tinker with this soon and write about my latest experiences. So, hold on for “playing emulator games with my ps3 with linux pt. 3″ :)