How I hacked an android game with Python and OCR!

Math Workout is a famous android game. In fact, it features in the top 5 of google listings for many math game + android related queries. The objective of the game is very very simple. It will fire simple math questions one after the other and you'll have to tap in the correct answer. Its a race against time among other users of the app in the world.

Here's how the app looks like and a few screenshots of questions:

As you can see, the game is fairly straigtforward. So its the time that you have to beat. A naive approach to that would be having a calculator or a computer near by and feeding in the questions to determine the answer and feeding it back to the phone. Totally manual!

Thats when the programming neurons of my brains started itching me that this could be automated and cheated by some mean. Come on think, think! So i sat on to solve this problem during my weekend and started thinking about ways i could attack this problem.

These are the steps that came into my mind in the first thought:

1. Grab a screenshot of every question
2. Crop the screenshot so that only the question is visible
3. Run the cropped image through an OCR engine
4. Parse the result and evaluate it
5. Identify the co-ordinates of the resulting number and appropriately simulate touch events in the phone

Bummer! Every step looked a bit complex in itself at first sight. Then came along a bit of googling, and voila, i found the perfect tool that i needed to perform steps 1, 2 and 5. It is the monkeyrunner tool that comes along with the Android SDK. It opens up a Python API through which i can grab and crop screenshots, simulate touch events given an (x,y) co-ordinate. Exactly what i wanted.



Now, I have the cropped image that has the question in hand. Next step is to run it through an OCR engine. Again googling told me that ocrad is an useful OCR command line tool that was available as a part of the GNU project. I installed it and found that it cannot process png images. So i had to run the image through a converter before passing it to ocrad. This small piece of shell script helped me accomplish that:


To keep things simple, the shell script is invoked from python using os.popen(). Now, I have the actual expression as a python string. As you can see from the sample screenshots, few questions can be solved by a direct "eval" whereas others require some processing. Basic operations like addition, subtraction, multiplication and division can be solved using "eval". Whereas questions like "10% of 20", "square root of 9" needs some processing. Thats what this following if else block does:



Now that the expression is evaluated and we have the result in hand, all that's left is to go through the result character by character and simulate touch events in corresponding positions in the screen. I managed to identify the co-ordinates of each number in the screen by trial and error and hard coded those values within two functions named getx() and gety() which will take a character and return its corresponding x and y co-ordinates respectively, and the simulation happens. Here is the code snippet:



To orchestrate this whole process and play the game fully automatically other cosmetic additions like coping up with the frame rate of the phone and taking care of screenshot/ocr lags are to be considered. These are handled by minor if conditions and sleeps for very small amounts of time.

The end result is as you see in the below screenshot :-P

Here is a video of how the game looks like when it is being played by my script:

Though these steps seem like computationally a bit expensive, in practice i found them to be really fast. The script was able to answer approximately 2 questions per second (with an explicit sleep of 0.2 seconds between two questions - which leads to 2 questions every 0.8 seconds). A C/C++ program might run faster than this, but i stopped here as i have accomplished what i wanted. Overall it was a fun filled Sunday! :-)

Here is a link to the full source code of the automated script: auto_math_workout.py (you can find ocr.sh from the gist above in this page - rest of the source code is in the link)

Any comments/feedbacks are welcome! :-)

-Vignesh

Replacing Lightweight Web Services with Twitter Bots!

Foresight
Here is a vague idea. I'm not sure if i can even call it an idea. I always had the tendency to create lightweight information portals as usable web-based services with a very simple user interface. It just used to get things done and nothing more.

Practical Use Case
For example, let us consider a simple problem of "PNR Status Enquiry" in Indian Trains (for the uninitiated: PNR status is nothing but the current status of a waitlisted ticket in Indian Trains). First of all, such a system is really useful because the official Indian Railways websites are a bit clogged and they don't provide any alert services. Also, their websites are not so catchy and mobile compatible, and we definitely don't want to switch on the computer just to check the PNR status.

Before today, if i was asked to build such a system, i would go for an elegant google-like web page, where there is just a text box and a button for the user to enter the PNR number and click go. The resultant page will be again a simple HTML table with the ticket details (of course all of them scrapped from one of the railway websites - pretty sure that this isn't legal, although i am not aware of any laws against site scraping).

Do we really need web for this?
But this thought stuck me today. Web is a wonderful platform, at times too good to host silly and redundant stuff like this. Tomorrow i may have to build a similar system for Buses or Aeroplanes requiring me to add more and more pages with almost same functionality but different information sources. We don't need the web for such silly things. My idea is that, why not use "Twitter" as a platform for such web services. I've heard of facebook as a platform before for many applications and games (Farmville), so why not twitter as a platform too?

Twitter as a Platform
So what exactly do i mean by twitter as a platform? Let us redesign the same PNR status enquiry system using what i mean by twitter as a platform. Instead of having a web page for inputs, lets host a twitter bot, say @pnrbot. Now, whenever you need to enquire the status of a PNR number, all you have to do is post a tweet mentioning that bot, for example "@pnrbot 1234567890" (where 1234567890 is the PNR number you wish to enquire).

Now, as long as your tweets are public, the bot can read your tweet almost instantaneously thanks to the vast amount of real time APIs provided by twitter. Now the bot does the usual site scraping from the railways website for your PNR status and it posts it as a reply to your tweet. Simple isn't it? To take it a level further, the bot can also autotweet your PNR status every day until your journey date, which is not very easy in the case of a web based app. Also, since you will not be following the bot and the bot will not be following you, you will not clutter any of your friends' timeline with this tweet as it won't appear in their timeline.

If any of the input data is sensitive and not to be exposed, then the same design can be adopted by just switching the term "tweet" with "Direct Message" (although in that case, both you and the bot need to be following each other, which can be easily accomplished).

Pros of the Platform
Twitter is a part of our day to day life since the inception of mobile internet. So, you get many useful information from such bots interactively rather than opening your webpage and waiting for the page to load. Moreover, twitter is known for its notifications. If you have activated SMS alerts for @ mentions, then you don't even need to have internet to make use of such a bot. You can just send the tweet through an SMS and read the reply from the bot as SMS. Also, twitter has email notifications which may also be of good use. As mentioned in the example, you can have one input with multiple periodical outputs (like time based notifications, etc.) which is not so easy to implement in web based systems.

Developer Standpoint
So from a developer standpoint, what do we ultimately gain by choosing Twitter as a platform over Web for lightweight services? The answer is quite simple and really advantageous. For one, you need not host a web server for lightweight web services. All you need to do is run a script that will act as the bot. The script will also be really lightweight since twitter APIs does all the pushing for you (no polling). You offload majority of the user interface and load to twitter and do only the actual processing in your server.

From the implementation perspective, it'd be really great to have a good library/framework in a nice scripting language (like php or python) for building such a bot so that the possible features (like twitter API access, etc.) could be abstracted out thereby the developers actually have to write just the logic of their actual bot and nothing extra.

Closing Thoughts
I am pretty sure such bots already exists. By building more and more of interactive bots Twitter can really stand tall as a good platform not just for communication but also for instantaneous information retrieval.

P.S.: I am in the process of developing a simple such bot as a proof-of-concept. I am also highly determined to come up with a generic twitter bot library as i mentioned in the post. So as always, interested developers are welcome to carry on if you like the idea!

-Vignesh

Je L'ai Dit turns one!

Yesterday, may 31, 2011, my blog turned one. I have written around 60 posts altogether and I really feel good about it.

My heartfelt thanks to all who are following and managing to read all the rubbish I write. Thank you! :-)

-Vignesh

Orkut Deja Vu - The Technical Side!

Its been almost two months now since I first launched "Orkut Deja Vu" - A series of web applications and a chrome extension that helps you move your memories from orkut to facebook. Personally, I think the application was a good hit. In this article i will try and explain the technical side of it and the various hurdles faced on the due course of development.


Note: This article is for the technically inclined and if you don't want the technical details and just want to use the application, visit http://orkutdejavu.foamsnet.com!


Java and My server!


To begin with it, my server is a VPS running linux with a humble 700MB RAM and a shared processor. My search for an Orkut API ended with orkut os client - An official API provided by Google. This is more of a library than an API and only the java implementation of the library was very sophisticated (though there were php implementations, they weren't that good). Ever since i started learning computer science, if there was one thing i hated, it was java (now don't even get me started about perl).


So there lied the first problem in front of my eyes: Running java in my server. I didn't want to take up the pain of setting up JSP for this sake and decided to just call the java program that does the orkut calls from PHP using shell_exec. I know it isn't a safe option, but since there is no user passed data and the calls are hard-coded, i was sure that there was no injection vulnerabilities. Yet this approach isn't very efficient as it spawns a separate java vm for every exec call, but it was a compromise that i had to make for not setting up a java based server.


OAuth - The headache!


The orkut library's OAuth implementation was really messy. Fortunately, it had a method to explicitly set an OAuth access token obtained from elsewhere. That said, I used Zend OAuth library in PHP to perform the 3 legged OAuth and use that access token in the java programs. There were totally four java programs: one each to fetch the user's orkut name, albums list, photos and scraps. The java programs printed the output as JSON if it was successful or nothing if there was an exception. PHP then parses that JSON. I used JSON so that all the escaping will be taken care by the JSON libraries and thereby ensuring safety of data transmission.


Photo Album Migrator


The photo album migration was quite straight forward without much hurdles. All i had to do was integrate the following: Image gallery, Facebook API, Orkut API and write some simple javascript that made AJAX calls to transfer the photos. This was quite a cakewalk as i was already very familiar with the Facebook graph API.

Scrapbook Downloader


Again, the programming side of this was quite straight forward. But there was one major challenge involved in designing this. The java program generates a html file containing the scraps which is then converted to PDF. HTML to PDF conversion is CPU intensive and hence needs to be done with care. I could have used a resource manager like Sun Grid Engine, but i did not want to make things complex for a simple job to be done. Also, the conversion is not done programatically by a library as all the PHP PDF libraries were memory-wise very expensive (a file with ~100 scraps always exceeded PHP's memory limit of 64MB). So, I used a external command line utility (wkhtmltopdf) to accomplish this conversion. Again it was a simple shell_exec with hard-coded arguments.

Testimonials Migrator


This was the really challenging part. There was no Orkut API that gave access to users' testimonials. But i didn't want to give up. I really wanted to pull off a tool that can migrate testimonials to facebook. So I thought of attacking this problem from the heart of orkut - the orkut website. Obviously, the first thing that came to my mind was a Google Chrome extension. The extension will inject a javascript into orkut.com website and add a "Post to Facebook" button beneath eacch of your testimonials.


The first hurdle in accomplishing this was that there were two versions of orkut (old and new) with different page structures. I didn't want to write two different scripts to handle the versions. Instead i wrote another script that detects the version and if it is new, it prompted the user to redirect to the older version in order to use the extension. The next job was to understand orkut's DOM so that I can place the buttons. This ended up being quite an easy job too.


Another glitch in chrome extensions is that we cannot specify images directly in CSS for content scripts(for e.g.: background: url(a.jpg); is not possible), as the CSS will run in the scope of the website. So, it has to be either done programatically using javascript or encode the image in base64 and hard code it in the CSS. I chose the first option. From this point, it was fairly straight forward. When the user clicks on the post button, store the corresponding testimonial using HTML5's local storage and create a new tab where the user can choose the posting options. Again, the facebook authentication and API usage here were simple as i had enough exposure already.


Epilogue


Though it seems like a simple app, huge amount of thoughts are put into every single aspect in design of the application and the entire development process was a fun journey with a great learning curve. This application has made me feel my web presence. Web is really a great platform for amateur developers like me. I have got around 20 new followers in twitter and a person from brazil appreciating me for this application. I have really been motivated a lot by this and hope to continue the same stride in creating usable applications as this one!


This post will not be complete without a heartfelt thanks to all those who supported me on due course of development and all those who used and shared this with your friends!


-Vignesh

Introducing Way2Droid - Way2SMS for Android !

This is one of the very few android apps that I have managed to get into a publishable state. This is a simple app that allows you to quickly send SMS messages using your data plan/wifi for free with a valid Way2SMS.com account. It uses way2sms-php library (read below for description of the library)!


Features include:

• Add any way2sms account
• Easy to use editor
• Pick your contacts
• Set upto 9 speed dials
• Send messages of any length ( no 140 character limit )
• Log sent messages in stock SMS
• Send in background

Source Code: http://code.google.com/p/way2droid

Download APK: Link here!

Screenshots

Open Source Way2SMS API

I have developed and a free API for sending SMS using Way2SMS as the gateway. The API has been developed using PHP.

To use the API and send SMS, simply send a HTTP request to the following URL:

http://www.foamsnet.com/smsapi?username={username}&password={password}&to={destination}&msg={message}
The fields enclosed in curly braces are the parameters and they are fairly self explanatory!


A sample usage in Python is as follows:
>>> import urllib2
>>> f = urllib2.urlopen('http://www.foamsnet.com/smsapi?username=username&password=password&to=12345&msg=hello+from+api')
>>> print f.read()
Sent successfully to 12345
Yes. Its that simple! Source code for the library can be found at: http://code.google.com/p/way2sms-php

-Vignesh

Orkut to Facebook Testimonials Migration! - Google Chrome Extension

Following the quite a good success of the Orkut to Facebook Photo Album migrator, I now introduce Orkut to Facebook Testimonials Migration as a Google Chrome Extension. Due to the technical limitations in the Orkut API, This is not available as a web application. Download this simple Google Chrome extension and migrate your Orkut testimonials to Facebook!

Many of our beautiful memories are left out as testimonials in Orkut! Its time to move them to facebook too! Download this Google Chrome Extension and start migrating your testimonials now! :-) 

Visit: http://orkutdejavu.foamsnet.com (or) http://orkut.foamsnet.com to see the Instructions

Visit: Click here to download the Extension!

Please post your bug reports/suggestions to: vign...@gmail.com (click to expand)

-Vignesh

The Social Network - Build it Google, We will Come!

I recently read an article about Larry Page taking over as the CEO of Google and sending an internal memo to googlers about 25% cut in their bonuses if Google didn’t do well in social arena this year. I was disturbed a lot ever since I read that and wanted to pour out my thoughts about a dream social network from Google. Yes, I frankly think Google can still do much better than Facebook.

A headnote, this article is totally how i view a Google social network shall be made possible. I have tried to think of it as practically as possible. Some of it may sound silly/stupid for expert readers, so kindly bear with me.

Orkut was the trend setter! - What went wrong?

Orkut was the first and most famous "social network" of all time. It came during the period where the term "social network" was not really defined. Orkut gave that term a definition. And it was doing really well especially in places like India and Brazil. Ever since Facebook came in, Orkut started to lose.

I, personally, would say that the reason why Orkut never caught on was because Google tried to remain professional. They probably wanted to stick to the Google style of doing things and in the process forgot that people just wanted to have fun and party on in the social network unlike other Google services where professionalism kept people happy. The best example i would like to quote is, Orkut did not have scrap threading for a long time and that was available even for SMS by the time Orkut added that. Naturally, we don’t want to roam around in a blazer/suit 365x24x7. Facebook realized it and they just let people do whatever they wanted to do, literally no restrictions whatsoever.

Sign of Innovation - Google Wave

Google then launched wave and claimed that "it is simply going to change the way people communicate". Well, we know how true it is from the fact that wave has been axed by Google few months back and is now residing in its open source home of Apache.



Again, the problem with Google Wave has been that it was difficult for a layman to understand and use it on a day-to-day basis. Even when wave was axed, not a single layman cared about it, only computer professionals cared since the underlying technology and protocols were really well built. Yet again, Google had failed to capitalize the wonderful technology they built by coping it up to what everyone actually wanted. In short, Google tried to remain professional with wave too.

Sign of Desperation - Google Buzz

Then came Google Buzz, supposedly the "Twitter killer". One major factor that i feel Google doesn’t realize when building social products is that, You cannot force people down their throats to go Social. Yes, I love Twitter and can’t live without it but at the same time I don’t want to read my tweets in my Gmail inbox.



Similarly, Google Reader is one of the most wonderful software ever built. But with literally no connection whatsoever, Google decided to tie Buzz with Reader. This tie up lead to nothing but junk in both Reader and Buzz. And when people wanted to opt out of Buzz, it resulted in loss all their Reader follow list too. Making it opt-in is a really good vision by Google and at the same time they should also make sure that the opt-out is cleanly done.

What next? Social + Google = ??

So can Google ever surpass the mountain, that is Facebook, and succeed in social like they did in search? My answer is, yes they surely can. Sure Facebook has some 600 million users but Google is no poor lad when it comes to userbase too. Infact, Google is still a more prominent part of our life than Facebook is. Conduct a poll asking "Which of the following can’t we live without - Facebook or Google?" and i bet the winner will be Google.

Imagine a social network that revolves around all of Google’s services. Imagine a service that aggregates all of gmail, youtube, reader, maps, picasa, talk, news, voice and even orkut. Google, if you can build a rock solid service that does this and add your typical magical touch to it, then definitely you can climb the mount everest that is Facebook. Add to this the crazy, yet possible, thought of Google acquiring Twitter and integrating Buzz with it seamlessly. By recitifying all the mistakes that it did in the social arena in the past and building such a clean and fun service, it can surely overpower Facebook.

Google’s userbase is much more loyal when compared to that of Facebook. Google, as a company, with its policies is much closer to our hearts than Facebook is. Identify Facebook’s problems and fix it. Facebook’s main concerns today are spam and privacy. I can’t think of a product other than Gmail that can handle spam so near to perfection. Also, Google Talk has been an integral part of Gmail and Orkut for so many years now and not once we have seen any spam in it that can even be compared with the Facebook chat spam we have these days. Though Google have had their own share of controversies regarding privacy, they have somehow held on and been in the good books of their users when it came to privacy.




Of course all this is not going to be an overnight affair, it is going to take lots of effort and time. Given that Google is investing so keenly on getting that top spot in the social arena, proper focus, being unprofessional and learning from the past are the key factors that will decide the fate of Google in the social arena. They can’t afford to flop anymore. This may be Google’s last chance to make an impact. By heart, i sincerely hope that Google can just snatch that number one spot in the social arena!

So, my message is short and simple: "Build it Google - We will come!".

-Vignesh