Replacing Lightweight Web Services with Twitter Bots!

Foresight
Here is a vague idea. I'm not sure if i can even call it an idea. I always had the tendency to create lightweight information portals as usable web-based services with a very simple user interface. It just used to get things done and nothing more.

Practical Use Case
For example, let us consider a simple problem of "PNR Status Enquiry" in Indian Trains (for the uninitiated: PNR status is nothing but the current status of a waitlisted ticket in Indian Trains). First of all, such a system is really useful because the official Indian Railways websites are a bit clogged and they don't provide any alert services. Also, their websites are not so catchy and mobile compatible, and we definitely don't want to switch on the computer just to check the PNR status.

Before today, if i was asked to build such a system, i would go for an elegant google-like web page, where there is just a text box and a button for the user to enter the PNR number and click go. The resultant page will be again a simple HTML table with the ticket details (of course all of them scrapped from one of the railway websites - pretty sure that this isn't legal, although i am not aware of any laws against site scraping).

Do we really need web for this?
But this thought stuck me today. Web is a wonderful platform, at times too good to host silly and redundant stuff like this. Tomorrow i may have to build a similar system for Buses or Aeroplanes requiring me to add more and more pages with almost same functionality but different information sources. We don't need the web for such silly things. My idea is that, why not use "Twitter" as a platform for such web services. I've heard of facebook as a platform before for many applications and games (Farmville), so why not twitter as a platform too?

Twitter as a Platform
So what exactly do i mean by twitter as a platform? Let us redesign the same PNR status enquiry system using what i mean by twitter as a platform. Instead of having a web page for inputs, lets host a twitter bot, say @pnrbot. Now, whenever you need to enquire the status of a PNR number, all you have to do is post a tweet mentioning that bot, for example "@pnrbot 1234567890" (where 1234567890 is the PNR number you wish to enquire).

Now, as long as your tweets are public, the bot can read your tweet almost instantaneously thanks to the vast amount of real time APIs provided by twitter. Now the bot does the usual site scraping from the railways website for your PNR status and it posts it as a reply to your tweet. Simple isn't it? To take it a level further, the bot can also autotweet your PNR status every day until your journey date, which is not very easy in the case of a web based app. Also, since you will not be following the bot and the bot will not be following you, you will not clutter any of your friends' timeline with this tweet as it won't appear in their timeline.

If any of the input data is sensitive and not to be exposed, then the same design can be adopted by just switching the term "tweet" with "Direct Message" (although in that case, both you and the bot need to be following each other, which can be easily accomplished).

Pros of the Platform
Twitter is a part of our day to day life since the inception of mobile internet. So, you get many useful information from such bots interactively rather than opening your webpage and waiting for the page to load. Moreover, twitter is known for its notifications. If you have activated SMS alerts for @ mentions, then you don't even need to have internet to make use of such a bot. You can just send the tweet through an SMS and read the reply from the bot as SMS. Also, twitter has email notifications which may also be of good use. As mentioned in the example, you can have one input with multiple periodical outputs (like time based notifications, etc.) which is not so easy to implement in web based systems.

Developer Standpoint
So from a developer standpoint, what do we ultimately gain by choosing Twitter as a platform over Web for lightweight services? The answer is quite simple and really advantageous. For one, you need not host a web server for lightweight web services. All you need to do is run a script that will act as the bot. The script will also be really lightweight since twitter APIs does all the pushing for you (no polling). You offload majority of the user interface and load to twitter and do only the actual processing in your server.

From the implementation perspective, it'd be really great to have a good library/framework in a nice scripting language (like php or python) for building such a bot so that the possible features (like twitter API access, etc.) could be abstracted out thereby the developers actually have to write just the logic of their actual bot and nothing extra.

Closing Thoughts
I am pretty sure such bots already exists. By building more and more of interactive bots Twitter can really stand tall as a good platform not just for communication but also for instantaneous information retrieval.

P.S.: I am in the process of developing a simple such bot as a proof-of-concept. I am also highly determined to come up with a generic twitter bot library as i mentioned in the post. So as always, interested developers are welcome to carry on if you like the idea!

-Vignesh

Je L'ai Dit turns one!

Yesterday, may 31, 2011, my blog turned one. I have written around 60 posts altogether and I really feel good about it.

My heartfelt thanks to all who are following and managing to read all the rubbish I write. Thank you! :-)

-Vignesh

Orkut Deja Vu - The Technical Side!

Its been almost two months now since I first launched "Orkut Deja Vu" - A series of web applications and a chrome extension that helps you move your memories from orkut to facebook. Personally, I think the application was a good hit. In this article i will try and explain the technical side of it and the various hurdles faced on the due course of development.


Note: This article is for the technically inclined and if you don't want the technical details and just want to use the application, visit http://orkutdejavu.foamsnet.com!


Java and My server!


To begin with it, my server is a VPS running linux with a humble 700MB RAM and a shared processor. My search for an Orkut API ended with orkut os client - An official API provided by Google. This is more of a library than an API and only the java implementation of the library was very sophisticated (though there were php implementations, they weren't that good). Ever since i started learning computer science, if there was one thing i hated, it was java (now don't even get me started about perl).


So there lied the first problem in front of my eyes: Running java in my server. I didn't want to take up the pain of setting up JSP for this sake and decided to just call the java program that does the orkut calls from PHP using shell_exec. I know it isn't a safe option, but since there is no user passed data and the calls are hard-coded, i was sure that there was no injection vulnerabilities. Yet this approach isn't very efficient as it spawns a separate java vm for every exec call, but it was a compromise that i had to make for not setting up a java based server.


OAuth - The headache!


The orkut library's OAuth implementation was really messy. Fortunately, it had a method to explicitly set an OAuth access token obtained from elsewhere. That said, I used Zend OAuth library in PHP to perform the 3 legged OAuth and use that access token in the java programs. There were totally four java programs: one each to fetch the user's orkut name, albums list, photos and scraps. The java programs printed the output as JSON if it was successful or nothing if there was an exception. PHP then parses that JSON. I used JSON so that all the escaping will be taken care by the JSON libraries and thereby ensuring safety of data transmission.


Photo Album Migrator


The photo album migration was quite straight forward without much hurdles. All i had to do was integrate the following: Image gallery, Facebook API, Orkut API and write some simple javascript that made AJAX calls to transfer the photos. This was quite a cakewalk as i was already very familiar with the Facebook graph API.

Scrapbook Downloader


Again, the programming side of this was quite straight forward. But there was one major challenge involved in designing this. The java program generates a html file containing the scraps which is then converted to PDF. HTML to PDF conversion is CPU intensive and hence needs to be done with care. I could have used a resource manager like Sun Grid Engine, but i did not want to make things complex for a simple job to be done. Also, the conversion is not done programatically by a library as all the PHP PDF libraries were memory-wise very expensive (a file with ~100 scraps always exceeded PHP's memory limit of 64MB). So, I used a external command line utility (wkhtmltopdf) to accomplish this conversion. Again it was a simple shell_exec with hard-coded arguments.

Testimonials Migrator


This was the really challenging part. There was no Orkut API that gave access to users' testimonials. But i didn't want to give up. I really wanted to pull off a tool that can migrate testimonials to facebook. So I thought of attacking this problem from the heart of orkut - the orkut website. Obviously, the first thing that came to my mind was a Google Chrome extension. The extension will inject a javascript into orkut.com website and add a "Post to Facebook" button beneath eacch of your testimonials.


The first hurdle in accomplishing this was that there were two versions of orkut (old and new) with different page structures. I didn't want to write two different scripts to handle the versions. Instead i wrote another script that detects the version and if it is new, it prompted the user to redirect to the older version in order to use the extension. The next job was to understand orkut's DOM so that I can place the buttons. This ended up being quite an easy job too.


Another glitch in chrome extensions is that we cannot specify images directly in CSS for content scripts(for e.g.: background: url(a.jpg); is not possible), as the CSS will run in the scope of the website. So, it has to be either done programatically using javascript or encode the image in base64 and hard code it in the CSS. I chose the first option. From this point, it was fairly straight forward. When the user clicks on the post button, store the corresponding testimonial using HTML5's local storage and create a new tab where the user can choose the posting options. Again, the facebook authentication and API usage here were simple as i had enough exposure already.


Epilogue


Though it seems like a simple app, huge amount of thoughts are put into every single aspect in design of the application and the entire development process was a fun journey with a great learning curve. This application has made me feel my web presence. Web is really a great platform for amateur developers like me. I have got around 20 new followers in twitter and a person from brazil appreciating me for this application. I have really been motivated a lot by this and hope to continue the same stride in creating usable applications as this one!


This post will not be complete without a heartfelt thanks to all those who supported me on due course of development and all those who used and shared this with your friends!


-Vignesh

Introducing Way2Droid - Way2SMS for Android !

This is one of the very few android apps that I have managed to get into a publishable state. This is a simple app that allows you to quickly send SMS messages using your data plan/wifi for free with a valid Way2SMS.com account. It uses way2sms-php library (read below for description of the library)!


Features include:

• Add any way2sms account
• Easy to use editor
• Pick your contacts
• Set upto 9 speed dials
• Send messages of any length ( no 140 character limit )
• Log sent messages in stock SMS
• Send in background

Source Code: http://code.google.com/p/way2droid

Download APK: Link here!

Screenshots

Open Source Way2SMS API

I have developed and a free API for sending SMS using Way2SMS as the gateway. The API has been developed using PHP.

To use the API and send SMS, simply send a HTTP request to the following URL:

http://www.foamsnet.com/smsapi?username={username}&password={password}&to={destination}&msg={message}
The fields enclosed in curly braces are the parameters and they are fairly self explanatory!


A sample usage in Python is as follows:
>>> import urllib2
>>> f = urllib2.urlopen('http://www.foamsnet.com/smsapi?username=username&password=password&to=12345&msg=hello+from+api')
>>> print f.read()
Sent successfully to 12345
Yes. Its that simple! Source code for the library can be found at: http://code.google.com/p/way2sms-php

-Vignesh

Orkut to Facebook Testimonials Migration! - Google Chrome Extension

Following the quite a good success of the Orkut to Facebook Photo Album migrator, I now introduce Orkut to Facebook Testimonials Migration as a Google Chrome Extension. Due to the technical limitations in the Orkut API, This is not available as a web application. Download this simple Google Chrome extension and migrate your Orkut testimonials to Facebook!

Many of our beautiful memories are left out as testimonials in Orkut! Its time to move them to facebook too! Download this Google Chrome Extension and start migrating your testimonials now! :-) 

Visit: http://orkutdejavu.foamsnet.com (or) http://orkut.foamsnet.com to see the Instructions

Visit: Click here to download the Extension!

Please post your bug reports/suggestions to: vign...@gmail.com (click to expand)

-Vignesh

The Social Network - Build it Google, We will Come!

I recently read an article about Larry Page taking over as the CEO of Google and sending an internal memo to googlers about 25% cut in their bonuses if Google didn’t do well in social arena this year. I was disturbed a lot ever since I read that and wanted to pour out my thoughts about a dream social network from Google. Yes, I frankly think Google can still do much better than Facebook.

A headnote, this article is totally how i view a Google social network shall be made possible. I have tried to think of it as practically as possible. Some of it may sound silly/stupid for expert readers, so kindly bear with me.

Orkut was the trend setter! - What went wrong?

Orkut was the first and most famous "social network" of all time. It came during the period where the term "social network" was not really defined. Orkut gave that term a definition. And it was doing really well especially in places like India and Brazil. Ever since Facebook came in, Orkut started to lose.

I, personally, would say that the reason why Orkut never caught on was because Google tried to remain professional. They probably wanted to stick to the Google style of doing things and in the process forgot that people just wanted to have fun and party on in the social network unlike other Google services where professionalism kept people happy. The best example i would like to quote is, Orkut did not have scrap threading for a long time and that was available even for SMS by the time Orkut added that. Naturally, we don’t want to roam around in a blazer/suit 365x24x7. Facebook realized it and they just let people do whatever they wanted to do, literally no restrictions whatsoever.

Sign of Innovation - Google Wave

Google then launched wave and claimed that "it is simply going to change the way people communicate". Well, we know how true it is from the fact that wave has been axed by Google few months back and is now residing in its open source home of Apache.



Again, the problem with Google Wave has been that it was difficult for a layman to understand and use it on a day-to-day basis. Even when wave was axed, not a single layman cared about it, only computer professionals cared since the underlying technology and protocols were really well built. Yet again, Google had failed to capitalize the wonderful technology they built by coping it up to what everyone actually wanted. In short, Google tried to remain professional with wave too.

Sign of Desperation - Google Buzz

Then came Google Buzz, supposedly the "Twitter killer". One major factor that i feel Google doesn’t realize when building social products is that, You cannot force people down their throats to go Social. Yes, I love Twitter and can’t live without it but at the same time I don’t want to read my tweets in my Gmail inbox.



Similarly, Google Reader is one of the most wonderful software ever built. But with literally no connection whatsoever, Google decided to tie Buzz with Reader. This tie up lead to nothing but junk in both Reader and Buzz. And when people wanted to opt out of Buzz, it resulted in loss all their Reader follow list too. Making it opt-in is a really good vision by Google and at the same time they should also make sure that the opt-out is cleanly done.

What next? Social + Google = ??

So can Google ever surpass the mountain, that is Facebook, and succeed in social like they did in search? My answer is, yes they surely can. Sure Facebook has some 600 million users but Google is no poor lad when it comes to userbase too. Infact, Google is still a more prominent part of our life than Facebook is. Conduct a poll asking "Which of the following can’t we live without - Facebook or Google?" and i bet the winner will be Google.

Imagine a social network that revolves around all of Google’s services. Imagine a service that aggregates all of gmail, youtube, reader, maps, picasa, talk, news, voice and even orkut. Google, if you can build a rock solid service that does this and add your typical magical touch to it, then definitely you can climb the mount everest that is Facebook. Add to this the crazy, yet possible, thought of Google acquiring Twitter and integrating Buzz with it seamlessly. By recitifying all the mistakes that it did in the social arena in the past and building such a clean and fun service, it can surely overpower Facebook.

Google’s userbase is much more loyal when compared to that of Facebook. Google, as a company, with its policies is much closer to our hearts than Facebook is. Identify Facebook’s problems and fix it. Facebook’s main concerns today are spam and privacy. I can’t think of a product other than Gmail that can handle spam so near to perfection. Also, Google Talk has been an integral part of Gmail and Orkut for so many years now and not once we have seen any spam in it that can even be compared with the Facebook chat spam we have these days. Though Google have had their own share of controversies regarding privacy, they have somehow held on and been in the good books of their users when it came to privacy.




Of course all this is not going to be an overnight affair, it is going to take lots of effort and time. Given that Google is investing so keenly on getting that top spot in the social arena, proper focus, being unprofessional and learning from the past are the key factors that will decide the fate of Google in the social arena. They can’t afford to flop anymore. This may be Google’s last chance to make an impact. By heart, i sincerely hope that Google can just snatch that number one spot in the social arena!

So, my message is short and simple: "Build it Google - We will come!".

-Vignesh

Scripting vs Programming - Mastering the art of arts!

Why this post?

The line of difference between a script and a program has become very blurry these days to the extent that these terms are used interchangeably. Though there is no hard and tight way to theoretically define what a script is and say how it is different from a program, if you have done a lot of programming and if you are a person who loves to automate things, then you can definitely realize the clear line of difference between scripting and programming. In this article, I try to give my views on what scripting is, how it differs from programming, etc. (Whole of this article just represents my view and there is a good chance that some of it may be wrong).

What is a script?

A script is just a small piece of code, usually written in a non-traditional programming language (like bash or perl) that is used to get a job done. Well, sure you can’t see much of how it differs from the definition of a "program". That is what i try to explain on the due course of this article.

How does it differ from a program?
As i said earlier, there is no concrete set of rules to distinguish a script and a program. The main deciding factors are:

• the purpose with which its developed
• the design mechanism underwent on due course of development, and
• the person who uses it

Scripts are generally very specific to their task. They just do what is to be done, nothing more or nothing less. Whereas programs generally have a broad scope. They are more sophisticated and usually do much more than what they are supposed to do. This is the difference with respect to the first point.

Scripts generally don’t follow any design. Scripts are usually just written, with no design in mind whatsoever. Even if they are complex, they aren’t designed with care and concern. Whereas programs on the other hand are designed to work with a proper flow and fault tolerance. Scripts are usually used by programmers themselves internally, whereas programs are full fledged tools that are used by everyone from geeks to laymen. To state an example, Facebook is a "program" and if you build something that will scrape data off facebook in the format you want, then that could be a "script".

Scripting is usually considered as programming as a part of developing an actual program. For example, the recent android patent issue says Google used a "script" to clean up all the comments and other stuff from the kernel header files. So, scripts can be generally categorized as utility functions that helps you making your "program" development easier.

What’s a scripting language?

Again, a scripting language is a programming language that is generally non-traditional. Scripting language usually provides constructs for doing things in the quickest way possible rather than in the most efficient way possible (For e.g. most of the scripting languages usually have a sort() function to sort the data. Though they may not be the most efficient implementation of sorting data, it gets the job done without fuss). Another aspect is that scripting languages are usually interpreted and not compiled. This gives the assurance that the script dies if anything goes wrong, the script starts over all again, taking advantage of the fact that scripts need not provide any atomicity over what they do.

Uses of scripting

I love automating things i do and hence scripting is a very essential part of my online life. Scripts can help your online life get much easier and better. For example, i have various little chrome content scripts (google chrome’s equivalent of greasemonkey scripts) that helps me make my day-to-day browsing easier and more productive. Also, i extensively use sed and awk scripts to browse through log files generated by the programs i develop. I even have scripts that help me check if there are new episodes of my favorite TV Serials (BBT, HIMYM) available for download and alert me.

When it comes to automating day to day online tasks, scripting is your swiss army knife. You can accomplish things quickly and in the way you want them to be.

Scripting as an art

We all know that programming is more of an art than a science. Developing a perfect program is close to impossible. But scripting too is an art that can aid you in making your programming life much better. Learning a scripting language at the beginning may seem weird, but once you get used to the constructs, it will really be a cake walk and you will find it a lot useful to have it as one of your assets.

To sum up..

Scripting and programming always go together. Scripting aids in automating things that will make a program better than what it is now. Though only a beginner, my humble piece of advice to every programmer out there is "Never hesitate to automate things. You learn a lot while automating silly things! So next time when you come across a silly problem in your work or in your project, try to tackle it with a quick utility script rather trying to fix it manually!"

-Vignesh