Somewhere in the overlap between software development, process improvement and psychology

web stuff

Serverless event feedback processing and analytics using #aws

I recently was involved in organising a techie conference. We wanted a feedback mechanism so I made a serverless event feedback system. Here’s how it works…

Have a bunch of ipads/tablets with a nice feedback form:

feedback_form

Collect up the feedback and sent it to a lambda function via API Gateway:


const APIurl = "https://api.mydomain.net/eventFeedback?";

//upload to AWS and start looking for results
function sendFeedback(){

    apiDelay = 500; //milliseconds - how long to wait before each check


    var learnedValue = $("input:radio[name ='learnedRadios']:checked").val();
    var awesomeValue = $("input:radio[name ='awesomeRadios']:checked").val();
    var commentsText = $('#mainInput').val();
    commentsText = encodeURIComponent(commentsText);
    
    var apiCall = APIurl + 'learned=' + learnedValue + '&awesome=' + awesomeValue + '&comments=' + commentsText;
    showSpinnyThing();

    $.get(apiCall, function(data) {

        //Got some data
        console.log('server returned');                    
        hideSpinnyThing();    
        $('#form_wrapper').hide();
        window.scrollTo(0,0);
        $('#feedbackSubmitted').show();
        setTimeout(function(){                        
            resetForm();
        },3000);
    });            

}

The lambda function does sentiment analysis, entity and keyword extraction on the text comments before sending it all off to ElasticSearch:

function detectSentiment(callback, responseObj) {

    var params = {
        LanguageCode: 'en',
        Text: responseObj.feedback.comments
    };
    comprehend.detectSentiment(params, function(err, data) {
        if (err) {
            console.log(err, err.stack);
        }
        else {
            responseObj.Sentiment = data;
        }
    });

}

(full code on github)

Then I configured a kibana dashboard to display the data, set it to auto-refresh and you’ve got serverless real-time event analytics.

event_feedback_dashboard

Oh yeah, and the whole thing took less than 6 hours to build (with the help of the frankly brilliant AWS Amplify hosting service)

The client, and lambda code, are all available on github: https://github.com/macmike/lambda_feedbackAnalyser


Can’t open links in Firefox on Ubuntu when it’s open – Fixed!

This is a little obscure but there’s a number of people with this problem and too many “simple” answers that just don’t work. There are a number of situations that cause Firefox to respond with an error message when you click on a link from outside of the browser (like Thunderbird or any other application that try to launch a url). You get an error message saying:

Firefox is already running, but is not responding. To open a new window, you must first close the existing Firefox process, or restart your system

The normal reasons for this are things like locked profiles etc. which are well covered here. But some users get this error message even when Firefox is open and there aren’t locked profiles 😦 The solution is to edit the way that Ubuntu invokes Firefox.

Grab a terminal and go to ~/.local/share/applications this is Unity stores it’s information for launchers in the unity dock thingy. You should have a Firefox.desktop in here, you might have several in which case the one called “Firefox Web Browser.desktop” is probably the one that Ubuntu is using by default.

Edit the file and have a look at the “Exec=” line about 4 lines down. In my case the problem was caused because this line was referring to an old profile that I’d previously removed trying to fix the problem. If you’re not sure what it should look like set it to “Exec=firefox %u“, save, exit and click links once more 🙂


Wrote some REST code to access a site’s API and saw a job offer in the response

What an excellent way to advertise directly to your target audience 🙂

I was writing some code to get some info from a site from their REST api and saw an unusual header in the response from the API which said something along the lines of “if you’re reading this come and get a job” with a link. The clever thing about this is that it’s already only filtered in people who already have the right skills that the company is looking for and it was unusual enough for me to immediately follow the link. Excellent innovative recruitment!

And the site? wordpress.com although it’s changed now to “Oh, Awesome: Opossum”

I’m a big fan of wordpress.com for a number of reas0ns. First they host my blog, for free. Second the authoring experience is always awesome opossum.


A rose by any other name would smell as sweet

I’ve decided to rename my blog from “The MacDaddy” to the imaginatively titled “Mike MacDonagh’s Blog”.

When I first setup this blog I had no idea what direction it would go in and I’ve always been terrible at naming things. I decided on the MacDaddy as an ironic nod to Kriss Kross and because there’s a “Mac” in my name and I’m a Daddy. I don’t think anyone ever got the joke. Jump jump…

Anyway, I’ve known for a long time it was silly so I’ve finally  renamed it. I toyed with ideas such as “The Software Ninja” and “Software Kung Fu” in blatant references to mastery and the Kung Fu of Software Engineering but that’s a bit too software-y when many of my posts aren’t really about software. Putting “Business” in the title would make me want to give myself a good slap so I’ve ended up with an almost no-name name. Hopefully now I’ll get less confused visitors looking for a blog about Apple Macs.

If you’ve got a better idea for the name of this blog I’d love to hear it 🙂


What is cloud computing?

In any sentence including the word “cloud” if you substitute “network” the sentence still makes sense, and normally makes a lot more sense.*

I’ve recently been doing some stuff with cloud computing and thought I would offer the above insight. It’s slightly flippant however, in that there is a difference  (at least to me) between cloud services and network services and that is in their focus. “Cloud services” tend to be focuses towards consumers whereas “network services” are rendered by businesses to business, although as is normal in business they’re further obfuscated translated to b2b services or, if intended to portray an image of easy-consumption “b2b cloud services”.

You’re currently reading this blog on the cloud or syndicated by a cloud service (like google reader, netvibes (oh yes I see you), bloglines etc.) on a device of your choice.  I’m writing it on a cloud service and other cloud services (like google, bing, whatever) are providing search for curious folks to find this blog.

So what does all this mean? Is cloud pointless or the most important thing since 7 layered network protocols? Or all of the above?

Personally I’m leaning towards the third option. In many ways this is nothing new. I’ve been able to use network based search, hosting, virtual provisioning, remote servers, remote APIs for many years. In another way it’s totally new, I can provision a network based on a topology of my choice based on a collection of OSes for little or no money for whatever purpose suits me, storing data in a federated fashion and integrated with a number of web services after just a few clicks, both as a business or an individual.

There are many excellent cloud services out there, I personally use the Ubuntu One service to transfer files between my main desktop PC and my Android tablet, it’s actually more convenient than using a USB stick (and free) – this for me is the real distinction between “network” services and “cloudn” services (deliberately misspelt for those who do the copy/paste search/replace exercise). They’re so convenient and cheap (or free) you’d use them above anything else. Amazon is really trailblazing in terms of consumer and business cloud services.

I recently setup a website. I entered my credit card details and literally moments later had a linux server with the web server of my choice running. I was able to FTP and SSH to the box and do whatever I wanted as it appeared to be my box. Of course it’s not my box, it’s a virtual box hosted by a (probably virtual) hypervisor or similar on a bunch of racks distributed around a bunch of data centres in various places but that’s not relevant to me as a consumer, I just ask for a “box” and get a “box” in moments, in a convenient and cheap way.

That is what cloud computing is all about.


* Try this trick with this blog, copy and paste the blog to gedit/notepad/whatever and replace “cloud” for “network”


FireFox extension to view your WordPress.com blog stats

The MMD WordPress FireFox extension is finally publicly listed on the mozilla site. After battling problems with xul layouts on Mac’s (which was tricky because I don’t have a Mac) I’ve finally resolved the issues that were preventing my firefox stats toolbar from being fully reviewed on the main FireFox site.

If you’ve not seen it before, the MMD WordPress FireFox extension adds a tiny toolbar that shows you your current blog hits on your wordpress.com blog(s). You can then right click to get quick access to the full stats page, new post page and view your blob. Simple but quite useful. Download and install from mozilla

MMD WordPress Extension

 


Ngrams for nerds

Pictures that are worth 500 billion words!

Google Ngram Viewer shows graphs of how many times words or phrases have occurred in a set of 5 million books over the years. They’re a really interesting way of seeing trends in information and relative importance between words. It’s free and easy so check it out.

Here’s some I recently ran that I found interesting. I ran most of them from 1950 onwards and  the info only goes up to 2008.

Comparison of programming languages

Programming Languages

Ngram link – When looking at this you’ve got to mentally remove the baseline Java and Pascal references from the 1950 as they’re about coffee, islands and mathematicians. Interesting to see Java so dominant.

Programming paradigms

Programming Paradigms

Ngram link – I found this one really interesting. Compared to the others in my query “structured programming” had a lot more books written about it. I wonder how much this is a reflection of the rise of the internet… these days although there are lots of programming books the primary source for learning a language is online material?

Methodologies

Methodologies

Ngram link – I was a little surprised to see RUP so much more prevalent than agile but then I did have to add “software development” to the term to avoid including the bendy and stretchy. Also as with the previous one I suspect that there’s a difference here between a vendor driven process with supporting books and a more open source philosophy on agile as a generic umbrella for methodologies, and therefore more online sources. As Ivar Jacobson says: “No one reads process books

Shareware, Freeware and OSS

Shareware, Freeware and Open Source

Ngram link – This one speaks for itself 🙂 I wish I could have worked out how to add “expensive vendor products” to the query!

User Stories vs. Use Cases

User Story vs. Use Case

Ngram link – Ah yes, this argument again. Interestingly this dominance of use case over user story in written books correlates with query stats between user stories and use cases on by blog and the ivarjacobson.com site. Personally I think they’re both great and complimentary, I often use them together on software projects.

Windows vs. Linux

Windows vs. Linux

Ngram link – Yep, Linux beats Windows at every turn.

More Ngrams!

For more fun with Ngrams watch this very funny video explaining this stuff


UNTHINK – first impressions and screenshot

I saw a video for UNTHINK a while ago that appealed to me. I like the idea of total control of my data, of separation between friends, acquaintances and brands/interests. Also I’m a pathological early adopter. I like new things, and the shinier and more buttons the better. Sometimes I get carried away with links too.

I had an early invite to UNTHINK when it opened to public beta, clearly along with many other early adopters. First problem there, the servers couldn’t handle the pressure, there were too many early adopters and without the power of Google behind UT (not the best acronym huh?) the public beta was essentially a fine example of  a DDOS attack.

Anyway, a couple of days later and here’s my first impressions, now I can access the site.

UNTHINK

First of all I find it be gloriously complex. G+ has catered for the simplistic market, but I like complicated things, and this suits me. The UI is reminiscent of the early Facebook and MySpace.  G+ allows me to separate people via circles but I have to set that up, UT has it from the start. It’s all private by default and they tell me I own my content (the unthink emancipation deed). UT prompts me from the very start to control the privacy options of my content, unlike Facebook which decides for me frequently. Finally, it reminds me of uplink for some reason, and that’s double or possibly even triple awesome

But… the social networking market is saturated. Facebook is the Daddy Mac, it’s where everyone is. Some of my techie friends still aren’t on G+ so that’s quiet. UT is positively empty. I keep coming back to Facebook because it’s where people are, many of my friends have no appetite for multiple networks and as with G+ I wish I could cross post with FB.

Whether it catches on or not, it’s existence is a good thing as it will apply pressure on widely adopted networks to provide better features. It wasn’t long after the exposure of G+ circles that Facebook created “Close friends” and “Acquaintances”.


Google+ review

G+ is a social network from Google to rival Facebook, twitter, tumblr etc. I’ve been trying it out and kinda like it. It’s intuitive, easy to use but also a bit surprising. The best feature by far is the circles (screenshot below) which allow you to segment your social graph, or more simply put your contacts into groups. Then when you share something like a post or photo you can choose who to share it with. That means you can have things like family, work, acquaintances or just people you follow all in one place.

The interface is nice and shiny with lots of drag and drop and nice little features and animations, but when you first load it up it can be a little disconcerting how much it already knows about you. If you’ve already got a gmail account, an android account, picasa etc. then all of that stuff is already there in front of you. When I first logged in it had my picture, suggested various people members I should connect with (based on my email) and once I linked in my blog account it put in my blog intro into my profile. Clever, but a little annoying as well.

I’ve read that hangouts (a kind of chat platform) are the killer app for g+ but I’ve not checked it out. If I want to video call I tend to use skype and even then I don’t really want to video call, I’d rather IM, text or phone than video call.

Overall, I really like it. The ease of use, the simplicity of controlling where content goes, the nice simple little andriod app and the shiny interface have me drooling like a geek with a new toy. If only it was 5 years earlier and there were more people on it. I can’t imagine it replacing Facebook and twitter in the short or medium term as the stable door has been open far too long and there’s an emotional investment that people have in those sites.


Play a game with Google 2001

In honour of Google’s 10th birthday they’ve dusted off an old search index from 2001 and tied it up to the Internet Archive. The Web was a lot different in those days (no ipods, facebook etc.) and it’s fun to ego search and see the difference.

In Google 2001 if you search for “Mike MacDonagh” I think the first hit is my Dad at the Smithsonian, and my sister features on Page 3 (har har, not like that!) because of a school play she was in but I’m nowhere to be seen! At that time I was running Midnight Skulker Software of which there’s a whole page of hits, but I seem not to have put my name on it anywhere! An exercise in bad branding! I’ve still got all the old source for that stuff, and still use some of it myself 😀

More reading: http://googleblog.blogspot.com/2008/09/2001-search-odyssey.html


My Blog Stats

MMD WordPress Extension screenshot

I a blog stat fanatic. I’m obsessed. It’s so bad I wrote a FireFox extension to show me the current daily hit count of my blog. I started this blog in April 2008 and since then the readership has gone up month by month:

If you look at this on a weekly basis rather then on a monthly basis some more info is apparent. This graph below shows a slow start then some action starting when I started writing about IBM Jazz. Then the next peak was some RSDC blogging in June. All was then returning to the baseline readership level of about a 1000 hits/week until I posted some Google Chrome screenshots on the morning before Chrome was released. This led to over 3,000 people turning up in one day!

Finally the daily chart. Normally this goes up and down a bit (slow weekends, mid-week peaks etc.) Unfortunately the 3k Googlers completely flattened my graph 😦 At least that peak is about to drop off the side and the normal scale will resume 🙂


Twiffid: See What Your Twitter Friends Are Blogging About

Cool site that shows you the latest headlines of your Twitter friends’ feeds in a Twitter-style format. Of course it gives you that as an RSS feed as well so it’s a good thing to add to your aggregator:)

read more | digg story


Google Chrome Screenshots

Since Google Chrome is about to be launched, here’s some screenshots

Google Chrome Screenshot

Google Chrome Screenshot


Google Chrome: A new web browser that looks very interesting

Official Google Blog: A fresh take on the browser.

Chrome looks set to be a big contender in the browser market. It’s built from the ground up to be a different, more secure way of running web apps and browsing. If it lives up to it’s vision it’s likelt to be really impressive!

Check out the Google Comic explaining it. The thing that’s really interesting about this is the way that tabs and plugins are all in seperate processes. This makes me think of Chrome as similar to a Operating System kernel. In many ways Chrome looks like it’s an Operating System for distributed applications.

BTW it looks like the beta will be available today at some point

Google Chrome as an OS

MMD WordPress.com FireFox Extension v1.1

I’ve released a new version of the MMD WordPress.com FireFox extension, that fixes a bugette I had in the clickability of icons in the about box and has added a couple of new options to the status bar panel. If you have a WordPress.com blog you simply must have this extension.

I’ve also nominated the extension to become public on the mozilla firefox addons site. To help with this I really need people who download it and use it to review it. There have been over a hundred downloads so far and only 2 reviews! Add a review people, I shall send you an email full of e-karma if you do so 😀

Screenshot of v1.1

MMD WordPress Extension screenshot

MMD WordPress Extension screenshot

Please leave comments on the extension page here


Ubiquity – language integrated internet

Mozilla Labs » Blog Archive » Introducing Ubiquity.

Check it out, this is seriously cool stuff. I’m using the prototype and I’m impressed!


MMD WordPress FireFox Extension – know your hit count!

I often want to know how many hits my wordpress blog has had without loading the stats page, mainly so I can compete with a fellow blogger who sits opposite me in the office 😛

So I made the MMD WordPress FireFox extension, install it and you get a tiny status bar panel that shows your current daily hits. You can right click the panel and hit “View Stats Details” and FireFox will happily take you to your normal stats page on WordPress.com

To Install:

The MMD WordPress FireFox extension is brand new! That means it’s an “experimental” extension according to mozilla.org and so it needs people to try it out and post a review to become publicly listed on the main FireFox addon website.

You can install it here – please add a review!

The Future:

I’m thinking about adding some stuff to do with the normal browser right click menu to add:

  • Add link to BlogRoll (when hovering over a link)
  • Post link to blog (which would just open the normal WordPress posting page but prefill the form with stuff link the link, title etc. ready for you to edit)

And other suggestions please comment!


Cuil

A new internet search engine has been launched called Cuil (pronounced “cool”). Cuil has been built by a bunch of people that used to work for google and has a slight philosophical difference:

“Rather than rely on superficial popularity metrics, Cuil searches for and ranks pages based on their content and relevance. When we find a page with your keywords, we stay on that page and analyze the rest of its content, its concepts, their inter-relationships and the page’s coherency.”

Cuil also claims to have an index (over 121 billion pages) three times bigger than googles. Google have stopped publishing their figures but still claim they have a bigger index that Cuil.

I think Cuil has added a lot the page relevance stuff is more intelligent than google’s ranking, the categorised related links are useful and interesting not to mention the refreshing layout. But will people talk to “cuiling” for information in the future…. unlikely.


Great Web 2.0 presentation

Saw this presentation by Sacha Chua and thought it was really good 😀


Tweet Cloud

You can create tag clouds a bit like over on the right here from tweets using Tweet Cloud Beta

Here’s mine: linky

😀


It’s FireFox 3 Download Day – World Record

Firefox 3

It’s not released yet though, to find out when it’s released in your time click here

I’m a big fan of FireFox, and an extension developer as well, so I’ve been running the betas for a long time of FF3. Today is the day it’s released, and today is the day to download it even if you don’t want to install it today!

Set a Guinness World Record

Download Day

Enjoy a Better Web

Sounds like a good deal, right? All you have to do is get Firefox 3 during Download Day to help set the record for most software downloads in 24 hours – it’s that easy. We’re not asking you to swallow a sword or to balance 30 spoons on your face, although that would be kind of awesome.

With more than 15,000 improvements, Firefox 3 is faster, safer and smarter than before.


6 days to RSDC 2008

Just 6 days to go until the IBM Rational Software Developers Conference 2008. Those of us going fromIvar Jacobson International (IJI) (and there’s a few of us!) had a conference call on Friday to discuss our plan for the Conference. IJI is a silver sponsor this year and as well as our 7 presentations we’re going to be doing something special that will turn all of the conference attendees into R-Heroes 😀

It’s a big secret until the Tuesday of the conference though so I can’t say anything about it here. Make sure you come and visit our stand at booth #713. You can’t miss it, it’s the one at the bottom of the main escalators into the exhibition hall when you’re coming from the hotel lobby. You can come and talk to people like Ivar Jacobson, Ian Spence, Kurt Bittner, Robert Maksimchuk, David West, Eric Naiburg, and a bunch of others, including me of course. We also have two hand held GPS systems to give away if you register at our booth.

There’s going to be a lot of interesting announcements especially around Jazz at the conference, and for business partners there’s going to be a preview this Wednesday – but that will also be secret until the conference! At the conference you’ll be able to see demos of EssWork integrated with Eclipse, Visual Studio Team System or even Jazz Team Concert if you catch me. Our Team Concert integration is likely to be an exciting direction in the future as I think Team Concert is an ideal execution environment for EssUP.

Ivar Jacobson International

RSDC


What are twitter and hashtags?

With the explosion in social networking and the associated plethora of websites, not to mention the almost insurmountable technobabble, I thought I might spend a few moments explaining some of these things. Well, I haven’t got anything else to do while on the train!

Simply put twitter is a website for microblogging just like wordpress is a website for blogging. Microblogging is writing small sentences describing what you’re up to. You can see my twitter microblog here. Entries on it are called tweets. In fact you can also see my tweets on this blog over on the right.

As a twitter user you can choose to follow another user to see what your friends are microblogging/tweeting. All of this stuff can be subscribed to using rss of course, in fact that’s how I get my tweets on this blog. If you’ve got a facebook account you can even have twitter update your facebook status with your tweets. People can even subscribe to have your tweets as text messages!

Hashtags are a way of keeping track of a twitter conversation rather than just a particular user, for example the #rsdc hashtag.

If users follow a pretend user called “hashtags” then their tweets about various topics can be tracked to produce stats and graphs like this:

#rsdc graph

Personally I use a service called hellotxt from either the web or my mobile phone to update twitter and therefore this blog (through the feed on the right) and my facebook status all in one, not to mention any one subscribed to any of the above via rss.

Oh yeah and all this stuff is free 😀


Oooh… Mobile Blogging

I’m blogging from my new HTC phone courtesy of a little Windows Mobile app called moblog 8) Although writing html tags manually isn’t that great!

Unfortunately my gadgets can’t lower the temperature on this packed train. Or provide a seat :s

Meh, what’s a little comfort compared to gadgetry?

Posted from moBlog – mobile blogging tool for Windows Mobile