[Study] How well do you recognise faces if you first see them masked or unmasked?

Please take part in my wife’s academic study on facial recognition and mask wearing – takes about 10mins of clicking buttons and pressing keys on a website. Here’s my wife’s study: https://www.labvanced.com/player.html?id=34918

Serverless event feedback processing and analytics using #aws

I recently was involved in organising a techie conference. We wanted a feedback mechanism so I made a serverless event feedback system. Here’s how it works…

Have a bunch of ipads/tablets with a nice feedback form:

Collect up the feedback and sent it to a lambda function via API Gateway:

const APIurl = "https://api.mydomain.net/eventFeedback?";

//upload to AWS and start looking for results
function sendFeedback(){

    apiDelay = 500; //milliseconds - how long to wait before each check


    var learnedValue = $("input:radio[name ='learnedRadios']:checked").val();
    var awesomeValue = $("input:radio[name ='awesomeRadios']:checked").val();
    var commentsText = $('#mainInput').val();
    commentsText = encodeURIComponent(commentsText);
    
    var apiCall = APIurl + 'learned=' + learnedValue + '&awesome=' + awesomeValue + '&comments=' + commentsText;
    showSpinnyThing();

    $.get(apiCall, function(data) {

        //Got some data
        console.log('server returned');                    
        hideSpinnyThing();    
        $('#form_wrapper').hide();
        window.scrollTo(0,0);
        $('#feedbackSubmitted').show();
        setTimeout(function(){                        
            resetForm();
        },3000);
    });            

}

The lambda function does sentiment analysis, entity and keyword extraction on the text comments before sending it all off to ElasticSearch:

function detectSentiment(callback, responseObj) {

    var params = {
        LanguageCode: 'en',
        Text: responseObj.feedback.comments
    };
    comprehend.detectSentiment(params, function(err, data) {
        if (err) {
            console.log(err, err.stack);
        }
        else {
            responseObj.Sentiment = data;
        }
    });

}

(full code on github)

Then I configured a kibana dashboard to display the data, set it to auto-refresh and you’ve got serverless real-time event analytics.

Oh yeah, and the whole thing took less than 6 hours to build (with the help of the frankly brilliant AWS Amplify hosting service)

The client, and lambda code, are all available on github: https://github.com/macmike/lambda_feedbackAnalyser

Building a custom lambda runtime for anything? Even Pascal? Yes! #lambda #reinvent #aws

At AWS Reinvent 2018 today, Werner Vogels said it was now possible to use any language in AWS Lambda. I thought I’d put that to the test!

I thought it’d be interesting to add lambda support for Pascal, specifically the FreePascal variant found with Lazarus (the free, cross-platform version of Delphi). Mainly because it doesn’t really fit and it’s a compiled language, but I do have a remaining soft spot for the Lazarus project so I thought I’d give it a go.

Fair warning, this doesn’t make lots of sense. As a compiled language, you can’t initialise things in the runtime and then call them from multiple instances of functions, it also doesn’t deliver an amazing cold-start experience since it needs compiling on each run.

But… it does work! I based this on an AWS tutorial for creating a custom bash runtime.

You can write a Pascal lambda function like this:

begin
  writeln('{"status":"200", "message":"hello from fpc lambda"}');
end.

To do this, save the above as function.pas and then create an execution role:

To create an execution role

1. Open the roles page in the IAM console.
2. Choose Create role.
3. Create a role with the following properties.
   • Trusted entity – Lambda.
   • Permissions – AWSLambdaBasicExecutionRole.
   • Role name – lambda-role.

   The AWSLambdaBasicExecutionRole policy has the permissions that the function needs to write logs to CloudWatch Logs.

Then we can simply create a lambda function using the command line like so:

zip function.zip ./function.pas
aws lambda create-function --function-name fpc-hello --zip-file fileb://function.zip --handler function.handler --runtime provided --layers arn:aws:lambda:eu-west-1:743697633610:layer:fpc-runtime:20 --role arn:aws:iam:::role/lambda-role

This works because I’ve made the custom runtime layer public so anyone can use it.

But that doesn’t really do much, what about parsing and returning JSON? Easy…

uses
  fpjson, jsonparser, sysutils;

var
  lambdaEventData: TJSONData;
  lambdaEvent: TJSONObject;
  nameParameter, outputJSON: string;
begin
  //get incoming JSON and parse
  lambdaEventData := GetJSON(ParamStr(1));

  // cast as TJSONObject to make access easier
  lambdaEvent := TJSONObject(lambdaEventData);

  nameParameter := lambdaEvent.Get('name');
  outputJSON := format('{"status":"200", "message":"hello %s from fpc lambda"}',[nameParameter]);
  WriteLn(outputJSON);

end.

You can then invoke the function simply from the command line or AWS console:

aws lambda invoke --function-name fpc-lambda-event --payload '{"name":"Mike"}' response.txt
cat response.txt

How it works

The custom runtime is a simple linux executable that receives events from aws and publishes results. That means that things that are compiled on Amazon Linux 2 are generally going to work.

When lambda runs, it loads your custom layers into the /opt directory, so modifying the fpc.cfg and paths to take account of that is all that’s needed. This isn’t an especially lean implementation of fpc, it includes most libraries and a good chunk of the fcl.

Here’s my bootstrap code:

#!/bin/sh
set -euo pipefail

PATH="/opt/fpc3/bin:${PATH}"
export PATH

PPC_CONFIG_PATH="/opt/"
export PPC_CONFIG_PATH
#env
# Processing
while true
do
HEADERS="$(mktemp)"
# Get an event
EVENT_DATA=$(curl -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
REQUEST_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)

#RESPONSE=$(./function 'EVENT_DATA')

RESPONSE=$(instantfpc --set-cache=/tmp/ ./function.pas "$EVENT_DATA")

echo "fpc response: $RESPONSE"

# Send the response
curl -s -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/response" -d "$RESPONSE"
done

You can see the entirety of the code, including the runtime, FPC files and the 2 functions described above on github.

#aws #reinvent 2018 – my tips, tricks and photos

AWS re:invent is huge. So here’s my tips and tricks:

1. Good shoes. Not new ones, but comfy ones. There’s a lot of walking, the hotels are huge! This is about 1/6 of the registration hall at the Venetian. The scale is so big it’s hard to get it in a photo!
2. Room snacks. You’ll get a lot of free food and drink but sometimes you need a snack, and you definitely will need water. Rather than use the hugely expensive hotel shops I go to the shops literally next door on the strip to buy bottles of water.
3. Portable phone charger. Your phone will run out of batteries due to all the picture taking, tweeting and app shenanigans.
4. Schedule your sessions early. Yep, I know it’s too late now, but there’s still going to be a lot of open repeats, and the overflow rooms are actually pretty cool, with a silent disco vibe of coloured headsets.
5. Parties. There’s lots, follow @reinventParties for details!
6. How to guide: Watch the Youtube How To Reinvent Guide
7. Certification Lounge: If you’re certified, go to the certification lounge. There’s a slightly higher than 0 chance of the occasional seat – which is a remarkable thing at re:invent. And it’s full of doughnuts and retro games machines 🙂
8. Find a secret coffee shop. The main ones will have big queues and the generally available free coffee is ok, but I like to go to my secret coffee shop. Sorry, I won’t over expose it.
9. Expect change. There will be some huge announcements, and a sudden unveiling of loads of extra sessions so be prepared to change your schedule at the last moment. I tend to prioritise my sessions so I’ve already decided which ones I can drop and which ones I won’t. If you’re going to drop one, do it early so you can release the reserved seats. My account team are being tight-lipped, but based on the amount of stuff that’s been already announced, and their teasing – I’m expecting big things.
10. Follow your learning style. There are all sorts of different things like presentations, chalk talks, deep dives, hackathons, workshops, gamedays etc. Do what works for you not what everyone else is doing.
11. Painkillers. Take some, whether it’s flights, jet lag, crazy long days, too much partying or just the intensity of a big event, headaches are fairly inevitable. Or maybe that’s just me.
    

    
12. Take an Echo. For music in your room, naturally 🙂
13. Manage your swag! It’s possible to get too much swag. And then you need to buy another suitcase to bring it home. Doh!
    

    

Finally, keep track of what’s going on by using my reinvent twitter dashboard – I’ll be posting updates on twitter regularly.

Why getting #aws certified is different and worthwhile

I’ve done a bunch of professional certifications over the years in processes, languages and technologies. Usually because someone else wanted me to or was paying me to. Generally, I don’t think certifications are worth the paper they’re printed on as they rarely seem to be indicators of real world experience.

As an example, look at Certified Scrum Master. First you have to go on a mandatory 2-day in-person course taught by another CSM that cost around $1000 (pyramid scheme anyone?) and then correctly answer 24 out of 35 questions (68%) in an open-book google-able test.

Personally, I found AWS certification different. First, I did it for myself, because I wanted to. But most importantly, it’s quite cheap, and quite difficult.

AWS certification has no mandatory course and has closed-book, offline tests that cost $150. So it’s significantly cheaper, and having done them myself, I don’t think it’d be possible to pass with purely theoretical knowledge – the emphasis is on practical understanding, not theory. The questions are nicely technical, often with many correct answer where the point is to choose the “best” answer, not just one that’s factually correct. They don’t publish the pass mark, instead there’s a bell curve that you’ve got to be ahead of (but forums indicate it’s around 75%)

Personally I recommend using an online training provider (my favourite is acloud.guru) and then doing some free (!) labs, courses and tutorials on the main AWS site.

Interestingly I was getting job offers within an hour of getting certified almost a year ago!

I enjoyed doing it so much I’m going to do another one at this year’s AWS Re:invent conference 🙂

AWSome Twitter Sentiment Analysis for #aws #reinvent #reInvent2018

I built a dashboard that keeps track of the #reinvent2018, #awsreinvent, #reinvent twitter hashtags. It automatically translates non-English tweets into English, and does sentiment analysis (to detect emotional content) and keyword/entity extraction to see what people are talking about.

Over time it’s interesting to see how sentiment changes. Most tweets (97%) are neutral and so I don’t add them to the graph as the interesting lines would be squished.

1 – Artificial spike caused by me starting the analyis so all of the historic tweets got squeezed into the 10/10/2018

2 – Negative spike caused by the launch of reserved seating for sessions. 50,000 people tried to book seats at the same time and the user-experience on the website isn’t great so there were some errors and complaints. Those dies down pretty quickly.

3 – A positive spike. My analysis shows this is mostly down to some good vibes about the AWS Serverless Application Model from this tweet. The tag cloud when focussed in on these positive tweets is:

Follow me on twitter for more updates as time goes by 🙂

Here’s the live dashboard: https://soft-practice.com/awsreinvent.html (updates every 15 mins)

How to deploy static websites to S3 using AWS CodeCommit, CodeBuild and CodePipeline #aws

git repository -> S3 based website

1. Create a git repository for your html/css/js files
2. Add a buildspec.yml to tell AWS CodeBuild what to do
3. Create a new CodeBuild project to do your builds
4. Create a CodePipeline based on your CodeCommit repo that triggers whenever you push to the repo (skip the Deploy stage)

Push changes and sit back and watch the magic. (Note if you’re using CloudFront then you will need to wait to see the changes or invoke an invalidation – which you could also do as a post-build step!)

buildspec.yml (at root of repository)

version: 0.2

#env:
  #variables:
     # key: "value"
     # key: "value"
  #parameter-store:
     # key: "value"
     # key: "value"

phases:
  #install:
    #commands:
      # - command
      # - command
  #pre_build:
    #commands:
      # - command
      # - command
  build:
    commands:
      # - command
      # - command
  post_build:
    commands:
       - aws s3 sync . s3://[dest-bucket] --exclude .gitignore --exclude buildspec.yml --exclude .git/
      # - command
#artifacts:
  #files:
    # - location
    # - location
  #name: $(date +%Y-%m-%d)
  #discard-paths: yes
  #base-directory: location
#cache:
  #paths:
    # - paths<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>

Also, don’t forget to let your CodeBuild service role have the necessary permissions on S3 to avoid the build failing (it’ll need List, Get and Put permissions on the destination bucket).

If you wanted to kick-off a CloudFront invalidation as part of the build process you could add something like:

aws cloudfront create-invalidation --distribution-id [your_CF_ID] --paths /*.html

 

This is how the soft-practice.com serverless website works 🙂

How to use Lambda@Edge to redirect a CloudFront url #aws

You’ve got a published URL: http://mydomain/an_old_link and you’ve moved your site to AWS CloudFront and that published URL is causing you problems. The good news is that you can use Lambda@Edge to trap the incoming request and redirect to a file of your choice.

This means you can redirect http://mydomain/an_old_link to https://mydomain/a_new_link.html easily, and most importantly, transparently to the user.

This is useful because sometimes you’ve got old links published in paper media, or otherwise outside of your control.  CloudFront is great for http to https redirection, but it doesn’t do default files in subfolders 😦 Lambda@Edge can fix that problem.

Fist, create a lambda function (in us-east-1), it will be automatically replicated to edge locations through CloudFront.

Make sure it’s execution role can be assumed by “lambda.amazonaws.com” and “edgelambda.amazonaws.com”. Edit the “Trust Relationship” policy document of the function’s IAM execution role (not it’s permissions) to:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "lambda.amazonaws.com",
          "edgelambda.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

The lambda function can then be triggered from CloudFront on a behaviour of “an_old_link” using an “Origin Request” trigger. That means the lambda function is only called when the object isn’t in the CloudFront cache, only when CloudFront reaches back to the origin for the content.  Which means this function isn’t called every time, just when  there’s a cache miss.

Here’s the lambda code.

exports.handler = (event, context, callback) => {
    const request = event.Records[0].cf.request;

    if (request.uri === '/an_old_link') {
        request.uri = '/a_new_link.html';
    }
    callback(null, request);
};

 

Forward commands from PHP to LightwaveRF

I write this little forwarder recently so that other bits of software in our house/network could send commands to our LightwaveRF kit without needing to be registered on the hub, effectively giving local speed access to all software integrations.

I’ve got a little web server running on a Pi, that machine is registered with the lightwaverf hub (it runs my Logitech Harmony integrations). That server can then be used to forward commands like this:

<html>
<head>
<title>MMD PI Lightwave Control</title>
</head>

<body>
<h2>RPI Lightwave Command forwarder</h2>
<?php

$lw_command = $_GET['command'];
$server_ip ='192.168.0.10';
$server_port = '9760';
$message = '666,!'.$lw_command;

$socket = socket_create(AF_INET, SOCK_DGRAM, SOL_UDP);
socket_sendto($socket, $message, strlen($message), 0, $server_ip, $server_port);

?>

<strong>Sent command: </strong> <?= $message ?>

</body>

</html>

Now other bits of software on web pages or native clients can simply call urls such as

http://myserver/lwrf.php?command=R3D3F1

(room 3 device 3 turn on)

My Alexa skill is now live in the UK

Every month at my kung fu club, Chi Wai Kung Fu in Cheltenham UK, there’s a series of strikes of the month that are on rotation for extra attention. Also there’s a rotating black belt syllabus. It’s not the easiest thing in the world to remember all of these details, and whether or not you need to take a bo staff or set of escrima sticks to the club. So, I thought I’d write an Alexa skill to remind me 🙂

If you’re into Alexa in the UK, then join my UK Facebook group, it’s got over 4,500 members:

The basic skill is pretty simple, interrogating an array and returning the requested information. Getting it certified was a bit more fiddly because I wanted to use a single word invocation name “chiwai” which is against the rules normally, and because it’s a trademark infringement if it’s not official. Of course, in this case it was official as I developed it on behalf of the club. The Amazon skill dev support team were very helpful so now it’s live 🙂

Here’s a video of it in action:

 

To get it, if you’re in the UK, just say “Alexa, enable Chi Wai“.

You can also search for “ChiWai” in the Alexa app, or install direct from Amazon.co.uk.

You can then ask it questions like “what are the strikes of the month” or go a step further and say “what are the strikes of the month for October” or “what’s the black belt element in February”. Because of the way Alexa works, voice recognition on those second questions is greatly improved if you first start the skill by saying “Alexa, talk to Chi Wai” and then ask them. You can directly ask by saying “Alexa, ask Chi Wai for the …” but it’s not quite so accurate.

Also, try “Alexa, tell Chi Wai, Zoon Ching” 🙂

The code is on github here: https://github.com/macmike/chiwai-alexa-strikes