Somewhere in the overlap between software development, process improvement and psychology

Posts tagged “software

Building a custom lambda runtime for anything? Even Pascal? Yes! #lambda #reinvent #aws

At AWS Reinvent 2018, Werner Vogels said it was now possible to use any language in AWS Lambda. I thought I’d put that to the test!

I thought it’d be interesting to add lambda support for Pascal, specifically the FreePascal variant found with Lazarus (the free, cross-platform version of Delphi). Mainly because it doesn’t really fit, it’s a compiled language, but I do have a remaining soft spot for the Lazarus project so I thought I’d give it a go.

Fair warning, this doesn’t make ton’s of sense. As a compiled language, you can’t initialise things in the runtime and then call them from multiple instances of functions, it also doesn’t deliver an amazing cold-start experience since it needs compiling on each run.

But… it does work! I based this on an AWS tutorial for creating a custom bash runtime.

You can write a Pascal lambda function like this:

begin
  writeln('{"status":"200", "message":"hello from fpc lambda"}');
end.

To do this, save the above as function.pas and then create an execution role:

To create an execution role

  1. Open the roles page in the IAM console.
  2. Choose Create role.
  3. Create a role with the following properties.
    • Trusted entityLambda.
    • PermissionsAWSLambdaBasicExecutionRole.
    • Role namelambda-role.

    The AWSLambdaBasicExecutionRole policy has the permissions that the function needs to write logs to CloudWatch Logs.

Then we can simply create a lambda function using the command line like so:

zip function.zip ./function.pas
aws lambda create-function --function-name fpc-hello --zip-file fileb://function.zip --handler function.handler --runtime provided --layers arn:aws:lambda:eu-west-1:743697633610:layer:fpc-runtime:20 --role arn:aws:iam:::role/lambda-role

This works because I’ve made the custom runtime layer public so anyone can use it.

But that doesn’t really do much, what about parsing and returning JSON? Easy…

uses
  fpjson, jsonparser, sysutils;

var
  lambdaEventData: TJSONData;
  lambdaEvent: TJSONObject;
  nameParameter, outputJSON: string;
begin
  //get incoming JSON and parse
  lambdaEventData := GetJSON(ParamStr(1));

  // cast as TJSONObject to make access easier
  lambdaEvent := TJSONObject(lambdaEventData);

  nameParameter := lambdaEvent.Get('name');
  outputJSON := format('{"status":"200", "message":"hello %s from fpc lambda"}',[nameParameter]);
  WriteLn(outputJSON);

end.

You can then invoke the function simply from the command line or AWS console:

aws lambda invoke --function-name fpc-lambda-event --payload '{"name":"Mike"}' response.txt
cat response.txt

Selection_023

How it works

The custom runtime is a simple linux executable that receives events from aws and publishes results. That means that things that are compiled on Amazon Linux 2 are generally going to work.

When lambda runs, it loads your custom layers into the /opt directory, so modifying the fpc.cfg and paths to take account of that is all that’s needed. This isn’t an especially lean implementation of fpc, it includes most libraries and a good chunk of the fcl.

Here’s my bootstrap code:


#!/bin/sh
set -euo pipefail

PATH="/opt/fpc3/bin:${PATH}"
export PATH

PPC_CONFIG_PATH="/opt/"
export PPC_CONFIG_PATH
#env
# Processing
while true
do
HEADERS="$(mktemp)"
# Get an event
EVENT_DATA=$(curl -sS -LD "$HEADERS" -X GET "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next")
REQUEST_ID=$(grep -Fi Lambda-Runtime-Aws-Request-Id "$HEADERS" | tr -d '[:space:]' | cut -d: -f2)

#RESPONSE=$(./function 'EVENT_DATA')

RESPONSE=$(instantfpc --set-cache=/tmp/ ./function.pas "$EVENT_DATA")

echo "fpc response: $RESPONSE"

# Send the response
curl -s -X POST "http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/$REQUEST_ID/response" -d "$RESPONSE"
done

You can see the entirety of the code, including the runtime, FPC files and the 2 functions described above on github.

Advertisements

How to deploy static websites to S3 using AWS CodeCommit, CodeBuild and CodePipeline #aws

git repository -> S3 based website

  1. Create a git repository for your html/css/js files
  2. Add a buildspec.yml to tell AWS CodeBuild what to do
  3. Create a new CodeBuild project to do your builds
  4. Create a CodePipeline based on your CodeCommit repo that triggers whenever you push to the repo (skip the Deploy stage)

Push changes and sit back and watch the magic. (Note if you’re using CloudFront then you will need to wait to see the changes or invoke an invalidation – which you could also do as a post-build step!)

git_code_build

buildspec.yml (at root of repository)

version: 0.2

#env:
  #variables:
     # key: "value"
     # key: "value"
  #parameter-store:
     # key: "value"
     # key: "value"

phases:
  #install:
    #commands:
      # - command
      # - command
  #pre_build:
    #commands:
      # - command
      # - command
  build:
    commands:
      # - command
      # - command
  post_build:
    commands:
       - aws s3 sync . s3://[dest-bucket] --exclude .gitignore --exclude buildspec.yml --exclude .git/
      # - command
#artifacts:
  #files:
    # - location
    # - location
  #name: $(date +%Y-%m-%d)
  #discard-paths: yes
  #base-directory: location
#cache:
  #paths:
    # - paths<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>

Also, don’t forget to let your CodeBuild service role have the necessary permissions on S3 to avoid the build failing (it’ll need List, Get and Put permissions on the destination bucket).

If you wanted to kick-off a CloudFront invalidation as part of the build process you could add something like:

aws cloudfront create-invalidation --distribution-id [your_CF_ID] --paths /*.html

 

This is how the soft-practice.com serverless website works 🙂


How to use Lambda@Edge to redirect a CloudFront url #aws

You’ve got a published URL: http://mydomain/an_old_link and you’ve moved your site to AWS CloudFront and that published URL is causing you problems. The good news is that you can use Lambda@Edge to trap the incoming request and redirect to a file of your choice.

This means you can redirect http://mydomain/an_old_link to https://mydomain/a_new_link.html easily, and most importantly, transparently to the user.

This is useful because sometimes you’ve got old links published in paper media, or otherwise outside of your control.  CloudFront is great for http to https redirection, but it doesn’t do default files in subfolders 😦 Lambda@Edge can fix that problem.

Fist, create a lambda function (in us-east-1), it will be automatically replicated to edge locations through CloudFront.

Make sure it’s execution role can be assumed by “lambda.amazonaws.com” and “edgelambda.amazonaws.com”. Edit the “Trust Relationship” policy document of the function’s IAM execution role (not it’s permissions) to:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "lambda.amazonaws.com",
          "edgelambda.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

The lambda function can then be triggered from CloudFront on a behaviour of “an_old_link” using an “Origin Request” trigger. That means the lambda function is only called when the object isn’t in the CloudFront cache, only when CloudFront reaches back to the origin for the content.  Which means this function isn’t called every time, just when  there’s a cache miss.

Here’s the lambda code.

exports.handler = (event, context, callback) => {
    const request = event.Records[0].cf.request;

    if (request.uri === '/an_old_link') {
        request.uri = '/a_new_link.html';
    }
    callback(null, request);
};

 


Forward commands from PHP to LightwaveRF

I write this little forwarder recently so that other bits of software in our house/network could send commands to our LightwaveRF kit without needing to be registered on the hub, effectively giving local speed access to all software integrations.

I’ve got a little web server running on a Pi, that machine is registered with the lightwaverf hub (it runs my Logitech Harmony integrations). That server can then be used to forward commands like this:


<html>
<head>
<title>MMD PI Lightwave Control</title>
</head>

<body>
<h2>RPI Lightwave Command forwarder</h2>
<?php

$lw_command = $_GET['command'];
$server_ip ='192.168.0.10';
$server_port = '9760';
$message = '666,!'.$lw_command;

$socket = socket_create(AF_INET, SOCK_DGRAM, SOL_UDP);
socket_sendto($socket, $message, strlen($message), 0, $server_ip, $server_port);

?>

<strong>Sent command: </strong> <?= $message ?>

</body>

</html>

Now other bits of software on web pages or native clients can simply call urls such as

http://myserver/lwrf.php?command=R3D3F1

(room 3 device 3 turn on)


My Alexa skill is now live in the UK

Every month at my kung fu club, Chi Wai Kung Fu in Cheltenham UK, there’s a series of strikes of the month that are on rotation for extra attention. Also there’s a rotating black belt syllabus. It’s not the easiest thing in the world to remember all of these details, and whether or not you need to take a bo staff or set of escrima sticks to the club. So, I thought I’d write an Alexa skill to remind me 🙂

chiwai_alexa_out.png

If you’re into Alexa in the UK, then join my UK Facebook group, it’s got over 4,500 members:

echo_group_welcome

The basic skill is pretty simple, interrogating an array and returning the requested information. Getting it certified was a bit more fiddly because I wanted to use a single word invocation name “chiwai” which is against the rules normally, and because it’s a trademark infringement if it’s not official. Of course, in this case it was official as I developed it on behalf of the club. The Amazon skill dev support team were very helpful so now it’s live 🙂

Here’s a video of it in action:

 

To get it, if you’re in the UK, just say “Alexa, enable Chi Wai“.

You can also search for “ChiWai” in the Alexa app, or install direct from Amazon.co.uk.

You can then ask it questions like “what are the strikes of the month” or go a step further and say “what are the strikes of the month for October” or “what’s the black belt element in February”. Because of the way Alexa works, voice recognition on those second questions is greatly improved if you first start the skill by saying “Alexa, talk to Chi Wai” and then ask them. You can directly ask by saying “Alexa, ask Chi Wai for the …” but it’s not quite so accurate.

Also, try “Alexa, tell Chi Wai, Zoon Ching” 🙂

The code is on github here: https://github.com/macmike/chiwai-alexa-strikes


Launch: Holistic Software Engineering

How do we join up business strategy to agile development? Is program management relevant? Where do project managers fit in? What about architecture?

Holistic Software Engineering (HSE) answers all of these questions – for free.

Agile and continuous flow are great for small teams or a small number of inter-related small teams working on exploratory or maintenance work. But what if we’re spending 100s of millions on an IT strategy of inter-related products that need to work together to deliver business value. What is business value anyway?

H-Model To answer these questions (and more) my friend Steve Handy and I have distilled our collective 30+ years of software experience in a single, cohesive model of software development. We’ve developed the H model that moves on from the v-model and it’s siblings by combining:

…all elegantly combined and de-conflicted by release planning.

We’ve not invented very much, we’ve simply put a lot of good ideas from others together into a cohesive framework. We’ve drawn it all as a big picture and developed a website that shows how to get value from all of this stuff. Everything is clickable, everything has content.

The best bit: it’s free! There’s no paywall, there’s no private “full” version, you can just use it or not as you like.

We don’t believe in process zealotry, or putting academic concerns above clarity and usefulness. HSE is indicative, not prescriptive. You’re already doing it and if you use the big picture to draw red blobs on the bits that aren’t working well, or missing, in your organisation then you can use the model to make tangible improvements – immediately.

Using HSE doesn’t replace any of your existing processes, it provides the glue that joins them all up together in one simple, elegant and cohesive model.

Holistic Software Engineering
is for large/complex software organizations
who need to understand modern software engineering in a business context
our solution is to present a big picture of software business backed up with practical detail avoiding academic or heavyweight process documentation
that covers people issues, business strategy, portfolio, programme and project management as well as architecture, technical delivery and integration
unlike simple small team based processes such as RUP, Scrum etc.
The big picture of software engineering

Holistic Software Engineering

And if it’s too big, we’ve got a small picture, which is essentially “normal” agile development.

Please share HSE with your friends and colleagues.


No more Project Managers, bring in the Movie Producers

I was reading some course material recently that was trying to teach people something to do with software development and it was using the same old tired “ATM machine” example. I’ve worked with hundreds of projects, many in the finance sector and none of them are anything like an ATM machine.  One of the reasons that example is soooo tired is that it’s describing commoditised software development, it’s something I’d expect to buy off a shelf or go to a specialised vendor for. It’s not something I’d put my team of uber |33t  haxorz on.

Developing software products is a balance between a creative and scientific pursuit, so it’s a little hard to describe, especially when that software can range from a tiny smartphone app, to a website, to router firmware, to an enterprise hr system (urgh!), to a big data processing system of systems etc. You get the gist.

The things these types of system have in common with each other is how different they are. And yet the traditional approach to managing these diverse kinds of work has been classical project management with a one size fits all process (I’m looking at Prince2, RUP, Iterative, Agile etc.) or worse hiring a large company full of body-shop project managers. For me this is one of the root causes behind large scale IT disasters.

I once had a discussion with the leader of a PMO in a large organisation of thousands of people about the nature of Project Management for software and he assured me that “someone who knows how to manage a bit of work can manage software, it’s just a different type of work” and indeed I should “stop thinking of software as special”. I’ve seen this attitude resonate in many organisations and is to me best summed up by the same Head of PMOs statement “from a project management point of view building software is the same as building a bridge”.

Now then. I know a little about software development, but not that much about bridge building (if you exclude my illustrious Lego engineer career when I was 7). If I managed the building of a bridge next door to one build by someone with a track record of bridge building who’s bridge would you drive your family on? Not mine if you’ve got any sense.

There are so many problems in the software development industry caused by people not understanding it and applying bad metaphors. When people who do understand it start to get the point across (e.g. the early agile movement) they often get misunderstood by the old guard who dress up their normal ways of working in new language.

Many of our “best” metaphors come from manufacturing lines where the same thing is made over and over again. That’s nothing like software.

To me a software project is more similar to making a movie than anything else:

  • It’s a unique one off product (or a remake of a previous version) with new parts working together in new ways
  • Once made we’ll often duplicate and digitally distribute
  • It can cost a lot of money because it can be very complex and needs lots of specialist skills
  • You need a high level plan for how you’re going to make it
  • There’s no guarantee on return
  • What’s written down originally bares little resemblance to the finished product
  • We know when it’s good or bad but it’s hard to objectively quantify
  • There’s lots of different types that need different collections of specialist skills
  • Both involve people and time and so are subject to change, they have to be adaptive
  • You can tell when the people making it had fun
  • It’s not feasible that any team member can fit in any role
  • There’s almost always going to be some quality problems
  • You wouldn’t get a movie maker to build your bridge
  • You wouldn’t get a bridge builder to make your movie
  • You don’t make a movie faster by telling the actors to act faster

So, I don’t want a Project Managers that know how to work a gannt chart. I want movie producers that know how to work with a team holistically to get the best out of them both technically and artistically.