Validating MPAN Numbers

Photo: Tim Mossholder - Unsplash

This week I wanted to cover an energy domain specific problem and solution. The approach I took use’s one of my favourite .Net features, Extension Methods.

The problem is simple, validating an MPAN number.

Whoa, what is an MPAN number you ask? An MPAN number is a Meter Point Administration Number. It is the identifier used to label electricity meters in the UK. They are also sometimes called Supply Numbers or S-Numbers. There is an equivalent for Gas Meters. These are generally known as MPRN or Meter Point Reference Numbers.

The MPAN numbers comprise different elements relating to the supply. These include the Profile Type, Meter Time-Switch Code, Line Loss Factor and the Core Identifier. As it turns out you can validate legitimate MPAN numbers. This is achieved by generating a checksum from 12 of the Core Identifier digits. You then compare it to a check digit which is the final digit of the MPAN.

The checksum is calculated as follows:

• Extract the final 13 digits from the MPAN which is the Core Identifier.
• Remove the final digit for use later. This is the check digit which is used to validate the MPAN.
• Using the remaining 12 digits, start by multiplying the first digit by 3.
• Multiply the second digit by the next prime number, 5.
• Repeat this process for each remaining ten digits. But, miss 11 out on the list of prime numbers used to create the digit sums.
• Calculate a checksum by summing up all these products.
• Generate a check digit by using the checksum modulo 11 modulo 10.

The calculated check digit is compared to the last digit of the MPAN. If they match then you have a valid MPAN number.

There are solutions floating around the internet for this in many languages. It was a task that one of our more junior developers was attacking to ensure we had a single MPAN validator. It reminded me of a typical interview question or Friday fun team task and so opened it to the team. The only criteria were that it could take no longer than half an hour to code up. Below is my example MPANValidationExtensions class. It contains two simple methods, IsValidMPAN(meterIdentifier) and IsNotValidMPAN(meterIdentifier).

using System;
using System.Collections.Generic;
using System.Linq;

namespace MPANValidatior
{
    public static class MPANValidationExtensions
    {
        public static bool IsValidMPAN(this string meterIdentifier)
        {
            const int MPAN_LENGTH = 13;
            
            if (meterIdentifier == null || meterIdentifier.Length != MPAN_LENGTH)
            return false;
            
            var primeNumbers = new List<int>() { 3, 5, 7, 13, 17, 19, 23, 29, 31, 37, 41, 43 };
            List<int> digitCheckSumResults = new List<int>();
            
            int primeNumberIdx = 0;
            meterIdentifier
            .Substring(0, MPAN_LENGTH - 1)
            .ToCharArray()
            .Where(x => int.TryParse(x.ToString(), out int convertedInt))
            .Select(x => Convert.ToInt16(x.ToString()))
            .ToList()
            .ForEach(x => digitCheckSumResults.Add(x * primeNumbers[primeNumberIdx++]));
            
            var checkDigit = Convert.ToUInt16(meterIdentifier.Substring(MPAN_LENGTH - 1, 1));
            
            return checkDigit == (digitCheckSumResults.Sum() % 11 % 10);
        }
        
        public static bool IsNotValidMPAN(this string meterIdentifier)
        {
            return !IsValidMPAN(meterIdentifier);
        }
    }
}

(Always provide a negated version of a boolean Extension method. It really does improve readability.)

This is a fast, simple solution. It processes around 3000 MPANs in around 68 milliseconds. It is also provided as a string Extension. There is no need to instigate objects to provide this utility functionality. Include the namespace and just use where needed.

If you have questions around validating MPAN numbers or alternative solutions then contact me via twitter or email.

Start Building Sites With Hugo & Netlify

Photo: Unsplash

I am a big fan of Netlify. I use it to power this blog and to host some small project sites. It is free for single users, provides a simple static site deployment process and SSL for your sites. What is not to like?

I recently had a new site to set up and headed straight over to Netlify. In the past, I have tended to opt straight for Jekyll. This time I thought I would use Hugo instead. Hugo is a static site generator built using the Go programming language. What follows is a simple how to get Bitbucket, Hugo and Netlify up and running. (Note, Bitbucket can be swapped out for Github or Gitlab etc)

The first thing is to install Go via your package manager, Brew or on Windows from here. Then you need to install Hugo by following the instructions from here.

Now we want to use Hugo to create our static site. Switch to the directory you wish to create the site in and use:

$ hugo new site NAME_OF_STATIC_SITE

Obviously, replace NAME_OF_STATIC_SITE with the real name of your site.

Now, let’s switch into our new site and initialise a new git repository.

(To keep things simple I am going to make NAME_OF_STATIC_SITE MySite)

$ cd MySite
$ git init

I then copied into the MySite directory my README.md and a .gitignore file specific for Hugo that I found here. Now we have added the supporting repository files we can hook up Bitbucket:

$ git add --all
$ git commit -m "Initial Commit For MySite" 
$ git remote add origin https://url_to_mysite_repo
$ git push -u origin master 

Again, change the origin URL to match the one provided by your Repository.

The next thing is to apply a Hugo theme and to configure MySite.

To start, find the theme you want to use as a starting point for MySite on Hugo Themes. Next, you can follow the download link next to the theme to install it. You can add the theme as a git submodule which will allow you to get the latest changes and fixes from upstream. This is up to you. Most people I have spoken to use a theme as a starting point. If you want to then execute the following:

$ git submodule add https://url_to_theme_repo themes/THEME_NAME 

Either way you install the theme you will end up with a new directory in your MySite/themes directory. Most themes will outline how to use the theme’s example site files. This allows you to copy some styling and possible content and JavaScript files to the matching directories in your MySite directory. Make sure to ensure that your root config.toml file contains a line specifying the theme you are using:

theme = "downloaded theme name"

At this point you can navigate to the root of MySite and execute the following to test your site:

$ hugo server -D

This will result in some output such as

Building sites …
                   | FR | EN
+------------------+----+----+
  Pages            | 36 | 36
  Paginator pages  |  0 |  0
  Non-page files   |  0 |  0
  Static files     | 69 | 69
  Processed images |  0 |  0
  Aliases          | 14 | 14
  Sitemaps         |  2 |  1
  Cleaned          |  0 |  0

Total in 165 ms
Watching for changes in D:\Repositories\MySite\{content,data,i18n,layouts,static,themes}
Watching for config changes in D:\Repositories\MySite\config.toml
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For full rebuilds on change: hugo server --disableFastRender
Web Server is available at http://localhost:1313/project-test/ (bind address 127.0.0.1)

One of the great things with Hugo is that changes you make to your source code are hot loaded into the browser. You can test this by modifying one of the values in your layouts and see it appear immediately on the screen.

Once you have finished modifying the theme and configuring your site then you need to publish it. You should ensure all your changes are committed to git and pushed to your origin repository before continuing.

Let’s get Netlify running now. Login to Netlify and from the Sites page click “New site from Git”. It will ask you to select your git provider and authenticate your access. Next, it will ask you to select the repository with your site in. Select the repository and then enter the build command and publish directory details. For a simple site like we are building then hugo for the build command is fine. If it needs certain options passing we can come back and adjust this later anyway. The Publish directory should be public.

Under the Advanced build settings, you need to create an environment variable called HUGO_VERSION. This should be set to the same version of Hugo you are running locally. You can get that information by running:

$ hugo version

Enter the version into the environment variable value field and click “Deploy Site”.

You can then go to the Sites page in Netlify and see your new site has been created. It will have already started to deploy the site. While on the Sites page I generally rename the site name away from the random name they assign. I then click the Trigger Redeployment button to redeploy under the new name.

I then head to the Deploys page and ensure the site has been deployed correctly. If it has failed then you can click on the deployment log for a full overview of what has gone wrong. At least to get enough information to correct the issue or Google it!

You should now have a fully working site ready for you to start to build out and configure. Remember every time you push your changes to origin the site will deploy. Netlify has ways of previewing changes and handling different branching strategies. I will look at them more in an upcoming post.

If you have comments or views on developing sites with Hugo then contact me via twitter or email.

Rejected By An Algorithm

Photo: Andrew Neel - unsplash

This week I experienced a first. In my 41 years on this planet, I have never been rejected for anything. Actually, I should clarify. I have never been rejected for anything that requires a screening process.

This week I was rejected by an algorithm.

I went to the Carphone Warehouse to get a new contract phone. After researching the deals available right now they had the best data deal. It was with O2 and a reasonable price.

So I went to the local branch and started the purchase. I won’t focus on the horrendous customer service from the sales assistant. It seems to be common (at least in the UK). The process involved the standard data collection. Personal, bank and payment details. Credit check, all good. Then there seemed to be a problem with putting in my address details. They wanted all my addresses since birth which at 41 seemed a little excessive. I obliged but it would not let the assistant complete the process.

So the sales assistant deleted my details and started all over again.

Personal, bank and payment details, collected again. Credit check, all good a second time. Then on saving my address data results, a message flashed up. It said “Internal Security Check Failed. This Is A Final Decision And Irreversible”. Wait, what? What does that mean? A reasonable question you would think given the message presented by the system. At this point, the unhelpful sales assistant turned nasty. He waved his hand up and told me he would no longer talk to me as I was obviously a fraud. All this happened in front of other customers. I left feeling not only a little embarrassed but confused and angry.

As I drove home it became clear that I had been denied by an algorithm. A computer system had crunched my data and decided that I was a security and/or credit risk. I called their customer services the next day to find out more. They consider their system a fraud prevention system. Because of that, they would not disclose what the system deemed a risk. I know I had passed the credit check as it showed as a pass on the screen when I was looking at it with the sales assistant. Their customer service team told me it could be the way my bank held my address data. “Really, why?”, “I can’t tell you that”. Hmm, an odd thing to say. Reading up online I am not the first to suffer this fate. In fact far from it. A lot of people found that the format that the credit check company hold your address data is different to that your bank does. But I knew that I had passed that. The general advice online is to wait 90 days and try again. In the meantime check that your address details line up between your bank and the credit company.

Next, I filled in a Subject Access Request to get hold of the data they hold on me. Remember, GDPR is there to protect you and your data. But the request will be fulfilled within 30 days. The offer on the contract would likely not be available then. Anyway, I will update here if they ever come back to me.

I happened to go past another branch the following day and went in to see if anyone there could shed some light. I hit gold. An extremely helpful sales assistant had a look and could find no issue or record of why I could be rejected. He started the purchase process again. Personal, bank and payment details, collected. Credit check, all good. Enter the address details and success. Their system decided that a day later I was no longer a fraud but instead a welcome customer. I left confused but with the phone and contract, I wanted.

This raises so many questions with me. What are the data sources that Carphone Warehouse uses to validate your data? Who else may use them? Where has that data been collected from? Why, if there is any truth in it does the format of your address data need to be managed by the customer. Other than credit, what other checks are they performing? As a company do they have absolute faith that the system is capable of correctly validating potential customers? What is the margin of error? How much revenue has been lost down to the failures?

In discussing it with the second sales assistant he was certain that the information misentered by the first assistant caused the problem. So a system that is designed to validate your customers was thrown by 2 applications in a certain time period. Even though the applications were made by the same sales rep in the same store. While I can see some logic in that you would think an additional check could be added to prevent this issue.

I don’t have the answers. This is the kind of issue that this kind of systems poses and which AI will only exaggerate. However, businesses that screen data in this way need to weigh up the benefit over lost revenue. With GDPR people have the right to review the data held about them including the results of this type of processing. Do you want them to find that you offended and rejected them down to the inability to match address data?

If you have suffered rejection by a system I’d love to hear about it via twitter or email.

Shellcheck

Photo: Unsplash

I am going to take a noteworthy detour from i3wm this week. As part of updating my dotfiles repository, I wanted to highlight a very useful tool when developing scripts - Shellcheck.

I came across this utility while reviewing some scripts in Jess Frazelle’s repositories. Shellcheck is a static code analysis tool for bash and sh scripts. Developed by Vidar Holen, the tool can be used in a variety of ways including from the shell and via a web tool.

Side note - Jess’s repositories are an excellent learning source and her twitter brightens most days!

The aim of Shellcheck is to:

• Find and clarify typical beginner syntax. It will also make it easier to understand cryptic error messages that the shell can print.
• Make it easier to resolve statements that cause the shell to behave in a less than intuitive way.
• Point out more advanced issues that will cause the user’s script fail in particular corner cases.

The easiest way to try the application out is to head to shellcheck.net and paste a script snippet into the editor.

As you can see from the image above, as you enter your code the issues Shellcheck finds are output below. The issues that it lists are all defined by an Id and provide a small amount of data. It also tries to align the issues to the statement or variables that it refers to. This makes it easier to see which piece of code is the cause of the issue.

The next way to work with Shellcheck is via the command line by installing it locally. I installed Shellcheck (on any Arch-based system) with

$ sudo pacman -S shellcheck

Once installed you can get the same output as shown in the web interface by calling shellcheck with the script you want to check. Such as:

$ shellcheck myscript.sh

This will result in the output of Shellcheck being written to your terminal.

The last way in which you can use Shellcheck is to integrate it with your editor but I have yet to do this. I will look at this when I overhaul my Vim configuration. Full details are given on the Shellcheck site.

The best feature of Shellcheck is not it’s useful linting of your scripts but the excellent Wiki on its Github page. Thanks to the Id’s that Shellcheck assigns to each error you can easily look up the details on the wiki. Each error has clear details on what it has found and why it is a problem. It also provides an example of the solution it proposes. I have (re)learnt many core idea’s when scripting while using Shellcheck.

Needless to say, I now have a script in my dotfiles that runs Shellcheck against each script file.

This kind of utility is so powerful. They improve consistancy, remove basic issues and enforce a standard style. I highly recommend adding Shellcheck to your utility collection.

If you have tips or advice on Shellcheck then contact me via twitter or email.

Navigating & Using i3 Window Manager

Photo: Andy Crouch

Last week I shared the fact that I have migrated over to i3wm (i3) from a full desktop environment. i3 is a tiling window manager which is much lighter on resources than an environment such as Gnome. It also uses extensive keyboard shortcuts for navigation.

Depending on how you install i3 you may need to set up some basic configuration. If you use a distro like Manjaro i3 then when you log in you will have a working environment. If you install i3 via your existing distro then when you log in you will see a setup wizard. It will offer you the ability to use default options or start with an empty config. I recommend you use defaults. I will follow up the post next week with details on i3 and my configuration.

Now you have a working environment, how do you do anything? Where is the menu? Where are the icons?

The first thing to know is that all the keyboard shortcuts involve the $mod key. You will use the $mod key to open applications, move windows and change workspaces. Most distros tend to use the Windows key as the $mod key. The wizard mentioned above suggests either the Windows key or the left Alt key. The key can be configured in the i3 configuration file using:

set $mod Mod4

(The i3 configuration file is usually found at ~/.i3/config)

As an aside if you want to find a list of alternative mod key mappings to use then run the following in your terminal:

xmodmap

Now we have the $mod key configured how do we use it? The equivalent of a “Hello World” example for i3 is to execute the $mod+Enter command. This will open a terminal.

If you repeat the $mod+Enter command you will get a second terminal open to the right of the first terminal.

This is the “tiling” idea that is core to i3. You can divide up your workspace by tiling horizontally or vertically. By default in a workspace when you open a new application window it will open horizontally tiled. If you want to tile your next application under a current one then you need to use $mod+v to change the behaviour. You can use $mod+h to switch back to horizontal split mode.

You can change the focus of the open applications by using $mod+ one of the arrow keys. You can also move between windows using keys from the home row (like Vim). $mod+j moves to the left and $mod+; moves to the right. $mod+k moves up and $mod+l moves down. You can mix in the shift key to the arrows of home row keys to actually move the application window within the tiling layout of the current workspace. So if you have a browser and the terminal open split horizontally and hit $mod+shift+j then you will have the terminal and the browser instead.

When you launch a new i3 session you will start on workspace 1. There are 8 workspaces you can access using $mod+1 through to 8. Most configurations will only show the workspace indicator in i3 bar if you have an application open within it. If you open an application on workspace one and decide you want to move it to workspace 2 then you can use $mod+shift+ workspace number.

Obviously, tiling is very useful but there are times you need to full screen a window. To do that you can use $mod+f and to exit full screen you also use $mod+f. In order to close an application window, you use $mod+shift+q.

OK, so we can open terminals and move between them and move them to over workspaces. How do we open our applications? You use d-menu. d-menu is a dynamic menu system for X and is almost always used by default within i3. To open the menu you use $mod+d and then type the application name you want to open. To dismiss the menu you can use the Esc key.

The last main key combinations to know about are how to handle switching layout mode. i3 supports three layouts. Tiling is what we have been using up until now. It also supports stacked and tabbed layouts. You still move between the stacked application windows or tabs with the $mod+ arrow keys (or home row). You move to stacked layout using $mod+s. You move to tabbed layout using $mod+w or you revert to tiled layout using $mod+e. $mod+e will also change the tiling mode from horizontal and vertical.

So that sums up opening applications, navigation, moving between applications and workspaces and closing the applications down. To exit i3 you need to hit $mod+shift+e which will bring up the i3 nagbar and ask if you really want to exit. Ironically you have to click “yes” with you mouse! That will return you to your login manager.

I will leave you with some references for further reading on i3 and using it which I found useful:

• The Official Documentation.
• Fedora Magazines Getting Started With The i3 Window Manager.
• i3 Cheat Sheet.

I hope this makes clear the basic navigation and how to work within i3. If you have any questions then message me via twitter or email.