Andy Crouch - Code, Technology & Obfuscation ...

2018 Wrap Up

Festive Decoration

Photo: Joanna Kosinska - Unsplash

So 2018 is almost at an end. For those that read this site with any regularity, you will have noticed I have not been posting as much of late. This is purely down to a major project I am leading with Open Energy Market which has been dominating my time. This will be the last post of the year and I will attempt to get back to writing more in the new year.

2018 has been a massive year on many levels for Open Energy Market. From closing out an investment round to building the company to more than double its size. This was both financially and in people. A new office to set up and fill up and a big push on redefining the core technology we are providing within the market.

Personally, getting to travel more and seeing my son continue to grow and ask interesting questions have been fun. Spending time with my wife is what keeps me sane and I hope to do even more of that next year.

2019 promises to be an interesting year.

Merry Christmas all!

If you would like to connect or exchange Merry Christmas messages then connect with me via twitter or email.

Open Energy Report

Electricity Pylons

Photo: Matthew Henry - Unsplash

Recently the Federation of Small Businesses published a report on modernizing the Energy market. The report entitled “Open Energy - Using data to create a smarter, cheaper and fairer energy market” can be found here. It is a worthy read for any company involved in the Energy market.

The FSB was the organisation in the UK to push for the Open Banking standard back in 2011. It took some years but that standard is now supported by the 9 largest banks in the UK.

The reports main suggestions are

  • The government should give energy customers greater control over their smart meter data.
  • That customers should have easier access to tariffs.
  • That the Energy industry should standardize market price information into machine-readable formats to allow for autonomous comparisons of tariffs.
  • The industry should provide access to smart meter data via an open API.
  • That customer should have the option to delegate contract switching powers to Third party intermediaries.

Combined, the suggestions would form the basis of the Open Energy standard. The aim from the FSB’s point of view is to increase the number of customers switching tariffs to save money. They also see it as an opportunity for “ innovative uses of data, including for demand-management purposes that could increase the proportion of the total energy mix from renewables.”

The current state of data availability and exchange in the Energy Industry is terrible. There is no common data exchange format. Powered by FTP, suppliers silo their data in badly designed and isolated portals. The recommendations in this report are what Open Energy Market has been pushing for. In order for Customers to take control back of their energy data, suppliers need to be made to embrace these recommendations. Open and well governed API’s, customers owning their data and having access to it on demand. I suspect it will take significant time and involvement from Ofgem to make it a reality. But, look at Open Banking and the innovation that is occurring there thanks to the standard.

Customers are back in control and that is spurring competition.

If you would like to discuss the report and how it’s recommendations could be implemented then message me via twitter or email.

Linus Takes A Break From Leading Kernel

Pneguins In The Snow

Photo: Ian Parker - Unsplash

This week, Linus Torvalds has announced that he is stepping back from maintaining the project he founded over 25 years ago.

I haven’t been following the LKML as closely as I used to but I was hardly surprised by the news. A combination of factors have led to this point and have been brewing for many years. Linus is famous for his blunt opinionated emails on LKML. On occasion, he has headed into insulting and personal attacks. This has lead to some high profile maintainers and developers quitting the project. He will be the first to admit that he is not a people person. But, he holds a key position in one of the most visible Open Source projects in the world. More than anyone else in the community his every word and action will be scrutinized. He sets the tone for how other maintainers and contributors behave.

There are a couple of trains of thought about the whole situation.

The first is that the situation is repairable. But, it will take significant time for the changes to propagate. Linus himself is taking time to look at how he acts and interacts with people. He is seeking “professional help” to correct his behaviour. This is commendable and the community should provide the time and support he needs. It’s a massive admission he has made in a very public arena. Tiger Woods, Hugh Grant and Britney Spears have all been allowed to put pasts behind them. We can not tell someone that they are wrong and then not support them when they try to correct the issues. The major issue here is that that behaviour has built a community over 25 years. We have seen since the news broke a range of reactions and suggestions and infighting. Due to his leadership style which the community has followed it is divided and broken. When he returns, Linus will have to spend as much time readdressing the community and pushing new values as he will reviewing code. This will be a lengthy process. Anyone that feels a Code of Conduct will solve this is only very partially right. In fact, the author of the code of conduct could do with being less of an antagonist as a good starting point.

Do I think there will be knee-jerk reactions to this? No, I do not think many developers are going to suddenly leave the project or withdraw code. Perhaps the few developers that are so bothered by the Code of Conduct will leave and that might not be a bad thing. If you need a Code of Conduct in the first place then you probably should follow Linus’s lead and seek some help.

The second train of thought I have is around leadership and building communities. Specifically, the impact that this will have on people starting or wanting to maintain projects. I hope that situations like this do not put anyone off from starting new projects. It takes significant time and effort in growing communities. You need to be sure you want to take on that responsibility. That you want your every action to be reviewed and commented against. That you can build in inclusiveness and openness while keeping the negativity out. It’s a hard role which a few key people such as Jono Bacon have mastered.

It will be an interest few weeks to see how this situation unfolds and to watch the inevitable fallout. I hope that the key community members step up and it will be interesting to watch Linus’s development when he returns.

There will be a range of thoughts on this news so why not share them with me via twitter or email.

Mexico Trip

Sensatori Resort Riviera Swimming Pool

Photo: Andy Crouch

So I have been away from this blog for a month which was due in part to a project I am on and in part due to a trip to Mexico & Dallas.

This post is entirely about the trip and not the project!

This was our first time to Mexico and we stayed in the Sensatori Resort Riviera Cancun. The hotel was right on the beach and is one of the best hotels I have ever stayed in. The staff and their approach to customer service are brilliant. Our room was large, clean and well made every day. They offered a wide range of food throughout the restaurants. They also had Caribbean, Italian and Mexican restaurants on site. We tried the Italian and the Mexican, the latter of which was brilliant. Not only good food but amazing entertainment and waiters.

Sensatori Resort Riviera Beach Walk

Sensatori Resort Riviera Hotel Room

Sensatori Resort Riviera Wildlife

There was one thing which we were aware of before we flew out and that was the seaweed issue along the Caribbean coastline. There is a problem that has been affecting the area on and off for the last few years relating to large amounts of seaweed washing up on the beaches. The hotels attempt to clear it but due to wildlife in the area, it is now a manual process. Which meant that while you could sit and enjoy the beach bar and restaurant you could not swim or even get to the sea. For us, this did not bother us so much as we were happy around the pool (which was large and had plenty of room). But, you do question why the travel operators do not mention it or even acknowledge it when pressed.

Sensatori Resort Riviera Beach Seaweed

Sensatori Resort Riviera Beach Seaweed

On the whole, I would very much recommend Mexico from the experience we had. We are already planning further trips to see more of the country.

On the way home, we stopped in Dallas for 3 days to see a very good friend. Apart from our luggage turning up a day after us (to my wife’s horror!) we had a great time. Dallas is nothing like you imagine and you can instantly see the appeal for the many large companies moving there. We only got to see a fraction of the city but it is a mix of old and very new. The people were extremely friendly. The ladies seemed particularly taken with my son (“An English accent, blue eyes and good looks”). I will plan to return here for a much longer period soon as I felt there was a lot more to explore.

Texas A & M Building

Dallas Book Repository

Deep Ellum

If you have places or experiences in Mexico or Dallas you would recommend then please share them with me via twitter or email.

Entity Framework Database Call Issues

Numbers On White Background

Photo: Mika Baumeister - Unsplash

Entity Framework (EF) is an abstraction over database queries and interactions. It is billed as more than another Object Relationship Mapper. It has enabled developers to quickly create data-driven applications. But, as a wise person once said: “with great power comes the need to know how your queries are created and executed by Entity Framework”. Or at least something like that.

There are a host of issues that I actually have with the use of EF. These are not all entirely fair given that if you take your time to learn the library and its intricacies it is very powerful. One of the most useful conversations I had was interviewing someone whose best friend was in the EF team. But, a large number of less experienced developers I work with don’t do put the effort in to learn the library. They read the examples and start to build code. In providing the abstraction Microsoft has removed the need for developers to know about the database.

When you mix in the Linq helpers you start to see code like makes many database calls.

var items = _dbContext.Items.Select(x => x.Type == 1).ToList();
// Some Other code

var subSetOfItems = _dbContext.Items.Select(x => x.Type == 1 && x.SubType == 4).ToList();

I call this the “Getting it as you need it problem”. The developer is getting objects as they need them. There is one more DB call there than needed.

var items = _dbContext.Items.Select(x => x.Type == 1).ToList();
// Some Other code 

var subSetOfItems = items.Where(x => x.SubType == 4).ToList();

One less DB call and the same result (in this case).

The point here is not really EF doing anything wrong but that it makes throw away database calls to easy to not spot. If you are working with a set of objects throughout your Class or Method then get the largest subset of data once and use Linq To Objects to get your subsets.

The other problem that is far worse than I see is what I call “The Database Death Loop”

var items = _dbContext.Items.Select(x => x.Type == 1).ToList();

foreach(var item in items)
    var relatedItem = _dbContext.RelatedItems.Select(x => x.Id = item.RelatedItemId).FirstorDefault();
    // Do something with objects


Do you see the problem? I see this a surprising amount and when you point it out to developers they generally let out a nervous laugh. If you have not called an EF execution query execution method (such as ToList()) to execute the query, you instead create a deferred query. That is that a query will be executed for each matching item in the Select statement. This means that you return a list of 500 items and go off to loop through them and get the Related Items and you create 500 DB calls. I pity your database and network.

As a slight pro tip, if you are only reading data with no intention of writing it back then research the NoTracking option with EF. This might be a blog post itself soon.

These are only a couple of the common issues that I see with EF. I may write about the others soon. The point I am trying to make is that you need to take time and learn EF at a deeper level that then example articles. If your application requires performance of any kind then think about how to create your queries and execute them.

Please share your thoughts around the issues you see with inexperienced developers using EF with me via twitter or email.