Andy Crouch - Code, Technology & Obfuscation ...

Always Have A Development Process

Man Standing Looking At A Board

Photo: Unsplash

“The battlefield is a scene of constant chaos. The winner will be the one who controls that chaos, both his own and the enemies.” - Napoleon Bonaparte

When you start a company you have one basic aim, to build a product that your customers love which makes you money. There are many theories and methodologies to guide you toward this goal. The fact remains that the pressure is on to deliver features and profit. In the rush to deliver it is sometimes easy for new, inexperienced teams to work in a chaotic manner. If there is one piece of advice I would offer to any small, early-stage team it would be to create a development process.

I have used a slightly adapted development process in each of my three early stage company’s. It has evolved over the last 10 years due to new applications and forms of communications. I will present it here in rough form and you are welcome to discuss it with me on twitter.

The Tools For Supporting The Process

The process use’s two existing items. The Kanban workflow process and Atlassian’s Jira hosted web application.

Kanban is a method for visualizing the flow of work. It aims to balance demand with available capacity and spot bottlenecks. This product can be software, a sandwich or cars which is where Kanban originated at Toyota. The process aims to provide a clear way to visualise and organise where a task is within a defined workflow. The workflow comprises many states such as In Development or Completed. Kanban limits the amount of work in a given state so that bottlenecks can be identified and resolved. There are some good Kanban articles here and here.

Atlassian builds applications that provide various supporting functions for software development teams. Jira is a tool it provides to log and manage the flow of work through a development team. It provides a way to create a project and set up an agile based workflow to support the work of your team. Kanban is one of the workflows that it supports. At the time of writing Jira costs $10 per month for up to 10 users. An advantage of Jira is that it interacts well with other Atlassian software such as Bitbucket.

(I should disclose that I am no way affiliated with Atlassian. There are many project management and issue tracking applications available. I have found Jira to be an excellent tool but that is my opinion and you can build your process using any tool.)

The Process

The aim of the process I use is to provide enough guidance and support without getting in the way. So, I usually start with the following states:

  • Backlog - All items start here. They should get added and prioritised in the backlog until they are ready to be worked on by the developers.
  • Todo - Items in the Todo state are ready to be worked on. That means the requirements and acceptance criteria are confirmed. If given to a developer they can complete the task with little issue.
  • In Progress - Items In Progress are currently being worked on by the development team. Items may be in a testing phase.
  • Done - Items in the Done column are completed, tested and ready for release to Production.

It is really easy for everyone to understand these states. Especially if you are working with less technical business stakeholders. I find that these states work well before launching a product.

The key to getting this working is to visualise the process. Create a simple graphic and display it in the office and communicate it to everyone. Get a screen displaying the Jira workflow board in the office. Integrate Jira with Slack so that as work is moved through the process the whole team can see it.

A sprint is a period of time within which an agreed amount of work is completed. Agree on a sprint length and stick to it. Pre-launch, weekly sprints usually work really well. I have never agreed to sprints that last more than 2 weeks. Whatever you agree, stick to it and communicate it so that people in the business know when to expect things. Stick it on their calendars and mark the end of a sprint with a demo to the business team.

Include business team members in prioritisation and planning of the backlog. I’ve never had a Product Owner when working pre-launch. If you have one then you may find this approach too simplistic.

If you do have a Product/Project manager and once you have gone live things become more complicated. Especially when you have to start fixing and improving features. I tend to extend the number of states in the process as follows:

  • Backlog - All items start here. They should get added and prioritised in the backlog until they are ready to be worked on by the developers.
  • Accepted - This is the development teams Backlog. Items in this state have been reviewed and broken down appropriately and estimated by the development team. Any item in the Accepted state can be pulled into a sprint when capacity becomes available with no outside involvement of the development team.
  • Todo - Items in the Todo state are ready to be worked on. That means the requirements and acceptance criteria are confirmed. If given to a developer they can complete the task with little issue.
  • In Progress - Items In Progress are currently being worked on by the development team. Items may be in a testing phase.
  • In Review - This state allows for code reviews. Everything your team is completing should be reviewed before it is released for testing.
  • In Test - Items in the Test state have been deployed to an environment that the QA team and/or Business can review the feature and test it.
  • Done - Items in the Done column are completed, tested and ready for release to Production.

I can not overstate the importance on the success of your early stage team by having a process. I have gone through many investment discussion at each of the small company’s I have worked. Each time, the fact that we have a documented process in place is flagged as a major positive. Not only will it improve your ability to deliver under pressure but it will only help your growth.

If you’d like to discuss my thoughts on this topic then message me via twitter or email.

Turning 40

American Football Field

Photo: Martin Reisch - Unsplash

Tomorrow I turn 40. To some people, this may be a massive life event to celebrate (or commiserate). For me, it’s nothing more than another day. When people ask how I feel about my life I tend to think of the following quote from The Big Bang Theory.

“Just like everyone else’s. Subject to entropy, decay, and eventual death. Thank you for asking.” - Amy Farrah Fowler

While I am a little indifferent to turning 40 it has made me reflect on a few things. I have never taken advice well and I do not suggest you listen to me. But, I will write the things I have learnt in life so far and you can take from it what you will.

Surround yourself with interesting, brilliant people.

This is not groundbreaking advice. To grow as a person you need to surround yourself with people that will push you out of your comfort zone. Work with people that make you think and make you question everything you do. Find mentors and learn from them, ask them even the stupidest of questions. The moment you feel comfortable in a profession or a job then you need to rethink your career or find a new job.

Don’t stop learning.

This doesn’t refer only to being a programmer (or whatever you do as a day job). This means read. Read a lot. Read books. Read the internet. Take advantage of the vast quantity of online learning resources. Appreciate the opportunities and knowledge it provides but mindful it can make anyone an expert.

(Pause for ironic effect given the title of this post).

Do things that make you happy outside of work.

There is a theory that to be good at whatever you do, you need to be doing it 90% of your waking hours. This is not true. The key to staying focused on achieving is to sometimes walk away and do something else. There is a culture in some company’s to expect 70 hour work weeks. This is not productive. While you may be working for a prestigious company, you will not be doing your best work. Developers do not need to be building side projects out of work hours to prove they are excellent at what they do. Have hobby’s, go on holiday and enjoy time with your family.

Prioritise.

As you get older you become time poor. Learn to prioritise everything. Once you have a family and commitments you will not be able to go to every conference and hackathon. Learning every new language and JavaScript framework becomes pointless. Learn to focus on the important things and create some time to focus on them.

People come and go.

You will make friends throughout life. Social Media has bred the belief that you have to remain friends for life once you have conversed once. This is rubbish and you may outgrow friendships or they may outgrow you. That’s normal and while some take some getting over don’t look back. Don’t treat others badly or in a way in which you wouldn’t want to be treated yourself. Don’t harbour bad feelings and be open and direct with everyone important in your life.

Be your own person.

Finally, people have opinions and views. The only person whose opinion and views you have to focus on are your own. You will disagree with a lot of people but that’s normal. Be respectful and do not force your views on other people. You can try and educate them but they are their own person as well. We are all the result of the millions of conversations, interactions and experiences that have come before this moment. If you don’t feel good about something then don’t do it. If you wouldn’t say something to someones face then do not write it.

Thats it! Now where is the cake?

Single Responsibility Methods

Man Sitting On Clock Rug

Photo: Kevin Ku - Unsplash

The Single Responsibility Principle is a core element of the SOLID Principles. Wikipedia states that the principle stipulates that

“a class should have only a single responsibility (i.e. changes to only one part of the software’s specification should be able to affect the specification of the class).”

Most developers agree with this approach and structure their code accordingly. Then they create methods in those classes with words like “And” and “Or” in the name. This is a clear code smell that your method is doing too many things. Why structure our classes following SRP and then tool them with confusing methods?

As an example, I have often seen code like

public class DataFileParser
{
    public List<Entity> DownloadAndParse(string filePath)
    {
        // long code block that validates that the 
        // filepath exists & downloads it
        // and parses it to a list of objects of type Entity
    }
}

Now I am not interested in showing the actual implementation. What I am showing is that it is common to see a method that does more than one thing. In order for the method to complete it needs to do three discrete steps.

public class DataFileParser
{
    public bool FileExists(string filePath)
    {
        // validate that file exists
    }
    
    public File Download(string filePath)
    {
        // download file
    }
    
    public List<Entity> ParseToList(File file)
    {
        // parses it to a list of objects of type Entity
    }
}

As an aside, by refactoring in this method there is a clear violation of SRP at the class level. FileExists and Download methods should clearly be refactored into a separate class.

public class DataFileParser
{
    public List<Entity> ParseToList(File file)
    {
        // parses it to a list of objects of type Entity
    }
}

public class DataFile
{
    readonly File _file;

    public DataFile(string filePath)
    {
        _file = new File(filePath);
    }

    public bool FileExists()
    {
        // validate that code exists
    }
    
    public File Download()
    {
        // download file
    }
}

This simple refactoring results in two classes that now have a single responsibility. Each method has a single responsibility as well. This improves readability and more importantly makes the methods easier to test. It also makes clear the intent of the methods which reduces the ability for it to cause side effects. Side effects from badly designed methods have lost me countless hours over the years. A final reason to structure your methods in this way relates to reuse. By limiting the methods to a single responsibility you will end up increase reuse significantly as you can chain calls up in various ways that you just can’t when they are grouped together as one method.

To end I will include a quote from Robert C Martin on the subject of SRP

“Gather together the things that change for the same reasons. Separate those things that change for different reasons.”

If you’d like to discuss my thoughts on method design then message me via twitter or email.

Debugging The Right Problem

Man With Head In Hands

Photo: Unsplash

“Continuous effort - not strength or intelligence - is the key to unlocking our potential.” - Winston Churchill

Sometimes it helps to have experience.

A developer was having an issue with a SQL query running over a large data set. The query was straightforward and on their machine, it ran on under a second. On a similarly sized data set in the Staging environment, it took 50. That’s quite the difference.

The developer had look at all the obvious things. Reviewed the execution plan and tested the code that executed the query. They talked me through all that they had done and to be honest it was the exact approach I may have taken.

Yet, the one thing that stood out was the fact that the query was running on a new Staging database on Azure. So I suggested that we look at its configuration. Th developer was as surprised as me to see that the database had been restored on a Basic Azure pricing tier. Thus, the amount of DTU’s available for processing the query was limited to 5. We scaled the database up to a standard tier with an increased number of DTU’s that matched Production. Our query now matched the performance expected on running it.

Sometimes when looking at issues it is easy to assume it is the code that is at fault. It’s the most logical. Although I would argue that if you have tests based on specifications your code should rarely be the issue. Many a developer has been tricked by a SQL query that isn’t quite thought through but which works on a sample data set only to fail in production. But, it can be an environment issue or setting or something you would not immediately think of. This is where talking the issue through with someone will save you a lot of time. We stress to our less experienced developers that they should seek help with any issue if they have struggled for more than 15 minutes. Talking the issue through with someone (or a rubber duck) results in a faster solution.

If you’d like to discuss experiences of difficult problems you’ve had to debug message me via twitter or email.

Modern Data Exchange

Chimney Smoking In Country Setting

Photo: Unsplash

It’s 2017 and many of the partners that I deal with are still exposing their data via FTP. This makes me sad!

That (s)FTP is still a mainstream method of exchanging data demonstrates laziness on both sides of the connection.

From a supplier side, you are saying you do not trust your customers. Or you are saying your system is not designed with collaboration in mind. If trust is an issue and you are generating files for a customer to collect what is different to providing that data via an API endpoint? If you do not want customers accessing your application then you can segment an API. You are running an FTP server so why not use it to make the customer feel enabled?

(s)FTP is not as secure as Https. In the majority of cases that I have used it, (s)FTP is used to get files that are then processed. That means when processing you could suffer undesirable behaviour from the malicious code. You have no way of verifying that what was uploaded to the (s)FTP server has not been tampered with. (You are taking precautions right, I mean you are not storing data straight from an FTP file to your database?!!?) Using a secure, encrypted API endpoint reduces those risks.

From a customer side, why are we putting up with it? I am building modern, fast software. Why should I settle on working with company’s that insist on using such old (40 years) technology? Why should I have to carry the burden of checking a server for updated files? Why would I choose this over a company that provides mechanisms to notify me of changes (webhooks etc)? Why do I want to have to process files? I was exchanging XML back in 2005 and JSON simplifies things even more.

Here is a real example of a company that I deal with. They provide their data as CSV files. When we had the meeting to talk technical details they asked what version of the file I wanted. “Erm, isn’t there only one CSV file format?” You’d think! Turns out that the data is provided to utility company’s. They each insist that their data is formatted in a particular order, same data just different order. This has lead my partner to maintain over 90 versions of the same file. Each version contains the same data in just a different order. Now imagine how easy it would be to do this via an API?

If you have similar frustrations message me via twitter or email to vent.