Rejected By An Algorithm

29 January 2019

Photo: Andrew Neel - unsplash

This week I experienced a first. In my 41 years on this planet, I have never been rejected for anything. Actually, I should clarify. I have never been rejected for anything that requires a screening process.

This week I was rejected by an algorithm.

I went to the Carphone Warehouse to get a new contract phone. After researching the deals available right now they had the best data deal. It was with O2 and a reasonable price.

So I went to the local branch and started the purchase. I won’t focus on the horrendous customer service from the sales assistant. It seems to be common (at least in the UK). The process involved the standard data collection. Personal, bank and payment details. Credit check, all good. Then there seemed to be a problem with putting in my address details. They wanted all my addresses since birth which at 41 seemed a little excessive. I obliged but it would not let the assistant complete the process.

So the sales assistant deleted my details and started all over again.

Personal, bank and payment details, collected again. Credit check, all good a second time. Then on saving my address data results, a message flashed up. It said “Internal Security Check Failed. This Is A Final Decision And Irreversible”. Wait, what? What does that mean? A reasonable question you would think given the message presented by the system. At this point, the unhelpful sales assistant turned nasty. He waved his hand up and told me he would no longer talk to me as I was obviously a fraud. All this happened in front of other customers. I left feeling not only a little embarrassed but confused and angry.

As I drove home it became clear that I had been denied by an algorithm. A computer system had crunched my data and decided that I was a security and/or credit risk. I called their customer services the next day to find out more. They consider their system a fraud prevention system. Because of that, they would not disclose what the system deemed a risk. I know I had passed the credit check as it showed as a pass on the screen when I was looking at it with the sales assistant. Their customer service team told me it could be the way my bank held my address data. “Really, why?”, “I can’t tell you that”. Hmm, an odd thing to say. Reading up online I am not the first to suffer this fate. In fact far from it. A lot of people found that the format that the credit check company hold your address data is different to that your bank does. But I knew that I had passed that. The general advice online is to wait 90 days and try again. In the meantime check that your address details line up between your bank and the credit company.

Next, I filled in a Subject Access Request to get hold of the data they hold on me. Remember, GDPR is there to protect you and your data. But the request will be fulfilled within 30 days. The offer on the contract would likely not be available then. Anyway, I will update here if they ever come back to me.

I happened to go past another branch the following day and went in to see if anyone there could shed some light. I hit gold. An extremely helpful sales assistant had a look and could find no issue or record of why I could be rejected. He started the purchase process again. Personal, bank and payment details, collected. Credit check, all good. Enter the address details and success. Their system decided that a day later I was no longer a fraud but instead a welcome customer. I left confused but with the phone and contract, I wanted.

This raises so many questions with me. What are the data sources that Carphone Warehouse uses to validate your data? Who else may use them? Where has that data been collected from? Why, if there is any truth in it does the format of your address data need to be managed by the customer. Other than credit, what other checks are they performing? As a company do they have absolute faith that the system is capable of correctly validating potential customers? What is the margin of error? How much revenue has been lost down to the failures?

In discussing it with the second sales assistant he was certain that the information misentered by the first assistant caused the problem. So a system that is designed to validate your customers was thrown by 2 applications in a certain time period. Even though the applications were made by the same sales rep in the same store. While I can see some logic in that you would think an additional check could be added to prevent this issue.

I don’t have the answers. This is the kind of issue that this kind of systems poses and which AI will only exaggerate. However, businesses that screen data in this way need to weigh up the benefit over lost revenue. With GDPR people have the right to review the data held about them including the results of this type of processing. Do you want them to find that you offended and rejected them down to the inability to match address data?

If you have suffered rejection by a system I’d love to hear about it via twitter or email.