You Can Never Test Too Much

A lot of knowledgeable experts will be weighing in with their points of view about what went wrong with the Iowa caucus vote counting. There were many problems but it is clear as can be the core problem was the lack of testing. I learned decades ago about the importance of testing anything involving software. The other thing I learned is software always does exactly what you tell it to do, not what you want it to do. The only way to get software to do what you want it to do is testing, and you can never test too much.

My team at IBM built a website for the Atlanta Olympic Games of 1996. In 1995, there were not many people who knew a lot about how to build really large websites or, for that matter, any kind of website. The Olympic site was the largest in the world back then, and we learned a lot in building it. We were humble with our expectations. We didn’t know how many people would come to the site, when they would come, or what they might do when they got there. We learned many lessons, but I can summarize it in a simple mantra: Think big, act bold, start simple, iterate fast (with testing). Another way to say it is to take a lot of baby steps.

The Olympic Games website was successful. For the first time, people around the world had real-time access to the individual and country results from the Olympic Games. We had a lot to share about how the site was developed but, unfortunately the story was overshadowed by a glitch in an unrelated application (the word app had not yet been invented) developed in Atlanta. The application was designed to distribute news and results to the press. A software developer made a change to the application but failed to test it. The result was a disaster. If an application was to have a failure for a customer, the last place you would want it to happen is with the press. 

In software development parlance, testing happens in phases. First is the alpha test to see if the concept works. Then comes beta testing where you bring in a small number of users and get their feedback. Sometimes software can be in the beta phase of testing for months or even years. When it is finally ready to go live for all users, the software is released. As an extreme example, Google introduced gmail in 2004 and kept it in beta for five years, signaling the company was still making feature changes and testing them. As of 2019, there are 1.5 billion users.

Development of a test plan is at least as important as the plan for the product or service itself. In the innovation group at IBM, my team would develop a new application and conduct the alpha test among just our own team. When everybody was happy, we would roll it out to a small subset, by invitation. When the team was satisfied with the testing progress, more beta users would be added. When it was clear the application was working properly and the users were happy, the application would be released for all users. 

Almost 20 years later I heard about the impending launch on October 1, 2013 of I knew it would be a failure from the get go. A project as massive as which intends to serve all types of insured citizens in all 50 states can have many possible points of failure. It reminded me of the mantra I learned in 1996. It appeared the mantra for was “think big, act big, start big, fail big”. The site could have been introduced in one state for one type of insured. After successful testing, another state could be added. Then another type of insured. The proper testing would have spanned at least a year, in my opinion.

Clay Shirky, an American writer, consultant and teacher on the social and economic effects of Internet technologies, analyzed the failure in “ and the Gulf Between Planning and Reality“. He concluded the developer’s idea that “failure is not an option” but, with no testing, was a fantasy. The developers feared if they openly tested the system before releasing it to the public, politicians would attack the purpose and efficacy of the site. The lack of testing caused catastrophic results.

And now, in 2020, after a vast accumulation over decades of experience with the Internet, the leaders of the Democratic caucus released an app which had virtually no testing. People in 1,700 precincts were told to download the app for the first time on the night of the caucus. The app was not available in the app store because it had not had proper testing. The users in the precincts had to follow a special download process which was daunting for many. In interviews before the caucus, leaders said they had a backup plan if things did not go well. The backup plan was a call center which also served as the technical support center for users having trouble with the app. Some users were on hold for hours and finally gave up. 

How difficult would it have been to have an alpha/beta test plan? It would have been relatively easy. It could have started with 17 of the 1,700 precincts. If successful, another 17, then 170, and after all were happy, released to all. A solid test plan would have taken months. It is clear from the various news reports the Democratic leadership had little or no technical competency and the vendor who developed the app had little experience with apps. 

The leadership of the Democratic caucus did a terrible disservice to American democracy. With people already skeptical of Internet voting, the experts are now piling on saying “See, we told you so. Don’t even consider Internet voting”. The Internet had nothing to do with this massive failure. It all came down to one simple concept: Testing. 

Another root cause of the Iowa disaster is the lack of technical skills. Voting was delegated by the Constitution to the Secretaries of State most of whom are lawyers. The administration of voting takes place at the county level, 3,000 of them. The staff and volunteers are hard working and dedicated to enabling people to vote. Unfortunately, the system they have to work with is a 150-year-old paper based system, the staffers are not technical, and budgets to upgrade are limited. 

In 2016, 100 million people who could have voted did not. Why? There is a long list of reasons. Absentee and vote early paper-based voting is not the answer. In Election Attitude – How Internet Voting Leads to a Stronger Democracy, I described how our smartphones with biometric authentication and blockchain technology can automate the voting process with security, privacy, accuracy, and verifiability. If we can land a robot on Mars, we certainly can automate voting. We just need the right attitude politically and technologically and a lot of testing.

West Virginia has shown great leadership in this area. They have been working with a Boston based technology company called Voatz. Voatz has a team of technology and voting experts. They designed a mobile blockchain app which has proven to be secure and easy to use. Rather than roll it out at the last minute with no testing, West Virginia worked with 144 overseas military voters residing in 31 countries. Ballots were submitted via the app for the November 2018 election. Before that, there was a smaller pilot of the system in two West Virginia counties in May 2018. Voter satisfaction with Voatz Internet voting was high, and the vote was accurate and secure.

We should not let the mismanaged Iowa experience diminish the vision to enfranchise the 100 million people who didn’t vote in 2016. We need Internet voting to have a strong democracy. As Vint Cerf, one of the fathers of the Internet said, “We can do this.”