The first page of Al-Kindi's manuscript on deciphering coded messages.
By its very nature, codebreaking is a clandestine activity. The last thing a codebreaker wants to do is to let the enemy know that its code has been broken, because then the code will be upgraded and the codebreaker will be back to square one. Consequently, codebreakers have largely remained anonymous, the most famous example being the men and women of Bletchley Park who cracked the German Enigma cipher during the Second World War.
When filming The Science of Secrecy, one of my main aims was to try and the tell stories of the codebreakers whose brilliant intellectual achievements changed history, but who have received no recognition. For the first time ever, a GCHQ cryptographer spoke on film about his research into codes, which resulted in one of the greatest scientific breakthroughs this century. The series also includes the story of Room 40, Britain’s top secret codebreaking department in the First World War, and Charles Babbage, who is well known as a Victorian inventor, but whose codebreaking achievements went unrecognised until a century after his death.
Personally, the most satisfying episode from the hidden history of code breaking concerns the very first code breaker. His seminal contribution to the subject was lost for centuries until a historian uncovered an ancient Arabic manuscript that described how a ninth century polymath working in Baghdad had been the father of the science of codebreaking, otherwise known as cryptanalysis.
The early history of cryptography was dominated by the codemakers, who had been concocting an array of encryption systems for use by generals, kings, priests and alchemists The Spartans had invented the scytale, the first military encryption device, in the fifth century BC and the Kama-Sutra contains a recipe for rendering messages unreadable. Usually, encryption meant substituting conventional letters with other letters (or symbols), a system known as the substitution cipher. So instead of writing A the sender might write J, instead of B he might write W, and so on. The Swedish singing sensation of the seventies would then be encrypted as JWWJ.
The substitution cipher, at first sight, offers a high level of security. An enemy interceptor looks at the encrypted message and has to guess which of the 26 letters does A really represent? If by some fluke he guesses right, then he has to guess which of the other 25 letters for B really represent? And so on. The chances of correctly guessing all 26 letters is 1 in (26x25x…x1), or 1 in 400 million billion billion, or effectively zero. Checking all possible substitutions would take longer than the age of the universe.
However, there is a shortcut, which enables a codebreaker to crack the substitution cipher within a matter of minutes. Historians of science have long been trying to track down the inventor of the shortcut, and over the last decade Professor Mohammed Mrayati has at last made a breakthrough. Myrayati, an engineer and historian based with the UN in Lebanon, while sifting through the Sulaimaniyyah Ottoman Archive in Istanbul uncovered a hitherto unknown document entitled A Manuscript on Deciphering Cryptographic Messages, first published in around 850 AD. The author was Abu Yusuf al-Kindi, otherwise known as ‘the philospher of the Arabs’.
Al-Kindi was director of the House of Wisdom, a research institute and library, based in Baghdad. The House of Wisdom was also a centre of translation, and texts were brought from all over the ancient world in order to gain more knowledge. In some cases, the texts were encrypted, so al-Kindi’s motivation for codebreaking may have been his craving to access these encrypted secrets.
Al-Kindi’s breakthrough, known as frequency analysis, may seem obvious to modern eyes, but at the time it was a radical breakthrough that destroyed the security of the existing encryption system. He realised that letters of the alphabet appear with varying frequencies in written text, e.g., on average E accounts for 12.7 per cent of letters in an English text, whereas J, Q, X and Z combined add up to less than 1%. If encryption involves substituting each letter for a different one, then the new letters will take on the frequencies of the letters that they actually represent. Hence, al-Kindi advised codebreakers to count the frequencies of letters in an encrytped text, and then identify their true meaning according to the frequencies, e.g. the most common letter probably represents E.
Al-Kindi’s invention was based on new mathematical techniques that were being developed by the Arabs and on a deeper understanding of the structure of language and of writing which was driven by a desire to gain a deeper insight into the Koran. Indeed, when Mrayati read al-Kindi’s manuscript it became clear that it contained the earliest known discussion of statistics. We were the first film crew to have access to the manuscript and we all wondered how many other Arab discoveries were hidden in the Istanbul archives, where the cataloguing of thousands of ancient documents is an ongoing project.
Frequency analysis was eventually exported to or reinvented in Europe, where court code breakers used it to decipher the messages of their enemies. The Babington Plot was intended to assassinate Elizabeth I and put Mary Queen of Scots on the English throne, but the English codebreaker Thomas Phelippes used frequencies to crack the plotters’ cipher and read their correspondence.
The development of cryptanalysis drove the desire to invent stronger ciphers and instigated an intellectual arms race that continues to the present day. Code makers invents ciphers, code breakers crack them, so code makers invent new ones. Today, we live in a Golden Age of cryptography, in which the code makers have come up with apparently unbreakable ciphers. It is possible for ordinary citizens to download encryption software that cannot be broken even if the all the computers in the world worked for a billion years.
However, there is always the chance that a twenty-first century al-Kindi might one day find a flaw in these ciphers, undermining e-mail encryption, destroying the security required for e-commerce and allowing countries to eavesdrop at will on their neighbours. In fact, it might even be that a government laboratory has already cracked today’s ciphers and we do not yet know about it. After all, the science of secrecy is a secret science.