A few weeks back, I was watching with fascination a 10-year anniversary video of the iPhone put out by the Wall Street Journal. In it, the chief hardware and software engineers on the project — Scott Forstall, Tony Fadell, and Greg Christie — recount the arduous process of turning Steve Jobs’s vision into one of the best-selling products ever made.
Scott Forstall goes into some detail on how the soft keyboard was designed, noting that, after a bake-off of competing technologies, the team went with one that looked like a regular QWERTY keyboard, but had a lot of software magic behind it. Specifically, while the user saw a static image of soft keys in their proper places, the “live” region behind the keys changed in size depending on the likelihood that a given letter was next. For a likely letter, the input area was enlarged. In other words, early artificial intelligence techniques were being used to recover from fat-fingered inaccuracies.
To achieve this trick, the team did an analysis by vacuuming up a text corpus (presumably first English) and giving a ranking for any letter following any other one. This context-dependency also took into account multiple letters at a time. Thus, if the user tapped “t,” the underlying map would expand the “h” hit-area because “h” often follows “t” in English.
What intrigued me about this discussion: what the user saw on the screen and what was happening underneath were two totally different things, leading me to recall that, in fact, the relationship is arbitrary. It’s only by convention that when you tap a “t” the Unicode bit string often interpreted as the letter “t” is sent to some memory register and can be displayed on the screen almost immediately as a bitmap that looks like a “t.” It could, however, lead to a code point that represents the Arabic letter “ﺇ” (alef, isolated form) and print that instead. But because Apple was trying to make the iPhone usable, the “t” was a “t” and the “h” was an “h.” But that huge landing pad for your fat finger that grew under the “h” after you hit the “t” was unseen. That was the magic Apple conjured up to make typing text easier and more accurate. It was important that the user not be disturbed by this prestidigitation and just see an ordinary-looking keyboard so as not to be spooked by these digital acrobatics.
These days, 10 years later, this stuff is such old hat that Forstall et al didn’t even mind discussing it publicly. Everyone uses similar technology, and the iPhone is no more guilty of fooling anyone than the Samsung Galaxy or the moto Z Force is.
All smartphones (except the poor Blackberry) are pretty much slabs of glass. The glass does two things: it displays an image (or black) and it takes touch input. The two are only as related as the vendor wants them to be.
All of which got me to thinking about the smartphone as a spying device. It’s impossible to know what’s going on inside. The phone may appear to be off. It may have even gone through the cycle of shutting down that the user is used to seeing. It sits there, black obsidian interface ostensibly quiet. But behind it, anything could be going on.
Now, you could do a thermal analysis of the phone to see what spots on the motherboard appear to be lit up with activity, but even if you did, you’d have no idea what it was doing. By now, we’ve all heard about various spyware that does things like turn on the phone’s camera, its microphone, and its geo-location system. The phone may look innocent enough, but it’s a perfect platform for eavesdropping. A person carries it with them practically everywhere, helping the spy immeasurably. And not only can the pocketable device provide a continuous stream of data from its various sensors, but it also is the portal for communications of all sorts: phone calls, email, messaging, and social posts.
The CIA couldn’t have designed a better spying device. It’s perfect, and its users voluntarily place themselves under surveillance.
What can be done about it? Nothing, really. Under the old Eskimo adage, pike the seal when it comes up to breathe, the spy can sit right at the edge of the hole in the ice because the phone user will come through there at some point as surely as a mammal needs air.
Russians know about these things from their experience under the Soviet regime (and perhaps from more recent activity). When you really don’t want to be overheard, leave the electronics behind and take a walk with your interlocutor by some riverbank. You can exchange information then — as long as your “friend” isn’t wearing a wire.