Recently, Apple revealed some new technical measures in Messages, iCloud, Siri, and search that are meant to protect children from sexual abuse online. Apple says that its new blend of on-device and cloud-based processing will strike a balance between user safety and user privacy. But some cryptography experts aren't convinced, and worry that the measures could open the door to other privacy breaches and government surveillance.
This week on Gadget Lab, WIRED senior cybersecurity writer Andy Greenberg joins us to talk about how Apple's tech works, and the company's delicate balancing act between safety and privacy.
Show Notes
Read Andy’s story about Apple’s new technology.
Recommendations
Andy recommends the book Empire of Pain: The Secret History of the Sackler Dynasty by Patrick Radden Keefe, and also the new Mortal Kombat movie. Lauren recommends Vauhini Vara’s story “Ghosts” in Believer Magazine. Mike recommends Brian Raftery’s “Gene and Roger” series of The Ringer’s The Big Picture podcast.
Andy Greenberg can be found on Twitter @a_greenberg. Lauren Goode is @LaurenGoode. Michael Calore is @snackfight. Bling the main hotline at @GadgetLab. The show is produced by Boone Ashworth (@booneashworth). Our theme music is by Solar Keys.
If you have feedback about the show, or just want to enter to win a $50 gift card, take our brief listener survey here.
>How to Listen
You can always listen to this week's podcast through the audio player on this page, but if you want to subscribe for free to get every episode, here's how:
If you're on an iPhone or iPad, open the app called Podcasts, or just tap this link. You can also download an app like Overcast or Pocket Casts, and search for Gadget Lab. If you use Android, you can find us in the Google Podcasts app just by tapping here. We’re on Spotify too. And in case you really need it, here's the RSS feed.
Transcript
Michael Calore: Hi, everyone. Just a warning upfront. Our topic this week is a heavy one. We're talking about how tech companies monitor their platforms for images of child abuse. We don't get into anything graphic. The conversation is more about how the technology works and its privacy implications. But if the topic of child abuse makes you uncomfortable, we understand if you tune out this week. OK. Now onto the show.
[Gadget Lab intro theme music plays]
MC: Hi everyone. Welcome to Gadget Lab. I am Michael Calore, a senior editor here at WIRED.
Lauren Goode: And I'm Lauren Goode, I'm a senior writer at WIRED.
MC: We are also joined this week by WIRED senior writer, Andy Greenberg. Hi, Andy. Welcome back.
Andy Greenberg: Hi, nice to talk to you both again.
MC: Of course. We've brought you on because last week, Apple announced a new set of technological measures in messages, iCloud, Siri, and search. All of which the company says are meant to protect children from sexual abuse.
These systems use a mix of on-device and cloud-based processing that Apple says will serve to protect its users while also ensuring their privacy remains uncompromised. The response to Apple's announcement has been mixed. Child safety advocacy groups have praised the move, but some cryptographers and privacy experts worry the features could open the door to government surveillance. So we have asked Andy onto the show to first explain how the technology itself works. Later on in the show, we'll talk about its broader privacy implications. Andy, you wrote a story for WIRED last week that touched on both of those things. So let's start with the first part, the technology. Can you walk us through the steps that Apple announced last week?
AG: Sure. There are actually three parts to this and it's pretty complex, but there is only one of these that is super controversial. One is semi-controversial and one that is kind of uncontroversial. The uncontroversial part is that Apple is going to sort of detect potential searches that could turn up child sexual abuse materials. As we call this stuff, CSAM, and that's the acronym for it in Siri and search. So if they see somebody searching for these CSAM materials, as we call them, then they'll put up a little warning that says you seem like you're about to find something that is about child sexual abuse materials, and maybe you should seek help, that kind of thing. That one is like a kind of no-brainer. The second of these is an iMessage, and this is only a sort of opt-in feature for family iCloud accounts.
And it works by warning parents, and children as well, when they have been sent or are sending an image that contains nudity. This is basically like a way to prevent kids from where adults who might call this sexting for children, it could easily be a form of child abuse. And iMessage will now, if this feature is turned on in family iCloud accounts, it will both warn the kids you are, it looks like you are about to receive or send nudity in an image. And if you open this or if you send it, then we will warn your parents because we want you to be safe. And then the third feature is by far the most complex and the most controversial, it's a feature of all iCloud photo storage that essentially when someone uploads an image to iCloud now, the code on their device, on their phone, or iPad, or laptop will scan that photo and using this really nuanced and very unique and actually kind of unprecedented cryptography system, try to determine if they are uploading child sexual abuse materials.
And there's really like a very kind of complex Rube Goldberg machine that Apple has invented from scratch almost to determine if they're uploading a full collection of child sexual abuse materials, then it will not only send that material to Apple so that somebody can review it, but also send it to The National Center for Missing and Exploited Children who will in turn notify law enforcement, so that's a severe step. I mean, we're talking about actually reporting potential child abusers or consumers of this child sexual abuse material to law enforcements and getting them potentially charged and put in prison. I mean, this is a very serious topic, but it's also Apple's attempt, I think, to try to thread a really, really tight needle here. They are trying to both protect their users' privacy while also trying to, I think, throw a bone to advocates for children's safety who argue that they need to do something to prevent the very worst abuses that are taking place on their platforms.
LG: I want to focus on that third part, right? Because Apple has said some version of we're not actually looking at your photos, what's happening is there's this series of hashes that are being compared to another series of hashes from that database to determine if the data being shared is problematic in any way. You described it as a Rube Goldberg-like system, but can you take us through exactly how that works?
AG: Yeah. So first your phone essentially downloads a big collection of so-called hashes, and a hash is a series of random looking characters that have been derived from an image. They're not an image, but they're unique to an image. And then that is essentially a collection of child sexual abuse materials that The National Center for Missing and Exploited Children has already assembled, that is their kind of block list. That's what they're looking for as like known CSAM materials, but it's been turned into this collection of hashes so that you don't end up with those terrible, abusive photos on your device, just in this sort of checking mechanism.
But then when you upload photos to iCloud, they are also hashed in this kind of unique way that Apple has invented called NeuralHash so that the photo doesn't have to be an exact match. It can be slightly transformed. It can be cropped or the colors can be changed. Apple won't say exactly like how this works, but using this NeuralHash thing, they create a hash of your photo and they try to match it to the hash list that has been downloaded to your phone. Now this is where some really sort of cutting edge cryptography comes in.
MC: So how exactly does this system work?
AG: So the results of these hash comparisons are encrypted, but not just encrypted, they're encrypted in two layers of kind of special encryption. And the first layer of encryption is designed to use this sort of new-ish cryptographic technique called private set intersection. So basically like this package, this double encrypted package, that Apple calls a safety voucher, only if there's a match can that first layer of encryption be removed. And if there's no match, then actually Apple or whoever controls the server where this analysis is taking place, once it's been uploaded from your phone, they can't learn anything about the hash or the image. That's why it's a private set intersection. You only learn anything if there is a match in the first place, and then you can remove that first layer of encryption. Now there's the second layer of encryption that creates a kind of second safe guard.
So Apple didn't want this to trigger if it just spots a single child sexual abuse material image, it's trying to detect entire collections of them. They won't say how many exactly, but they're looking for certainly more than one, probably at least a handful of these images. And so that second layer of encryption uses a kind of thresholds cryptography. So that only if a certain number of matches are detected, do all of them decrypt, essentially. So that second layer of encryption sort of makes sure that it's a whole collection of abusive materials. And then if there's enough matches, they all unlock at once and they all get uploaded to Apple to do a manual check, and then to The National Center for Missing and Exploited Children, and then probably to law enforcement.
LG: Is this system working on photos that are stored locally on your iPhone? Like what happens if these photos have never been backed up to iCloud or never been shared via iMessage. What happens then?
AG: Yeah, that's exactly the right question because yes, this only applies to photos uploaded to iCloud and only in the United States. But the fact that it only applies to photos uploaded to iCloud is really weird because Apple does not have end-to-end encryption on iCloud photos. They could just scan all the photos in iCloud. They have the keys to decrypt that the kind of normal server encryption on those photos and look at them just like everybody else does, honestly, like Dropbox and Microsoft and every other cloud storage company does this kind of basic check-in in a much simpler way, scanning every photo for known child sexual abuse materials. But Apple instead has chosen to do this system that starts on your phone. And that is what has freaked everybody out, like this notion that they are scanning with code on your phone, that your device now has code on it that is designed to snitch on you if you are a, you know, child sexual abuser or possess these materials.
That is a new kind of line that they've crossed and just why they've done it remains a bit of a mystery because they could do it in this much simpler way. Now the theories about why they've done it are that either they're planning to expand it very soon to photos that are offline that you don't upload to iCloud. That's what really has every privacy advocate screaming 1984, like Apple you're about to do this thing where I have a photo that's just on my phone and this device that I bought from you, it's going to rat me out to the police. I mean, that is something that we've never quite seen before, or, but the other option is actually completely the opposite.
If we start to like try to theorize about what Apple is going to do next, it seems to me, the more likely option is that actually Apple is about to implement end-to-end encryption on iCloud, and they won't be able to scan their own cloud storage for child sexual abuse materials. And so they have to do it on your device before it's uploaded to iCloud and encrypted in a way where they can't decrypt it. But for some reason, they're not saying that. So that has left everybody to imagine the worst. And as part of, I think what's in my world of security and privacy people made this extremely controversial and the response has been, I would say 90% negative in my Twitter feed at least.
MC: So what has Apple said about how they're going to deal with false positives in this system?
AG: Well, they say there'll only be one in a trillion cases that is a false positive. When they flag something as a collection of child sexual abuse materials, only one in a trillion cases will they be wrong. And that sounds just kind of like marketing, but actually so, like I said, there's this sort of thresholds that they haven't defined. That's the difference between a single image and a small number of them and a full collection, which is what they're detecting. And they say that they're going to adjust that number to make sure that false positives stay at one in a trillion or less. So I don't actually think that like the scariest part of this is that there is going to be false positives.
I do kind of believe Apple, that they're going to find what they're looking for. I think that the scarier part for privacy folks everywhere is that at any time, Apple could flip a switch on behalf of the Chinese governments or who knows even just the Indian government who has pressured a lot of companies about encryption to start searching for other kinds of material. And they may be just as accurate when they're looking for, you know, political content or something that, you know, perhaps the Kingdom of Saudi Arabia considers controversial, but we consider free speech.
MC: All right, well, that's a good point to take a break because when we come back, we have a lot more to talk about around those privacy concerns.
[Break]
MC: Welcome back. Andy, in your story, you talked with privacy experts and cryptography experts who worried that this move by Apple, while seeming well-intentioned, might have more complex motives. For one, as you mentioned in the first half of the show, it can open the company up to pressure by law enforcement or governments to reveal user's private information. How valid are those concerns?
AG: I think that they're valid. The issue is Apple has now introduced a technology that could be used if it's adapted just the bids to scan for photos that you have not shared with anyone on your device, and then tell law enforcement about that. Now, in this case, this initial use of it is to spot child sexual abuse materials, which is I think really important. I think privacy and security people like to describe this as like, oh, think of the children. And that's not a valid argument somehow. I think the child sexual abuse materials are a huge problem across technological platforms and should not be underplayed at all. But the problem is that we are trusting Apple in their relationships with governments around the world. In some cases we've seen Apple take an extremely hard line stand against government encouraging on its users' privacy.
Like when they in 2015, in this standoff with the FBI, when they refused very strongly to decrypt or even to help the FBI decrypt the encrypted iPhone of the San Bernardino shooter Syed Rizwan Farook. And maybe they will kind of hold the line here, set the line to child sexual abuse materials, and then stand up with that same kind of fervor against any kind of incursion beyond it.
But we've also seen Apple cave sometimes like to China when China demanded that they host Chinese users' data centers in China, which leaves them vulnerable to Chinese surveillance. And Apple said, oh, sorry, that is Chinese law. We're going to have to abide by that. And it's hard to be sure that Apple won't have the same answer when China demands that it starts scanning for, for instance, like, you know, photos related to Tiananmen Square on people's iPhones. And I think that that's security and privacy people never want to be in a position of trusting a company to protect user's privacy from what could very well be a kind of an invisible silence form of surveillance when we could be technologically protected in a way where we have to trust no one.
LG: Right and the Electronic Frontier Foundation has come out pretty strongly against this move by Apple. And you know, we've said it before, we'll say it again. It's impossible to build a client-side scanning system that will only be used for sexually explicit images involving children, right? That like every well intention effort has the ability to be abused. The EFF also points out that once a certain number of photos are detected that are problematic, those photos will be set to human reviewers within Apple, who then determine if they should be sent to authorities, like based on whether or not they match the CSAM database. So it seems like there's a certain amount of, this is super private. This is super private. Don't worry folks from Apple and then with privacy advocates pointing out, yeah, but then there are all these exceptions to those rules or potential exceptions to the rules that could make this inherently un-private.
AG: Well, I think that with the current system, it is really carefully designed and you know, the Stanford professor, Dan Bernay, who's a really renowned cryptographer helped to build this, actually. And I think he has not tried to take too much public credit for it since it's become so controversial, but this, the whole private set intersection thing is very clever. It's designed so that although there is scanning that happens on the phone, nothing is sort of like made visible until this safety voucher is sent to Apple's servers, until there's an entire collection of CSAM detected, so that if Apple is not just outright lying about this one in a trillion false positive thing that they have designed the system around, then I do believe that in this current implementation, the only privacy violations should happen for child sexual abusers.
The problem is that Apple has built a slippery slope and it's kind of just like waiting for a government to give it a push and send it down that slope that is maybe the EFF's strongest point here is that this system is technically described in exactly the same way. If it's for spotting photos at a protest or using facial recognition to find political dissidents in photos or something, the only difference is just that hash list, and what is on that list. So is that list The National Center for Missing and Exploited Children's list or is it going to be China's equivalent list? And is that list going to have government ties? What is going to be put on that list? Are they going to sneak in things that have more political ramifications?
MC: So you mentioned Dropbox before, what other companies are doing this and in what ways, like what is Facebook or Google or Microsoft doing?
AG: Well, I think that the kind of defaults among all of those companies, in fact, is to do a much simpler form of cloud-based scanning. This system called photo DNA is really common among tech companies that also takes a hash list basically from The National Center for Missing and Exploited Children, and just looks in the cloud, that the photos that these companies have on their servers and tries to find matches. That's simple and easy. Apple considers that a privacy invasion. And in my background call with them, they didn't really spell out why they consider it more of a privacy invasion when it happens in the cloud than on your phone. I mean, that's kind of the opposite from how everybody, like the sort of normal person thinks of this. It's like your phone is your phone. Even when I put my photos on your server, it's still your server and people are kind of used to the idea.
I think that those photos get analyzed that if you uploaded child sexual abuse materials to, I don't know, Dropbox, then it's not a big surprise that Dropbox is going to spot that and maybe even report you, but Apple is actually trying to do something that's more privacy protective, in their kind of strange, overachievers mind here. They, I think are, my theory is that they're about to end-to-end encrypt iCloud so that they technically cannot do that simple system that everybody else does. That means they have no access to your photos when you upload them to i Clouds. And if the FBI sends them a subpoena and asks them for your picture of a big pile of cocaine on your home office desk or something, they will not be able to give it over. But at the same time, if they're going to do that and try to create a way for child sexual abuse materials to be spotted, they have to do this kind of overly clever thing where they look at the photos before they're sent to the cloud on your phone.
So that's what all of this is about, I think, is that Apple is about to actually take a good step for privacy, which is that they're going to end-to-end encrypt iCloud, but to do that, they had to carve out this really technically complex exception for child sexual abuse materials. And if I were Apple's PR team, I would have made this whole child sexual abuse materials thing a footnote to the announcement that I'm end-to-end encrypting iCloud, but it seems like for some reason they rolled it out in the other order and wanted to I think maybe show governments around the world, or at least the US government, look, we've created a way to spot these terrible abusive materials, even when we encrypt iCloud. So now we're going to do it. Now you can't complain when we switch on that really strong encryption for all of our cloud storage.
MC: Andy, thank you for walking us through all of that. It is crystal clear to me now how all of this works and I'm sure all the listeners feel the same way.
AG: Perfect.
MC: Let's take a break. And when we come back, we'll do our recommendations.
[Break]
MC: All right, welcome back. This is the last part of our show, where we go through our recommendations for things that our listeners might enjoy. Andy, you are our guest, so you get to go first. What is your recommendation?
AG: Well, I hope it's OK. I actually have two recommendations. I have a highbrow recommendation and I have a lowbrow recommendation. My highbrow recommendation is a book I just read by Patrick Radden Keefe of The New Yorker. It's called Empire of Pain, and it's this truly like amazing, very fat volume that is the entire history of the Sackler family. This family that's basically created the opioid epidemic. I think it's fair to say it by running this small pharmaceutical company, Purdue Pharma, and just absolutely popularizing the use of Oxycontin in America. And as they even describe it kind of like raining pills on everyone in the country and getting millions of people addicted to this incredibly harmful drug. And it's just a wonderfully reported and told epic book. That's kind of like succession, that is to show succession, but over multiple generations and with this incredible underlying, like, very high impact historical value.
MC: Ah, very cool.
AG: My lowbrow recommendation. I know Lauren is a Peloton fan, I believe, or at least a Peloton critic, reviewer. I don't know.
LG: Cult member.
AG: Cult member, yeah. I have like my own janky Peloton-type setup where I put my bike on a trainer and then I like watch very violent movies on an iPad. And I recently watched Mortal Kombat, the new Mortal Kombat in this setup and it was just kind of exquisitely gross. And there was just like a minimum of talking between people hitting each other and tearing each other's limbs off and spines out and things. So, yeah. That's my second recommendation.
LG: Who needs Cody Rigsby when you have Mortal Combat?
MC: Is Cody one of the Peloton people?
LG: Obviously.
MC: Obviously, yes.
LG: I love that, Andy.
MC: That's great. Lauren, what's your recommendation?
LG: My recommendation is an incredible story by Vauhini Vara in Believer Magazine. This week, it's called "Ghosts." We'll link to it in the show notes. And she basically contacted OpenAI, which we've written about in WIRED a bunch before, and OpenAI has developed this machine learning model called GPT-3 that has gotten a lot of attention lately. And basically what it allows you to do is you plug in some texts and it spits out text for you, it writes things for you in a very human-like way. And Vauhini's story is that her sister died when they were young, when they were both in high school. And she writes about despite being, you know, a professional writer, how she's never really been able to write honestly about her sister's death and how traumatic it was for her. And so what she did is she used GPT-3 to like start sections of the story.
She would author certain sentences at the top and then GPT-3 would fill in the rest based on notes that she was feeding it. And the resulting story is kind of this multi-chapter, really hauntingly, beautiful and sad story. And it's not only sort of remarkable because of Vauhini's approach to writing this and the fact that she's using technology, but that, that technology has captured this experience in such a moving way. And so Mike, hat tip to you because you shared it in our slack thread this week. And I just, I thought it was really remarkable. So that's my recommendation this week, check it out. And also Mike, a lot of my smartphone reviews too, from this point forward will just be written by GPT-3.
MC: I look forward to that.
LG: I can't even say it. That's fine. It'll correct itself. So Mike, what's your recommendation this week?
MC: My recommendation is a podcast series. That's part of The Ringer's, The Big Picture podcast. The publication does this big podcast about movies. And it's very good. They've been breaking form a little bit over the last month and inserting this mini-series. It's an eight episode mini-series hosted by a former WIRED one, Brian Raftery. It is called "Gene and Roger," and it is a mini-series about Siskel and Ebert. So if you don't know Gene Siskel and Roger Ebert, you are probably younger than me because when I was growing up, these two guys were on television every week talking about movies. And this was like, pre-internet, the only way to find out whether or not a movie was any good was to watch Siskel and Ebert. And they had this really interesting rapport, two film critics, two guys from Chicago, middle-aged men in sweaters, sitting in a theater, talking about movies.
And they were like passionate and intelligent. And they had all these wonderful things to say. And of course they always gave the thumbs up or the thumbs down. I was like riveted by this show. I never missed it for years and years. The whole time I was growing up. And Roger Ebert passed away I think about eight years ago, Gene Siskel passed away about 22 years ago. So they've been gone for a while, but their shadows still loom large over American criticism. Like if you're reading film reviews now, if you're reading cultural criticism now, there is like no doubt that the person that you're reading and the words that you're reading were influenced by these two guys. The podcast takes you back to how they grew up, how they met their first attempts at making the show, which were bad. And then their later successful attempts at making the show.
And then their sort of ascendancy into icon-dom, of icon-hood, icon-dom of American pop culture. It's really fascinating. So if you have fond memories of Gene Siskel and Roger Ebert, and if you have fond memories of growing up thinking that the most intelligent thing that you could do was to speak fondly about art and culture, then you will love this podcast. So check it out. All you have to do is subscribe to The Big Picture and the episodes show up in The Big Picture feed. So Gene and Roger hosted by Brian Raftery. That's that's my rec.
LG: Excellent.
MC: Would you say thumbs up?
LG: I would give it two thumbs up.
MC: Good.
LG: Yeah.
MC: Good. Perfect. All right. Well that is our show. Andy Greenberg, thank you for joining us as always and explaining all this deeply technical stuff.
AG: Thank you for having me as always.
MC: And thank you all for listening. If you have feedback, you can find all of us on Twitter. Just check the show notes. This show is produced by Boone Ashworth. We will see you all next week. Goodbye.
[Gadget Lab outro theme music plays]