Apple Privacy

Apple has announced it will provide fully encrypted iCloud backups, meeting a longstanding demand by EFF and other privacy-focused organizations.

We applaud Apple for listening to experts, child advocates, and users who want to protect their most sensitive data. Encryption is one of the most important tools we have for maintaining privacy and security online. That's why we included the demand that Apple let users encrypt iCloud backups in the Fix It Already campaign that we launched in 2019.

Apple's on-device encryption is strong, but some especially sensitive iCloud data, such as photos and backups, has continued to be vulnerable to government demands and hackers. Users who opt in to Apple's new proposed feature, which the company calls Advanced Data Protection for iCloud , will be protected even if there is a data breach in the cloud, a government demand, or a breach from within Apple (such as a rogue employee). Apple said today that the feature will be available to U.S. users by the end of the year, and will roll out to the rest of the world in "early 2023."

We're also pleased to hear that Apple has officially dropped its plans to install photo-scanning software on its devices , which would have inspected users' private photos in iCloud and iMessage. This software, a version of what's called "client-side scanning," was intended to locate child abuse imagery and report it to authorities. When a user's information is end-to-end encrypted and there is no device scanning, the user has true control over who has access to that data.

Apple's image-scanning plans were announced in 2021 , but delayed after EFF supporters protested and delivered a petition containing more than 60,000 signatures to Apple executives. While Apple quietly postponed these scanning plans later that year, today's announcement makes it official.

In a statement distributed to Wired and other journalists, Apple said:

We have further decided to not move forward with our previously proposed CSAM detection tool for iCloud Photos. Children can be protected without companies combing through personal data, and we will continue working with governments, child advocates, and other companies to help protect young people, preserve their right to privacy, and make the internet a safer place for children and for us all.

The company has said it will focus instead on "opt-in tools for parents" and "privacy-preserving solutions to combat Child Sexual Abuse Material and protect children, while addressing the unique privacy needs of personal communications and data storage."

Constant scanning for child abuse images can lead to unwarranted investigations and false positives. Earlier this year, the New York Times reported on how faulty scans at Google led to false accusations of child abuse against fathers in Texas and California. The men were exonerated by police but were subjected to permanent account deletion by Google.

Companies should stop trying to square the circle by putting bugs in our pockets at the request of governments, and focus on protecting their users, and human rights. Today Apple took a big step forward on both fronts. There are a number of implementation choices that can affect the overall security of the new feature, and we'll be pushing Apple to make sure the encryption is as strong as possible. Finally, we'd like Apple to go a step further. Turning on these privacy-protective features by default would mean that all users can have their rights protected.

Apple Inc. announced this week that it will offer full encryption for data in its cloud storage system worldwide, which not surprisingly has been met with consternation from hackers, thieves & snoopers but joy from customers.

The move means that all content, chats, photos and videos, will have end-to-end encryption under Apple's Advanced Data Protection feature. For the average consumer, this is a win, and for the average privacy advocate, it's a victory in an ongoing fight with the authorities.

We applaud Apple for listening to experts, child advocates, and users who want to protect their most sensitive data, the Electronic Frontier Foundation wrote today. Encryption is one of the most important tools we have for maintaining privacy and security online.

The foundation was equally pleased that Apple also stated that it had finally decided not to implement its CSAM photo-scanning child protection technology.

The authorities, on the other hand, have denounced the move, especially the FBI, which has a history of battling with Apple over trying to get its hands on user data.

Apple will also likely run into problems concerning the U.K. government and its online safety bill. The bill gives the U.K. government broad powers to force companies to ensure content on their platforms aligns with what it calls internet safety. Many privacy advocates have condemned the bill as a new kind of censorship.

Apple is set to roll out a snooping feature that scans messages for nudity to UK iPhones.

The feature uses AI technology to scan incoming and outgoing messages.

For the moment it is otional and allows parents to turn on warnings for their children's iPhones. When enabled, all photos sent or received by the child using the Messages app will be scanned for nudity.

If nudity is found in photos received by a child with the setting turned on, the photo will be blurred, and the child will be warned that it may contain sensitive content and nudged towards resources from child safety groups. If nudity is found in photos sent by a child, similar protections kick in, and the child is encouraged not to send the images, and given an option to Message a Grown-Up.

All the scanning is carried out on-device, meaning that the images are analysed by the iPhone itself, and Apple never sees either the photos being analysed or the results of the analysis, it said.

As originally announced in summer 2021, the communication safety in Messages and the search warnings were part of a trio of features that proved extremely contentious, and Apple delayed the launch of all three while it negotiated with privacy and child safety groups.

Of course having implemented the feature as an option it won't be long before it becomes an option that can be turned on by law enforcement in the name of seeking out terrorists, racists, anti-vaxers etc

Apple has announced on its website that it will delay it implementation of device software that snoops on users' images nominally in the name of child protection, but could used be for anything that authorities demand. Apple said:

Update as of September 3, 2021: Previously we announced plans for features intended to help protect children from predators who use communication tools to recruit and exploit them and to help limit the spread of Child Sexual Abuse Material. Based on feedback from customers, advocacy groups, researchers, and others, we have decided to take additional time over the coming months to collect input and make improvements before releasing these critically important child safety features.

Apple's new program for scanning images sent on iMessage steps back from the company's prior support for the privacy and security of encrypted messages. The program, initially limited to the United States, narrows the understanding of end-to-end encryption to allow for client-side scanning. While Apple aims at the scourge of child exploitation and abuse, the company has created an infrastructure that is all too easy to redirect to greater surveillance and censorship. The program will undermine Apple's defense that it can't comply with the broader demands.

For years, countries around the world have asked for access to and control over encrypted messages, asking technology companies to "nerd harder" when faced with the pushback that access to messages in the clear was incompatible with strong encryption. The Apple child safety message scanning program is currently being rolled out only in the United States.

The United States has not been shy about seeking access to encrypted communications, pressuring the companies to make it easier to obtain data with warrants and to voluntarily turn over data. However, the U.S. faces serious constitutional issues if it wanted to pass a law that required warrantless screening and reporting of content. Even if conducted by a private party, a search ordered by the government is subject to the Fourth Amendment's protections. Any "warrant" issued for suspicionless mass surveillance would be an unconstitutional general warrant. As the Ninth Circuit Court of Appeals has explained , "Search warrants . . . are fundamentally offensive to the underlying principles of the Fourth Amendment when they are so bountiful and expansive in their language that they constitute a virtual, all-encompassing dragnet[.]" With this new program, Apple has failed to hold a strong policy line against U.S. laws undermining encryption, but there remains a constitutional backstop to some of the worst excesses. But U.S constitutional protection may not necessarily be replicated in every country.

Apple is a global company, with phones and computers in use all over the world, and many governments pressure that comes along with that. Apple has promised it will refuse government "demands to build and deploy government-mandated changes that degrade the privacy of users." It is good that Apple says it will not, but this is not nearly as strong a protection as saying it cannot, which could not honestly be said about any system of this type. Moreover, if it implements this change, Apple will need to not just fight for privacy, but win in legislatures and courts around the world. To keep its promise, Apple will have to resist the pressure to expand the iMessage scanning program to new countries, to scan for new types of content and to report outside parent-child relationships.

It is no surprise that authoritarian countries demand companies provide access and control to encrypted messages, often the last best hope for dissidents to organize and communicate. For example, Citizen Lab's research shows that--right now--China's unencrypted WeChat service already surveils images and files shared by users, and uses them to train censorship algorithms. "When a message is sent from one WeChat user to another, it passes through a server managed by Tencent (WeChat's parent company) that detects if the message includes blacklisted keywords before a message is sent to the recipient." As the Stanford Internet Observatory's Riana Pfefferkorn explains , this type of technology is a roadmap showing "how a client-side scanning system originally built only for CSAM [Child Sexual Abuse Material] could and would be suborned for censorship and political persecution." As Apple has found , China, with the world's biggest market, can be hard to refuse. Other countries are not shy about applying extreme pressure on companies, including arresting local employees of the tech companies.

But many times potent pressure to access encrypted data also comes from democratic countries that strive to uphold the rule of law, at least at first. If companies fail to hold the line in such countries, the changes made to undermine encryption can easily be replicated by countries with weaker democratic institutions and poor human rights records--often using similar legal language, but with different ideas about public order and state security, as well as what constitutes impermissible content, from obscenity to indecency to political speech. This is very dangerous. These countries, with poor human rights records, will nevertheless contend that they are no different. They are sovereign nations, and will see their public-order needs as equally urgent. They will contend that if Apple is providing access to any nation-state under that state's local laws, Apple must also provide access to other countries, at least, under the same terms.

'Five Eyes' Countries Will Seek to Scan Messages

For example, the Five Eyes--an alliance of the intelligence services of Canada, New Zealand, Australia, the United Kingdom, and the United States-- warned in 2018 that they will "pursue technological, enforcement, legislative or other measures to achieve lawful access solutions" if the companies didn't voluntarily provide access to encrypted messages. More recently, the Five Eyes have pivoted from terrorism to the prevention of CSAM as the justification, but the demand for unencrypted access remains the same, and the Five Eyes are unlikely to be satisfied without changes to assist terrorism and criminal investigations too.

The United Kingdom's Investigatory Powers Act, following through on the Five Eyes' threat, allows their Secretary of State to issue " technical capacity notices ," which oblige telecommunications operators to make the technical ability of "providing assistance in giving effect to an interception warrant, equipment interference warrant, or a warrant or authorisation for obtaining communications data." As the UK Parliament considered the IPA, we warned that a "company could be compelled to distribute an update in order to facilitate the execution of an equipment interference warrant, and ordered to refrain from notifying their customers."

Under the IPA, the Secretary of State must consider "the technical feasibility of complying with the notice." But the infrastructure needed to roll out Apple's proposed changes makes it harder to say that additional surveillance is not technically feasible. With Apple's new program, we worry that the UK might try to compel an update that would expand the current functionality of the iMessage scanning program, with different algorithmic targets and wider reporting. As the iMessage "communication safety" feature is entirely Apple's own invention, Apple can all too easily change its own criteria for what will be flagged for reporting. Apple may receive an order to adopt its hash matching program for iPhoto into the message pre-screening. Likewise, the criteria for which accounts will apply this scanning, and where positive hits get reported, are wholly within Apple's control.

Australia followed suit with its Assistance and Access Act, which likewise allows for requirements to provide technical assistance and capabilities, with the disturbing potential to undermine encryption. While the Act contains some safeguards, a coalition of civil society organizations, tech companies, and trade associations, including EFF and--wait for it--Apple, explained that they were insufficient.

Indeed, in Apple's own submission to the Australian government, Apple warned "the government may seek to compel providers to install or test software or equipment, facilitate access to customer equipment, turn over source code, remove forms of electronic protection, modify characteristics of a service, or substitute a service, among other things." If only Apple would remember that these very techniques could also be used in an attempt to mandate or change the scope of Apple's scanning program.

While Canada has yet to adopt an explicit requirement for plain text access, the Canadian government is actively pursuing filtering obligations for various online platforms , which raise the spectre of a more aggressive set of obligations targeting private messaging applications.

Censorship Regimes Are In Place And Ready to Go

For the Five Eyes, the ask is mostly for surveillance capabilities, but India and Indonesia are already down the slippery slope to content censorship. The Indian government's new Intermediary Guidelines and Digital Media Ethics Code (" 2021 Rules "), in effect earlier this year, directly imposes dangerous requirements for platforms to pre-screen content. Rule 4(4) compels content filtering, requiring that providers "endeavor to deploy technology-based measures," including automated tools or other mechanisms, to "proactively identify information" that has been forbidden under the Rules.

India's defense of the 2021 rules , written in response to the criticism from three UN Special Rapporteurs , was to highlight the very real dangers to children, and skips over the much broader mandate of the scanning and censorship rules. The 2021 Rules impose proactive and automatic enforcement of its content takedown provisions, requiring the proactive blocking of material previously held to be forbidden under Indian law. These laws broadly include those protecting "the sovereignty and integrity of India; security of the State; friendly relations with foreign States; public order; decency or morality." This is no hypothetical slippery slope--it's not hard to see how this language could be dangerous to freedom of expression and political dissent. Indeed, India's track record on its Unlawful Activities Prevention Act , which has reportedly been used to arrest academics, writers and poets for leading rallies and posting political messages on social media, highlight this danger.

It would be no surprise if India claimed that Apple's scanning program was a great start towards compliance, with a few more tweaks needed to address the 2021 Rules' wider mandate. Apple has promised to protest any expansion, and could argue in court, as WhatsApp and others have, that the 2021 Rules should be struck down, or that Apple does not fit the definition of a social media intermediary regulated under these 2021 Rules. But the Indian rules illustrate both the governmental desire and the legal backing for pre-screening encrypted content, and Apple's changes makes it all the easier to slip into this dystopia.

This is, unfortunately, an ever-growing trend. Indonesia, too, has adopted Ministerial Regulation MR5 to require service providers (including "instant messaging" providers) to "ensure" that their system "does not contain any prohibited [information]; and [...] does not facilitate the dissemination of prohibited [information]". MR5 defines prohibited information as anything that violates any provision of Indonesia's laws and regulations, or creates "community anxiety" or "disturbance in public order." MR5 also imposes disproportionate sanctions, including a general blocking of systems for those who fail to ensure there is no prohibited content and information in their systems. Indonesia may also see the iMessage scanning functionality as a tool for compliance with Regulation MR5, and pressure Apple to adopt a broader and more invasive version in their country.

Pressure Will Grow

The pressure to expand Apple's program to more countries and more types of content will only continue. In fall of 2020, in the European Union, a series of leaked documents from the European Commission foreshadowed an anti-encryption law to the European Parliament, perhaps this year. Fortunately, there is a backstop in the EU. Under the e-commerce directive, EU Member States are not allowed to impose a general obligation to monitor the information that users transmit or store, as stated in the Article 15 of the e-Commerce Directive (2000/31/EC). Indeed, the Court of Justice of the European Union ( CJEU) has stated explicitly that intermediaries may not be obliged to monitor their services in a general manner in order to detect and prevent illegal activity of their users. Such an obligation will be incompatible with fairness and proportionality. Despite this, in a leaked internal document published by Politico, the European Commission committed itself to an action plan for mandatory detection of CSAM by relevant online service providers (expected in December 2021) that pointed to client-side scanning as the solution, which can potentially apply to secure private messaging apps, and seizing upon the notion that it preserves the protection of end-to-end encryption.

For governmental policymakers who have been urging companies to nerd harder, wordsmithing harder is just as good. The end result of access to unencrypted communication is the goal, and if that can be achieved in a way that arguably leaves a more narrowly defined end-to-end encryption in place, all the better for them.

All it would take to widen the narrow backdoor that Apple is building is an expansion of the machine learning parameters to look for additional types of content, the adoption of the iPhoto hash matching to iMessage, or a tweak of the configuration flags to scan, not just children's, but anyone's accounts. Apple has a fully built system just waiting for external pressure to make the necessary changes. China and doubtless other countries already have hashes and content classifiers to identify messages impermissible under their laws, even if they are protected by international human rights law. The abuse cases are easy to imagine: governments that outlaw homosexuality might require a classifier to be trained to restrict apparent LGBTQ+ content, or an authoritarian regime might demand a classifier able to spot popular satirical images or protest flyers.

Now that Apple has built it, they will come. With good intentions, Apple has paved the road to mandated security weakness around the world, enabling and reinforcing the arguments that, should the intentions be good enough, scanning through your personal life and private communications is acceptable. We urge Apple to reconsider and return to the mantra Apple so memorably emblazoned on a billboard at 2019's CES conference in Las Vegas: What happens on your iPhone, stays on your iPhone

   Apple intends to install software, initially on American iPhones, to scan for child abuse imagery, raising alarm among security researchers who warn that it will open the door to surveillance of millions of people’s personal devices.

The automated system would proactively alert a team of human reviewers if it believes illegal imagery is detected, who would then contact law enforcement if the material can be verified. The scheme will initially roll out only in the US.

According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a safety voucher saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities.

The scheme seems to be a nasty compromise with governments to allow Apple to offer encrypted communication whilst allowing state security to see what some people may be hiding.

Alec Muffett, a security researcher and privacy campaigner who formerly worked at Facebook and Deliveroo, said Apple's move was tectonic and a huge and regressive step for individual privacy. Apple are walking back privacy to enable 1984, he said.

Ross Anderson, professor of security engineering at the University of Cambridge, said:

It is an absolutely appalling idea, because it is going to lead to distributed bulk surveillance of . . . our phones and laptops.

Although the system is currently trained to spot child sex abuse, it could be adapted to scan for any other targeted imagery and text, for instance, terror beheadings or anti-government signs at protests, say researchers. Apple's precedent could also increase pressure on other tech companies to use similar techniques.

And given that the system is based on mapping images to a hash code and then comparing that has code with those from known child porn images, then surely there is a chance of a false positive when an innocent image just happens to the map to the same hash code as an illegal image. That could surely have devastating consequences with police banging on doors at dawn accompanied by the 'there's no smoke without fire' presumption of guilt that exists around the scourge of child porn. An unlucky hash may then lead to a trashed life.

Apple's official blog post inevitably frames the new snooping capability as if it was targeted only at child porn but it is clear that the capability can be extended way beyond this narrow definition. The blog post states:

Child Sexual Abuse Material (CSAM) detection

To help address this, new technology in iOS and iPadOS* will allow Apple to detect known CSAM images stored in iCloud Photos. This will enable Apple to report these instances to the National Center for Missing and Exploited Children (NCMEC). NCMEC acts as a comprehensive reporting center for CSAM and works in collaboration with law enforcement agencies across the United States.

Apple's method of detecting known CSAM is designed with user privacy in mind. Instead of scanning images in the cloud, the system performs on-device matching using a database of known CSAM image hashes provided by NCMEC and other child safety organizations. Apple further transforms this database into an unreadable set of hashes that is securely stored on users' devices.

Before an image is stored in iCloud Photos, an on-device matching process is performed for that image against the known CSAM hashes. This matching process is powered by a cryptographic technology called private set intersection, which determines if there is a match without revealing the result. The device creates a cryptographic safety voucher that encodes the match result along with additional encrypted data about the image. This voucher is uploaded to iCloud Photos along with the image.

Using another technology called threshold secret sharing, the system ensures the contents of the safety vouchers cannot be interpreted by Apple unless the iCloud Photos account crosses a threshold of known CSAM content. The threshold is set to provide an extremely high level of accuracy and ensures less than a one in one trillion chance per year of incorrectly flagging a given account.

Only when the threshold is exceeded does the cryptographic technology allow Apple to interpret the contents of the safety vouchers associated with the matching CSAM images. Apple then manually reviews each report to confirm there is a match, disables the user's account, and sends a report to NCMEC. If a user feels their account has been mistakenly flagged they can file an appeal to have their account reinstated.

This innovative new technology allows Apple to provide valuable and actionable information to NCMEC and law enforcement regarding the proliferation of known CSAM. And it does so while providing significant privacy benefits over existing techniques since Apple only learns about users' photos if they have a collection of known CSAM in their iCloud Photos account. Even in these cases, Apple only learns about images that match known CSAM.

Expanding guidance in Siri and Search

Apple is also expanding guidance in Siri and Search by providing additional resources to help children and parents stay safe online and get help with unsafe situations. For example, users who ask Siri how they can report CSAM or child exploitation will be pointed to resources for where and how to file a report.

Siri and Search are also being updated to intervene when users perform searches for queries related to CSAM. These interventions will explain to users that interest in this topic is harmful and problematic, and provide resources from partners to get help with this issue.

These updates to Siri and Search are coming later this year in an update to iOS 15, iPadOS 15, watchOS 8, and macOS Monterey.*

Update: Apples photo scanning and snooping 'misunderstood'

13th August 2021. See article from cnet.com

Apple plans to scan some photos on iPhones, iPads and Mac computers for images depicting child abuse. The move has upset privacy advocates and security researchers, who worry that the company's newest technology could be twisted into a tool for surveillance and political censorship. Apple says those concerns are misplaced and based on a misunderstanding of the technology it's developed.

In an interview published Friday by The Wall Street Journal, Apple's software head, Craig Federighi, attributed much of people's concerns to the company's poorly handled announcements of its plans. Apple won't be scanning all photos on a phone, for example, only those connected to its iCloud Photo Library syncing system.

It's really clear a lot of messages got jumbled pretty badly in terms of how things were understood, Federighi said in his interview. We wish that this would've come out a little more clearly for everyone because we feel very positive and strongly about what we're doing.

 

 

Update: Apple offers slight improvements

14th August 2021. See article from theverge.com

The idea that Apple would be snooping on your device to detect child porn and nude mages hasn't gone down well with users and privacy campaigners. The bad publicity has prompted the company to offer an olive branch.

To address the possibility for countries to expand the scope of flagged images to be detected for their own surveillance purposes, Apple says it will only detect images that exist in at least 2 country's lists. Apple says it won't rely on a single government-affiliated database -- like that of the US-based National Center for Missing and Exploited Children, or NCMEC -- to identify CSAM. Instead, it will only match pictures from at least two groups with different national affiliations. The goal is that no single government could have the power to secretly insert unrelated content for censorship purposes, since it wouldn't match hashes in any other database.

Apple has also said that it would 'resist' requests from countries to expand the definition of images of interest. However this is a worthless reassurance when all it would take is a court order for Apple to be forced into complying with any requests that the authorities make.

Apple has also states the tolerances that will be applied to prevent false positives. It is alarming that innocent images can in fact generate a hash code that matches a child porn image. And to try and prevent innocent people from being locked up, Apple will now require 30 images to nave hashes matching illegal images before the images get investigated by Apple staff. Previously Apple had declined to comment on what the tolerance value will be.

Apple has announced impending changes to its operating systems that include new protections for children features in iCloud and iMessage. If you've spent any time following the Crypto Wars, you know what this means: Apple is planning to build a backdoor into its data storage system and its messaging system.

Child exploitation is a serious problem, and Apple isn't the first tech company to bend its privacy-protective stance in an attempt to combat it. But that choice will come at a high price for overall user privacy. Apple can explain at length how its technical implementation will preserve privacy and security in its proposed backdoor, but at the end of the day, even a thoroughly documented, carefully thought-out, and narrowly-scoped backdoor is still a backdoor.

To say that we are disappointed by Apple's plans is an understatement. Apple has historically been a champion of end-to-end encryption, for all of the same reasons that EFF has articulated time and time again. Apple's compromise on end-to-end encryption may appease government agencies in the U.S. and abroad, but it is a shocking about-face for users who have relied on the company's leadership in privacy and security.

There are two main features that the company is planning to install in every Apple device. One is a scanning feature that will scan all photos as they get uploaded into iCloud Photos to see if they match a photo in the database of known child sexual abuse material (CSAM) maintained by the National Center for Missing & Exploited Children (NCMEC). The other feature scans all iMessage images sent or received by child accounts204that is, accounts designated as owned by a minor204for sexually explicit material, and if the child is young enough, notifies the parent when these images are sent or received. This feature can be turned on or off by parents.

When Apple releases these client-side scanning functionalities, users of iCloud Photos, child users of iMessage, and anyone who talks to a minor through iMessage will have to carefully consider their privacy and security priorities in light of the changes, and possibly be unable to safely use what until this development is one of the preeminent encrypted messengers.

Apple Is Opening the Door to Broader Abuse

We've said it before, and we'll say it again now: it's impossible to build a client-side scanning system that can only be used for sexually explicit images sent or received by children. As a consequence, even a well-intentioned effort to build such a system will break key promises of the messenger's encryption itself and open the door to broader abuses.

All it would take to widen the narrow backdoor that Apple is building is an expansion of the machine learning parameters to look for additional types of content, or a tweak of the configuration flags to scan, not just children's, but anyone's accounts. That's not a slippery slope; that's a fully built system just waiting for external pressure to make the slightest change. Take the example of India, where recently passed rules include dangerous requirements for platforms to identify the origins of messages and pre-screen content. New laws in Ethiopia requiring content takedowns of misinformation in 24 hours may apply to messaging services. And many other countries204often those with authoritarian governments204have passed similar laws. Apple's changes would enable such screening, takedown, and reporting in its end-to-end messaging. The abuse cases are easy to imagine: governments that outlaw homosexuality might require the classifier to be trained to restrict apparent LGBTQ+ content, or an authoritarian regime might demand the classifier be able to spot popular satirical images or protest flyers.

We've already seen this mission creep in action. One of the technologies originally built to scan and hash child sexual abuse imagery has been repurposed to create a database of terrorist content that companies can contribute to and access for the purpose of banning such content. The database, managed by the Global Internet Forum to Counter Terrorism (GIFCT), is troublingly without external oversight, despite calls from civil society. While it's therefore impossible to know whether the database has overreached, we do know that platforms regularly flag critical content as terrorism, including documentation of violence and repression, counterspeech, art, and satire.

Image Scanning on iCloud Photos: A Decrease in Privacy

Apple's plan for scanning photos that get uploaded into iCloud Photos is similar in some ways to Microsoft's PhotoDNA. The main product difference is that Apple's scanning will happen on-device. The (unauditable) database of processed CSAM images will be distributed in the operating system (OS), the processed images transformed so that users cannot see what the image is, and matching done on those transformed images using private set intersection where the device will not know whether a match has been found. This means that when the features are rolled out, a version of the NCMEC CSAM database will be uploaded onto every single iPhone. The result of the matching will be sent up to Apple, but Apple can only tell that matches were found once a sufficient number of photos have matched a preset threshold.

Once a certain number of photos are detected, the photos in question will be sent to human reviewers within Apple, who determine that the photos are in fact part of the CSAM database. If confirmed by the human reviewer, those photos will be sent to NCMEC, and the user's account disabled. Again, the bottom line here is that whatever privacy and security aspects are in the technical details, all photos uploaded to iCloud will be scanned.

Make no mistake: this is a decrease in privacy for all iCloud Photos users, not an improvement.

Currently, although Apple holds the keys to view Photos stored in iCloud Photos, it does not scan these images. Civil liberties organizations have asked the company to remove its ability to do so. But Apple is choosing the opposite approach and giving itself more knowledge of users' content.

Machine Learning and Parental Notifications in iMessage: A Shift Away From Strong Encryption

Apple's second main new feature is two kinds of notifications based on scanning photos sent or received by iMessage. To implement these notifications, Apple will be rolling out an on-device machine learning classifier designed to detect sexually explicit images. According to Apple, these features will be limited (at launch) to U.S. users under 18 who have been enrolled in a Family Account. In these new processes, if an account held by a child under 13 wishes to send an image that the on-device machine learning classifier determines is a sexually explicit image, a notification will pop up, telling the under-13 child that their parent will be notified of this content. If the under-13 child still chooses to send the content, they have to accept that the parent will be notified, and the image will be irrevocably saved to the parental controls section of their phone for the parent to view later. For users between the ages of 13 and 17, a similar warning notification will pop up, though without the parental notification.

Similarly, if the under-13 child receives an image that iMessage deems to be sexually explicit, before being allowed to view the photo, a notification will pop up that tells the under-13 child that their parent will be notified that they are receiving a sexually explicit image. Again, if the under-13 user accepts the image, the parent is notified and the image is saved to the phone. Users between 13 and 17 years old will similarly receive a warning notification, but a notification about this action will not be sent to their parent's device.

This means that if204for instance204a minor using an iPhone without these features turned on sends a photo to another minor who does have the features enabled, they do not receive a notification that iMessage considers their image to be explicit or that the recipient's parent will be notified. The recipient's parents will be informed of the content without the sender consenting to their involvement. Additionally, once sent or received, the sexually explicit image cannot be deleted from the under-13 user's device.

Whether sending or receiving such content, the under-13 user has the option to decline without the parent being notified. Nevertheless, these notifications give the sense that Apple is watching over the user's shoulder204and in the case of under-13s, that's essentially what Apple has given parents the ability to do.

It is also important to note that Apple has chosen to use the notoriously difficult-to-audit technology of machine learning classifiers to determine what constitutes a sexually explicit image. We know from years of documentation and research that machine-learning technologies, used without human oversight, have a habit of wrongfully classifying content, including supposedly sexually explicit content. When blogging platform Tumblr instituted a filter for sexual content in 2018, it famously caught all sorts of other imagery in the net, including pictures of Pomeranian puppies, selfies of fully-clothed individuals, and more. Facebook's attempts to police nudity have resulted in the removal of pictures of famous statues such as Copenhagen's Little Mermaid. These filters have a history of chilling expression, and there's plenty of reason to believe that Apple's will do the same.

Since the detection of a sexually explicit image will be using on-device machine learning to scan the contents of messages, Apple will no longer be able to honestly call iMessage end-to-end encrypted. Apple and its proponents may argue that scanning before or after a message is encrypted or decrypted keeps the end-to-end promise intact, but that would be semantic maneuvering to cover up a tectonic shift in the company's stance toward strong encryption.

Whatever Apple Calls It, It's No Longer Secure Messaging

As a reminder, a secure messaging system is a system where no one but the user and their intended recipients can read the messages or otherwise analyze their contents to infer what they are talking about. Despite messages passing through a server, an end-to-end encrypted message will not allow the server to know the contents of a message. When that same server has a channel for revealing information about the contents of a significant portion of messages, that's not end-to-end encryption. In this case, while Apple will never see the images sent or received by the user, it has still created the classifier that scans the images that would provide the notifications to the parent. Therefore, it would now be possible for Apple to add new training data to the classifier sent to users' devices or send notifications to a wider audience, easily censoring and chilling speech.

But even without such expansions, this system will give parents who do not have the best interests of their children in mind one more way to monitor and control them, limiting the internet's potential for expanding the world of those whose lives would otherwise be restricted. And because family sharing plans may be organized by abusive partners, it's not a stretch to imagine using this feature as a form of stalkerware.

People have the right to communicate privately without backdoors or censorship, including when those people are minors. Apple should make the right decision: keep these backdoors off of users' devices.

Apple has announced 2 new snooping capabilities (initial used for US users only) that will be added to its operating systems in the near future.

The first will be to analyse the content of pictures on users' devices sent in messages. Apple says that this system will only be used to inform parents when their under 12yo children attempt to send sexual content. No doubt Apple will come under pressure to scan images of all users for an ever expanding list of restrictions, eg terrorism, covid memes, copyrighted images etc.

The second scan is to match any photos being uploaded to Apple's iCloud Photo Library. If these images match a curated list of child abuse image hashes then Apple will decrypt flagged images and judge for themselves whether it is an illegal image, and then inform the police. Apple claims that they will avoid the life shattering possibility of a false positive by only investigating if several images match hashes.

Update: WhatsApp responds: A setback for people's privacy all over the world

7th August 2021. See article from dailymail.co.uk

The head of WhatsApp tweeted a barrage of criticism against Apple over plans to automatically scan iPhones and cloud storage for images of child abuse. It would see flagged owners reported to the police after a company employee has looked at their photos.

But WhatsApp head Will Cathcart said the popular messaging app would not follow Apple's strategy. His criticism adds to a stream of criticism of Apple's new system by privacy campaigners who say it is the start of an infrastructure for surveillance and censorship. Cathcart said:

I think this is the wrong approach and a setback for people's privacy all over the world.

Apple's system can scan all the private photos on your phone -- even photos you haven't shared with anyone. That's not privacy.

People have asked if we'll adopt this system for WhatsApp. The answer is no.

Instead of focusing on making it easy for people to report content that's shared with them, Apple has built software that can scan all the private photos on your phone -- even photos you haven't shared with anyone. That's not privacy.

We've had personal computers for decades and there has never been a mandate to scan the private content of all desktops, laptops or phones globally for unlawful content. It's not how technology built in free countries works..

Will this system be used in China? What content will they consider illegal there and how will we ever know? How will they manage requests from governments all around the world to add other types of content to the list for scanning? Can this scanning software running on your phone be error proof? Researchers have not been allowed to find out. Why not? How will we know how often mistakes are violating people's privacy? What will happen when spyware companies find a way to exploit this software? Recent reporting showed the cost of vulnerabilities in iOS software as is. What happens if someone figures out how to exploit this new system?, Cathcart listed as concerning questions.

There are so many problems with this approach, and it's troubling to see them act without engaging experts that have long documented their technical and broader concerns with this.