The Storage Situation: Removable Storage

http://commonsware.com/blog/2014/04/09/storage-situation-removable-storage.html

There is a lot of confusion regarding Android’s storage model. That confusion has only increased with Android 4.4’s changes to that storage model. There are countless StackOverflow questions and the like where they clearly do not quite grok the various pieces of Android’s storage model.

This is the third post in a five-part series covering this storage model, to help clarify what is going on. Today, we will look at removable storage, and the source of an unfortunate amount of angst.

What Your Users Think “Removable Storage” Means

Many of your users will have a device that has some sort of removable media. Often times this is a micro SD card. Some tablets or docks have a full SD card slot. Plus, USB mass storage is possible via USB On-The-Go connectors (not to mention devices or docks with a full USB host port).

Your users will think that they can work with removable storage much like they can with a desktop or notebook.

Unfortunately, your users are largely mistaken, and are even more mistaken with Android 4.4+. That’s because Google’s approach towards removable storage is… odd.

What Google Thinks “Removable Storage” Means

In the beginning, as was noted yesterday, external storage was often in the form of a removable micro SD card. At that time, many developers got in the habit of thinking that external storage == removable storage.

However, as Android 3.0 and higher started rolling out en masse, developers soon started to realize two things:

External storage != removable storage on most of those devices
There’s nothing in the Android SDK for removable storage

Wait, Wut?

That’s right: until Android 4.4, there was no official support for removable media in Android. Quoting Dianne Hackborn:

…keep in mind: until Android 4.4, the official Android platform has not supported SD cards at all except for two special cases: the old school storage layout where external storage is an SD card (which is still supported by the platform today), and a small feature added to Android 3.0 where it would scan additional SD cards and add them to the media provider and give apps read-only access to their files (which is also still supported in the platform today).

Android 4.4 is the first release of the platform that has actually allowed applications to use SD cards for storage. Any access to them prior to that was through private, unsupported APIs. We now have a quite rich API in the platform that allows applications to make use of SD cards in a supported way, in better ways than they have been able to before: they can make free use of their app-specific storage area without requiring any permissions in the app, and can access any other files on the SD card as long as they go through the file picker, again without needing any special permissions.

But… But… But… What About All These Apps That Use Removable Media?

They fall into three buckets:

Some are just relying on MediaStore indexing. So, for example, a video player can find out about videos on all available media by querying the MediaStore, and if the device manufacturer hasn’t broken the MediaStore indexing of removable media, the player will be able to play back videos on removable media.
Some are apps that ship with the hardware. The hardware manufacturer knows the device and what the rules of the game are for that device. The hardware manufacturer is also far less concerned about cross-device compatibility, as their apps aren’t (usually) shipping on the Play Store. Hence, a hardware manufacturer has carte blanche to work with removable media.
Some are apps written by developers who decided to go past the boundaries of the Android SDK. There are various recipes online for examining various Linux system files (and file-like substances) to determine what “mount points” exist, and from there apply some heuristics to determine what represents removable media. While reliability across devices could easily be an issue, beyond that, these techniques worked… until Android 4.4, when everything changed.

What Happened in Android 4.4: The Good News

Two things happened in the timeframe of Android 4.4 that affect removable media.

On the plus side, we gained some measure of official Android SDK support for removable media. Specifically getExternalFilesDirs() and getExternalCacheDirs() (note the plural form) will not only return directories that we can use on “real” external storage, but also will return directories that we can use on any available and supported removable media. Our apps do not need any specific permissions to use any of those directories.

Also, the Storage Access Framework gives device manufacturers some options for exposing removable media to our apps in a more controlled fashion. Quoting Jeff Sharkey:

However, apps can create/write files outside of their package-specific directories on secondary external storage devices by using the new CREATE_DOCUMENT intent, which involves the user in the file creation process.

What Happened in Android 4.4: The Angst-Riddled News

Since Android 4.2, there has been a request from Google for device manufacturers to lock down removable media. Generally, this was ignored.

For Android 4.4, Google amended the Compatibility Test Suite (CTS) that device manufacturers must comply with in order to ship a device containing Google’s proprietary apps (e.g., Play Store, Maps, Gmail; otherwise known as “GMS”). Quoting Dave Smith:

However, new tests were added in CTS for 4.4 that validate whether or not secondary storage has the proper read-only permissions in non app-specific directories, presumably because of the new APIs to finally expose those paths to application developers. As soon as CTS includes these rules, OEMs have to support them to keep shipping devices with GMS (Google Play, etc.) on-board.

As a result, apps can read files on removable media using the various undocumented and unsupported tricks for finding removable media. However, apps cannot write to or otherwise modify such removable storage. Note that device manufacturers themselves may have ways of dealing with this, but ordinary app developers do not.

The android-platform Google Group has been the home of a rather epic discussion thread regarding Google’s decision here, including Dianne Hackborn’s entry linked to earlier in this post.

Despite the hand-wringing, I do not expect Google to backpedal on this issue. For every user that complains about how Android 4.4 makes their micro SD card useless, there’s a pundit complaining about how Android lets any app run amok and read stuff on storage, raising privacy and security issues. In fact, I would not be the least bit surprised if read access to removable media is blocked in the future, beyond an app’s own directory on that media.

Where Can I Learn More?

Two of the better resources for this topic are ones I linked to earlier in this post:

Dianne Hackborn’s explanation of what’s going on
Dave Smith’s explanation of how this was implemented

What We’ve Got Here Is Failure to Communicate

This episode is yet another in a continuing series of episodes that demonstrate communications breakdowns: from Google to developers, from Google to users, and from developers to users. I will explore those breakdowns in the remaining two posts.

The Rest of the Posts

The entire blog post series covers:

Internal storage, as defined by users and Google
External storage, as defined by users and Google
Removable storage, and where things have gone off the rails
Where Google went wrong with all of this
Where developers went wrong with all of this\

—Apr 09, 2014 Tweet