Do you want it easy? Or do you want it secure?
One of the reasons things have to change so drastically on the security-front is because of Unix-style user and group permissions, and the file system of your SD card. You expect an SD card to just work when you plug it into a computer. Any computer. Because something like 96 percent of all the computers out there that aren't some sort of server run Microsoft Windows, you need to make sure your SD card will work with them.
If you want removable storage that can be read anywhere, understand that it's not going to be secure.
FAT-based file systems — that's what your SD card is formatted to — are universal on Windows, Apple OSX and (most) Linux machines. Makes sense. What good is a portable drive that can't be read, right? But there's a catch. FAT-based systems don't support file and folder permissions. If I'm an app, I can't say "This is my folder. I will allow you to look inside, but you are not to put anything new in there or change any of my stuff!" on this sort of file system. It's an all-or-nothing mess, and one that even Microsoft has moved away from in its newer versions of Windows. But FAT still needs to be used, because anything else will require a lot of finagling for people using legacy versions of Windows, and that is a large number of people.
Note that this has nothing to do with accessing your phone storage — internal or external — via a cable attached to your computer. That uses a completely different protocol and method, that has nothing to do with the actual file system of a removable piece of media.
If SD cards were set up with the same file system as the rest of the partitions inside your Android (EXT, for those of you keeping track at home), none of this would matter. Third-party apps could be given permission to individual folders, other folders could be marked "hands-off," and everyone would be happy. But they are not, and can not be without modification of a lot of computers, or a stand-alone program you run on your computer just to access the SD card you pull out of your Android.
This is a mess that nobody can fix right now.