Not on a horse, but I basically have to mountain bike alone. I stick to "open space preserves" that are pretty well traveled by bikers, hikers, and equestrians on the weekends so I'm never really totally out in the woods "alone". I did bike alone in much more remote areas when I was an undergraduate as the college grounds provided GREAT trails, but were rarely traveled (well documented and well known by the college safety personal however).
Suggestions:
Always let someone know where you are going and when you expect to be back. This person should have at least a general idea of the trails/direction you are heading on and at most an actual map of your route. They should also have a time to call for help if they haven't heard from you. I tended to always be within a few minutes of cell phone signal, so I always made this time a close call because if I wasn't quite home yet I could just give a quick "running a little behind, 10min out" call if I needed to. If you don't have cell signal you might want to give a little bit of buffer so they aren't sending out search parties just because you stopped to take some photos ;)
Carry your legal id, insurance information, credit card, $20 cash, and your cell phone (with emergency contacts labeled in it) ON YOUR PERSON. If you have any allergies or medical problems have a list of those on you as well. This gives emergency personal enough info to get you in a helicopter and to a hospital, and gives you some means of helping get yourself out of trouble if you get lost, etc.
Carry at least a rudimentary first aid kit for you and the horse (gauze, padding, and wraps for bandaging wounds and stopping bleeding are a minimum) and a rudimentary tack repair kit and a knife. On this vein a basic first aid class might not be a bad idea, but some basic research on what to put in a kit and how to use it is probably more than sufficient (I'm wilderness first aid qualified; ie: overkill, but I also find myself backpacking 5-6 day's walk from ANY cell coverage or help!!! (see sig) ;) ).
Educate yourself as much as possible about the area you are riding. If you can get a map keep it with you.
Carry more water than you need, some food (high carb, high sugar), and if anything overdress in layers in case you get lost/stranded and end up out longer than expected (LED headlamp and "Space blanket" thrown in a saddle bag aren't bad ideas either and take up virtually no space).
Always wear a helmet!
Consider carrying pepper spray or other defense if you are in an area where you expect there may be a chance of someone/something messing with you, but KNOW how to use it and know the risks! Have your horse trained to deal with it if that might be an issue. (this is really a low probability scenario as most wild animals are not going to mess with a healthy horse and human and unless you are in an area known for human issues that has a small chance as well).
Most (all?) modern cell phones have 911 GPS that's based off of satellites, some you can even set to allow family and friends to view coordinates as well. If you get reasonable signal where you are going you are probably covered by that. If not you can consider the emergency GPS locators, but I believe they are still VERY expensive at this point, and probably overkill in any sort of non-backcountry area as they are marketed to those that are hours or maybe days from any sort of cell phone signal or other help (if you follow a powerline road and let someone know roughly where you are they'll be able to find you pretty quick without such advanced systems).
I don't ride/hike/bike during hunting season if it is allowed ANYWHERE NEAR where I am riding. It's just not worth the risk to me. Wearing visible clothing is a good idea however just in case you are thrown off the trail. If you are riding during hunting season make sure your horse is also outfitted with dayglow orange (they make saddle pads, quarter sheets, bridles, etc for this).
Marking your name and phone number (and another emergency contact number, perhaps your barn or vet) on the cheekpiece of your bridle, halter, saddle or all three might be a good idea as well just in case you and the horse are separated. That way someone that finds the horse knows what to do to get him safely back to you.
Follow your gut. If you aren't comfortable don't do it (that includes going out at all. There are days when I just don't feel quite right and I don't get on the bike and head to the woods, period. I get on the roadie instead and do a few laps of the very safe base I live on.)! I'm ALWAYS more careful when alone and all my falls have come while in groups, but horses are less predictable than my bike, so listen to what your instincts are telling you :)
Even if you go in a group you should follow the above suggestions; it's possible that something could happen leaving you both/all in some sort of trouble.
Have fun :) This sounds like a lot of info, and maybe a bit overwelming. Really now that I'm set up it takes me maybe 30-40 minutes total to get ready to go for a ride (bike) and I don't have to think much of it. I haven't had to use more than bandaids out of the first aid kit, but it sure makes me feel better knowing it's there! :)
