I started Nova last year as a 3 yr old. In late spring/summer he went to some lessons with me and I worked a lot at the walk and trot and got some steering and brakes installed (mostly). He had part of the summer off as I focused on our other gelding, but in September I started with him again. He was put into steady work from Sept. - Nov. By steady, I mean usually 1-2 lessons a week, with probably 1-2 other 'trail' rides during the week. Most of our riding is road riding so he got exposed to a lot of things - trash cans, cars, bikes, dogs running out, people, etc. During that period in fall we also introduced the canter to him - which he took to easily. I just want to say that while our lessons were about an hour long, we weren't working for the full hour, and he always had some breaks where he got to hang out (and fall asleep almost) while someone else was working.
Nova then got all winter off (Nov. - April) with just two short rides during that time. The weather this year hasn't been very nice, so he hasn't been worked as steadily as I would have liked. But, when I got on him again for our first lesson, he was exactly where I had left him last year. We have been making steady progress in each lesson. Last week I took him to the mountains and we did a week of trail riding there. He was awesome! We didn't put in killer days, but he probably used muscles he didn't know he had! He had to walk through puddles, through mud, across creeks and rivers, climb hills, go down some relatively steep hills, watch where he put his feet (both up and down), and just generally be ridden for 4+ hours every day.
I think that this particular senario worked out well for Nova. Other horses it might not, but he felt really mature underneath me (both mentally and physically) and he has taken everything in stride. Of course, he's a pretty easy-going guy...
ETA: Whoops, didn't answer the question really! I wouldn't start a horse before 3, and would wait until 4 if they didn't seem ready.