If I were you, I would first work on getting the horse out of my space. You don't need a 1000 pound animal crowding you, especially if they're spooky. They could possibly spook and run over the top of you, no matter how much they trust or respect you. It's just like a horse that totally repsects the electric fence until one day he spooks, bolts, and runs right through it. He'll be using the reactive side of his brain and won't think to see what's in his way.
I'll use four main methods to get a horse out of my space.
One: Flop your elbows up in the air. You should be able to do this without hitting your horse, and if you do hit him, oh well--it was his fault. A herd leader will similarly shove with his shoulder if another horse gets into his space.
Two: Move that front end. Pushy horses benefit a lot from moving their front end. They learn that you can push them around, but they can't push you around. Using a stick (lunging stick, training stick, dressage whip, or just a stick from your backyard) tap the air right behind your horse's ear four times. Lean over a little to show your horse you want him to do something. Use your free hand (with the rope) to block forward movement by holding it up between his eye and nose, palm facing him. If he doesn't respond, tap him gently four times right behind the ear. Make sure to keep a constant rhythm. Gradually increase the amount of pressure you use, whacking him if necessary, until he takes one step by crossing his inside foreleg (the one nearest you) overtop of his outside foreleg. Take all the pressure off, including straightening up, so he gets a reward. Then continue. You want to work to where you can push his forehand all the way around, on either side. Ask for one more step at a time and work up to this. Then, anytime your horse gets pushy, move his front end around in a full circle to show him who's boss--just like a herd leader would do.
Three: Get his feet moving. If a horse begins to push on you, you can move his front end, lunge him, or back him up. You said that you have a hard time lunging him now; I would push him away with my stick, then point and begin to ask him to move by smacking the ground really hard and clucking, blocking any movement towards you with your hand. If he doesn't move, smack his rump. When he moves, release all the cues, but if he stops, pick them back up again. He'll learn that, number one, he can't push on you, and number two, if he moves his feet when you ask him, he gets relief. It takes practice to learn how to lunge properly, and to get the muscle memory to put your hand up to keep the horse from coming towards you. I recommend practicing on a well-trained horse if at all possible.
Four: Back him up. You need to teach your horse how to back up first. Horses don't back up naturally. You don't see them backing up on their own in a field, head flexed, carrying their weight just right over their hind end. Teach your horse to back up by wiggling the rope, then swing your arms back and forth (with a stick in one hand), and walk toward the horse. Get your horse to where you can wiggle the rope and he'll back away from you. Whenever he crowds you, you can wiggle the rope and get him away from you, so he learns not to crowd you.
Never reinforce a horse crowding by giving them attention. Make them move their feet instead, so they learn that crowding means moving, and standing obediently means attention. With a little effort, you should be able to get your horse back to normal. If not, seek professional help.