Is my music teacher too interested in me?

Yeah, I think you're right to be concerned. Next time he keeps you back, tell him flat out that he's making you uncomfortable with his attention and that he especially shouldn't touch you again, then leave. If his intentions are good (or even if they're not but he has any sense of self-preservation), that should be enough to get him to back off. If he doesn't, then go talk to the principal about it. You might not want the fuss, but if he tries this on you, he might try it on someone who's even less able to handle the attention and might get hurt.