Why do we say "Oh man!" when we are let down?

Remember when, perhaps as a child, you were told to do something you didn't want to do? When the parent says "Do It Now!" you would go "oh man! and trudge off? Where did this come from?