Siddhartha Gotama returns to the palace at night, when his wife and young son were sleeping. He was married and had a young infant son.
He had decided he would leave the palace in the middle of the night starting his journey to a better understanding of life and how to help people escape their misery.
He did not understand his mind as much as he would later in life, once he attained Enlightenment but, he understood it enough that he felt if he had woken up his wife and young son to say good-bye, he would not be able to leave. He thought it would be too tough for him and he might give up on his pursuit to help people out of misery and despair, so he leaves in the middle of the night.