Is it ok for me to leave my retainer out for 4 days?

I got my braces off 2 months ago and have consistently worn my retainers; however, the other day my orthodontist tightened one side of my retainer so tight that it is pushing one of my front teeth backwards and messing up my teeth. My mom called him and he said to not wear it till I see him Monday morning.....Will my teeth shift in 4 days?