Realistic Enough to Learn, Distant Enough to Protect