Small designers feel particularly vulnerable, although bigger brands, like those that work with Andrew Rosen, would also feel the pinch.