Rising interest rates, geopolitical tensions and more cautious Chinese consumers have created a problematic market and pullback in demand that continues to spook luxury groups.