The owner of Tommy Bahama, Lilly Pulitzer and Johnny Was lowered its guidance for fiscal 2024.