The owner of Tommy Bahama and Lilly Pulitzer also pulled down its projections for the year.