The British American designer is leaving after three-and-a-half years to focus on his eponymous brand.