The brand got its start in shorts and has since expanded into swimwear, polos, pants and an NFL license.