I just had this problem in a different tagset - the problem is that straight and curly apostrophes are different characters, so they make different fandom tags. You'll have to delete the version of the fandom with fewer tags under it and then re-associate all those relationships to the other version of the fandom in either the Edit screen or as a batch load.