Olympics announcer breaks silence after being removed from coverage for comment about Australian women's swimming team