May and September to early October: warm enough for swimming, fewer crowds than July–August, and accommodation is easier to book.
May and September: hotel rates drop 20–30% from peak, weather is still good for swimming, and the harbourfront bars are busy but…
Go in May or September for the best balance of good swimming weather and manageable crowds.