May, June, and September: warm but not hot, fewer tourists, and the Newark Festival (May) or Heritage Open Days (September) add interest.
April and October: pleasant enough for walking, hotel rates up to 30% lower than August, and the castle grounds are quiet.
Go in June or September for the best balance of decent weather, manageable crowds, and lower hotel prices.