May (spring, mild, cherry blossoms), September (autumn colours, fewer tourists), October (crisp air, harvest festivals).
April and November: hotels drop 30-50% off peak rates, weather is transitional but manageable, fewer people.
Go in September for the best balance of clear weather, autumn foliage, good food, and reasonable prices.