When do you work on your olive bonsai? When it is nice and warm. Olives like to have a warm period for repotting. Not 40C mind you, but a gentle 25C summers day is great for repotting olives to get them all vigorous again.
Yellowing leaves on an olive, and a lack of vigour may mean it is time to repot your olive bonsai.