Yes, knowing when to harvest limes can be a little tricky! Since limes are cold sensitive, part of the answer as to when to harvest is before a cold snap or cold weather.

And limes are typically harvested before they are truly ripe (i.e. yellow in colour), which is another reason it can be hard to know when to harvest them. The Mexican/Key limes I am familiar with are harvested when they turn light green.

Another way to know when to harvest (especially if you desire limes with a dark green colour), is to just test some out- since lime trees typically produce throughout the year (and depending where you are, they can have a couple of pulses in spring and fall), you can use a few limes as indicators, such as when a few turn yellow, then harvest the rest OR just harvest and test a couple of limes every day. Lime trees typically produce more limes that you need, so you can probably afford to harvest a lime or two when you think you are close to harvest to see if they are juicy enough for your preference (or just grow them until they are ripe and yellow!)