Covid-19 France: can gardener work at second home?

The government has made it clear that anyone can go to work (obviously banned sectors such as a non-takeaway restaurant are closed) if they can do so whilst respecting the health barrier recommendations.

Gardeners working on their own in an empty property could easily comply with the barrier requirements, keeping a metre between people and regularly washing hands.

Your gardener can refuse to work if he chooses, but there is no restriction on him not to work because you do not live there permanently.