The government has made it clear that anyone can go to work (obviously banned sectors such as a non-takeaway restaurant are closed) if they can do so whilst respecting the health barrier recommendations.
Gardeners working on their own in an empty property could easily comply with the barrier requirements, keeping a metre between people and regularly washing hands.
Your gardener can refuse to work if he chooses, but there is no restriction on him not to work because you do not live there permanently.
Stay informed:
Sign up to our free weekly e-newsletter
Subscribe to access all our online articles and receive our printed monthly newspaper The Connexion at your home. News analysis, features and practical help for English-speakers in France