You've nailed it on the head when you mention the distance to get anywhere fun. For me it was at least two hours to ride to some moderately fun gravel roads towards Kinmount & Bancroft, so it was a minimum of four hours just to get to and from the fun stuff.
In contrast, the trip to Ganaraska takes me an hour each way, and I usually spend 4 hours of hard riding there before I am exhausted. The ratio of travel time to fun is just so much higher. It sounds like you're even closer to Ganaraska, so I'd highly recommend spending a day with TrailTours, as it might drastically shift your bike selection.
In contrast, the trip to Ganaraska takes me an hour each way, and I usually spend 4 hours of hard riding there before I am exhausted. The ratio of travel time to fun is just so much higher. It sounds like you're even closer to Ganaraska, so I'd highly recommend spending a day with TrailTours, as it might drastically shift your bike selection.