It's not that easy. There are parts of the 'market' that are paid for by user, and parts that are not. In some cases, infrastructure in particular, it makes sense to provide it so economic activity can happen. Without roads, ports, railways etc, economic activity is challenged or locked up.
Look at the pipeline issues out West, or the lack of a transport system through the mineral-rich 'ring of fire' in Ontario. Users can't build that type of infrastructure, nor will they pay for it without enormously expensive guarantees, so the economic opportunity is locked. Build a pipeline, port, rail or road so goods can move, and the supplemental economic benefits pay for the absent user fee.
If you think user fees are a great idea, let's sell high-traffic Toronto bicycle lanes to the 407 Corp.