the layer of fat only really makes up like 1" of your mid section. Your "stomach" includes all your abdominal muscles, as well as the actual organs used in digesting food etc.
In effect your abdominal muscles act like a 'bra' holding everything up, supporting everything and make it look good. When they're weak, everything is sagging and it doesnt look flattering.
If you dont like the posture of your mid section(sagging etc), doing core strengthening exercises will do a lot to help with that.
More importantly you'll never have back pain/leg/sciatica and your riding stamina will improve.
(I devote a significant amount of time torturing my core muscles on a regular basis and now have a titanium core)