What is wrong with enjoying one's food or a householder enjoying their marriage bed or any of the many beautiful and wonderful things in this human life? Surely they all belong?
This is exactly what I mean by wanting to have your cake and eat it too. Phrasing the rhetorical question in terms of right and wrong is misleading. Of course, there is nothing "wrong" with sensual enjoyment per se. That is not the issue.
The issue is one of results. The realization of the Unconditioned, nibbana, is only to be had by a disengagement with the Conditioned, samsara. All the "beautiful and wonderful things" are still marked by dukkha and anicca (misery and impermanence) and engagement in even the most refined objects of the senses lead only to rebirth in the sensual desire sphere. A moral life but one still enmeshed in sensuality can, at best, lead only to the heavenly realms.
The middle path as defined in the Dhammacakkapavattana Sutta is between the ignoble extremes of sensuality and ascetism. It doesn't mean a moderate enjoyment of sensuality, but a contented non-seeking after either pleasure or pain.
The realization of nibbana only comes from disenchantment and dispassion towards the conditioned objects of sense. You simply cannot have your cake and eat it too.