The movie explains that the curse means everyone living in the town has no memory of the prince and his servants, but there is no clear explanation as to why they don't age despite time passing. Until now.
A reddit fan theory explains that the village is stuck in a time loop, just like Groundhog Day.
"The village of Villaneuve is caught in the Enchantress' curse as well," the user explains. "It's stated outright that she erased the memories of the Prince, his lands, and servants from the villagers' minds, but if you pay attention she also locked the village in a time loop."
And why aren't Belle and her father Maurice, seen as outcasts, affected? They moved to the village after the curse was placed.
Everyone is living the same day over and over. Makes sense with Belle's opening song too... pay attention.