Despite turning down a sequel to Elf, Ferrell told PEOPLE recently the Christmas comedy will always have a place in his heart. "It's really quite special," he said. "I mean, it continues to grow ...