Warning: spoilers from the Harry Potter: Wizards of Baking finale ahead.

James and Oliver Phelps, known for portraying Fred and George Weasley in the Harry Potter movie franchise, are reflecting on their experiences as the hosts of Food Network's Harry Potter: Wizards of Baking.

Speaking with PEOPLE recently, the twin brothers, 38, shared some of their favorite moments from the set of the six-episode series, which premiered last month and followed nine teams of bakers as they competed to create enchanting Harry Potter-inspired bakes. The onscreen wizarding brothers were joined by real-life chefs Carla Hall and Jozef Youssef, who served as the judges, plus a parade of Harry Potter alumni who dropped in.

Hosting the competition, the brothers shared, came with its fair share of both pressure and pleasure. In the series finale, filmed on the set of the Great Hall from the original Harry Potter movie franchise, Oliver recalled a moment before announcing the winners of the competition, where he felt the need to triple-check the names -- due to just how good both teams who made it to the finale were.

Related: Harry Potter's James Phelps Recalls the Lesson Maggie Smith 'Subtly' Taught Him During the First Film (Exclusive)

"Just before we went on set to announce the winner, we were again saying, 'So who has won?' Can we triple-check that we've got the right answer? Because you don't want to call the wrong one," he told PEOPLE. "But when they did make the final decision and we announced the winners, you kind of just realize how much it means for them to get that far. It just goes to show their talent, but also their sheer will and determination for what they do."

Zoe Burmester and Jordan Pilarski won the series with a magnificent treat that just edged out the competition. Winning the show means that the two earned the first-ever Wizards of Baking cup and will be featured in an upcoming Harry Potter-themed cookbook.

Despite the high-stakes environment for the bakers, James recalled that there were never any tense moments like he might have expected from the bakers.

"No one ever got, like, panicky and flappy or anything like that," he said. "Which I think that's where the professionalism comes in. Like, they're used to hyper-high stakes kind of thing."

While there were more serious moments on set, the pair also leaned into the personas of their Harry Potter characters, Fred and George, who own a wizarding joke and sweet shop in the original films, sharing that they consumed their fair share of sweets on the set.

"There's a reason I've just bought baggy shirts and I'm sitting down for every interview these days," Oliver joked. "Yeah, my sweet tooth was very, very sustained on this whole exercise, but it was well worth it."

James, on the other hand, made a twin-like attempt to one-up his brother, adding, "Oliver hates me telling this story because I'm a bit smug, but I forced myself to do a 5K every morning. So then, when I went to work, I could eat whatever I wanted. It was probably not the best nutritional diet I've ever done, but it definitely paid off."

Never miss a story -- sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from celebrity news to compelling human interest stories.

And, the insane skills of the bakers on the show rubbed off on the brothers during their time hosting as well.

"I considered myself a baker before the show, and then I went on and saw actual bakers do what they do, and I realized they're playing a completely different sport to me," James joked. "But when we would get the screeners for the episodes, I was very keen to get into the kitchen and try my hand at different things.

Related: James and Oliver Phelps Say Filming Wizards of Baking on the Harry Potter Set Felt Like 'Returning to High School' (Exclusive)

Hosting for the brothers did go beyond just excessive treat consumption, despite what Oliver's new wardrobe might say.

"We've always taken it with the notion that you know a good host is someone who, they're the link between the audience and the contestants and judges," Oliver said. "Hopefully, we can do some more stuff like [Wizards of Baking] in the future."

Harry Potter: Wizards of Baking is available on Food Network GO, and to stream on Discovery+ and Max.