Chris Columbus’ children are the ones who got him into the “Harry Potter” movies – and now they’re getting him out.

After directing last year’s “Harry Potter and the Sorcerer’s Stone” and the second film in the series, “Harry Potter and the Chamber of Secrets,” Columbus will step aside for No. 3 to spend more time with his wife and their children, aged 13, 10, 8 and 5.

“My daughter, Eleanor, was the person who forced me to read the books and told me it would make a great movie,” the 44-year-old filmmaker said. “And then I realized after 2 1/2 years of not seeing them for dinner during the week that I wanted to take them to school in the morning and see them for dinner.”

Columbus will serve as a producer on “Harry Potter and the Prisoner of Azkaban,” due in theaters in June 2004. But he concedes it’ll be harder to watch director Alfonso Cuaron take control of the third movie than it would have been to direct it himself.

“To keep my mouth shut is a hard thing for me to do, but I’m gonna have to,” Columbus said. “I have to give Alfonso his space to make the film he needs to make.”

While Columbus recognizes that he’s best known now for adapting J.K. Rowling’s “Harry Potter” book series, he’d already made his name directing the “Home Alone” movies. Before that, he wrote the scripts for “Gremlins” and “The Goonies.”

Chris Columbus is the director of the first two Harry Potter movies. Columbus won't be in the director's chair for the third film, although he will remain as a producer.

But audiences and critics alike have responded with groans to some of Columbus’ movies, including “Nine Months,” “Bicentennial Man,” and his first, “Adventures in Babysitting” :quot; complaining that they’re too sappy. Ever the optimist, Columbus said such bombs make a person stronger.

“Unfortunately, you need those failures, desperately,” he said. “You get them anyway because you will sort of get comfortable in a sense with your success and something will happen that will trigger either a failure or a misstep, and then you need to bounce back from it.”

From here, Columbus plans to write another original screenplay, which he hasn’t done since 1991’s “Only the Lonely.” Based on his love of rock ‘n’ roll, he said he’d love to do a musical. And he hasn’t ruled out the possibility of directing the fourth film in the series, “Harry Potter and the Goblet of Fire.”

“I have to sit down with a yellow pad and a pencil and say to myself, ‘OK, what am I going to write?”‘ he said. “That’s actually probably more terrifying than walking onto a set with 350 people waiting for me to tell them what to do.”