WASHINGTON (AP) — High-profile supporters of President Donald Trump are turning on special counsel Robert Mueller, the man charged with investigating Russian interference in the U.S. election and possible collusion with Trump’s campaign.

As Mueller builds his legal team, Trump’s allies have begun raising questions about the former FBI director’s impartiality, suggesting he cannot be trusted to lead the probe. The comments come amid increasing frustration at the White House and among Trump supporters that the investigation will overshadow the president’s agenda for months to come — a prospect that has Democrats salivating.

Trump friend Chris Ruddy, the CEO of Newsmax, went so far as to suggest the president was already thinking about “terminating” Mueller from his position as special counsel.

“I think he’s considering perhaps terminating the special counsel,” Ruddy said in an interview with Judy Woodruff of “PBS NewsHour.” ″I think he’s weighing that option.”

Former House Speaker Newt Gingrich said Tuesday that Trump called him Monday night and that he doesn’t think the president will seek Mueller’s dismissal.

In an interview Tuesday morning on CBS, Gingrich said Trump is confident that Mueller will find he did nothing wrong. Gingrich, however, worries. Trump called because “I have been very clear” about Mueller and the lawyers he’s hired amounting to a “rigged game,” Gingrich said.

“It’s a mistake to pretend that this is going to be some neutral investigation,” Gingrich said.

Under current Justice Department regulations, firing Mueller would have to be done by deputy attorney general Rod Rosenstein, not the president— though those regulations could theoretically be set aside.

Rosenstein may be asked to address the issue when he speaks at a Senate subcommittee hearing Tuesday morning.

Attorney General Jeff Sessions recused himself from all matters having to do with the Trump-Russia investigation because of his own conversations with Russian officials during the Trump transition.

Rep. Adam Schiff, the top Democrat on the House Intelligence committee, said such a move would “be the last straw” for many in Congress and would have “echoes of Watergate,” when President Richard Nixon dismissed special prosecutor Archibald Cox over Cox’s subpoenas for White House tapes.

Schiff told MSNBC’s “Morning Joe” that “I don’t think the Congress would sit still and allow the president to pick his own investigator.”