After Hermione and Penelope Clearwater are petrified, Harry and Ron go to Hagrid’s hut and watch from under the invisibility cloak as Cornelius Fudge and Lucius Malfoy serve the orders (CS14).
Remember that Hagrid had been expelled from Hogwarts because Tom Riddle led people to believe Hagrid was responsible for the monster attacks, and that Hagrid was allowed to stay at Hogwarts and be trained as gamekeeper because of Dumbledore. Without Dumbledore there to protect him, it would have been much easier to take Hagrid away. That likely factored into the decision to remove Dumbledore. -BB