Creating a government department has a certain amount of overhead attached to it. You only create such a thing if the nature of the mission is separate enough and there is enough useful work that would be improved by the bureaucratic separation so that the improvement outweighs the overhead increases from the separate bureaucracy.
That's fine as good government theory but how do we know that the departments that exist now are fulfilling that requirement? The truth is we don't. We don't measure the good that is being done in any rigorous way and how much that good costs in money, time, and opportunities. We don't routinely measure whether a separate existence serves us better or whether some of these institutions would be better folded back into the department that they calved off from years ago. This sets up a situation where growth of government is favored even when it doesn't actually make sense. Inertia carries useless departments forward, year after year.
This insight is at the heart of "shut it down" and "defund" conservatism. The idea is get rid of the rotten parts and the useful parts will tend to be salvaged and absorbed by other departments. We don't have to go through the hard work of sorting the wheat from the chaff ahead of time. And that last bit is exactly where conventional conservatism goes wrong.
We do need to go through that hard work. We need to make going through that work cheaper and easier. We need to do that work routinely to establish an ethic of getting rid of government we no longer need. Until we do, conservatism will always be an exercise in failing slowly.