Yes, there are issues with the Coast Guard, the Army Corps of Engineers and even various state-directed national guard units. I don't think you will be able to find a perfect agency, anywhere.
As for "At which point it would not be FAA except in name," that is my point. It really does need a break from the existing FAA structure. But, where to put it? None of the 3 groups you listed knows anything about the issues involved in space launches, other than clearing the areas that somebody has determined might get dangerous. How to prevent those dangerous things from happening is not in their areas of expertise.
So, I think the proper agency culture will need to be built from the ground up. But, that won't start with educating kindergarten children - it will begin with transfers and hiring. The important point is to get a good architect for building the program, so an appropriate culture and expertise assortment can be acquired and applied with an appropriate fresh mandate.
The regulatory agency that I once worked for had a lot of "rules of thumb" as well as archaic regulations that I was involved with updating and replacing to more realistically address risks and avoid pursuing trivia. I have had almost zero involvement with the FAA, but reading the news stories indicates to me that they have some similar problems with addressing the realities of space program and facility development.