LOL, sorry, sorry, I completely misstated what I was getting at. I didn't mean that galaxies, stars and brown dwarfs collapse into black holes, I meant that galaxies collapse into black holes, stars, and brown dwarfs. I'm just talking about ordinary star formation - when you get a sufficiently large and dense cloud of gas, parts of it will start collapsing into denser objects like stars. My argument is that the same processes that make baryonic matter in galaxies gravitationally collapse into stars should also apply to dark matter. The difference is that non-gravitational processes are at work preventing normal matter from collapsing into black holes, but there would be nothing equivalent supporting a similarly sized collection of dark matter. Main sequence stars are held up by photon pressure from the fusion reactions going on (caused by a combination of EM, strong and weak nuclear forces.) White dwarfs are held up but electron degeneracy pressure (again, electromagnetism at work.) And neutron stars are held up by neutron degeneracy pressure (nuclear forces.)
I think we were just using the terminology slightly differently when it comes to friction. The reason something like an accretion disk can lose energy to friction is because collisions between atoms in the disk will energize their electrons, and when when those electrons return to their ground state they emit photons. It's the emission of EM radiation that makes the disk lose energy. Those collisions and emissions are governed by electromagnetism.
So my analysis is that if you consider two clouds of gas the same mass and density, one made of hydrogen and the other of non interacting dark matter, they should both undergo gravitational collapse in roughly the same way up until the hydrogen cloud gets hot and dense enough for it to start emitting significant amounts of EM radiation, at which point the hydrogen cloud will start collapsing much faster then the dark matter. But once the hydrogen gets dense enough to start fusing, in other words becomes a star, it will stop collapsing. The dark matter, on the other hand, will just keep collapsing at its slower rate until it becomes a black hole.
I would also like to point out that the gravitational evaporation process I described early should occur faster the denser the cloud is, since higher densities mean more frequent and stronger interactions between the particles, increasing the chances of a particle being ejected. So as a cloud of dark matter undergoes this sort of process, the collapse should accelerate as it progresses.
The real question in all of this is what happens when two dark matter particles collide. Classical physics would suggest that because gravity is only ever attractive, the particles ought to collapse into a miniature black hole. Presumably this doesn't actually happen, since collisions of particles is the domain of quantum mechanics. So ultimately we're stopped again by the lack of a theory of quantum gravity.