It would be harder maintaining a large collection of software outside of one of the major Linux distros. Debian/kFreeBSD might have a chance thanks to the GNU userland, but the archive is less well supported than with the Linux kernel.
Quite the opposite. This isn't about the userland, it's about the kernel. The linux kernel is where the high-performance cluster computing advances have been. The GNU userland is just a clone of the BSD userland, nothing special there. Moreover, FreeBSD and OpenBSD's packages are diverse, well-maintained, and current. But neither OS's userland nor any of their packaging systems have anything to do with high-performance cluster computing.
I may be wrong but they probably have a support contract with RHEL or similiar or they are using a sponsored distribution that caters to their specific needs such as Scientific Linux. I don't know if similiar projects/teams exist for BSD distros.
I doubt CERN has a formal run of the mill support contract with RH like other large organizations. Scientific Linux is developed (in-house) in partnership with Fermilab.[1]
For offical usage in the context of Oracle production databases and eventually other applications requiring commercial support there is a limited number of Red Hat Enterprise Linux 6 Server, Red Hat Enterprise Linux 5 Server licenses available at CERN. For all other tasks standard CERN Linux distribution: Scientific Linux CERN shall be used.
BSD kernels generally do not have such good support for hard real time applications. It's been done though, so I don't know if it would have a real impact.
No need for hard real time, first level detector output goes through FPGAs which only pass potentially interesting data to the Linux powered systems which do further data filtering.
