One Jesse Eisinger considers this question at Market Movers. Since the credit derivatives business is essentially an insurance business as pointed out by Eisinger, we have to ask how skilled are the insurers(swap sellers) at underwriting(doing due diligence) on the asset they are insuring; and whether the insurance sellers keep any minimum amount of assets in storage to allow them make good on their policies(swaps). My belief (and Eisinger's) is that the answer to question one is "not very" and the answer to question number two is "probably not".
So if a particular credit defaults and the swap seller can't make good on its policy, the swap(insurance buyer) loses both the cash they paid for the insurance plus the cash lent to the borrower. It is as if a Florida resident bought a house and storm insurance, and a hurricane comes through, blows their house down, and the insurance company sold so many policies that its claims exceed the value of the assets it has been holding in anticipation of claims. The resident loses the policy payments they had made plus their house; and are probably still stuck paying the mortgage on a now nonexistent house. That's a triple whammy!