Regulators are trying to get a handle on offshore reinsurance once investment firms took an ownership stake and deals boomed.