The UN High Commissioner for Human Rights, Louise Arbour, said the justice system needed to address its lack of training, independence and integrity.
Her comments came at the end of a five-day visit to Cambodia.
Relations between the UN and Cambodia have been tense following earlier reports criticising the country's human rights record.
The timing of Ms Arbour's comments could not have been more pertinent, says the BBC's Guy Delauney in Phnom Penh.
The past two weeks have seen lively debate about the qualifications of Cambodia's legal officials, especially those set to preside over the trial of officials linked to the former Khmer Rouge regime.
Some of the judges only hold the equivalent of high school certificates.
Others completed their legal training in Soviet-bloc countries in the 1980s when Cambodia had a communist government, backed by Vietnam.
"The problems with the judicial branch of governance are profound," Ms Arbour said.
"Lack of professional training, insufficient guarantees of independence and lack or perception of lack of integrity are at the heart of what needs to be addressed, both by legislation and by a change of the culture."
Cambodian Prime Minister Hun Sen had earlier described UN officials in his country as long-term tourists and had accused a special envoy of being deranged.
On Thursday however, his spokesman denied reports that the government was planning to close the UN's human rights office in Cambodia.
"What the government wants is for the human rights groups to tell the truth about the work the government is doing, to show the facts and avoid just viewing Cambodia as hell," Eang Sophalleth quoted Hun Sen as saying.
In March, Hun Sen called UN envoy Yash Ghai "deranged" after he suggested too much power lay in the hands of the prime minister.
The row erupted in March when Mr Ghai said Cambodia's government was not committed to human rights, and power had been too centralised around "one individual".
Hun Sen said Mr Ghai should be sacked by UN Secretary-General Kofi Annan and dismissed UN rights staff in Cambodia as "long-term tourists".
Mr Sophalleth said Hun Sen had told Ms Arbour Mr Ghai's comments were out of order.
"The prime minister said it was not acceptable. If the prime minister does not have power, how can he lead the country?" he said.