Patients who receive care in a specialist unit have a 50% reduced risk of death and better long-term recovery.
National guidelines published in 2001 recommend all patients are treated in a specialist unit but similar guidelines were not introduced in Wales until 2006.
The College said there had been significant improvements but some hospitals had failed to recognise that stroke patients needed "21st century care".
The audit, funded by the Healthcare Commission, found the number of eligible hospitals in England with stroke units increased from 82% in 2004 to 97% in 2006.
But only 28% of patients in Wales were treated in a stroke unit compared with 64% in England and 73% in Northern Ireland.
Only 15% of patients are admitted to a stroke unit on the day they arrive at hospital.
And although brain scans are needed to determine treatment, patients admitted on a weekend often have to wait until the next working day.
Those with minor strokes who are in hospital for less than two days are least likely to have access to specialist services.
Dr Tony Rudd, chair of the Intercollegiate Stroke Network, said more than 90% of patients should be treated in stroke units.
"And not to have done imaging by 24 hours is not really good enough. We need to differentiate between bleeds and blocked arteries."