Half of the 10,000 pupils in the study were put in sets based on their ability - measured by test results at age 11.
The other half were grouped on a number of other issues, predominantly social class, an educational conference heard.
The study matters because earlier research suggests that pupils who are put in too low a set for their ability get worse GCSE results than those in the right set.
The findings will come as a blow to ministers who are keen to ensure that all pupils, no matter what their social background, are given an equal chance of succeeding.
A spokesman for the Department for Children, Schools and Families said the study it commissioned had only looked at a small number of schools and was not representative of the national picture.
The joint research team from the Universities of Manchester and Sussex aimed to look at how pupils in low attainment groups could best be taught.
Professor Judy Sebba said: "What we found was that school teachers, and in some cases pupils, think that they are set on the basis of ability or prior attainment - usually based on Key Stage 2 results.
