​EA20, reviewing and language tests

Language tests as a tool in asylum investigations were questioned in Swedish Television show “Uppdrag Granskning” broadcasted 4^th March 2015. Cases at the Swedish Migration Agency, including cases where the questioned analyst ea20 had been involved, were scrutinized.

The quality requirements used by the Swedish Migration Agency for language tests may not be complete, but they certainly stipulate qualifications of the analysts.

”…the majority of the analysts holding a degree from a university and preferably within linguistics or languages.”
(Swedish Migration Agency: Invitation to Tender 3.2.1.-2014-26959)

This requirement seems not to be met by the current supplier of language tests.

This does not rule out, however, that the outcomes of the test are nevertheless acceptable. As a consequence of the alleged shortcomings of analyst ea20, the Swedish Migration Agency commissioned a third party review of language tests involving the Somali language.

Likelihood of errors

For the third party review, the supplier was asked to select 10 cases. The audio recordings were then independently analysed by the corresponding authority in the Netherlands, thus enabling a comparison of the outcome. But what does a test of similarity in outcome with a sample size of 10 show?

One way to express it, is by estimating the probability that no error will be found in the sample.

For instance: If there is, on average, 1 error per 50 cases, then the likelihood of the sample failing to contain a false conclusion (disregarding whether the test will detect the false conclusion or not) is a staggering 82%. Clearly the test is designed to react only to a much higher concentration of errors. 1 error out of 20 still gives a 60% likelihood of the sample not containing an error to be detected and 1 out of 10 gives 35% likelihood. At 1 error in 5 there is 11% likelihood the test in spite of this will come out clean.

Methodology

- There is a trade-off between the efforts spent to obtain the sample and the concentration of errors that can be detected with any reasonably likelihood. This test seem unfit at least for error concentrations under 15 or 20 percent, says Lars Johan Lundberg, Director Research & Development at linguistic consultancy Verified.

In order to properly assess the accuracy it is important to also look at the methodology. By validating components and see how they are brought together in the method will give a fair picture of its strengths and limitations – when it can be useful and when not.

Lars Johan Lundberg emphasizes that the conclusion must be built on clearly indicated observations and that such observations must be made by a native speaker of the linguistic variety narrowly defined and that each observation must be an instance of a discriminating trait with references to the exact place in academic literature where the phenomenon is described.

Can a language test be trusted when the origin of an asylum claimant shall be determined? If so, how shall they be used? Anders Eriksson, Emeritus Professor of Phonetics:

- Let us first put language tests in its correct legal context. The result of a language test is a form of expert evidence. It serves the same purpose as other types of expert evidence – DNA, finger prints, ballistics etc. – namely to increase the possibility of reaching a legally secure decision by supplying a potentially crucial piece of evidence, Anders Eriksson says.

- A criticism sometimes heard is that evidence that is not totally reliable should not be used. But this is to throw out the baby with the bath water. It is true, of course, that language testing is not 100 percent reliable, but the same holds true for all other types of expert evidence including DNA, and applying a requirement of complete reliability would mean that no expert evidence at all would be admissible. In what way legal rights would benefit from such a principle I leave for the proponents of this type of criticism to explain.

Validity of language testing

- The TV show presents the view that the scientific community almost unanimously reject the validity of language testing. Such a claim is totally unfounded. Language testing is part of forensic speech science and is a form of forensic linguistics that is accepted by those who are professionally active in the field, Anders Eriksson says.

- There are two international organisations representing practitioners and researchers in the field – International Association for Forensic Phonetics and Acoustics (IAFPA) and the International Association of Forensic Linguistics (IAFLA). Both organisations accept members who work with language testing and there is simply nothing controversial about that.

- There is a debate going on between some members of the two organisations on how to best conduct language tests, but that does not mean a rejection of language testing as such. So the answer from the part of the scientific community professionally active in the field is the complete opposite of what was suggested in the TV show “Uppdrag granskning”.

- I am personally a member of IAFPA since many years and have been part of a committee examining language testing. This led to a position statement, now the official policy of the IAFPA concerning the subject, and there is not a word in that statement suggesting the rejection of language testing, Anders Eriksson says.

For further information, please contact:

Per Lunqe, Press Officer

per.lunqe@live.se
+46 (0)735 106010