In praise of context: A little background on that "secret government survey" of successful Tamil refugees

It would be fair, I think, to describe my first reaction to the story that was front page news throughout the Sun chain today as somewhat -- brow-furrow-y. Not, I should note, over the "BOGUS REFUGEES" headline plastered over a picture of Tamil migrants disembarking from the Sun Sea, but specifically, the apparent source for that particular claim: 

A secret government survey reveals the majority of successful Tamil refugees travel back to Sri Lanka, raising questions about the legitimacy of their refugee status ... The survey of Sri Lankan nationals was conducted in early August. A total of 50 people were surveyed, 31 of them had successfully obtained refugee status and 22 had returned to Sri Lanka

First of all, what exactly is a "secret government survey"? Does that mean a poll? A focus group? A voluntary questionnaire? Were respondents randomly selected from the total Tamil immigrant population - not that I'm sure how, exactly, one would do that -- and then weighted so as to be representative of the whole? How was it conducted -- and by whom? What was the margin of error?

According to the Sun, CBSA refused to release the full study, although it did confirm the numbers obtained by QMI. The story did not, alas, provide any further detail on the methodology, so I sent a query off to CBSA to see if they'd be willing to give me a little more information. Less than two hours later, I received the following response:

Per your request: travel patterns of Sri Lankans returning to their homeland after being granted refugee status in Canada, we can confirm that a small sampling was done (50 persons) and here are the results:

Of the 50 files reviewed the following information was identified:

*Number of sponsors not previously deemed Convention Refugees - 19
*Number of sponsors formerly deemed Convention Refugees - 31
*Within the aforementioned subset of 31 sponsors formerly deemed Convention Refugees who, according to the FC1 applicant, had subsequently returned to Sri Lanka - 22

As this review was done based on statements applicants made during their immigration interview and not based on any formal entry control system, this information must be considered anecdotal in nature.

Numbers were counted within the sample group and no statistical modeling or statistical process was applied. As such, CBSA has no ability to state that this sample is statistically representative of any pattern that may or may not exist beyond this sample. Rather, these numbers are solely indicative of an observed pattern within the small sample reviewed.

Since there has been no analysis beyond what was on the immigration file in Colombo, these findings should be considered as context only.

Additionally based on the size and statistical validity of this sample, it should be noted that no conclusions should be reached on the comments/observations made on this file.

So it seems that this was not, in fact, a "secret government survey," but a review of a sampling of files from Sri Lankan nationals, some of whom were formerly found to be Convention refugees, who now want to sponsor family members to come to Canada. No weighting was done, and no claim has been made that the findings are representative of the total population. In fact, as far as CBSA is concerned, it "must be considered anecdotal in nature."