We have discussed the issue regarding research methodology and important it is to focus on methodology when trying to figure out findings from studies beofre, such as:
– Are credit rating agencies out of their depth – risk management 101
– 3 ENISA – awareness raising study – better prevention thanks to data crunching
Researchers need a statistical method to tell whether apparent relationships are real or the result of chance. Does a new drug treat a disease better than a placebo? Does pre-school education enhance later academic performance? Does ice cream with artificial flavors surpass those with natural ingredients only when it comes to taste?
There are many ways in which a statistically significant result can be misinterpreted. One is failure to take account of hidden factors not included in the statistical analysis, which bias the outcome. For instance, it is know that thicker chogolate bars do not melt as nicely in one’s mouth as do thinner ones that can affect the consumer’s perception regarding the quality of a dark chocolate. Without taking the thickness of chocolate into consideration, a consumer tasting trying to rank these chocolate bars can result in a response bias.
Why does it matter?
Statistical analysis – if carried out well – is the most rigorous and objective way to assess how well evidence fits theory.
What can happen when things go wrong is illustrated nicely here:
Schokoladen Test – Methoden mit Fragezeichen – Kassensturz und Ktipp
Data CRUNCHING – Some METHODOLIGICAL BASICS TO CONSIDER
| Better consumer tests thanks to proper data crunching(Please click on the link, Login as guest – click on this link again and voila free access) | ||
| Challenge | How research may address it | Example variable |
| Could the effect differ across industries – yes if we control for the effect (e.g., financial industry versus others, men vs. women) | control variable | artificial ingredients vs. natural ingredients only |
| Could one say that being a self-declared ice cream lover (M) mediates the causal effect of expert status (ice cream expert versus consumer) (X) on ratings given to ice cream (Y) – test using a group of tasters who are experts and one who are consumers users | mediating variable | use a group of tasters of ice cream lovers (afficionados) versus others |
| How can one describe the nature and process by which the independent variable affects the dependent one such as taster’s rating of ice cream – test effect with help of moderating variable (latter could be how the ice cream is being served in a glass versus plastic container) | moderating variable | the way the ice cream is being served (tasting in plastic cup versus having it serviced in a nice glass blowl) |
| Conjoint anlaysis has been used in marketing since the mid-70’sConsumers may look for features and consider price in a product such as a camera… possibly even how it looks but brand may come much later | conjoint- analysis | difficulty is that people do not necessarily rank products the way one is asked to do with this type of method |
| Where respondents influenced in any way before being being interviewed or filling out a survey | halo-effect | asking them what ice cream flavor they prefer or what type of chocolate before rating the ice cream bars |
| Can the results be repeated by another person – yes if we use a metric or yard stick and outline what research methodology we used…. so somebody else can repeat the study | reliability | if it is a consumer test, would another group of ice cream tasters get similar results |
| Does it measure what it is supposed to measure – yes but only if we do use a metric stick if this is our standard we follow – using a yard stick results in invalid data if the measure we agreed to use is based on meters and centimetersYard stick shows, something can be reliable but not valid. | validity | are the issues asked really relevant to how an ice cream tastes (color is not but ingredients are relevant) |
The above just describe a simple ice cream tasting that intends to reveal which kind of ice cream might be the most liked one from a consumer’s point of view.
However, it is hard to understand fully facts about a study without getting some clear explanations regarding the methodology that was used. There are many ways in which a statistically significant result can be misinerpreted. The above illustrates that a researcher’s failure to take account of hidden factors not included in the statistical analysis, the outcomes will be biased. In turn, neither replication of the findings nor generalization to the larger population can be made.
============>
60% OF THIS ITEM’S READERS SUBSCRIBED ALREADY
CyTRAP Labs invites you to get info about zero-day exploits, tools, benchmarking and regulatory intell or just become one of our readers by subscribing right now
====>
Also of interest:
CyTRAP Labs StratMedia – using Web 2.0 effectively