Hello,

One week ago, I posted (here) asking for recommendations for the upcoming Webinar and some of you recommended the topic of questionnaire development. Further, I posted the suggestions to my YouTube channel community, and still, the highest recommendation is the questionnaire development dilemma (voting here). So I decided to discuss it briefly in this blog post.

The commonest misconception I found in the medical community is that some people consider the case record form (CRF) of ANY clinical research study as a questionnaire. This is NOT true. There is a big difference.

A CRF is a form that includes all demographic, clinical, laboratory, and imaging variables that you will collect from the patient for a clinical study - all variables in one place as a hard copy (called CRF) or electronic format (eCRF). People who work in the operation department of pharmaceutical companies and contract research organizations are more likely to use this term (it is less used in academia).

A questionnaire is a set of questions that are scored as a psychometric measure for psychological variables and other variables that are felt but can not be seen. 

My mom is a super talented chef (this is a fact, not a compliment; she never reads these posts anyway), but I still find the Pizza of PizzaHut more delicious than the home-made ones, and there is nothing personal about it. Every time someone speaks to me about their *revolutionary* new home-made questionnaire, I remember this Pizza thing because 

Needless to say that if there is no standard reliable validated questionnaire, then you are doing a good service to your patients and to the research community by try to develop a new one.

First, you can not develop a questionnaire alone. A team of physicians and researchers with long-standing expertise in evaluating this condition develop the questions. The team should include a psychometrician.  

Construct validity is the extent to which the questionnaire measures the theoretical or the psychological outcome of interest. To establish the construct validity, you need to run the principal component analysis (abbreviated as PCA) and the factor analysis (sometimes called confirmatory ... or CFA). The minimum number required to conduct this analysis is 60 responses. This analysis allows you to investigate how much correlation exists between the different questions, how many psychological factors play a role in the responses of these questions, and whether one question is likely to be unrelated to the outcome of interest. All these advanced calculations are done by statistical software (I briefly explained this here).

If the questionnaire will be used to diagnose a condition (i.e. anxiety), then it is important to plot the sensitivity and 1-specificity to determine the optimal cut-off value at-which the questionnaire performs the best in classifying the participants. The receiver operating characteristics curve (ROC curve) is used for this purpose, it shows the trade-off between sensitivity (or TPR) and specificity (1 – FPR). Usually, Youden's index is used to determine the optimal cut off value (I briefly explained this here).

Accuracy is another different form of validity. It is a quantitative measure of how the questionnaire is accurate in ruling in and ruling out the condition of interest. This can be done by calculating the overall accuracy, sensitivity, and specificity against the gold-standard evaluation. PPV, NPV, +LR, and -LR is less common and less useful in questionnaires (I discussed this in detail in more than 5 papers on PubMed - look for them if you want).

There are many levels of reliability that should exist in the questionnaire:

(A) Internal reliability
Questions should have the same direction. This internal reliability is measured by Cronbach's alpha. This internal reliability should be no less than 80% (the higher the better). Further, if the internal reliability is less than the 80% threshold, you can perform sensitivity analysis to detect which questions should be omitted to fix these reliability issues and you can change the questionnaire accordingly (I also mentioned this briefly with an example here).

(B) Test-retest reliability
Reliability here means that every time you ask the same questions you get the same answers irrespective of how much these answers are accurate (the accuracy is a different parameter as I mentioned above). In order to investigate the test-retest reliability, it is advised that a set of pilot responders answer the same questionnaire on two different occasions separated by a two-week period to allow the responders to forget their responses to the initial questionnaire. These data are analyzed similarly to the "inter-rater reliability analysis" of clinical scores. To clarify more, you consider these two sets of responses from the same patients as two different raters, and you calculate the Cohen's Kappa test of the agreement to determine your K value (the higher the better). I'm not aware of a minimum threshold for that and have not read a specific guideline but a questionnaire sounds reliable for me if the K is above 0.6.  

A good questionnaire should be socially acceptable, feasible to answer, and easy and simple to understand. For this purpose, adopting a standard questionnaire to your community might requires TRANSLATION (to overcome the language barrier and make it easy to understand) and CULTURE ADAPTATION (to make it socially acceptable and feasible to answer). A major problem here is that if you introduced substantial changes to the questionnaire itself, it might lose its reliability. Therefore, if you plan to adopt a questionnaire and translate it to your own language for the research purpose, you have to do the following:

It is also recommended that:

1- First, you publish the processes that you did to make the questionnaire ready for use. For example, if you have developed a new questionnaire from scratch, you should first publish the "development & validation" process as an article. Similarly, if you have translated the questionnaire, you should publish the "translation & reliability assessment" process as a research article before you publish any results you measured by this questionnaire.

2- If the questionnaire, you use, is not valid or reliable, no one can count on or trust your findings. Therefore, the first thing you should do is to validate the tool and declares its validity and reliability to the scientific community, then you can start using it and publish whatever you want by applying it to your population.

3- Data of the pilot phases MUST BE EXCLUDED from the final analysis of your work.

I realize this post is a bit long, but the topic is very important and interesting. I hope this helps. Please, let me know in the comments if you suggest that I write about any other topics of interest. 

Thank you ... AN