Validity without a Gold Standard

This study is closed to enrollment.

To view the Final Progress Report, Click Here.

To view a Comparative Study of Psychiatric Diagnostic Interviews paper, Click Here.

To view Bayesian semi-parametric ROC Analysis paper, Click Here.

Purpose of the study

This proposal will examine the usefulness of mental health screening instruments for children and adolescents. We focus on the validity of screens because they are much more widely used in the real world than are full psychiatric interviews. There is also little work has been done to validate them. This application has three phases. The aims of Phase 1 are to assemble and analyze a representative collection of data sets relevant to the validity of screens for child psychiatric disorder. Here validity is interpreted as screening efficiency relative to a structured psychiatric interview. Using statistical methods developed for cancer research where there is no agreed gold standard for diagnosis, we will construct a “multi-screen/multi-interview” matrix that will provide measures of the efficiency of a range of screens, relative to one or more interviews, in various settings (epidemiological, primary care, school, clinic, juvenile justice, etc.), and where possible by age, sex, race/ethnicity, and diagnosis. The advantages of this approach are quick access to several informative data sets. Disadvantages are that many of the “criterion” measures are now out-of-date or were inadequate as indicators of validity. Phase 2 of this program of work will begin the task of rigorously testing the most promising screens, in relation to 3 state-of-the-art diagnostic interviews (Diagnostic Interview Schedule for Children (DISC-IV), Child and Adolescent Psychiatric Assessment (CAPA), Developmental and Well-Being Assessment (DAWBA). We propose to begin this work (which must be done separately in each type of setting where screens are used) using a primary care sample, for reasons of public health importance, generalizability, and cost. In phase 3, we will use the diagnostic data sets generated in phase 2 to compare three approaches to psychiatric diagnosis: respondent-based (DISC-IV), interviewer-based (CAPA), and best estimate (DAWBA). We will examine the performance of each relative to a “nomological net” of validators: school performance, service use, impact on family, functional impairment, family psychiatric history, maternal depression, etc.

Background and Significance. In the absence of pathognomic tests for the current taxonomy of child psychiatric disorders, clinicians and researchers have made extensive use of question-and-answer methods ranging from brief questionnaires to lengthy interviews, sometimes combined with observations and “objective” neuropsychological tests (although the latter two methods have never been mapped onto the current taxonomy in any organized way). Different assessment methods make different demands on respondents and resources, for time, effort, clinical expertise, laboratory resources, data management and analysis, and ultimately cost. We tend to assume that the amount of investment in these resources is reflected in the quality of the outcome, in this case the accuracy of the diagnosis; that the more we invest in assessment, the closer we will come to the “gold standard” (although in practice there is remarkably little empirical evidence for this).  We use the term “screens” in this application for a wide range of instruments administered to children, or parents and teachers about children. The reason is that our long-term goal is to encourage a more nuanced approach to the diagnosis of child psychopathology; one that includes both non-specialists (parents, teachers, primary care providers) and specialists via a graded series of investigations with a known relationship to one-another in terms of sensitivity, specificity, cost, etc. A model for this might be the “pyramid” of increasingly costly screens for breast cancer, from self-examination to physician exam to mammogram to biopsy. Recent research has thrown doubt on the validity of some steps in this pyramid, but this is good in the sense that cancer researchers and public health officials have a program set up to evaluate and improve the screening process. We have no such program for child psychopathology. If we were to develop one, it would inevitably rely heavily on questionnaires, etc., as screens at various levels. So, more work on screens is crucial.

Design and Procedures for Phase 2 – Youth aged between 9 and 16 (N=1800) will be recruited from the Duke University Medical Center primary Care Pediatrics and Durham Pediatrics (a part of the Duke University Health System). There will be 8 cells of 150 each, counterbalanced by sex, White/African American race/ethnicity, and age (9-12, 13-16).  Participants (parent and child) will complete the chosen set of screening measures twice. Each screening will be followed by a different psychiatric interview. Using complete randomization, each participant will receive 2 of 3 interviews; 400 will thus receive each combination. Screening efficiency will be calculated relative to each interview separately, and averaged over all 3. Dennis Clements MD Ph.D., Medical Director of Duke University Medical Center Primary Care Pediatrics, has agreed to let us use the Duke Children’s Primary Care pediatric clinic in Durham, North Carolina as the recruitment site for this study. Barbara Keith Walter Ph.D., is a co-investigator on this proposal, and she will be responsible for subject recruitment.

New data from other birth cohort studies around the world have highlighted the importance of the intrauterine and perinatal environment as a predictor of risk for adult diabetes, cardiovascular disease, schizophrenia, ADHD, and other chronic diseases. In light of the growing importance of this information for understanding etiology, we will call the mothers/fathers whom have previously completed the study and ask a few additional questions about the perinatal period. They are familiar with the study its goals and methods. We propose that the study interviewers will make contact by telephone, ask for verbal informed consent (see attached script), and complete the 12-item questionnaire, which will not take more than 10 minutes at the maximum. The telephone script has been included.

The Great Smoky Mountains Study has from the beginning collected information on birth weight, pregnancy, and birth difficulties. The parents have been interviewed regularly since 1993 about the target child. We will use a similar telephone script, and use the same questionnaires and protocol.

Recruitment procedures for subjects – The recruiter will access the daily provider schedules and the “Same Day Clinic” using IDX, Duke’s Scheduling Database. The recruiter will make a list of the clients whose age is 9-16. When the patient arrives and is taken back to the examination room, the recruiter will give the nurse a brochure that describes the study. The nurse will make first contact with the patient and give the brochure to the parent/guardian. After the parent/guardian has read the brochure, the nurse will ask if the parent/guardian would like to speak with the recruiter. If the parent/guardian declines or refuses the opportunity to speak to the recruiter, the nurse will inform the recruiter and there will be no contact made by the recruiter. This will allow the patient the opportunity to refuse participation without ever meeting the study recruiter.  If the parent/guardian agrees to talk to the recruiter, the recruiter will enter the examination room, introduce him/herself, and confirm the child’s name and age. The recruiter will then explain the study and the risks and benefits of participating in the study to the potential subject. If the parent/guardian and the child agree to participate, the recruiter will schedule a day and time and the desired location for the interview to take place.  The recruiter will then obtain the address and directions to the subject’s home. The recruiter will obtain written informed consent for participation. Consent will include permission to review the child’s medical chart for indications of mental health service use or referral. The recruiter will then enter subject information into our database (Fast Track) and generate a contact sheet for the interviewer. If the parent/guardian is bringing more than one child between these ages of 9-16 to see the pediatrician, the recruiter will consult pre-computed random selection tables to select the child to be the target of the study. There will be some situations in which a parent or legal guardian will not be accompanying a minor patient: Patient is 16 years old and drove himself/herself; patient is with someone other than parent/guardian – grandmother, aunt, or uncle. In these situations, the nurse will ask the child to give the brochure to his/her parent/guardian and ask permission to call the child’s parent/guardian. If consent to contact is not given by the child, the nurse will inform the recruiter and no contact will be made by the recruiter. If consent is given, the recruiter will collect contact information and call the child’s parent/guardian to explain the study and attempt to recruit.

Statistical methods for data analysis- The statistical approaches that will be implemented in Phase 1 and Phase 2 are based on the Receiver Operating Characteristic Curve (ROC), and its variants QROC (quality ROC) QCROC (quality-cost ROC). ROC analysis is a technique frequently used by engineers in signal detection problems and by medical researchers in comparing the value of different tests, or different cutpoints on the same test. It is a geometric approach to aggregating information from the Se, Sp, PVP, and PVN of multiple tests in a single two-dimensional space. The QROC is an extension of ROC analysis that adjusts Se, Sp, PVP, and PVN for the prevalence of the disorder of interest in the population being studied, and the cutpoint (“quality”) used for the screen. The QCROC  takes into account the quality of the screen, the cost of administering a test, and the relative costs of false positives and false negatives to the patient. QCROC will be our basis for comparing a range of screening tests in their ability to identify children with diagnosed disorders. In Phase 3, we will use mainly Latent Class Analysis (LCA) and Item Response Theory (IRT). Other statistical modeling techniques such as logistic regression, multiple regression, etc will be used whenever appropriate.

Interviewer recruitment and training-The study design requires 3 different interviewing styles: DISC (respondent-based), CAPA (interviewer-based) and DAWBA (respondent-based plus supplementary questions and probes). In order to avoid contamination across interviewing styles, 4 interviewers will be hired and trained for each type of interview. Interviewer will be trained to criterion before recruitment of subjects begins. Thereafter the twelve interviewers will work 1/3 time. On the basis of more than a decade of experience, we know that each interviewer will be able to complete 100 interviews a year. This gives the necessary 1200 interviews (300 families) per year, for a total of 4800 interviews (1200 families) over four years. We expect each interview to last about 2 hours. All interviewers will have the same credentials, i.e., a four-year degree in psychology or a related discipline. Every effort will be made to ensure that the ethnic distribution of the interviewers reflects that of the participants. Clinicians for the DAWBA best-estimate procedure will be recruited from the large pool of child clinical psychologists and child psychiatrists who work in the Durham-Chapel hill area. They will be paid at a rate that will make the work competitive with clinical practice. Certain characteristics of training will be common to all interviewers. All will be trained to explain the consent forms, and to administer chosen set of screening questionnaires in a standard manner. They will all learn how to deal with suspected abuse or suicidality. Then 4 interviewers will be assigned to training in each of the 3 interviews.

Procedures in the event of identifying at-risk of suicide or homicidal behavior- When child is identified who appear to be at serious risk of suicidal or homicidal actions, or who are the subject of physical or emotional abuse, the interviewer consults at once with the project coordinator.  If the case meets the criteria laid down in the guidelines specified by the study’s clinical faculty (Drs Angold and Egger), and if it is determined by the clinician that intervention is mandated, the interviewer calls and explains to the subject and parent the reasons why it is deemed necessary to obtain a clinical consultation and offers referral to the appropriate agency or, in the case of suspected abuse, files the necessary Child Protective Services Law report.

Potential risks and benefits- There are no mental or psychical risk to the subjects (parent and child) that will be involved in this study. Every attention will be given to the parent and his/her child to prevent any discomfort due to fatigue or any other reason that may occur during the interview. If necessary, interviewers will interrupt the session so that the parent or his/her child can take a break. The potential benefits of this study are the identification of the “best” screening in terms of cost-effectiveness and diagnostic efficiency relative to the expensive diagnostic procedures, and the collection of valuable data on three major diagnostic tools for future investigations. One of these, DAWBA, has never been used in United States before.

Compensation and responsibility to subject for costs- There will no cost to the subjects for taking part in the study.  We will pay subjects for any travel costs that they may have in order to participate in the study.  We will pay each $25 to the parent and child after the second interview.  If the subject (parent/child) withdraws from the study after completing the first interview, we will pay the subject (parent/child) $12.50 each for completing the first interview.

Funding for this study will be provided by the National Institute of Mental Health in response to MH-02-005, specific request for applications (RFA) titled “Validation of Child and Mental Health Diagnostic Measures.”

Confidentiality- We instituted a series of checks at several levels of data handling, to maintain its quality and confidentiality: (1) Interviewers will be given extra training in the importance of confidentiality in psychiatric interviewing.  (2) Protocols will checked by the interviewer, with reference to the tape if necessary, before being sent to the coordinator for further checks for completeness and accuracy.  The data clerk will then enter them. The protocol will be identified by number only; from this point they cannot be traced back to an individual except by the project coordinator. (3) At Duke, the interviews will be double-checked using specialized programs design to detect any inconsistencies, i.e., subject age is out of the interval 9-to-16. Computer files will have no names or addresses in them, and individuals will not be identifiable through them. (4) Protocols (and tapes) are kept in a locked storage room. (5) Written reports and papers do not identify subjects or make identification possible, without the written consent of the persons concerned.