Measuring Success
How Do We Measure the Success of Activities and Interventions for People with Dementia?
The Hearthstone Research Division’s professional team of trained researchers led by Michael Skrajner uses a variety of tested and validated measurement tools to gather data in field situations such as residences, classrooms, and training sessions.
To understand more about Hearthstone’s research – on which the Institute staff training programs are based – you will find here a partial list of principal research tools used to measure the impact of I’m Still Here® techniques. For a list of our published research in peer-reviewed scholarly journals, click here.
Traditional medical research often uses the ‘double-blind placebo control trial’. In this type of study, one group of participants is given a drug, another matched group is given a placebo, and the difference in outcomes between the two groups is measured to calculate the efficacy of the intervention. In working with non-pharmacologic interventions, using this approach is not typically possible, for a variety of reasons. So, researchers are faced with a special challenge: how can you measure the efficacy of a non-medical intervention? How do you measure behavioral change, such as ‘greater enjoyment’ or ‘more engagement’ between matched groups of participants? And if you record new behaviors, can you assume that they are attributable to a changed psychological state such as reduced anxiety?
To ensure study validity and reliability, the Hearthstone Research Division utilizes reliable, validated, and standardized measurement tools that collect objective data. The team has also authored several valid measurement tools that are now in use in the industry. Utilizing these types of data collection tools avoids the bias of a single observer and the subjective perception of the individual researcher.
These tools fall into one of two general categories of measurement scales used to evaluate the impact of a given activity or intervention.
Proximal Measures examine the effects on participants while the activity/intervention is occurring. Proximal Measures evaluate the immediate effects of an intervention on participants: for example, are residents smiling/laughing while they are taking part in the activity?
Distal Measures evaluate the impact of an intervention on participants over a longer period of time or more generally on behavior. Distal Measures are used, for example, when we wish to find out whether participants become less anxious after participating in a new program, not only during the activity but also at other times.
A partial list of measurement tools used in our research is provided below. For a list of our published research using these measurement tools, click here.
-
Proximal Measures
Engagement/Affect
The Menorah Park Engagement Scale (MPES)
An observational tool that assesses engagement and affect displayed by PWD taking part in activities, the MPES has been used in many studies examining the impact of non-pharmacological interventions (Camp, Skrajner, & Gorzelle, 2015). The MPES divides engagement into four distinct types: Constructive Engagement (CE), Passive Engagement (PE), Non-Engagement (NE), and Other Engagement (OE). CE is defined as any motor or verbal behavior exhibited in response to the target activity, e.g., turning the pages of a booklet, responding to a question posed by the leader, etc. PE is defined as listening and/or looking in response to the target activity, e.g., listening to a discussion, watching someone pointing to a picture in a book, etc. NE is defined as staring off into space, keeping one’s eyes closed, or sleeping during the activity. OE is defined as either self-engagement (engagement with one’s own body, clothes, or personal effects, such as biting one’s nails or fidgeting with one’s shirt while ignoring the activity) or engagement unrelated to the target activity, such as watching a nurse dispense pills to a client in an adjacent room, chatting with a friend while ignoring the activity, etc. The MPES also includes items involving Pleasure (clearly observable smiling and/or laughing).
The Observational Measurement of Engagement Assessment (OME)
This measure was specifically developed to assess the levels of engagement of persons with intellectual disabilities (Cohen-Mansfield, Dakheel-Ali, Jensen, Marx, & Thein, 2012; Cohen-Mansfield, Marx, Regier, & Dakheel-Ali, 2009). OME data are gathered through direct observation and entered onto a handheld computer with special software developed for this purpose. Outcome variables on the OME include:
- Duration: The time (in seconds) that the participant is engaged with the stimulus,
- Attention to the stimulus: Examples include eye tracking; visual scanning; facial, motor, or verbal feedback; eye contact; and touching the stimulus.
- Attitude to the stimulus: Examples include positive or negative facial expression, verbal content, and physical movement toward the stimulus.
Refusal rates for activities also are measured. Inter-rater reliability of the OME was assessed during 48 engagement sessions with long-term care residents.
Communication Behaviors
Observational Verbal Communication Scale (OVCS)
The OVCS measures the length of a person’s verbal communication behaviors over a five-minute time period. The OVCS also assesses the nature of the person’s verbal communication behaviors. For example, is the person saying things that are intelligible? And to whom are the person’s statements directed?
Subjects are observed for five consecutive minutes during which all clearly-observable signs of verbal communication are recorded. The exact length of time for each ‘utterance’ is documented, and at the end of the observation period, the total length of time is calculated by adding up all the times recorded. After five minutes, a mean is calculated for the average length of time participants communicated during the program.
An “utterance” is defined as any vocal sound made by a person, with the exception of semi-autonomous bodily functions such as sneezing or coughing. If the person says a word while simultaneously doing one of these things, this is considered an utterance.
-
Distal Measures
Quality of Life
DEMQOL
DEMQOL is a patient-reported outcome measure made up of a 28-item scale that assesses quality of life (Smith et al., 2005). Examples of items include: “In the last week, have you felt cheerful?” and “In the last week, have you felt full of energy?” Each item is rated on a score of one to four (i.e., not at all, a little, quite a bit, or a lot). Total scores range from 28-112, with higher scores indicating higher quality of life. The DEMQOL is typically conducted via direct interview. However, if a person has advanced dementia, a proxy version can be used.
The DEMQOL has undergone rigorous evaluation, which demonstrated psychometric properties comparable to the best available dementia-specific measures.
Quality of Life—Alzheimer’s Disease (QOL-AD)
The QOL-AD measures five domains of quality of life: interpersonal, environmental, functional, physical, and psychological (Logsdon et al., 1999). The scale includes thirteen items: physical health, energy level, moods, living situation, memory, family, marriage, friends, overall self, ability to do chores around the house, ability to do things for fun, money, and overall life. There are two versions of the scale: a patient version and a caregiver version. The total score is calculated separately for the patient version and the caregiver version, with possible scores ranging from 13 to 52.
Alzheimer Disease Related Quality of Life (ADRQL)
The ADRQL was designed to address aspects of quality of life that caregivers and clinicians regard as important and to detect changes over time (Rabins et al., 1999). The ADRQL is a disorder-specific measure of health-related quality of life for use in evaluating therapeutic interventions for persons with AD across various care settings and various stages of the disease. It may be used as an outcome measure in determining efficacy and effectiveness of behavioral interventions, environmental settings and drug treatments.
The scale measures both positive and negative behaviors across five domains: Social Interaction, Awareness of Self, Feelings and Mood, Enjoyment of Activities, and Response to Surroundings. The majority of items measure observable behaviors and actions, although some rely on assessment of subjective and internal states. Caregiver respondents are used for the ADRQL.
Behavioral /Neuropsychiatric Symptoms
Behavioral Pathology in Alzheimer’s Disease (BEHAVE-AD)
The BEHAVE-AD is a global measure of behavioral psychological symptoms of dementia (BPSD) with 25 items grouped into 7 major categories: paranoid and delusional ideation, hallucinations, activity disturbance, aggressiveness, diurnal rhythm disturbances, affective disturbances and anxieties and phobias (Reisberg, Auer & Monteiro, 1997). . It can detect clinically significant changes in BPSD and has become widely adopted for drug trials.
The revised BEHAVE-AD-FW adds a frequency-weighted severity score to improve the accuracy of identifying abnormal BPSD. The E-BEHAVE-AD is an adaptation that allows the clinician or staff member to rating the person’s behavior based on direct observation.
The Cohen Mansfield Agitation Index—Short Form (CMAI-SF)
The CMAI-SF was developed to assess agitation—i.e., “inappropriate verbal, vocal or motor activity”—in long-term care residents, including those with dementia (Cohen-Mansfield, 2008; Cohen-Mansfield, Marx & Rosenthal, 1989). The CMAI-SF consists of 14 items that are posed to a proxy (caregiver). Examples of items includes “during the past two weeks did the resident display cursing or verbal aggression?” and “Did the resident display constant request for attention for help.” Each item is rated on a scale of one (behavior never occurs) to five (behavior occurs a few times an hour or continuous for a half an hour or more). Therefore, total scores on the CMAI-SF can range from 14-70, with higher scores indicating higher levels of agitation.
Neuropsychiatric Inventory – Nursing Home (NPI-NH)
This measure is designed for use with patients with AD and other dementias, to evaluate frequency and severity of 10 neuropsychiatric symptoms, often referred to as challenging behaviors (Cummings et al., 1994; Wood et al., 2000). These include: apathy, agitation, irritability, dysphoria, disinhibition, anxiety, hallucinations, delusions, euphoria, and abnormal motor output.
Each behavior is rated as absent or present. In addition, if the behavior is present, its frequency is rated on a scale of one to four, and its severity is rated on a scale of one to three. For each item, Frequency is multiplied by Severity to create an F x S score for that item. A total score on the NPI-NH is calculated by adding up all of the F x S scores. Therefore, total scores on the NPI-NH (F x S) can range from 0-120, with higher scores indicating increased frequency and severity of neuropsychiatric symptoms / challenging behaviors.
Cognition
Mini-Mental State Examination (MMSE)
The MMSE is the most commonly used instrument for screening cognitive function (Folstein, Folstein & McHugh, 1975). It can be used to indicate the presence of cognitive impairment, such as in a person with suspected dementia or following a head injury. The MMSE is more sensitive in detecting cognitive impairment than the use of informal questioning or overall impression of a patient’s orientation.
The MMSE measures orientation, registration (immediate memory), short-term (but not long-term) memory as well as language functioning. The total possible score on the MMSE is 30, with lower scores indicating more severe cognitive impairment
Montreal Cognitive Assessment (MOCA)
A one-page 30-point test administered in approximately 10 minutes (Nasreddine et al., 2005). MoCA assesses several cognitive domains. The short-term memory recall task (5 points) involves two learning trials of five nouns and delayed recall after approximately 5 minutes. Visuo-spatial abilities are assessed using a clock-drawing task (3 points) and a three-dimensional cube copy (1 point). Multiple aspects of executive function are assessed using an alternation task adapted from the trail-making B task (1 point), a phonemic fluency task (1 point), and a two-item verbal abstraction task (2 points).
Attention, concentration and working memory are evaluated using a sustained attention task (1 point), a serial subtraction task (3 points), and digits forward and backward (1 point each). Language is assessed using a three-item confrontation naming task with low-familiarity animals (3 points), repetition of two syntactically complex sentences (2 points), and the aforementioned fluency task. Finally, orientation to time and place is evaluated (6 points).
Alzheimer’s Disease Assessment Scale-cognitive (ADAS-cog)
The ADAS-cog is one of the most frequently used tests to measure cognition in clinical trials. More thorough than the Mini Mental State Exam, it primarily measures language and memory. The ADAS-Cog consists of 11 parts and takes approximately 30 minutes to administer (Rosen, Mohs & Davis, 1984) The ADAS-Cog was developed as a two-part scale: one to measure cognitive functions and one for non-cognitive functions such as mood and behavior.
The original version of the ADAS-Cog includes (1) Word Recall Task, (2) Naming Objects and Fingers, (3) Following Commands, (4) Constructional Praxis, (5) Ideational Praxis, (6) Orientation, (7) Word Recognition Task, (8) Remembering Test Directions, (9) Spoken Language, (10) Comprehension, and (11) Word-Finding Difficulty.
Points for each task are summed for a total ADAS-Cog score. The greater the dysfunction, the higher the score. A ‘normal’ score is 5, whereas 31.2 has been found to be the average score for those who have been diagnosed with probable Alzheimer’s disease or mild cognitive impairment.
Depression
Cornell Scale for Depression in Dementia (CSDD)
The CSDD is a 19-item instrument that measures depressive symptoms in patients with dementia (Alexopoulos, Abrams, Young, & Shamoian, 1988). It can be administered either to a patient or proxy (caregiver). Information is gathered through semi-structured interviews which focus on the prior week’s depressive symptoms and signs. Many items can be completed after direct observation of the patient.
Each item on the CSDD is rated on a scale of 0-2 (0=absent, 1=mild or intermittent, 2=severe), and ratings are summed to determine a total score. Scores on the CSDD can range from 0-38, with higher scores indicating increased depressive symptoms. A score greater than 10 is indicative of major depression; a score greater than 18 indicates definite depression; a score less than 6 is associated with an absence of any significant depressive symptomology.
Geriatric Depression Scale-Short Form (GDS-SF)
The GDS-SF is a self-report measure of depression in older adults (Sheikh & Yesavage, 1986). Subjects respond in a “Yes/No” format. The GDS was originally developed as a 30-item instrument but was revised down to 15 items, chosen because of their high correlation with depressive symptoms in previous validation studies.
Of the 15 items, 10 indicate the presence of depression when answered positively while the other 5 are indicate depression when answered negatively. This scale can be completed in approximately 5 to 7 minutes, making it ideal for people who are easily fatigued or are limited in their ability to concentrate for longer periods of time.
Overall Activity Participation
Activity Participation Scale (APS)
The APS is a six-item measure of long term care residents’ involvement in activities (Lawton et al., 1998). The APS is implemented through interviews with proxies (caregivers/activity staff), and has been used extensively in collaborative studies on the effects of environment in special care units. Examples of items include “how often does the resident take part in planned social activities (games, music, parties, group crafts, etc.?” and “how often does the resident take part in unplanned non-solitary, social activities (games, etc.)?”
Each item is scored on a scale of one (never) to five (daily or more often). Therefore, total scores on the APS can range from 6-30, with higher scores indicating increased involvement in activities. The APS provides a wider time frame for examining engagement than is available in 5-minute observation windows of the MPES.
Improve your residents' lives and your market position today.