I built an evaluation plan and a technology adoption model to measure whether two mental health programs really help veterans and their families.
In 2013 I planned how a veterans medical center could judge a self-care mental health website and a mobile app for post-traumatic stress by more than page views. The plan ties each mission goal to a question, each question to a method, and every interview and survey question to a model of what makes people use health technology.
Client
A veterans medical center
I changed names and details to respect my nondisclosure agreements.
My role
Product researcher
Focus
Evaluating mental health technology for veterans
Result
A ready evaluation plan built on a technology adoption model
Two programs promised to help veterans with their mental health, and nobody could yet say whether they did.
The first program was a self-care website for service members, their families, and veterans in the months after a deployment. The website offered self-assessments, workshops, a library, videos, and referrals to care. The second program was a free mobile app for people living with post-traumatic stress. The app taught people about the condition, offered self-assessments and coping skills, connected people to crisis support, and guided them toward treatment.
Both programs had a clear mission. Neither had an agreed way to show whether the mission was working. Visits and downloads could show that people arrived, but they could not show whether a veteran felt better, sought care, or felt pushed away by the stigma around mental health. My first question for the app's developers was simple: were goals set for the app, and has anyone measured them?
I organized the plan into six steps, from mission to evidence.
I built the plan on the DECIDE framework from Interaction Design: Beyond Human-Computer Interaction, so every step leads to the next, and the team could brainstorm one step at a time. Click or tap a card to turn it over.
I built a model of what predicts whether a person uses health technology.
A download count tells you that someone tried an app. The count says nothing about whether that person trusts the app, can use the app in private, or would open the app again after a hard night. I wanted the evaluation to measure the reasons behind use, so I drew a model with actual use at the center, measured through a person's intent to keep using the program.
Eight factors feed that center. I wrote interview and survey questions for each factor, worded for veterans and their families, so the answers would sort themselves by factor at analysis time. The questions about stigma and privacy carried the most weight for this audience.
- Beliefs cover whether a person expects good or bad outcomes from seeking care online.
- Attitudes cover whether a person judges mental health care and self-help as harmful or beneficial.
- Related behaviors cover habits close to the program, such as finding health information online.
- User characteristics cover health, demographics, and impairments such as a brain injury.
- Perceived ease of use covers whether the program is clear and easy to learn and remember.
- Perceived usefulness covers whether the program saves time and gives people more control over their health.
- Social influence covers whether doctors, commanding officers, family, or peers would change a person's mind.
- Facilitating conditions cover access to a device, a private place to use the program, and help when needed.
Three kinds of experts looked for three kinds of problems.
Expert reviews are fast, but a single expert sees only what that expert knows. I asked for two or three outside experts in each group, so the team could compare and contrast their findings. Click or tap a card to turn it over.
Explore the plan, the methods, the model, and the survey.
My 2013 plan lived in slides. I rebuilt the plan as a clickable tool under the fictional Haven Health name, with AI writing the code from my slides. The program names are fictional, and every step, method, factor, and survey statement comes from the original plan.
Things to try
- Step through the six parts of the plan with the Next step button.
- Filter the methods by what each one tells you, and open a method to read the details.
- Choose a factor in the adoption model to see the questions that measure the factor.
- Rate the app in the post-test survey, then read how to interpret the summary.
Pine green, warm paper, and a calm serif keep a heavy subject steady.
I styled the tool for a health program serving veterans: a deep pine for structure and actions, a clay accent for the program labels, and warm off-white paper instead of clinical white. A serif sets the titles and a plain government-style sans sets everything else, and every text color passes current contrast standards.
Color
- Pine
#1E4A3DThe top bar, primary buttons, and the selected step - Deep pine
#143329Button hover states and text on sage - Sage
#E7EFEASelected methods, the model's center, and chips - Clay
#9A421CProgram labels, method types, and focus rings - Ink
#1A2420Titles and body text - Moss gray
#56635DLabels and secondary text - Warm paper
#F6F4EFThe page background and tiles
Type
Titles
Six steps from mission to evidence
Source Serif 4
Labels
Read the numbers
Public Sans
Body
Do you have a private, secure place where you can visit the portal?
Public Sans
Components
DECIDE stepper
Six lettered steps show where the team is in the plan, and Back and Next buttons move one step at a time.
Method cards
Each card names the method, the kind of question the method answers, and opens a panel with measures and limits.
Adoption model
Eight factor buttons surround actual use, and each factor opens what the factor covers and the questions that measure the factor.
Rating scale
Five large, labeled choices per statement work by mouse, touch, or keyboard, and an error names every statement still unrated.
Principles
- Start from the program's own mission, not from the data that happens to exist.
- Ask about stigma and privacy before asking about buttons.
- Match each question to the method that can answer the question.
- Ask about strengths as often as weaknesses.
The team got one plan that connects each program's mission to evidence a person can trust.
This was planning work, so the page has no launch numbers. The plan itself was the deliverable. Click or tap a card to turn it over.
I bring the same rigor to teams who need to know whether a product really helps people.
As a principal designer offering AI-native product research and design, I help public and private teams decide what to measure before they measure it, from a health program to an AI assistant. If your team needs to show that a product helps the people the product serves, I would be glad to plan that evaluation with you.
There is plenty more where this came from.
Names, brands, and data in my case studies are changed to protect my clients.
Tell me about your project.
Tell me a little about it on my contact form, and a real person will get back to you soon.