Measuring for Outcomes

The data is measured and collected directly impacts the value and quality of the insights gathered from running a trial.

Types of Data Collected

There are two main types of data collected during a trial. Most of the information collected will be qualitative, which is information collected through a structured interview. Qualitative data comes from open ended questions.

Qualitative Prompt: Were there any things in particular that you quite liked about the process?

We also collect quantitative data. Quantitative data is used to attach numbers to the data collected. This means that well be able to make comparisons and show trends over time.

Quantitative Prompt:

For your last accreditation, how long did it take your organisation to submit the information from when you started the pre-assessment process?

What percentage of your time was dedicated to the pre-assessment during that time?

Quantifying Time Commitments

We want to understand how much time is spent on tasks so we can articulate how much time has is saved with different feature additions to Accreditron. Time savings is a great way to tangibly show benefit.

Our aim Trial 1 was to gather quantifiable information about how much time providers spend on different processes, particularly their pre-assessment process. Unfortunately, providers participating in the trial we're not able to put numbers to their time commitment, instead they would answer with qualitative statements like:

We spend two weeks with our staff in the office until 10pm at night preparing for an Accreditation.

or

We spend a lot of time working on putting all of our information together.

As we move into future trials we will trial asking different questions that can help us quantify how much time is saved when using Accreditron. In trial one we ask about total time spent on the process. We will still ask this, but will not expect quantitative answers.

Asking about total time:

For your last accreditation, how long did it take your organisation to submit the information from when you started the pre-assessment process?

Would you say more or less than a week..?

Additional follow up to understand, and start to quantify. Is this a one person job, is it a two person job, do the spend 50% of their time working on their pre-assessment?

Asking about percentage of contribution:

What percentage of your time was dedicated to the pre-assessment during that time?

Were any other people involved?

To gather quantitative information, we will begin to ask about first times successful submissions. Are we reducing the number of back and forth emails from a provider not providing all of the right information up-front.

Asking about events and first time completion rate: For your last pre-assessment were you asked to provide additional information after? How many times were you asked for additional information? This might of looked like an email. During your pre-assessment on Accreditron we're you asked to provide additional information after submission?

Checking completion status of a pre-assessment and ensuring that all of the correct information is submitted first try is a better indicator of success compared to reducing time spent on the process.

Accreditron has a goal of shifting as much work as possible to a pre-assessment where it can reviewed at a desk instead of during a site visit. And since all information is saved for a providers next assessment on Accreditron we will expect to see the majority of the times savings occur during their second pre-assessment process.

Comparing Insights from Previous Trial

We will continue to review and gather insights relating to:

1. Login and Registration

2. Expectation Setting

3. Question and Content Comprehension

4. Navigation

Themes Introduced in Trial 2

Additional themes that will be included in Trial 2

5. Dashboard Functionality

6. Internal action management

7. Staff file management