The QSE for Assessments applies to the activities of monitoring the Quality Management System. If the activities are not monitored regularly, how would you know that they are being achieved? Monitoring quality is the means to determine the effectiveness of the quality system.

The other term that is commonly used for this QSE is Quality Assurance (QA). The main goal of quality assurance is to verify that all of the laboratory’s services are being delivered in such a way as to contribute to the overall delivery of excellent patient care. The policies, processes and procedures that relate to Assessments, according to the CLSI,^[1] are listed below.

The Policies included in Assessment include

• External quality management or accreditation assessments (ie, peer reviews)
• External quality assessment (ie proficiency testing)
• Quality indicators
• Internal audits
• Benchmarking and other quality comparisons
• Blood utilization, where applicable

The Processes that are part of this QSE include

• Mechanisms for the organization to participate in external assessments, such as accreditation or proficiency testing
• Identify and implement quality indicators to regularly monitor the effectiveness of patient care
• Develop an internal audit program
• Establish a mechanism for benchmarking to outside organizations and/or best practice
• Identify opportunities for improvement
• Develop mechanisms for regular reporting to management and the quality function
• Processes for blood utilization

Procedures included in this QSE

• Develop indicators to measure aspects of the path of workflow
• Collecting, analyzing data, and monitoring quality indicators
• Benchmarking to best practices by comparing the laboratory’s performance data with those of other labs
• Reviewing internal and external assessment information
• Preparing quality reports for management review
• Preparing blood utilization data

Assessments can be broken out into 2 types of assessments – External and Internal. Both types are equally important to the effective and smooth functioning of the laboratory.

External Assessments include proficiency testing, accreditation surveys, and benchmarking while Internal Assessments include indicators, audits, and self assessments.

External Assessments

Proficiency testing is carried out in clinical laboratories in Canada because it is required by accreditation. However, proficiency testing was also in place in most labs in the country long before accreditation became mandatory. Proficiency testing provides a way for laboratories to compare their test results to those of other laboratories with similar methodologies and instrumentation. Specimens for proficiency testing are analyzed in a similar manner to regular patient specimens that come into the laboratory. There are certain rules that must be followed when performing this testing to ensure that the results are being obtained under the same conditions as the patient test results.

Requirements of Proficiency testing

1. Samples for proficiency testing must be incorporated into the routine workflow
2. Test values and samples are not to be shared with other labs
3. Testing is to be performed by the technical staff who conduct the patient testing
4. No extra repeats or duplicate testing can take place other than that which would be done on patient samples
5. Testing should be completed within the same timeframe as routine testing on a patient would be completed.

Once the results of the proficiency samples have been obtained, they are reported back to the program that provided the specimens. The results are summarized and reported to the testing laboratories so that laboratories can compare their results with those of peer laboratories doing similar testing. When discrepant results are found for any analyte, a thorough investigation is conducted by the performing laboratory to determine where the problem may have occurred. It could be something as simple as a dilution/multiplication or transcription error, or it may be the result of a deeper rooted problem such as a calibration problem, a systemic error, or an error involving technical staff. Once the source of the problem has been identified and corrected, the laboratory must file a report stating what they found, how the problem was addressed, and what the final outcome was.

When no proficiency testing program is available for comparative testing of an analyte, the CLSI identifies several other alternatives that can be used to verify the accuracy and reliability of the testing performance.^[2]

1. Examinations on samples that have been split with another lab
2. Periodic testing of stored stable aliquots with known values
3. Analysis of data used for inter-laboratory QC
4. Re-testing by a second person or persons
5. Direct observation of the performance of technique dependent tests6.Culture of surrogate organisms for dangerous organisms

Accreditation

Accreditation surveys are a second component of external assessments. As part of the accreditation process, surveyors from the accrediting body come to the clinical laboratory to perform detailed assessment of the quality management system of that laboratory. Effective 2008, accreditation is mandatory for all clinical laboratories in Canada. Accreditation Canada, formerly known as CCHSA (Canadian Council for Health Services Accreditation), is the accrediting body for all hospitals in the country. Laboratory Services is included in this accreditation and must be assessed each time the hospital accreditation is up for renewal. If laboratories use another accrediting body, such as CAP or AABB, Accreditation Canada accepts that accreditation and the laboratory is accredited based on that process when the hospital survey takes place. The accreditation surveys are called peer reviews and are generally carried out by experienced laboratory professionals from other provinces.

The final form of external assessment takes place when a laboratory goes through a benchmarking process. Benchmarking is the process whereby a laboratory compares its’ performance to another organization that is considered in the industry to follow best practice. The benchmarking laboratory then rates itself to that performance excellence. It is a means of measuring performance against that of world class organizations so that the laboratory can see where they fit on the scale of excellence. For example, the Lab at the QEII might compare a method to the performance of the same testing at the Mayo Clinic, a lab that is recognized around the globe as providing excellent quality testing. If they discovered that their test results were taking a longer time to report on average, they would then look at ways of improving the turnaround time to improve it to the same level as that of the Mayo Clinic. In order to find these different ways of operating, they could talk to staff at the Mayo Clinic about their procedures, or they may visit that lab and spend time observing the tests being performed to look for variations from the way the QEII is currently performing the testing procedure. They could also review their current process to see if there are ways they can improve by simply working more efficiently.

In order for a laboratory to manage external assessments and inspections, there are several activities and responsibilities that must be in place. These activities and responsibilities include:

• Scheduling the assessments and/or inspections to take place
• Pre-assessment paperwork
• Conducting the actual assessments and inspections
• Providing closing summaries of the assessments
• Follow-up responses to any assessments and/or inspections that have taken place
• Implementing any required corrective actions
• Verifying the effectives of any changes that have been implemented in response to the assessments/inspections.^[3]

Internal Assessments

Internal assessments include both internal audits and indicator reporting. Internal audits can take many forms and are done for a variety of reasons. An audit is a type of assessment that is defined as a systematic investigation that is carried out within the laboratory service to determine if the actual activities and practices are being performed according to the written policies and procedures of the organization. An audit is generally a self-assessment tool that helps the laboratory identify that it is meeting set targets and process controls.

The self assessments that laboratories perform in order to complete their preliminary accreditation report are used to help the laboratory prepare for the accreditation process. In many labs, a preliminary audit or assessment takes place to help the lab identify where they need to improve. Once the identified improvements have been completed, a formal internal assessment is completed and reported to the accrediting body. This preliminary report is used by the accrediting body to determine if they think the lab is ready for the external survey to proceed.

A second type of internal assessment is the quality audit. An audit may be performed on an area of the laboratory where management want to know if the service is being delivered successfully to provide the optimum quality of service to their patients. This is a proactive type of audit that is being conducted to identify whether there may be a problem that has yet to be identified. Examples of this type of audit would be a wait-time audit in the Outpatient Blood Collection area or a turnaround time audit for testing from the Emergency Department.

Sample Blood Collection Wait Time Data Collection Sheet

• How would you decide if there was a problem in the process that slowed down the wait time?
• What is the average wait time?
• How would you design an audit data collection sheet to track Troponin TAT from collection until resulting for patients in Emergency?

There are many other types of audits that may be performed throughout the laboratory. How many can you list?

Pre-examination Audits:

Examination Audits:

Post-Examination Audits:

Quality Indicators are the other types of internal assessment. They can be designed to measure any aspect of the Laboratory’s service. Two important considerations when determining which indicators a particular laboratory may want to implement are known risks and their potential consequences.^[4] Quality indicators are usually reported as a ratio. One example of such a ratio would be the number of accessioning errors per 1000 requisitions accessioned. This indicator would be displayed numerically as 10/1000 or 1% error rate. The point of this indicator would be to ensure that the error rate in the accessioning area is not growing or that the accessioning skills of the staff are improving over time. The management team of the laboratory generally determines which aspects of the service they want to report on regularly. Although it is important to make sure all aspects of the quality system are operating effectively, there are too many aspects to monitor continuously. Therefore, management must choose what to measure and for how long. For an example of how an organization might report on their quality indicators, follow the link below and you will be connected to Capital Health’s Strategic Indicator Reports.^[5]The aspects of laboratory operations to monitor are chosen for one of four reasons:

• To check a specific, normally stable function to ensure it is continuing to operate as expected, for example, the qc variation for a specific analyte over a month
• To check a complex process that has many inputs or activities that can introduce variability. An example would be the turnaround time for STAT testing for the Emergency department. The ratio being measured would include a number of steps from the original order through till results are received by the physician.
• To check the effectiveness of improved processes in operations to monitor the consistency of the improvement. One example might be monitoring accessioning after the accessioning staff completed a refresher training course.
• To explore potential gaps in quality within the service.^[6]

Patient experience and patient safety are indicators that are often tracked in most Healthcare facilities in Nova Scotia. Planning for all aspects of care is impacted by these 2 indicators. In the survey above on Patient Experience, Outpatient Blood Collection experience would be included in this measurement. Often the organization would break this report out further to see if any particular area continued more than others to the percentage of responses indicated by the green boxes. By doing this type of breakdown, the organization has a better indication of services that may be in need of improvement initiatives.

Looking at indicators in a graphical format helps to show easily where an organization is meeting targets and where they are not. Histograms are particularly helpful in simplifying this data. The tables (shown below) Appendix B: Patient Safety Scorecard, shows the different components of safety that are measured and whether or not these indicators are meeting current targets. The scorecard gives a very easy to read of problem areas and areas where the organization is meeting and exceeding expectations.

System Indicators

• Cost/Benefit ratio – to determine the cost of a test per patient outcome
• Customer satisfaction – to determine if the patient and/or the physician are satisfied with lab services
• Safety indicators such as needle stick occurrences
• Others?

Pre-examination Indicators

• Accuracy of patient information on requisitions
• Adequacy of samples
• Accuracy of sample information
• Correct sample labelling
• Identification of patients not wearing wristbands
• Patient consent appropriately obtained
• Others?
  

Examination Errors

• Accurate and timely qc monitoring
• Accuracy of POC testing
• Competency of staff
• Effectiveness of “Rules” used to detect abnormal results
• Sample contamination
• Surgical path slide review
• Instrument to instrument correlation
• Others?
  

Post examination Errors

• Auto-verification errors
• Client satisfaction audits
• Consistency of critical values reporting
• Proportion of corrected reports
• Result reporting accuracy
• Others?
  

Turnaround time

• Time until physicians receive autopsy reports
• Compliance with internal turnaround time targets to Emergency Dept.
• Cytology turnaround time
• Frozen section turnaround time
• Turnaround time for ac glucose results when patients are awaiting GTT
• Clinical Chemistry turnaround time for forensic reports
• Others?
  

Another source of indicator selection may come from the executive level of the organization’s management. Financial indicators will tell them if the laboratory is operating effectively. An example of a financial indicator would be cost per test. They are also interested in whether the laboratory staff is meeting organizational targets for such things as attendance, professional development, and retention of employees.

When deciding to start collecting a particular indicator, there are a number of factors that must be considered. The following questions must be answered:

• What information is going to be collected?
• Where will the collections take place? In the LIS, manually, by auditing requisitions, etc?
• Who will be responsible for collecting it?
• How often will it be collected? Monthly, quarterly, yearly?
• How will the data be collected and by whom?
• What target values are set to establish acceptable performance?

The answers to these questions will help in the decision-making about whether to proceed with the monitoring of this indicator and will also help to determine how long the indicator will be collected.

Once quality indicators have been selected, a target performance must be determined and monitored, as well as benchmarking performance improvements must be established. In order for an indicator to be used effectively, the laboratory’s management must know what they are trying to accomplish and they must also be able to develop improvements to their process if they hope to reach the benchmark in the industry. Milestones or interim targets can be established to help monitor progress along the way.Tools that are commonly used to collect data and analyze the data collection in determining indicators include the following (CLSI Document GP35-A, p.16):

Control Charts, such as the Levy-Jennings Control chart – allow the independent variables to be observed and recorded. They can be checked visually to see if they are within acceptable limits.

Pareto charts – a simple bar graph used to focus efforts on the problems that offer the greatest chance for improvements. The variables are displayed in order of descending frequency. This way, the team can focus on the areas that have the greatest potential to improve the overall process.

Histograms – provides a graphical representation of the frequency distribution in a bar format. When evaluating the histogram, the team can determine if the process is centered on the target value, how much variation there is within the distribution and whether the data forms a normal Gaussian distribution.

Scatter diagrams – used to show whether or not there is correlation between two variables.Peer Review for Blood Utilization is an internal assessment process in Transfusion Medicine where the Laboratory ensures that they are using safe and appropriate transfusion practices. It includes a review of blood ordering practices and transfusion policies, processes and procedures.

Periodic reporting to Laboratory Management is the final internal assessment component. Management needs to gather data and information about the QMS and its overall performance. Internal and external assessments, non-confirming events and customer feedback are all valuable sources of how effectively the whole system is working. The following information on Hand Hygiene compliance would be one example of this type of monitoring.