We have identified an issue with the date period that was previously used when calculating the number of patients/surveys included each month which is causing discrepancies among some of the figures reported. The charts on the website are generated from the “Collection date” that was submitted with the data item and that is used to group the data and display as months. We have found that this is different from the way that the data was previously updated and grouped into the excel tools. Previously the “Submission Dates” were used as the monthly cut off. This would mean that any data submitted early in the month would be counted against the previous month, which is counter-intuitive to how a monthly collection should work.

Next Generation’ NHS Safety Thermometers: changes in monthly reporting

Background

The ‘Next Generation’ NHS Safety Thermometers (Medications, Maternity, Mental Health and Children and Young People) have been developed to enable the collection and use of data in specific areas to interest to be collected at the point of care across the health economy in acute hospitals, community hospitals, intermediate care, care homes and district nursing services.  Any organisation is welcome and encouraged to join the collection and start collecting data to use in local improvement programmes.

The NHS Safety Thermometer programme is owned and overseen by NHS Improvement, with development and management subcontracted to a third party supplier.  Previous to April 2017 the data collected through the ‘Next Generation’ NHS Safety Thermometers was published via an Excel based dashboard that was downloaded by users from the Safety Thermometer website.  In April 2017 the NHS Safety Thermometer programme came under the auspices of the Patient Safety Measurement Unit and became the responsibility of a new supplier.  Under the requirements of the new contract, the ‘Next Generation’ NHS Safety Thermometers are being moved towards consistency with the larger NHS Safety Thermometer ‘Classic’ data collection, formerly run by NHS Digital, through the use of online charts instead of the Excel download.

During the transition process an issue has been repeatedly raised with regards to comparative differences between the online charts and Excel based download.  The underlying reason for these differences has been identified; the remainder of this paper explains this and outlines possible remedial actions.

Issue explanation

The differences apparent between the online charts and the Excel download tools are due to a difference in how the data collected is assigned to months.  In the Excel downloads, any data added to the system in the first week of a calendar month was counted against the previous calendar month (e.g. patients surveyed on the 3rd March would be counted in February data).  In the online charts, any data added is counted against the month indicated (e.g. patients surveyed on 3rd march would be counted in March data). 

The worked examples below illustrate scenarios for three organisations submitting data to a ‘Next Generation’ NHS Safety Thermometer over the course of three months, with the impact for each explained.

 

Next steps and recommendations to organisations

Whilst the ‘Next Generation’ NHS Safety Thermometers have been collecting data for a number of years we do not propose to revert to the approach taken for defining months used in the Excel download tools.  This is for a number of reasons:

  • In the NHS Safety Thermometer ‘Classic’ collection data is assigned to months on the basis of month surveyed as per the online charts for the ‘Next Generation’ NHS Safety Thermometers. Maintaining this approach ensures consistency across all NHS Safety Thermometer data collections.
  • On interpretation of the data it is much simpler to discuss month on month variation when there is clarity on what time period the data is submitted within
  • All other NHS data collections (e.g. incident reporting, SUS data, audit) are generally made available where months are grouped on a basis of when the activity/incident etc. actually happened (i.e. if an incident occurred in March it is counted and analysed within March). As triangulation of data sources is an important part of building an holistic view of patient safety, it is helpful to have reporting on a common footing wherever possible.

In the longer term, the transition from the Excel based tools to the online charts will work itself out of the system.  However, it is appreciated that there will be short to medium term difficulties in moving from the prior system to the website resources.  To assure individual organisations and relevant stakeholders that the data is accurate the following actions are recommended:

  • Work through the examples in this paper to understand the impact for your organisations (if any)
  • Review the raw data behind each chart on the website and compare against the relevant data on the latest Excel download made available to assess overall accuracy
  • Contact the NHS Safety Thermometer helpdesk (details below) to discuss if you are unclear on any element of this issue or if you find an additional problem.

NHS Safety Thermometer helpdesk:

Call us: 0300 123 6260

E-mail us: This email address is being protected from spambots. You need JavaScript enabled to view it.