Seminar: Auditing for Quality Improvement

Title: ‘Improving the Quality of Care using audit’

Why are auditing practices important for realising Quality Improvement in maternal and newborn health services? In particular, what is the role for standards-based audit among other approaches and how can it be effectively implemented?

At this seminar, held at DFID Whitehall, on 28th October 2015, our guest speakers were the Nynke van den Broek and Dr Charles Ameh (Director and Deputy Director) of the Centre for Maternal and Newborn Health (CMNH) at the Liverpool School of Tropical Medicine.

Their presentations stress the importance of effective measurement of Quality of Care, describe good practice around auditing processes and highlight CMNH work in practice, using case-study material from Kenya.

View the presentation

Presentation  by Nynke van den Broek and Dr Charles Ameh at DFID Whitehall, October 2015

Find out more

Speaker bioPresentation summaryDiscussion summaryExplore further

Professor Nynke van de Broek is a recognised international expert in global maternal and newborn health.

She has designed and conducted large population based randomised controlled trials of single interventions for improved maternal and newborn outcomes. She has used this experience to develop complex packages of interventions and to design and conduct operational research programmes in multi-country settings in both Asia and sub Saharan Africa.

Impact has been ascertained through the development and application of new monitoring and evaluation frameworks and indicators to measure quality of care and maternal morbidity.

At LSTM she established and leads the Centre for Maternal and Newborn Health (CMNH).

Dr Charles Anawo Ameh (PhD, MPH, MBBS, DRH, FWACS-OBYGN) is a Senior Clinical Lecturer and deputy Head of the CMNH, at the LSTM. He has over 15 years professional experience working in the health sector in resource poor countries and specialising in Obstetrics/Gynaecology and Public Health.

He is experienced in the design, implementation and evaluation of programmes to build capacity of skilled birth attendants at both pre and in-service levels. He has been involved in obstetric skilled based training and research in low and middle since 2007.

His research interest are in eclampsia/serve-preeclampsia, Emergency Obstetric and Newborn Care, Quality of maternal and newborn care, maternal death audits, near miss reviews and confidential enquires into maternal deaths.

Summary prepared by: Adrian Bannister and Sue Newport, HEART-IDS

Professor van den Broek commenced the presentation by stressing that the majority of the Centre for Maternal and Newborn Health’s (CMNH) focus is on strengthening health care facilities within health systems and that she and Dr Charles Ameh were pleased to have the opportunity to share some ideas around improving quality of care (QoC).

The presentation focused on standards based audit as CMNH’s key methodology, the linkages with maternal death audits and also the importance of effective measurement, including the outcomes of improved quality.  Dr Ameh presented a case study from Kenya which reinforced many of the key points made by Prof van den Broek.

Throughout the presentation Prof van den Broek advocated for different ways of thinking about QoC, based on the assumption that women understand what is available to them and make decisions on this basis.

Traditional approaches have focused on increasing access; however is it now time to think about what should be done at health facility level to make sure women come to the facility? There is limited research but it is clear that if the QoC is poor, women and their families are less likely to visit a facility. The following is a summary of the presentations and the subsequent discussion.

Terminology: The terms audit and review are often used interchangeably.  In this presentation the term audit is used although the actual methodology is considered to be a review rather than “a critical inspection”.  The presentation noted the complexity of defining QoC and highlighted one definition which recognised the need to think at both individual and population level, to provide  timely, appropriate and evidence based care, safety and within the context of basic human rights.

Methods and tools for improving quality of care: Three key methodologies/tools were highlighted.

  1. Maternal audit/review: which is accepted across most of sub-Saharan Africa and South East Asia and which is something to work on moving forward. (The Kenyan case study provides an example of how this can be catalysed at district level and brought together at national level to demonstrate areas with sub-standard quality).
  2. Perinatal death audits: which is much less practised.
  3. Standards based audits (sometimes called clinical or criterion based audits): a well appreciated method.

There is good evidence that a combination of all three is an effective methodology for improving QoC, although the linkages may not be immediately obvious especially to health workers in the field. Death audits are used to identify specific areas of substandard care for which standards can be developed or implemented and which are the basis for standards based audits.

Standards based audits: take a multidisciplinary approach across all areas of diagnosis and treatment, management and administration and human rights and dignity.  Not all standards should relate to clinical care.  The focus is on finding solutions rather than just problems, which can be more motivational.

Key learning about standards based audits:

  • They need to be introduced on the back of the accepted processes of audits or death surveillance and review.
  • The standards need to be evidence based and informed by recent developments in care; the process of developing standards allows the opportunity to review potentially outdated processes and procedures (for example the view that all women are given an episiotomy).
  • The standards need to have a very clear objective and defined criteria.
  • The criteria should be divided into: structure (what you need): process (what you do): and outcome (what you expect).
  • The criteria need to be developed through discussion and debate amongst healthcare providers.
  • It is very important to involve all professional groups, including at the higher professional association level, from the start as they will determine how well the standards are adopted.
  • Team work and leadership is important.
  • Standards need to be SMART and desirable.

The presentation includes an example of standards and criteria for meeting the objective of: All women with obstructed labour are delivered by Caesarean Section within one hour of diagnosis (Kenya).

The most important learning is that the standards need to be locally developed in order to be relevant and to be owned by those working with them.  This requires significant time to enable the required discussions, but it is very valuable in overcoming concerns amongst those who may be unfamiliar with the process and/or who may conceive it as an inspection exercise.

Some challenges:

  • Ensuring that the development of standards is not theoretical or imposed; they need to be the result of everyone working together to examine current evidence, to share expertise and to agree relevant standards.
  • Ensuring that the standards are locally developed, relevant and owned.  It is difficult to get people enthusiastic about auditing standards if they are not locally relevant or locally attainable.
  • Ensuring that the process is viewed as a continuous cycle of improvement, not just a one off audit.  The findings from an audit should be the basis identifying solutions and to plan for changes after which the status can be re-evaluated.  This is not yet understood by everyone. Examples of the outcomes of this process in Kenya (Ollier, L. and Stanton, C, 2011)(delivery of obstructed labours and access to antibiotics) and Malawi (Kongnyuy EJ, Mlava G and van den Broek N, 2009a, 2009b, 2009c; Kongnyuy EJ, Mlava G and van den Broek N, 2008a, 2008b; Kongnyuy EJ and van den Broek N, 2008; Kongnyuy EJ, Mlava G and Broek N, 2009) (referral systems) are provided in the presentation.
  • Scepticism amongst participants that it is not possible to address identified areas of sub-standard care due to a lack of resources.  In reality there are a lot of improvements that can be made without additional funding or without necessarily having the structure in place.  The presentation provides an example of where local teams in Kenya have successfully decreased the time between diagnosing obstructed labour and delivery using simple solutions such as jumping the queue for theatre and repairing the lift.

It is easy to challenge the feasibility of developing and implementing standards based audits. Taking action is critical; as the Kenya and Malawi examples demonstrate.  Local quality improvement committees are also important for ensuring that action is taken and there is evidence that they are working.  A recent Cochrane review (Ivers et al, Cochrane Review 2012) into audits confirmed the role of champions to ensure that agreed activities are actioned. CMNH have developed a workshop package.

Links between data and audits: The two are very much interlinked although some suggest that poor data will be a barrier to effective audits.  Audit actually presents an opportunity to illustrate the value of good data and to get people collecting, thinking about and analysing both qualitative and quantitative data.

Evaluating the effectiveness of audit as a quality improvement method: The presentation noted that there is currently insufficient robust evidence but there are methods such as the new indicators from WHO which CMNH have contributed to.


CASE STUDY: Setting up Confidential Enquiry into Maternal Deaths in Kenya

Dr Charles Ameh, Senior Lecturer and Deputy Director of CMNH

Background: Despite an increased spotlight on improved quality of care (QoC) and accountability, Kenya still has an unacceptably high maternal mortality ratio (MMR) of 400 maternal deaths per 100 000 live births.   The country has recently gone through a process of devolution of government and Departments of Health at county level now hold a lot of power.  A recent survey by the Centre of Population Studies and the University of Nairobi established huge disparities in maternal health between the counties.  Out of 47 counties, 15 contributed 98.7% of maternal deaths and MMRs across the country ranged from 600 to 4000:100 000.

In 2004 Kenya made maternal deaths notifiable and a paper based system was developed.  In 2007 CMNH undertook a review of the system and found gross under reporting of maternal deaths and a lack of standardized reporting of causes of death due to differing interpretations of the WHO ICD-10 to deaths during pregnancy, childbirth, and the puerperium: ICD-Maternal Mortality (ICD-MM) coding by service providers.

The data was effectively meaningless and it was difficult to determine any centralized action.  The review also found huge disparities with data reported through the HMIS.  As a result the government revised the guidelines and tools in 2009 but a subsequent review in 2011 found little change in the reporting and, at health facility level, weak leadership, no quality improvement (QI) committees and continued differing interpretations of the causes of maternal deaths.

CMNH supported quality improvement in Kenya: The introduction of the confidential enquiry into maternal death system is the latest in an ongoing programme of QI. The presentation included a diagram showing progress in Kenya since the introduction of institutionalised death audits in 2003 across the two key areas of place of death (moving from the starting point of government health facilities through to all health facilities) and scale of coverage of the reviews (from districts / counties to national coverage).  Additionally the diagram shows the increasing depth of the review processes over time towards the confidential enquires that are being introduced.  The diagram illustrates a framework for ‘starting small’ and expansion into a very comprehensive process; which clearly takes time.

The DFID funded Making it Happen programme allowed CMNH to support the QI programme to address the findings and recommendations of the previous reviews.  This included the establishment of QI committees and the conducting of maternal death reviews at health facilities in three of Kenya’s eight regions; the focus was on public health care facilities and a few high-volume mission hospitals.  The committees were trained in audit and have a designated lead.

The maternal death and surveillance response (MDSR) system was introduced in 2012. This was in response to the fact that the maternal death reviews were taking place in health facilities but that there was no national response to feed the findings into.  MDSR provides the opportunity to count the number of deaths, to identify the causes of deaths and to identify contributing causes whilst also helping to realise strategies to address issues such as inequality and accountability.

The MDSR system was linked to the electronic infectious disease surveillance system and all review reports were uploaded electronically.   Comparison of the electronic data with HMIS indicated that the disparity between information was greater than previously seen with the paper based system; a process challenge.  It was also discovered that those facilities supported by donor-funded projects had better reporting capacity; suggesting a dependence of external support.  A 2015 Ministry of Health (MoH) commissioned review of the system is ongoing.

Addressing challenges:  Whilst there is a high level of political will towards improving maternal health and strong precedents in and learning from countries such as South Africa and Malaysia, there are some challenges to the QI programme. Critical challenges included weak central level coordination to date and poor accuracy of information and data along with a need for improved, in-depth analysis and reporting in order to transform accurate maternal death data into an effective response.  Key activities to address these and to implement the QI programme included:

A National Secretariat for MDSR: This is the result of a coordinated approach involving the MoH, members of the UN family, professional associations and main referral hospitals to advocate for an effective review system.  After consultation a central committee was formed to lead in this area, to oversee the implementation of the QI strategy and to disseminate findings.  National assessors were identified and health workers were sensitised at county level to ensure good buy-in. The National Secretariat was subsequently established within the MoH and its remit includes organising the central review of all maternal deaths and reporting on this.

Developing the capacity of the Secretariat: Working in association with the UK National Perinatal Epidemiology Unit capacity has been built in the confidential enquiry approach.

Developing an electronic audit system: This has been developed based on the very successful MaMMAS software used in South Africa.  It can store and rapidly aggregate information and produce timely reports.  It also allows for in-depth analysis by variables such as demographics, initial diagnosis which can be compared with final diagnosis of the reviewers, underlying causes of death and contributing or associated factors in deaths.

Other challenges included suspicion about the process and hesitation in releasing information such as case notes.  The Secretariat has a dedicated team working with the MoH to build on previous sensitisation activities to improve access.

Outputs to date: The initial few months were mostly dedicated to advocacy to ensure the required buy in and to the development of the MaMMAS tool. By the end of the first year the programme hopes to have trained 40 assessors and to be supporting a minimum of four review meetings per year; a total of 780 deaths will be reviewed.  The first interim report will be available at the end of month 15.

The programme has been catalytic in several ways such as increasing ownership of the process and increased capacity around data collection and management at health facility level.  The MoH’s capacity to manage the confidential enquiry process and to utilise outputs has been increased and there is strong potential to increase the accountability of the MoH.

Key learning:

  • Long process: Advocacy requires time
  • Rethinking: Surveillance system needs a more innovative approach
  • Catalytic
  • Essential ingredientsfor an effective/successful MDSR:
    • Strong structures put in place
    • National committee supported by competent and motivated assessors
    • National ownership
    • Political will and funding
    • Ownership at level of MoH and professional associations

The following is a précis of the discussion that was had between those present:

Data: Questions were raised about the accuracy and use of data as reflected in the Kenya case study; in particular a request for feedback on how data was used and how the impact on QoC was demonstrated in order to ensure a willingness to improve the accuracy of data.

The response confirmed that audits/reviews predating the programme were not undertaken consistently and that the quality was not adequate. The 2012 DFID funding allowed for the QI programme to be put in place, for training in how to conduct proper audits to be undertaken and for other recommendations from previous reviews to be actioned.  Central level coordination was also strengthened in order to build a national picture of the causes of deaths and contributing factors.  Reviews to date have shown that 50% of deaths by haemorrhage were Caesarean Sections.

Further investigation shows that senior obstetric staff were involved in less than 20% of those cases. This was feedback to the providers who were unaware of the situation and is an example of the value of accurate data. The programme allowed a national perspective which requires a national policy response.  This is the value of supporting at health facility level and a central coordination for the whole process.

With regard to the quality of data one study looked at the information in case notes.  It noted that many health workers are not trained to correctly identify the cause of death and may not understand the ICD MM classification and therefore don’t extract the information from the case notes; even where it is available.

Advocacy:  Questions were asked about the advocacy activities involved in the Kenya programme; what role did civil society movements and or community groundswell play?  Why did the MoH have such a level of commitment?

The Beyond Zero campaign is a significant factor; the involvement of the wife of the President has really pushed MNH issues to the front. The free MN health care service was put in place soon after the new government came to power partly as a result of her advocacy, resource mobilization and fundraising.  As an example of the political will the MoH “will listen to you” and it is possible to access senior officials to discuss the programme, they also attend meetings and demonstrate their support.

Both a top-down and bottom-up approach are needed.  High level support is important in moving to a national position but the bottom-up needs to happen at the same time.  Health workers in facilities are desperate to improve quality, they just sometimes lack the know-how, they haven’t heard of the ICD MM classification and see MDSR as complicated.  This needs addressing at the same time as the ‘top level’ advocacy and activities are underway.  The inclusion of women is also ideal in setting standards.  There is evidence that they want to be involved in key decisions relating to their health and healthcare etc.

The role of communities has become a political issue in some countries that will bring pressure to governments.  An example in India has been documented where a case was not well managed and the community involved the media.  This is a disadvantage of advocacy; it often goes to the press and it usually becomes judgemental.  Sometimes rightly so, but it destroys the self-evaluation and improvement aspects. The Sierra Leone example of the Health Centre Management Committees with exiting links to the health system works without destroying a positive process.  That is possible.  There has not been much to date in the programme but this would be a good platform to involve the community.

Connections between free services and quality: Touched on already as delivering an influx of clients.

Inclusion of the non-public sector in the Kenya programme: Were both public and private sector involved? Was there any different in the response between the different providers?  How interested are the private sector? Is there a difference in the type of maternal deaths happening in private facilities?

A few mission hospitals were involved which serve as referral centres in their locality; at the request of the MoH.  But started with high volume public sector.  There is another organisation working with the private sector.  They use financial incentives and this may be a good way to enter the private sector – they actually invested in helping them raise their quality although the funds had to be paid pack (in kind through service provision).  Generally the private sector doesn’t promote deliveries as they are not so lucrative, Caesarean Sections are different.  Definitely a need to improve the QoC in private sector.  There are concerns about over medicalisation in some facilities/countries so a different approach is needed and this has to come through the finances. Both a carrot and a stick.

We don’t know if there is a difference in the types of deaths happening in private sectors and the relationship with possible over medicalisation and, for example, whether this reduces the number of deaths.  It is difficult because it is a selected population using the facilities and often the facilities are not under the remit of the government; the government may have no say over what happens or whether standards are adhered to.

Monitoring compliance with regulations may be within the government’s remit and legislation is often in place but sometimes the systems don’t work, it’s not a priority or it may not actually be written into specific roles etc.  With the private sector burgeoning this needs consideration.  There is a process underway in India looking at accreditation, ie your facility has to be accredited in order to take patients, but there are many questions about the process, who manages it? What are the components/criteria?  etc.  The government wants to manage it and they may be able to get some control over facilities, but at the moment they do not have control. As mentioned financing mechanisms may also play a role.

Sustainability: Often audits need to be driven by a champion or is a financing system more effective?  The South Africa system is reliant on volunteers, how does this relate to your work? Were midwives involved in the Kenya programme?

CMNH has no experience of results-led financing but it has worked well in countries such as Bangladesh (Raven J, Utz B, Roberts D and van den Broek N, 2011)in improving QoC.  Not sure what criteria are incentivised but it is well documented.

The South Africa system does rely on voluntary contributions from clinicians to work as assessors and it is seen as part of their professional duty.  But it is acknowledged as additional workload for which some want remuneration.  As a matter in principle (as a UK training obstetrician and gynaecologist) it should be part of your duty to examine your QoC and it is already embedded in many aspects of existing workloads such as meetings.  It should be part of professional practice but that is unlikely in some places.

We are advocating for the assessor role to be linked to professional development schemes (eg CPD or CME) and to involve professional organisations more.  This is being introduced in some places, but slowly.  There are various models but no central international agreement on exactly how this could work.  There is some discussion within the private sector with insurers; if the insurers insist that a facility is accredited only if it undertakes audits it will be implemented very quickly.  So there are a number of levels which need to be integrated into the system.

We can also borrow from other QI approaches.  For example recognition of good audit practices through an award scheme, Presidential recognition or a trophy etc.  Sustainability within the Kenya context has been well considered.  There are challenges, despite good sensitisation and buy-in from clinicians many are too busy and have private health work considerations – it’s a hard sell to advocate for participation in reviews.  The BJOG supplement includes case studies from South Africa, Malaysia, the UK and elsewhere.

In Kenya the national secretariat is embedded within the MoH and is funded until May 2019 and the programme has been based on ensuring national ownership and on developing national capacity which will be retained after this programme.  Reassurances have been given by the current government that they will sustain this in the future.   There are 3 years to make sure that this is planned for.

Midwives have been involved at all levels including the Nurses and Midwifery Council.  In fact the midwives took off earlier than the obstetricians.  You have to have the midwives involved and in many instances they are the deputy chairs or chairs in some facilities.

Wider perspective: How reflective is the Kenya example of the global situation?

Globally there is agreement of the need for a new emphasis on quality improvement.  We may be in the lead a little bit in how to actually do that, with all respect.  The USA has a different system to ours with different terminology; it’s essentially the same as the audit cycle but it’s more single-approach than consensus approach.  So that’s two different approaches moving forward. UK is very much in the lead with regard to confidential enquiries.  This is not implemented across the USA; maybe three states us it.  No one is talking about perinatal deaths.  There is a call to UK professional associations to help with this but it is a sensitive issue, even at state level and requires time and know how.

So in that sense Kenya is ahead of the game.  The only other countries with functioning MDSR systems are Malaysia, Sri Lanka (although nothing is published), South Africa (the shining example for Africa) and Malawi is starting.  India has done in Kerala and Tamil Nadu states.  The BJOG supplement provides more information. Various countries have approached CMNH and the know-how exists but there is still a need for catalytic investment to get it in place; this is the focus of advocacy.

This seminar relates to a number of key themes and issues outlined by the learning resource:

Maternal death surveillance and response (MDSR) approaches Professional strategies (Section C)
Accountability mechanisms Regulation, multifaceted and other strategies (Section C)
Health information and vital registration systems Data sources (Section D)
Standards-based auditing for QI Norms and Standards for quality (Section D)
Indicators of quality Capturing health outcomes (Section D)

The following resources provide more information about this specific seminar, and related examples:

  • BJOG: An International Journal of Obstetrics & Gynaecology. Issue: International Reviews: Quality of Care, September 2014, Volume 121, Issue Supplement s4 Pages 1–171 – read full text
  • Raven J, Utz B, Roberts D and van den Broek N (2011) ‘The ‘Making it Happen’ programme in India and Bangladesh’, BJOG: An International Journal of Obstetrics and Gynaecology, vol. 118, no. S2, pp. 100-3. – read full text
  • ‘Making it Happen’ – project website
Seminar: Auditing for Quality Improvement was last modified: February 23rd, 2016 by Adrian Bannister

References - View all