Home > Musings > Measuring ‘Development’

Measuring ‘Development’

October 26, 2010

People are obsessed with measurements. Each one of us wants to constantly measure everything around us. Whether it is our own personal wealth, or our academic performance or a movie that we want to see or just saw, or the service that we received in a recent flight that we took; we want to constantly measure and rate. It is now become such an integral part of our lives that we do not pause to ask why are we measuring and what are the metrics that we are using. Being in the Development sector, I have been fascinated by the evolving obsession of practitioners, donors, academia and the community in measuring and evaluating. The tools, methodologies and the people involved in this activity are getting better and better. Everyone seems to be so preoccupied and engaged that many consider a program a failure or bad, if some acceptable form of measurement is not undertaken. I have written numerous proposals and implemented many projects in different sectors of health, education and community development that I have found myself questioning not just the validity but also some of the metrics and the fundamental premise that drives these measurements. I am not saying that measurements by themselves are wrong. All that I am trying to point out is that we need to understand the program being measured, the competence of the people measuring, the tools deployed and the metrics of measurement and more importantly the context – before one indulges in this activity.

I would like to explain this a little more clearly with a personal example. I first came to the tribal area in the forests of Heggadadevanakote in Mysore district of South India in 1987 and began running a small dispensary. Obstetrics was one of my favorite subjects, and it was only natural that my focus was on maternal health and mortality. Being concerned by the high maternal mortality in the area, I started exploring how I could bring it down. Public health knowledge and practice has established maternal mortality to be an important indicator of the health of a community. It was also the time when everyone (including the WHO and Government of India) focused heavily on maternal health and safe motherhood initiatives. The whole health sector was being pushed towards Institutional deliveries and we also got caught up in this excitement. Over the next many years, we campaigned for Institutional deliveries and ensured that we created adequate facilities for safe childbirth at our hospitals. We even had a World Bank funded project in 2001-02 to ensure improved maternal & child health outcomes amongst the tribals in the area.

A few weeks ago, my wife Bindu, an obstetrician who has been long associated with the program was remarking how she was seeing a huge positive change w.r.t. the health-seeking behaviors of tribal women. She told me how challenging it was 20 years ago to motivate the tribal women to have Institutional deliveries but now the women sought the hospital on their own. Institutional deliveries, which were non-existent when I first came to the area, had grown to 40% by the end of the World Bank funded project in 2002. Today, virtually every tribal woman in the area comes to our hospitals to deliver their babies. Our Reproductive and Child Health (RCH) programs have been written about and studied; World Bank considers the RCH project that they funded us as one of their best; Public Health practitioners and academicians are impressed with the falling maternal mortality and improved health outcomes. It is indeed very reassuring when so many people and institutions with their sophisticated tools and methods call us a public health success.

A year away from our projects has given me the space, the energy and the willingness to re-look critically at all that we have been doing. I was assessing what it was that I first came to the tribal areas to do and how I had done it. Using the metric of improved health outcomes and the falling maternal mortality and morbidity rates, we were definitely an unqualified success. But was this the right metric to measure our work and intent? Can this metric capture everything that exists in this ecosystem? How honest would it be if we did not try to engage ourselves in outcomes that unintentionally emerged because of our programs, but were not given any attention, as they were not readily visible or worthy of measurement? Or that we were ignorant of the metrics that one needed to deploy?

In our intent to reduce the maternal mortality by increasing institutional deliveries, had we not unintentionally taken away the community’s ability to cope and manage this natural phenomenon without any dependence on people or a system outside their community and tradition? We today have a generation of young women who have mostly delivered their babies in our hospitals but who have neither the knowledge, the attitude nor practice to ensure that they can continue their century old tradition of delivering children at home. What if we changed the metric to building the capacity and competence of the community to have cost-effective and rational health practices that did not need an expensive health care system that they could neither afford not sustain with their resources. Isn’t building the capacity and competence of communities to ensure a workable health system that they can run and sustain with their own resources and abilities more important than running a sophisticated health care program that needs doctors, nurses and managers to come from faraway cities?

Well, the metric of measuring what is important for communities is what the development sector needs to focus on rather than what is easily measurable or merely the programmatic and managerial aspects. And the metric needs to be something that attempts to capture what is happening in the whole ecosystem rather than the just the piece that is the most evident.

Balu

Categories: Musings
  1. Balu
    November 1, 2010 at 2:55 am

    Sudheer,

    While appreciating the enormously good work that is being done at Saragur, i am only trying to communicate that measurements as traditionally understood may not capture all that is happening within the Eco-system. We may need to keep this in mind and understand that complex dynamics occurring within communities and determine how to capture and measure them. I also do not agree that Primary Care has come down, but would like to qualify that the level of Institutional care has enhanced. It is not about the care, but about the inherent capacities in communities. I am again not sure how you have inferred that there is any suggestion of going back to any old era. I am sure that the Gadchiroli model working well is also limited by your determining the metric that you are deploying to call it good. Knowing this geographical area and the work of the Bangs, i am certain that the metrics of measuring their work may also need to be redefined to be comprehensive enough to capture all that could be happening as intended and unintended consequences.

  2. Dr.A.P.Sudheer
    October 31, 2010 at 8:57 pm

    Dear RB Sir,
    Your observations on the current metrics of measuring development goes to the extremes. You are correct that the current model we have developed to provide maternal care at Sargur is not sustainable. This is because as I used to say previously we have ignored Primary health care after moving out from Kenchanahally and focusing on providing specialised care only. If you look at the data you can see that most of the pregnancy related problems in tribal women was due to the cascade effects of malnutrition, severe anaemia and late diagnosis of pre-eclampsia, which finally requires services of OBG person, anaethesists,blood bank etc.A good primary health care system like the Gadchiroly model of Dr.Abhay Bhang which had successfully trained local dais and other local community members to provide quality basic primary care and was able to improve maternal and child health will be cost effective and sustainable in the long run. We can not deny the benefits of modern medicine to the tribals and other deprived sections saying it is not sustainable but we should find ways to provide it in a sustainable way. There is no point in going back to the old era were population was stabilised by high birth rate and high death rate by the nature and human beings were left to their fate.

    • Arun Karpur
      November 1, 2010 at 4:47 pm

      Dear Sudheer:
      Having worked at SVYM at the stage where these discussions were passionately pursued, I would differ from your views that there has been a departure in their commitment to Primary Health Care. I believe the growth and evolution of SVYM continues to embody the basic principles of prevention, even in the formal institutional settings. So I strongly disagree with your perception on this.

      But picking up on what Balu has discussed in this post, I think it is very important that we contextualize a metric that is central to determining a success or failure of an approach. A very narrowly defined metric without contextual dimension is as good as an article published in a journal.

      Another side of this discussion that I see is that it is important for us to embrace the tenets of positivism in our approach to public programs, bringing much needed objectivity through the lens of the community and not from what we feel or what someone else believe that to be true. It is really difficult to be dispassionate in science, but we should consider employing tools that cuts through our passions and provide us a more objective view of all the issues that will ultimately lead to community’s well-being.

      Whether Dr. Bangs approach of training local dais worked or whether SVYM’s model for promoting high quality institutional care in resource limited settings worked from the community’s context are some of the questions that need to be answered in dispassionate scientific inquiry.

      I would like to resound Balu’s post encouraging the scientific community to question not only the validity and reliability of measures, but also question the assumed theory of a proposed model and look at the “appropriateness” of a metric or a measure from the beneficiary’s point of view. Any measure is just a number as long as it is not reflection of the ‘people’ that make it!

      Thanks for a good discussion.
      Arun

  3. Balu
    October 26, 2010 at 5:43 pm

    Great views Arun and i fully agree with you. As we go along with the GRAM experiment, we need to work together to ensure that we could bring in this kind of critical thinking into this whole process. Again, great comments.

  4. Arun Karpur
    October 26, 2010 at 4:50 pm

    Balu: This is such a relevant discussion in the context of increasing interest in development among several for-profit and non-profit entities. I am of a belief that many times groups of professionals inject their personal beliefs in the way of developing “objective” measures. Let me give you an example from the field of disability. There is a fascinating indicator called “DALYs” – i.e., Disability adjusted life years. The World Health Organization defines this as “The sum of years of potential life lost due to premature mortality and the years of productive life lost due to disability.” This definition equates number of years lived with a disability by any individual to “potential life lost.” Meaning that lives of people with disabilities is inferior in quality compared to people with no disabilities. Being in this field from more than 7 years, I have come across extraordinarily achieved individuals who have some significant disabilities as well as some common citizens with disabilities engaged in work, community, and life. None of them complained that their quality of life was inferior as many critical things in the community were accessible. So why does WHO, an agency that promotes a more holistic view of disabling experience as an interaction between individual, physical and social environment, continue to use a metric that may not make sense and sometimes demean the lives of people living with a disability.

    As academicians and practitioners grapple with the idea of measuring the impact of investing our resources, it is very important that we bring the voices of people into our thinking of these measures. Having worked along-side you and Bindu on the World Bank project, I think it is important we built structures within the communities that are sustainable and providing women in local communities with tools and knowledge to have a safe home delivery would be a key strategic move towards permanent reduction in maternal mortality.

    But again, we academicians are driven by an indicator which is measured as an aggregate number. So even one incident of maternal mortality would dent the number, because it is measured as number of maternal deaths per 100,000 live births. Here the denominator averages out the overall experience. No one would have cared to learn from each mother who would have received services from a local Dai (traditional birth attendent) or their mother or relatives in the birthing process.

    Another problem we as academicians and evaluators have is that we do not understand the paradigm of implementation science. The fidelity with which a proposed program is supposed to be implemented and measuring the extent of this implementation is still an elusive science for many of us in academia. Many, if not most, program suffer an unfair judgment without taking into account how each component was implemented. Coming back to your example of empowering women in communities for normal home delivery, the existing data shows that a program run by the Government to train local birth attendants was a failure. But no one is ready to ask why this happened? why local birth attendants did not practice safe birthing practices as was taught by “doctors”? what beliefs of the practitioners needed to be changed or strengthened before having them as community-based resources? what were the issues with their supplies, as simple as scissors needed to cut the umbilical cord? what types of training needs they had on a continual basis?

    I believe, if we can develop contextually relevant indicators by combining program fidelity data with the measures then we can get a better idea of how a public program is continuing to benefit the people it is intended to serve.

    Also, another paradigm that is very important for us to learn is to develop a theory of change model that looks at a program as a set of conceptual paradigms that interact in a particular manner leading to improved health or social outcomes. This theory of change is critical in developing practical intervention strategies as sustainability is central to the feasibility of a proposed concept of change.

    At this time, I do not mean to suggest that we have all these tools readily available for measuring program effects. But encouraging a discussion in academic and practitioner groups to explore reasonable mechanisms to incorporate these ideas into their program efforts. Contextually relevant and sustainable programs will be the true success of any population-based intervention and that means academicians and practitioners alike will have to learn the art of listening to people across different socio-cultural context and work with their systems to bring change. Otherwise, the problems will continue and we all will have the same jobs!

    My two cents to your fascinating discussion.

  1. No trackbacks yet.
Comments are closed.