Teaching Excellence Framework (TEF) – Analysis of Metrics Via Scatter Charts

Since it was announced in the 2015 Conservative party manifesto the TEF (a measurement of excellence that will allow high performing colleges and universities to increase tuition fees) has caused quite a stir in the world of higher education.  There has been a lot of opposition, non more so than the National Union of  Students (NUS) which has discussed  encouraging a boycott.  Despite this, it is set to go ahead and institutions are preparing by trying to predict their scores.  Beyond these first results they will have to decide how to improve their scores and which metric to target first.  However, which metric should they start with and are all the metrics related so that a positive impact on one will have a positive impact on the others?  Using Scatter Charts in QlikView we take a look using UNISTATS data.

For the first part of this post will look into the trends for those interested in higher education.  Then, at the bottom, I’ll explain the Qlik techniques used for those interested in Qlik.

The metrics that the TEF will use are:

  • Teaching on my course (National Student Survey)
  • Assessment and feedback (National Student Survey)
  • Academic support (National Student Survey)
  • Non-continuation (HESA and ILR data)
  • Employment/further study (Destination of Leavers from Higher Education survey)
  • Highly skilled-employment or further study (Destination of Leavers from Higher Education survey)

We could easily argue, as many have, that these do not accurately reflect teaching excellence accurately, that such a multifaceted and subjective occupation cannot be measured in numbers, but institutions can at least play the government at their own quantitative game by targeting some of the above metrics so long as it doesn’t impact on their values or indeed any of the other metrics.

The boring but important bit – How I’ve collapsed the 6 metrics down to 3:  The data we’ll be using comes from UNISTATS.  You can download the same data set from www.HESA.ac.uk.  This is not the exact same data that will be used in the TEF but it’s as close as can be using public data.  The data does contain all 3 National Student Survey questions but to save on time and space I’ve gone with the “Overall Satisfaction” question which encompasses the above 3 questions.  The non-continuation score has been replicated (and flipped around to make a positive score) by using the percentage of leavers data and subtracting it from 100% to give a continuation score.  The 2 Destination of Leavers from Higher Education survey metrics have been combined and given an average score (percentage in work or further study added to percentage in professional or managerial roles and divided by 2)

OK, on with the analysis.  The first thing that stands out when you plug the data into Qlik is that the better an institution performs in one metric the better they perform in the others.  This is the good news we were hoping for because it means that institutions won’t have to write off one metric to compensate for another.  The one metric that had the most impact was continuation.  In the blow chart it is compared to the employment score. NOTE:  IN ALL CHARTS THE DOTS REPRESENT INDIVIDUAL INSTITUTIONS.

continuation-vs-employment-or-study-score

The fact that continuation has the most impact is also good news because it’s something that institutions can work on from the first day that students walk through the door.  It’s the reason why ‘learner analytics’ is a buzz word in the sector and why JISC are working with over 50 universities to develop an learner analytics app – see www.JISC.ac.uk.

In the next chart we look at continuation’s impact on the remaining two metrics combined (Y axis ‘score’).  The institutions have been coloured to show an above or below average Overall Satisfaction score.

Continuations vs Score.JPG

It’s clear that as continuation rates improve the rest of the TEF metrics also improve.

Another way that institutions may be tempted to improve their TEF score is by targeting students with better tariffs (better grades).  It would be easy to assume that such students would be more likely to stay in education and to go onto professional and managerial jobs.  Whilst that does seem to be the case, the pay off for such a huge effort to rebrand an institution and compete for higher performing students doesn’t appear to be worth while:

tariff-vs-score  For instance, to increase the average intake tariff from 100 to 140 would only gain you around 5 more points.  It’s far better and cost effective to concentrate on the students you’ve already got and improve promote continuation and career advice.  The charts below break down the impact of increasing tariffs.  The spreads of the points in the charts are much greater, indicating an element of risk in raising tariffs.

Tariff.JPG

It must be acknowledged that some institutions are often small and highly specialised and this causes outliers in our charts.  Such institutions will be more concerned with benchmarking themselves within a smaller group.  However, for the larger institutions the trends from this short piece of analysis offer hope.  Positive changes in one area of assessment breed positive changes in other areas and even if you disagree with the TEF or increasing fees these changes this competitiveness should serve to improve the student experience and make it pay back on their investments with better jobs and, as shown below, better salaries.

score-vs-salary


 

The Technical Bit:

The above entirely consists of scatter charts as these are often the best way to compare different measurements and establish trends.  You can compare 2 measurements (expressions) by putting them on the X and Y axis and grouping the results with a dimension.  wp1

You can size your data points (the dimension) by adding a third expression.  For example if I wanted to size the data points by salary I would add a third expression avg(Salary).  In the presentation tab you can resize the points.

If you want to use colour to distinguish your data points you don’t do this by adding yet another expression.  Instead, expand the first expression and select the background colour.  In here enter your formula for calculating the colour.  You can use some of the standard colours as below or you can always use the RGB() command and create your own colours.WP2.JPG

To add a trend line make sure to select the second expression and use the tick boxes in the trendlines box in the expressions tab.

In the charts we’ve used we’ve broken some golden rule, but with good reason:

The first is not using ‘forced zero’ on the axis.  This may change the angle of trend lines but none of the information is lost and in this case it actually makes it much easier to distinguish the data points.  In all cases where a measurement has been used in multiple charts it has the same axis starting point to maintain consistency.

The second is invoking what some people call the Skittles Effect.  Essentially this means giving every data point a different colour.  In cases with a large spread of data points this just distracts the eye from the lesson the chart is trying to teach.  However, with tight clusters of data points we think it actually helps distinguish the data points.

If you have any questions about how to produce scatter charts please feel free to get in touch.

The Unistats Dataset is reproduced with the permission of the Higher Education Funding Council for England (HEFCE), the Higher Education Statistics Agency (HESA) and their licensors. The Unistats Dataset may be accessed in its original form here: http://www.hesa.ac.uk/support/tools-and-downloads/unistats.  All copyright and other intellectual property rights in the Unistats Dataset are owned by HEFCE, HESA and their licensors.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s