On Diversity and Tolerance: James Fitzjames Stephen vs John Stuart Mill

First of the Month readers might not warm to a Victorian criminal lawyer and judge who believed that law and morals were inseparably linked and for whom capital punishment was the bedrock of an effective system of justice. Offenders would emerge from the court presided over by James Fitzjames Stephen (1829-94) with their character in shreds and facing either a long and harsh period of incarceration, or the gallows.  The judge, Stephen believed, was merely the servant of the public’s sense of righteous indignation, and was duty bound not to disappoint.

However, law enforcement was only the day job for Stephen; he doubled as a prolific and combative writer who waded into every controversy – political, literary, theological, and scientific – that unsettled his age. He has recently featured in Russell Jacoby’s perceptive book, On diversity: the eclipse of the individual in a global age, as the robust critic of John Stuart Mill. When Mill published his liberal manifesto, On Liberty, in 1859 advocating maximum individual liberty, Stephen was initially sympathetic. But over the next thirteen years, including the last three spent in India, he sensed trouble with some of its leading claims, particularly its reduction of liberty to a ‘simple principle’. He went in for the kill in a series of essays brought together in 1873 as Liberty, Equality, Fraternity.

In his book, Jacoby hails Mill’s understanding of diversity in terms of ‘individuality’ against the counterfeit model of today’s diversity champions centred on ‘choices’ and ‘inherited characteristics’. By individuality, Mill meant the pursuit of a distinctive plan of life, entailing relentless self-development, self-improvement, and self-dissatisfaction in roughly equal measures. Awesome, especially to the fragile modern psyche. But for Mill this was the essence of a progressive society, a useful reminder of the constantly shifting meaning of that much lauded term.

What individuality most required was the removal of all restraints on conduct, expression and belief that affected individuals only. In particular, received wisdom should be questioned with impunity, and conformity renounced.

Stephen, however, believed that these measures would prove fatal to society, particularly in subverting religion, the mainstay of public order and social cohesion in turn. He also challenged Mill’s doctrine of liberty for flying in the face of experience. All educators, Stephen asserted, seek to close down debate in order to mould and shape the minds of their charges; look no further than Mill’s father, who notoriously indoctrinated his precocious son with the Utilitarian creed associated with his friend, Jeremy Bentham.

Rulers were no different from educators in this regard. Readers may have noticed the slippage in Stephen’s argument here from ‘is’ to ‘ought’, what exists to what should be, or at least, cannot be otherwise. The evils of this slippage was precisely Mill’s point.

However, before condemning Stephen as a vile instrument of Victorian reaction, no more suited to an age dominated by BLM and widespread insistence on democratic ‘inclusion’ than the egregious Trump, let’s see if there is not an edgy side of him. Mill’s saintly status as the ultimate champion of diversity might take a hit at the same time.

We must first get past the oppressive baggage that Stephen brings with him into the twenty-first century, pausing only to issue the now customary trigger-warnings.

First up is his background, which exuded advantage of the kind now firmly on the back foot. He was the son of a leading civil servant, Sir James Stephen, who administered the colonial empire almost single-handedly, an oppressor par excellence in BLM terms, redeemed only – and perhaps not even then – by his responsibility for drafting the 1833 legislation that abolished slavery in British colonies.

An Eton and Cambridge education and membership of the innermost Cambridge elite, ‘the Apostles’, opened up extensive networks of access and influence to Fitzjames – ‘Uncle Fitzy’ to his mocking niece, Virginia Stephen, later the writer and novelist Virginia Woolf.

As a young barrister, JFS made ends meet through hack journalism, although of the superior Stephen kind associated with his father (who also moonlighted in the literary trade) and his brother, Leslie, father of Virginia and her equally talented sister, Vanessa Bell.

The Stephen brand of journalism collapsed the boundaries between serious thought and letters, in Fitzjames’s case injecting it with a high dose of literary vigour. This did not always endear him to his contemporaries, who were no snowflakes themselves in the literary department.  To one of his critics, Frederic Harrison, Stephen’s ultra-masculine style had more than a touch of ‘brawniness’ about it; he was always flexing his biceps in the manner of the playground bully. Stephen’s sins were further compounded in Harrison’s agnostic eyes by his belief in ‘hell’ – any ‘hell’ would do so long as it was sufficiently hellish to deter the would-be criminal through fear of ending up there.

Alas, we must also acknowledge Stephen’s vehement belief in the natural inequality of men and women to which Mill seemed impervious in his advocacy of equal rights. As the weaker sex, women enjoyed the protection of men, which would no longer be assured if men and women were placed on the same legal and educational footing. It is easy to see through Stephen here as a Victorian patriarch affronted by Mill’s attack on male domination, which to him was like a red rag to a bull.

Finally, we should acknowledge what many would regard as his shortcomings as prosecution counsel in the Governor Eyre case of 1867, recruited by the Jamaica Committee of which Mill was chairman. The case was brought against two military officers in Jamaica for the murder of an opponent of colonial rule following the brutal suppression of an uprising, with Eyre also in the firing line as the source of the order. The two officers were acquitted, helped by Stephen’s insistence on addressing them as fellow gentlemen, different from those ordinarily on trial for murder. While Mill and the Committee persisted in their attempt to bring Eyre to justice, Stephen walked away, believing that the legal case had run its course, in which he was proved right; the moral dimensions of conduct in a public capacity did not seem to interest him.

It sounds as if Stephen should be cast into the furnace of hell himself, the hell reserved for offenders against liberal self-righteousness. But that’s just where he starts to become interesting. For surely, if we are to embrace ‘diversity’ in full, that modern deity needs some credible opposition, certainly to its prevailing liberal face. While Mill never imagined that ‘diversity’ would become a new orthodoxy, Stephen showed some insight into how this might come about.

How could Mill have created a minefield that has blown up in liberalism’s face? We need to look first at the ‘simple principle’ that for Stephen lay at the root of the problem with Mill’s thought.  Briefly summarised, it maintained that the sole ground for interfering with a person’s liberty of thought and action is ‘self-protection’ and ‘the prevention of harm to others’. This was a noble aim, although one beset with difficulties, not least arising from the question, what constitutes harm to others?

For example, in matters concerning freedom of speech, to what extent does the ‘harm principle’ cover the giving of offence as opposed to causing physical injury? Attacks on both conventional and unconventional ways of thought and behaviour can be wounding; some of those affected suffer in silence, others protest, sometimes violently as in the Charlie Hebdo case. Does this mean that the pain threshold in respect to freedom of expression must be set at its lowest level? If so, might this not make freedom of expression a hollow liberty?

This is certainly a point Stephen made in his broadside against Mill. Freedom of thought would not lead to the easy ‘win-win’ situation his opponent seemed to think.  While Mill recognised the problem of inflammatory speeches delivered in locations likely to stoke physical violence, he was seemingly blind to the distress caused by onslaughts on people’s most cherished beliefs.

Stephen was not. In his view, the sufferings of persecution in former times compared favourably with the devastating effect on individuals of their beliefs and practices being exposed to sustained ridicule. There was something heroic about dying for one’s faith by the rack and the stake; but the weapon of eloquence could be equally deadly, slowly destroying from the inside while denying its victims the honour of martyrdom. This was especially true of those who were not able to give as good as they got.

So did he want to return to the days of the Inquisition? Not at all. He was as committed to the value and importance of freedom of speech as Mill, albeit for those who had taken the trouble to acquaint themselves with the subject-matter in hand. How this would work in practice he failed to spell out, although the suggestion of a duty to inform oneself fully before exercising the right to free speech is perhaps not without merit.

However, unlike Mill, he accepted that freedom of expression could never be free of harmful consequences. But at the same time, he believed that it was impossible to mitigate the harm without compromising the principle itself. If the less articulate members of society had to be abandoned to their fate, there would seem to be even less scope in Stephen’s argument for justifying the ‘no-platform’ policy which is now commonplace on university campuses.

Oddly, ‘no-platforming’ is deployed to defend ‘diversity’, which for Mill was integral to freedom of speech.  If diversity has now trumped – or is threatening to trump – its Millian root, this suggests the need for rethinking the relationship between diversity (or ‘variety’) and freedom.  We may not like Stephen’s account of how they can be held in balance; but it might repay attention in considering how freedom can regain some of the ground it has lost recently to the diversity culture.

Chief among Stephen’s argumentative strategies was something we’ve touched on already: his appeal to ‘experience’ as the touchstone of truth.  He maintained that great shifts in thought had never been the result of individuals reaching their own conclusions through the application of rational principles. Instead, these movements reflected powerful swings of support behind a few leaders capable of taking the world by storm. Against the force of these tidal waves of conviction, Mill’s ‘simple principle’ of liberty was akin to ‘blowing against a hurricane with a pair of bellows’. Human beings were highly suggestible, and Mill’s vision of a humanity much improved by the spirit of tolerance belonged to the realm of fantasy.

For Stephen, Mill was himself imbued with the leading shibboleths of the age: liberty, equality, and fraternity. We have already noted Stephen’s disdain for equality as applied to the sexes; in his eyes, fraternity fared no better as the ‘worship and service of humanity in the abstract’. What did he have against liberty? Unlike Mill, he maintained that it was neither an absolute value nor a universal ideal; like fire, it was good in some circumstances, bad in others. Strikingly, Stephen painted a sympathetic portrait of Pontius Pilate who he likened to a colonial governor in India, faced with the rise of a new religious movement led by a charismatic reformer that threatened the foundations of civil order.

Stephen’s contemporaries were not impressed by this demotion of liberty from its elevated status as the highest spiritual good, that which alone makes human life worthwhile.  One contemporary – R.H. Hutton, editor of The Spectator – was moved to write that ‘Mr Stephen’s theory tramps over the most delicate blossoms of human life and character with a heavy, elephantine tread’.

Perhaps the most baffling thing about Stephen is that he claimed to be writing as a liberal, although one who believed that liberalism was fast becoming corrupted by radicalism. How could he treat liberty in such a cavalier manner? If liberty could be given and taken at will by the sovereign power depending on ‘circumstances’, what distinguishes ‘liberal’ rulers from their authoritarian counterpart?

The obvious answer might be, not much in these days of pervasive and extensive government intrusion into people’s personal lives, justified as a necessary price for maintaining a free society.

But what if liberty is associated with a set of values such as individuality, diversity, energy, and spontaneity rather than a cover for engaging in mass surveillance? Can these values be maintained while recognising the necessity at times for narrowing the sphere of liberty? Stephen thought so. Indeed, against Mill he maintained that individuality expands in inverse proportion to the sphere of liberty. Emancipated from rules, discipline, hardship, and fear, individuals would not take the trouble to exert themselves. ‘If you wish to destroy originality and vigour of character, no way to do so is so sure as to put a high level of comfort [i.e. liberty] easily within the reach of moderate and commonplace exertion’.

For Stephen, liberty was certainly the key to individuality and to a harmonious life; but only when controlled by a regime of law that at the same time recognised its limits in preventing ‘harm’. As Jeremy Waldron has argued (The Harm in Hate Speech (2014)), laws against ‘hate speech’ should be confined to the denial of a person’s capacity for equal citizenship on grounds of religion, gender, sexuality, race, ethnicity or some other identity. Speech that some find offensive but does not touch the citizenship issue is not ‘hate speech’. Without the capacity to express disagreement, liberal society is seriously imperilled.

This is very much in the spirit if not the letter of Stephen’s thought. He relished intellectual combat and the freedom of expression that made it possible; but he insisted that clashes of opinion should be conducted in a spirit of ‘fair play’ rather than policed heavily by law. To his mind, the need to strike a balance between unfettered liberty, on the one hand, and war to the finish, on the other, was the essence of liberalism. This is perhaps worth revisiting in the social media age, plagued as it is by both extremes.