What happens to a service which becomes disconnected from the preferences of those who need it? We know the answer to that one: we get something like Communist medicine, or milder versions of the same effect e.g. nationalised utilities. But at least in most such cases, users have ways of expressing dissatisfaction outside the supplier-user framework (e.g. by getting their complaints reported by the press) and a clear incentive to do so.
What if the service doesn't have any users who really need it, so that there is no such incentive? And what if there is no easy way of telling whether the service is usable? You are liable to find that the service develops in strange ways, according to the complex internal workings of the supplier group, and has increasingly little relation to anything that is actually useful or wanted by anyone outside the supplier group.
Judging by my experience of working for accountancy firms, this disconnect between suppliers and users occurs even for services which do face competitive pressures. A surprisingly large part of what happens inside professional firms (meetings, courses, human resources initiatives, etc.) seems to have precious little to do with what will actually provide clients with what they want — or 'need', though I avoid the use of this word, as it is often used to imply that suppliers are better placed than consumers to decide what's best.
However, competitive pressure at least places a limit on how far this disconnect can go, before a firm goes bust because its clients decamp to another one which can do the necessary work for less money and/or with fewer distracting frills. How much worse can it get when there is no competitive pressure from users?
Imagine the following scenario. A bunch of intelligent people get together and create — using funding that is more or less unconditional — a system for generating intellectual output. However, this output does not have to pass any particular test except whether a majority of system insiders agree it is worthy. So the members of the system are entirely insulated from assessment other than their own. Like any social group, they create a hierarchy of rank, in which some are allowed to progress to the top of the ladder depending on criteria which the group as a whole decides on. What is the likely outcome? And what happens if there also starts to be an ideology which places pressure on them to produce results which fit with, rather than go against, that ideology?
It depends on the different motivations present, and which are strongest on an aggregate basis. The desire to (a) advance knowledge probably doesn't occur that frequently, but perhaps we should allow that it occurs among at least some of the members. However, there is also likely to be (b) the usual range of motives which in a collective setting typically outweigh any more idealistic aims: desire for career advancement, desire for power, desire to do down your rivals, the formation of power cliques, it mattering more whom you know than what you know — and all the other usual aspects of office politics. Of course, the 'business' in question being intelligence and creativity, the criteria for advancement will be superficially based on cleverness, innovation, and so forth. It's just that this won't necessarily have much to do with real usefulness or progress.
Given the implausibility that (a) will triumph over (b) and continue to do so, what is the likely outcome? Probably something that looks rather like contemporary academia, as in the following description of management studies.
Chief executives ... pay little attention to what business schools do or say. As long ago as 1993, Donald Hambrick, then president of the US-based Academy of Management, described the business academics' summer conference as "an incestuous closed loop", at which professors "come to talk with each other". Not much has changed.With the exception of applied sciences, this description probably applies reasonably well to most other academic disciplines.
[An academic writing in] The Academy of Management Journal says: "Most of what we publish isn't even cited by other academics." ... In article after article in the Journal, the business school professors lament their inability to research and write about their work in a way that real-life business people understand.
... A scroll through the most recent issues demonstrates why managers may be giving the Journal a miss. "A multi-level investigation of antecedents and consequences of team member boundary spanning behaviour" is the title of one article. Why do business academics write like this? ... to win tenure in a US university, you need to publish in prestigious peer-reviewed journals. Accessibility is not the key to academic advancement.
Similar pressures apply elsewhere. In France and Australia, academics receive bonuses for placing articles in the top academic publications. The UK's Research Assessment Exercise, which evaluates university research and ties funding to the outcome, encourages similarly arcane work. But even without these incentives, many business school faculty prefer to adorn their work with scholarly tables, statistics and jargon because it makes them feel like real academics. (Financial Times)
But it's not much good expecting confirmation on this score from insiders. "Yes, I am a paid academic, dependent for my career on the approval of my peers, but I can tell you good people at the Times/Guardian/whatever that my department is useless at promoting genuinely progressive work, rather than work which will reinforce the prevailing paradigms." Not very likely to happen.
Part 2