Names are typically the best way of quickly achieving identification, especially if the person is already know to everyone. But sometimes context demands that you do something other than the ordinary. TV programs are a great modern example of this. I remember one show where a girl ran off to look for distant relatives in eastern Europe. Instead of using some real country that everyone had heard of, the writers made one up: Baltslavia or something like that. There was a problem, though. Since no one knew where it was (or even that it was a country) EVERY TIME they mentioned it (which was a lot) they had to reintroduce it: “She went to Baltslavia, that small European country.” Their choice to make up a new country made the name fairly meaningless because they could not rely on our knowledge of the world for the quick identification associated with names.
The same kind of thing happens with John’s reference to the “other disciple” as in John 18:15–16, and elsewhere during his description of Jesus’s death. We can’t know exactly why the author made the switch, but the consensus view is that John is the intended referent of the expression.
So why switch names? Generally speaking, named people tend to play a bigger role than unnamed ones, like those referred to with an epithet like “the other disciple.” This principle is clearly not a rule, as we see Mahlon, Chilion and Elimelech in Ruth 1 all named, but as Adele Berlin describes them, they are names without faces. In the NT we see the same with some of the apostles who are named, but never participate in dialogue or activity as individuals.
There are also characters who play big roles but who also lack a name, like the woman at the well (John 5) or the man born blind (John 9). Both of these people play significant roles, so the omission of names here is likely attributable to no one knowing them.
So where would obscuring John as “the other disciple” or “the one whom Jesus loved” fit in? It makes most sense to attribute it to keeping the spotlight on Peter by pushing “the other guy” into the background by removing his name. However, the length and complexity of these expressions almost has the opposite effect. More often we see expressions like this purposely used for thematic reasons to recharacterize the person, like shifting from “Judas” to “the one who was betraying him” in Mark 14:44.
Choice implies meaning. John made the choice not to use a proper name to refer to himself during Jesus’s last night, and we associate meaning with his choices. But the nature of the expression draws elements from two different devices: backgrounding by avoiding a name, and thematic highlighting by changing the referring expression to a thematically loaded one. I think the former was the intended function, but the latter the unintended side effect of the decision.
If you have had some Greek and are longing to dig deeper into issues like these, then it’s time to dive into discourse grammar. And there is no easier or more effective way to get started than the New Testament Greek Discourse Bundle.