Thomist wrote:
The real problem is rejecting the mandate that Jesus so clearly gave.
The authority from Christ is crystal clear:
All four promises to Peter alone... Sole authority to Peter alone... Later to the twelve also... Thus any attempt to degrade Christ’s mandate to His teaching authority through St Peter is false.
If by "crystal clear" you mean unassailable I think you might be overstating the case.
I am in general agreement with your interpretation of Matthew that he is applying it to Peter personally, but exactly how that relates to the other Apostles and to subsequent Bishops of Rome and bishops of other locations is - among the Orthodox - understood differently. This is the Eastern Orthodox POV:
OCA.org: In Matthew 16:18, the word “rock” refers to Peter’s confession of faith, and not to Peter himself, despite the fact that Peter/rock is a play on the word for rock in Aramaic [petros] and Greek [petra]. As we read in 1 Corinthians 10:4, “...they drank of that spiritual Rock that followed them, and that Rock was Christ,” Who alone is the very foundation, or true Rock, upon which stands the Church. It is on Jesus Christ, the Rock, that the Church’s unchanging faith and confession is firmly rooted.
GOArch.org: (We) can discern in the New Testament an early tradition which attributes a special position to Peter among Christ's twelve apostles. The Church built its identity on them as witnesses, and responsibility for pastoral leadership was not restricted to Peter. In Matthew 16:19, Peter is explicitly commissioned to "bind and loose"; later, in Matthew 18:18, Christ directly promises all the disciples that they will do the same. Similarly, the foundation upon which the Church is built is related to Peter in Matthew 16:16, and to the whole apostolic body elsewhere in the New Testament (cf. Eph. 2:10). It is thus possible to conclude that, although the distinctive features of Peter's ministry are stressed, his ministry is that of an apostle and does not distinguish him from the ministry of the other apostles.