I've not been involved in this thread, but just for what it's worth, the question surprised me, because I've never even thought of it. My own view has always been (and I see no reason for it not to be) that no one is infallible save God alone, and that the only reason the apostles and prophets were infallible when they wrote Scripture was precisely because they wrote and taught under divine inspiration. Now, on the assumption (which is quite an assumption!) that the words we have in the Scriptures are verbatim accounts of what was said by the individuals, then I suppose it's easy enough to conclude that the verbal utterances were, at that time, inspired, and as such would have been infallible.
It is clear, however, that no individual (save Jesus) was infallible at every moment. Peter implicitly taught error when he refused to sit with the Greeks in Galatia, and the Acts 15 council suggests that there was genuine division until James gave his final judgment. If Peter (or any other of the apostles) spoke infallibly in general, then there would be no such disagreement at all. Moreover, Peter himself seemed confused by God's revelation in Acts 10 regarding the cleanness of all foods. It would seem an odd doctrine to me to suggest that words that come from the mouth are infallible, but the thought that produced them is not. Yet if Peter was confused, he clearly was holding at that moment to untrue propositions, meaning his mind was not infallible at that moment.
So, again, I conclude with what seems to me an obvious proposition: the apostles (or anyone else, pope, bishop, church, prophet, etc.) is only infallible when they speak/teach/communicate under the inspiration of the Holy Spirit, for it is really the case that only the Holy Spirit is infallible and that, in those moments, it is He that teaches us, not Peter or anyone else.
I think you have pretty well nailed how Catholics view infallibility. For instance, we say when there is unanimous consent among the Fathers -- who either learned the Faith directly from an Apostle or from somebody who learned the Faith from an Apostle -- that is an infallible teaching. Not everything the Fathers say and teach is infallible, but when they are in agreement on some doctrine of faith or morals, that is infallible.
With the Bishop of Rome, he is only infallible when publicly teaching the entire Church, and binding the Faithful to a certain doctrine. Otherwise, he is as capable of thinking and privately professing error as any other man -- exactly as you say Blessed Peter did with the Judaizers (though we Catholics have a different understanding of that event).
But none of these guys are infallible in and of themselves. They only exercise that gift, which is a manifestation of the guidance of the Holy Ghost promised by Christ, when God so ordains that they should exercise that gift. They are ordinary men called by God to do extraordinary things, and given the graces and the gifts to carry out His will, which is to spread the one true Gospel to all men throughout all time. And if these men are so called to preach the one true Gospel, then that preaching must have a divine guarantee against error, otherwise the Gospel that they preach may not be true. But it must be true, for God Himself has commanded us that we should believe their preaching, and God cannot command us to believe that which is not true.
I mean, think of the chaos that would ensue if a bunch of people just started preaching the Gospel on their own authority and without the divine gift of infallibility. In no time flat you would have several different gospels being preached, and Christians splintering from each other and setting up churches in opposition to each other.