Problems with the Allegory of the Cave

Although I think the Allegory of the Cave is an interesting hypothetical much like Zhuang Zhou's dream of a butterfly, I'm not so sure about the leap to government by the philosophers.  Plato says that the man who has seen the outside the cave should return and try to influence those still in the cave to go out and see the light or, at least, to guide them as his new knowledge dictates.  Within the context of the hypothetical, this certainly seems reasonable.  However, where I don't really follow Plato is when he says that the man who has been outside the cave is comparable to a philosopher in real life.  The problem with this comparison as I see it is that the man who is released from the cave gains a tangible experience not available to anyone else--he has real information that nobody inside the cave has.  A philosopher merely uses experiences available to everyone to come up with theories about things.  It seems to me like Plato would be more like a particularly thoughtful person inside the cave, who is just as likely to think that he is merely a figure in a butterfly's dream as he is to realize that he is in a cave.  That is, in real life there is no clear, demonstrable delineation between people who have "correct" philosophies--i.e., those who know they are in a cave--and people who do not; we cannot leave the cave.  From this perspective, the conclusions of the Allegory (that philosophers like Plato are the only ones who see the world as it is and thus the only ones deserving of leadership) seem rather self-aggrandizing.  Besides, I have never really liked the idea of the tangible world being a shadow of some "more real" world--I think Occam's razor provides a much more practical and elegant way of thinking about things.