Freudian Literary Criticism

I really don't understand why literary critics are so enamored with Freudian analysis.  I admit that Freudian elements are there in books like The Metamorphosis.  I always wonder what the point of this kind of literary criticism is, though.  Kafka makes a psychological novella (or is it?) that includes ideas from Freud, and then literary critics take it upon themselves to sniff out every faintly oblong object (phallus!) or ambiguous wording and connect it to Freudian ideas (repressed anger!).  What, though, is the point of all of this?  Supposedly literature, like all art, is a reflection and observation of life, human nature, or whatever.  How do critics contribute to any of this by describing their deliberately obscure discoveries?  I think that the point Kafka was trying to make is not contained in some cryptic series of symbols and psychological patterns, but in the overall impression of the book and the way the way the reader feels upon reading it.  That can't be captured or clarified by debating whether Kafka intended multiple levels of symbolism.

My particular problem with the use of Freud for literary criticism is that it takes a theory that is supposed to be grounded totally in the real world (psychoanalysis) and proceeds to use it in a way completely isolated from the real world.  The same sort of thing is done with theories like Marxism.  Critics seem to take terms from Marxism and Freudian analysis and see how they can apply them to works of literature without actually considering their meaning--Gregor may or may not be self-alienated, but Marxism is supposed to reflect the general feelings of real people, not a fictional character created and freely manipulated by an author.  To me it's a weird juxtaposition of supposedly empirical theories and created, manipulated abstractions that doesn't even make internal sense, let alone apply in a useful way to the real world.

An even better example: for my independent study last year, I read The Turn of the Screw.  It was a moderately enjoyable ghost story by the American/British author Henry James, and his obvious intention was to create a subtle, ambiguous ghost story in which the characters never have understanding or control of the forces working against them.  Despite the fact that few people apart from literary critics and students have ever read the book, there are literally volumes of criticism about it, many finding Freudian elements in a book written in 1898 by an author who likely read Freud only much later in his life if at all.  I remember bemusedly reading in one paper that a turret briefly mentioned for geographic convenience in a description of a medieval house was "an obvious phallic symbol".  Papers hotly debated whether the protagonist, the governess, was hallucinating the entire story out of repressed sexual desire for the landlord--never mind that the latter is mentioned only once on page 17--or her male charge has an Oedipal complex causing him to hallucinate ghosts.  Anyway, back to my point, how does any of this help anybody?  Literary criticism as anything other than an exercise to facilitate learning seems not only like a waste of time, but a distraction from the real point of the work, which can best be conveyed by reading it.