Mussorgsky was in many ways the most rebellious of the Handful. He supported their nationalism, but unlike them endeavored to find a completely Russian expression of it. He borrowed or invented a lot of Russian folk melodies and he scorned Western styles like the sonata and symphony. Moreover, the impression most people got of his music was roughness: it is unrefined, showing a clear scorn for conservatory practices in its rhythm, its development (or lack thereof), and its use of tonality. It is so unusual that it has only recently been played again in its original form. Rimsky-Korsakov, the most traditionally educated of the Handful, edited it into a more palatable form; he especially took liberties with Boris Gudonov and Pictures at an Exhibition.
Because of the unrefined quality of his music, The Mighty Handful were never terribly supportive of Mussorgsky; Rimsky-Korsakov put forward a typical criticism of his music, calling it "sloppy and incoherent" while more generous critics deemed it "willfully eccentric". I think that in some ways, like Dostoyevsky's writing, M.'s music prefigures realism: it tries not to sentimentalize, and it tries to shock and surprise the audience. Here, for example, is a recording of his piano suite Pictures at an Exhibition, one of my favorite pieces, which I spent a lot of time working on last summer:
Pictures, performed here in approximately its original form, was written in 1874 and is supposed to represent Mussorgsky walking through an exhibit of the works of his recently-deceased friend Viktor Hartmann. The initial theme represents him walking through the exhibit, and is repeated in strikingly different forms throughout the piece. To give a few highlights of the most relevant movements, in their original languages:
1. Gnomus: The first full movement, starting at 1:27, portrays a lopsided gnome lurching about--like the Underground Man, it is deliberately abrupt and repulsive; musically, it is characteristic of Mussorgsky in that it is very onomatopoeic (it sounds like what it represents) and pays little attention to tonality or thematic development. The quiet, pained interruptions hint at a tortured, sympathetic character, something of a favorite theme of Mussorgsky's.
2. Il Vecchio Castello, 4:31: In something of a parody of Romantic nostalgia, it portrays a decaying castle with a broken, repetitive, and similarly abrupt melody. Like the gnome, it is supposed to be somewhat ugly.
4. Bydlo, 10:03: Referencing Romantic depictions of the countryside, it shows a massive ox-cart lumbering over a trail, driven by a peasant whose song interrupts the piece. The heavy, similarly onomatopoeic left hand accompanies a melody is simply repeated with no development before it fades away.
8. Con Mortuis in Lingua Mortua, 18:20: Represents Hartmann in the Paris catacombs. It is the most avant-garde movement; it essentially lacks tonality with only a brief hint at a melody near the end. Once again, it is purposefully ugly, pessimistic, onomatopoeic, and nothing like typical Romanticism.
9. Baba-Yaga, 21:38: Named for the witch from Slavic folklore, this is one of the few truly frightening pieces of music ever written. It is an exaggerate repetition of Gnomus, as it is tonally similar and similarly depicts a grotesque, monstrous figure. The subject shows off M.'s nationalist, traditionalist credentials.
10. The Bogatyr Gate at Kiev (my copy only has Cyrillic letters for the original Russian), 25:18: The last movement is triumphant and majestic, representing a planned gate to celebrate legendary Russian heroes. It recalls to some extent Dostoyevsky's "positive solution" of a return to Russia's roots (Kiev was the namesake and effective capital of the original Russia, the Kievan Rus). My favorite section of the whole suite is the rendition of the Promenade theme at 27:56; we leave the "Exhibition" triumphant despite the increasingly weighty minor-key renditions of the Promenade that lead up to it.
To sum up this ridiculously long post (sorry) I think Mussorgsky's unsentimental, pessimistic music, which only reaches a triumphant end with a celebration of ancient Russian heroes, is very similar to Dostoyevsky's philosophy, which probably arose from their similar community in St. Petersberg; they hung out with the same crowd and drew the same conclusions. They both distanced Russian arts from the West and concluded that only nationalism, not western Romanticism, could solve Russia's problems.