Why doesn't the magma in the Earth's core cool down and solidify? What is the source of energy for maintaining its molten state?
The Earth's core does, in fact, cool down over time, and eventually it will solidify completely. Since the Earth's magnetic field (which protects the atmosphere and biosphere from harmful radiation) is generated by molten iron in the core, the solidification of the core might seem quite foreboding. Fans of the 2003 sci-fi film The Core will know what I'm talking about. Fortunately, the reader need not worry: let's see why.
First things first, let's establish the facts. The diagram below shows the interior structure of the Earth as a whole. Right now, the Earth's core has both solid and liquid components, which respectively make up the inner and outer core (shown in light and dark grey).
When the Earth formed, it would have been entirely molten due to the release of gravitational energy; at this time, the Earth became chemically differentiated, meaning that heavy elements (notably iron, but also scarce species like uranium) mostly sank to the center to form the core while relatively light elements remained in the mantle and crust. The energy released by the formation and differentiation of the Earth is often called primordial heat.
It turns out that, if primordial heat had been all the Earth had to work with, the core would have completely solidified long ago, which is inconsistent with observation. As the question suggests, something else must provide additional heat to slow the solidification of the core; that something so happens to be radioactivity.
As we noted before, heavy elements mostly ended up in the Earth's core. This means that the core contains most of the Earth's budget of radioactive substances. As they decay, these atoms release energy as radiogenic heat in the core, which helps to keep it molten. The timescale for the core to cool and solidify is therefore related to the half-lives of the species that supply this heat, which range between 700 million and 14 billion years. For comparison, the Earth is currently about 4.57 billion years old, so there is plenty of "fuel" left to maintain a semi-molten core.
(By the way, don't worry about the radioactivity in the core -- it is in no way dangerous to you, your loved ones, or your cat.)
In summary, the Earth's core is cooling very, very slowly; some of it has solidified, but it will take many billions of years for the rest to follow suit.