Why doesn't the magma in the Earth's core cool down and solidify? What is the source of energy for maintaining its molten state?
The Earth's core does, in fact, cool down over time, and eventually it will solidify completely. Since the Earth's magnetic field (which protects the atmosphere and biosphere from harmful radiation) is generated by molten iron in the core, the solidification of the core might seem quite foreboding. Fans of the 2003 sci-fi film The Core will know what I'm talking about. Fortunately, the reader need not worry: let's see why.
First things first, let's establish the facts. The diagram below shows the interior structure of the Earth as a whole. Right now, the Earth's core has both solid and liquid components, which respectively make up the inner and outer core (shown in light and dark grey).
When the Earth formed, it would have been entirely molten due to the release of gravitational energy; at this time, the Earth became chemically differentiated, meaning that heavy elements (notably iron) mostly sank to the center to form the core while relatively light elements remained in the mantle and crust. The energy released by the formation and differentiation of the Earth is often called primordial heat.
It turns out that, if primordial heat had been all the Earth had to work with, the core would have completely solidified long ago, which is inconsistent with observation. As the question suggests, something else must provide additional heat to slow the solidification of the core; this alternative heat source so happens to be radioactivity.
As we noted before, heavy elements mostly ended up in the Earth's core. One might think, therefore, that the core contains most of the Earth's budget of radioactive substances. It turns out, however, the most important radioactive species on Earth -- uranium-235 and -238, thorium-232, and potassium-40 -- are lithophilic or 'rock-loving.' Lithophilic elements readily form chemical bonds with oxygen, which helps them to remain in the crust and mantle while others sink to the core.
As they decay, radioactive atoms release energy as radiogenic heat in the mantle. Much as an electric blanket keeps you warm on a cold winter's night, radiogenic heat has allowed Earth's core to remain hot and molten far longer than primordial heat. Specifically, the timescale for the core to cool and solidify is related to the half-lives of the species that supply radiogenic heat, which range between 700 million and 14 billion years. The Earth is currently about 4.57 billion years old, so there is plenty of "fuel" left to maintain a partly molten core.
(By the way, don't worry about the radioactivity in Earth's interior -- it is in no way dangerous to you, your loved ones, or your cat.)
In summary, the Earth's core is cooling very, very slowly; some of it has solidified, but it will take many billions of years for the rest to follow suit.
Correction: An earlier version of this article stated that most of the Earth's radioactive elements were in the core rather than the mantle. The Curious Team apologizes for this error.