A while ago I threw together this high voltage generator. In total, it cost me about 4 dollars. I don't actually know what the output voltage is, because it varies a lot and I don't actually have any good tools to measure it
Here is a video of it in action

If you're wondering how this only cost 4 dollars, here's a cost breakdown:
This device is what is known as a relay inverter. To explain it simply, the relay switches on and off really fast, which also switches the power into the transformer, which creates short high voltage pulses on the output.