What is the JIT? JIT stands for "Just In Time," and we use it to describe a Dalvik JIT compiler, which was added to Android with the 2.2 release. It compiles bytecode into native machine code at runtime. Essentially it takes the code for an app, analyzes it and converts it into something that runs faster. It does all this while the application is running, and that's where the "just in time" tag comes from. The JIT compiler designed for Android also can do this with a very short "warm up" time, meaning it doesn't take very long to analyze the code before it starts working. It stores information in a cache in your phone's RAM, which means it's not an ideal solution for devices with low memory. It's been optimized to have a small footprint -- about 100K per process -- but even that is enough to impact performance on older models like the G1 or HTC Magic. This is why most phones that came before the Nexus One never got an official version from Google -- hardware limitations.
You hear terms like "runs faster" or 4 to 5 times performance increase anytime you talk about Android's JIT compiler. A JIT compiler simply saves CPU cycles -- more work can be done for each clock cycle. This means applications that were throttled by CPU performance get faster, and apps that are "rate-limited" (run until they are finished without taxing the processor to the maximum) finish faster and use less battery because of it. Not all applications see a significant speed increase, and most applications written with the NDK or in native code won't see an increase at all, as they don't use the Dalvik virtual machine.
If all this reads like Greek to you, that's OK. To the end user (that's you and me) all we should see is better performance in most of our applications. We certainly saw that when Froyo was released for the Nexus One, and things have just gotten better with every release. For the more technically inclined, check out the video from Android engineers Ben Cheng and Bill Buzbee during Google I/O 2010 where the JIT compiler was introduced.