No, songs don't have to stick to one tonal center (key signature). Play what sounds good but listen to see if each part progresses naturally to the next part. Changing keys without a plan or structure can make a song sound like several different songs shoved into one. A good tool is to keep an aspect of the previous key when you change keys. Such as a common rhythm or dynamic. This tactic links the two contrasting parts into one chain.

Yeah you could use a bass fill in the first riff over E. The fourth riff however will probably sound more like F minor. E can be used as a harmony for V in F minor, and then F the tonic. Classical views on modulation say that when a change of key is done, it should be done at a cadence or have a cadence immediately following to firmly change the tonal center of the music.

This seems right because the first riff right of the bat is in E minor, the last note A# (or as I'll put it Bb) is the 7th in a C7 chord. A C7 chord in E minor (bVI7) is a common harmony, but more advanced in it's understanding. Blues and some rock music uses it quite often. Back to where I was going. Since it's going to F minor in the fourth riff. Bb and E (first note in the fourth riff) both are in the the C7 chord. The next note that follows is F, the tonic note of the new key. Your cadence and a secure key change. However this depends on what you are actually playing, so use your own judgement.

There's also theories that don't stick with classical views of key signatures, such as chromatic harmony and atonal music. I say play what sounds good and you won't go wrong.