The tools available now to help you write a compiler are very good, so it isn't really that difficult. Tools to help with scanning and parsing are particularly obvious. There are some tools to help with AST handling (treecc, for example, is a simple AST-nodes-with-multiple-dispatch-operations tool).
Perhaps more importantly, there's back-ends like LLVM.
If you look at the LLVM tutorial (http://llvm.org/docs/tutorial/) you'll find it starts with a simple pure functional language, then adds mutability etc later. The reasons relate to the Static Single Assignment form which is, I believe, quite widely used in compilers - not just in LLVM. Though I'm no expert, so I could easily be wrong.
So it probably does make a difference which kind of language you start with for reasons of intermediate-code handling, as well as the parsing issues that other answers have already mentioned. The easiest first steps probably depend on what kind of intermediate code you generate, and the abstract machine model it's based on. And starting with a pure functional language is probably a good idea in general.
BTW - "pure functional" here doesn't necessarily imply first-class functions, higher-order functions etc. Evaluating arithmetic/logic expressions may be all you need before you start adding imperative stuff.