An MMU plays two major roles: it translates virtual addresses into physical addresses, and it allows the possibility of trapping (also known as faulting and various other names), i.e. executing code (in the kernel context) when a process dereferences certain virtual addresses.
There is a third related capability, that of having different mappings and different traps for different processes. This is a consequence of executing code to change MMU settings during a context switch. This doesn't require anything special about the MMU (but most processor designs have features in the MMU to make this more efficient).
Some processors (typically high-end microcontrollers, such as the ARM Cortex-M3) do not support translations but nonetheless support trapping: they have a memory protection unit (MPU). An MPU allows the kernel to prevent certain addresses from being accessed. Thus you can enforce memory separation between processes at runtime with only an MPU, you don't need an MMU.
If you have an MMU, you can use it as an MPU. It's up to you to design your kernel such that every virtual address is either always mapped to the corresponding physical address or unmapped. (An unmapped address triggers a trap.)
There is a version of the Linux kernel that supports processors with an MPU but no MMU: µClinux. The code is integrated in the Linux source tree under
nommu architectures. The lack of an MMU brings some limitations, in particular the impossibility of implementing fork.
An important thing that you give up if you have no virtual memory is the ability to load every program in the same address space. This makes it considerably easier to compile programs. For example, with virtual memory, you can choose to always load the code of a program at a certain predefined address, store global variables at a predefined address, etc. If every process is stored at a different address, you need to generate position-independent code all the time.
Another limitation of having no virtual memory is that memory management becomes more difficult, because of fragmentation. Suppose you have four consecutive physical memory pages of 4kB each, numbered 0,1,2,3, and you first allocate four page-sized objects, one in each page; then you free the objects in pages 1 and 3. With virtual memory, you can allocate an 8kB object made of pages 1 and 3 by mapping them at consecutive addresses. With a direct physical mapping, page 1 is lost until you need a single-page object or the surrounding objects are freed.
If you want, you can use your processor without activating its MMU. All processors boot with the MMU turned off, because you can't do anything useful until the MMU tables have been initialized to something sensible. With no MMU, you can't enforce any isolation between processes. You also can't swap, or have memory-mapped files.
With address 0 mapped, take care if you program in C or most other languages. There are a lot of runtime environments and non-fully-portable C code that assume that an address of all-bits-zero is the
NULL pointer, which does not point to any object.
Depending on the architecture, there may be no way to access some hardware without mapping it into memory. This requires having the MMU up and going. You can still pretty much ignore the MMU by mapping the physical RAM from address 0, mapping devices above the last RAM address, and never changing the MMU mapping.
All in all, by eschewing virtual memory, you're simplifying some parts of the kernel, but making it a lot more difficult to write programs that do anything useful. There's a reason why all high-end processors have an MMU: it's difficult to do anything sophisticated without it. MMU management is not the most difficult part of writing a kernel if you keep it simple; if you find it “mind-boggling”, you have a lot to learn about low-level programming. Writing a kernel with MMU management would be a very good learning exercise.