It really depends on what you like. Micro-controllers are always usually one of the first things you could go for if you are willing to shell out some money.
Micro-Controllers
Arduino is probably the most popular choice and has a large fan-base. As such, if you are the type who would need help or direction then this is definitely the choice. I personally have had fun working with a Velleman PIC and "experiment board". You can actually get those from some RadioShacks. Although with this board, you will have to have some decent soldering skills because it requires you to assemble it yourself.
If you are not so much looking to work with hardware, you can always go the route of an emulated older computer.
Old Systems
Obviously you won't have to spend money on this so that is always a plus. There are a way to many systems to list but some of the more well known ones are the Commodore 64 or Apple II. Finding the required files for these two should be no problem. If you want to work with something like the Atari 2600 or NES, that is always a possibility. These will usually entail assembly, but C is still a possibility.
OS Development
Rolling your own OS is a really fun, just don't expect to get some über-special end-result just like you wouldn't expect to write Crysis 2. Maybe you'll achieve a cruddy DOS-like product if you're lucky and pour some time into it. A good place to start if you have no idea what to do for making your own OS is this decent tutorial.