You will need the following:
- A reason for creating a new language
- A Philosophy
- A Semantic Definition
- A lexical description of your tokens
- A Syntax Analysis definition
How will your language be different? What is its mission? Is it functional? Is it object orientated? Is it a meta-language? What are its unique features? What will it give the world that doesn't exist (or exists in an ugly way)? How do you want to change things? Is it compiled or interpreted? A DSL or general purpose language? This is your philosophy and dictates alot about your language's design.
Next, work on scratching out rough syntax and semantics on paper. This will be your semantic definition ... writing fake code is a great way to develop your thoughts. Read "The C Programming Language" for an excellent example of how this is done. Play with it.
You will then need to define your tokens and syntax in some way. Programs then process these into automata capable of reading in strings and processing the syntax. Yacc and Bison use Regular Expressions and a BNF style syntax for lexical and syntax analysis respectively. There are also Yacc and Bison like tools in for other languages.
You will also need a grounding in language theory/compilers to know what NOT to do. Examples include ambiguous grammars, AST generation and manipulation problems and generally how to make life simple for yourself. Knowing the theory is very important. I would consider getting the following to start off:
Compilers: Principles, Techniques and Tools (Dragon Book)
Modern Compiler Implementation in C or Modern Compiler Implementation in Java