In the earlier processors, not all registers were equal:
- There was not enough space on the chips to have an adder unit for each register.
- With 8 bits, there weren't enough opcodes available for all possible combinations of source and destination.
So assuming that one specific register was always implicated when the adder was involved, made the chip less complex and opcodes shorter.
E.g. the 6510 (used in Commodore 64) could only add using register A, and indexing used either X or Y. There are INC X and INC Y instructions, but no INC A.
As registers had different usages, mnemonics were chosen reflecting their usage. E.g. the A, X, and Y in the 6510 (instead of A, B, and C).
The names in the 8086 are chosen to reflect their usage as well. With 4 general purpose registers, it was logical to name them AX, BX, CX, and DX. Additional indexing registers were called BP and SP (mnemonic: Base Pointer, Stack Pointer).
As many opcodes were extended to 16 bits, there was some space to indicate which one out of four register was used. However, some of the historical reasons still applied, as CX was a bit special: REP and the likes, which are 8 bit opcodes, always use CX as the counter. A simple mnemonic, CX = Counter, helps to remember which one is used.
The opcodes for successors to the 8086 had to be backwards compatible, and are a mess as a result of the variable length opcodes. When 32 bit busses became more common, processors with fixed opcode length were tried. This simplifies the decoding part of the CPU, which freed up space that could be used for e.g. more registers.
Processors that followed this line of thought are called RISC processors (Reduced Instruction Set CPU), to contrast with the CISC (Complex Instruction Set CPU).
More registers results in less spill-over to memory. Basically, registers are the fastest cache available, so increasing the number of registers is a good idea, even nowadays. The lack of specialized instructions was (hopefully more than) compensated by the faster through-put of simple instructions.
Fixed length 32 bit opcodes have enough space to include a source, a second source, an operation, and a destination. SPARC managed to wringle 5 bits for each of the source, second source, and destination, and therefor had 32 registers visible at the same time.
32 registers are too many to use letters, and they were mostly equal anyway, so numbering them was the obvious choice. The 'R' was used to distinguish them from the constants 0..31, and 'R' was an easy mnemonic for Register. Therefore: R0..R31.
Over the years, the Pentium and its successors have maintained backwards compatiblity. However, many of the more succesfull RISC ideas were incorperated as well. Frequently, those new, RISC-like instructions will run faster than the backwards compatible versions.
The number of registers was also increased by Intel, to reduce the number of memory accesses.
And apparently, Intel finally has started to use the R-notation. Backwards compatibility will ensure that AX, BX, ... will stay, but I would bet that AX is just a synonym for e.g. R0.
Disclaimer: The above is my view on history. It will be incomplete as I wasn't around to witness the earlier parts of history first hand. Nevertheless, I hope it is useful to some.