The question is a bit widespread - so I am mostly giving you pointers to think about which can be refined to be more specific.
This essentially applies to design of Application layer protocol (sometimes including session layer). Application layer protocol has following elements
A. definitions of messages:
How many messages do you need? HTTP has GET, PUT, POST as used mostly. What are the essential unique communications client-server needs to do? Which message followed by which other makes sense - and which ones don't?
B. design of structure and encoding of messages:
When sockets receive arbitrary length of bytes - the primary tasks is to find where does the new message start and when it ends? How are bytes represented - ASCII, binary, B64, ASN.1?
C. Is this stateful or stateless?
If there is a state to be maintained by the server during the execution of the message/command or across multiple requests - then that state machine must be consistent and robust; all events orders must be addressable by this state machine to ensure that the protocol doesn't go in the deadlock.
D. Is there a session, authentication or some identifier?
Identification of who's talking to whom, whether one is authorized and or authentication is valid (during session) and maintaining such sessions becomes key to ensure right entities get right information.
Any application layer protocol design really has the questions to be answered. These are things I had learned when I was exploring BEEP. I leave upto you if that is of some good use to you or not. However, but there is a great deal of learning from it; and you are right on the context of it.
1. An Overview of BEEP - The Internet Protocol Journal - Volume 5, Number 2
2. BEEP, understanding why it was created, Marshall T. Rose
3. Building port oriented applications and BEEP