First, try to find a format that is close enough to what you are about to build. In general, it is better to use someone's format than to invent your own, even if the format appears to be slightly more complex than what you need1.
If you cannot find a suitable ready-made format, see if you can build your own on top of an existing general-purpose format, such as XML or Binary XML. This should be possible in nearly all cases when you are about to start a new file format. Text-based XML takes more space, but gives humans some measure of readability. However, if you find yourself using Base-64 encoding inside an XML file, that's a clear indication that you should have used a binary encoding instead.
As far as good and bad practices go, make sure that you do not "bake in" the hardware feature of your initial target platform into the design of your file format. Specifically, make sure that your numbers are stored in a format that can be read correctly on platforms with endianness that is different from that of the writer, and that your user-facing strings are stored in UNICODE.
Another good practice is to include a header from which it is possible to determine the type of your file in case its extension is missing or incorrect. It is a good idea to include a version of your file format in the header. This would let you change the format later, and stay backward-compatible.
If possible, do not make your format dependent on the specifics of the default serialization mechanism built into your platform. For example, binary-serialized Java objects do not make a good file format2.
Finally, decide if your files need to be streamable. This introduces additional complexity, because one should be able to interpret individual "frames" of your file in isolation. In cases when you need streamability, however, you should almost always be able to locate a suitable file format that already exists.
On the other hand, you should avoid formats that require extraordinary efforts to support the complexity that your application requires.
2 This does not mean, however, that you should not attempt to custom-integrate reading and writing of your new format with the serialization scheme of your platform, only that you should not rely on the default mechanisms of serialization.