I suggest writing a full test suite in areas where it's sensible and practical to do. In less practical areas, write sanity checks.
In my experience, the overhead of a full set of test cases is certainly worth it in most cases, but realistically code coverage has diminishing returns. At some point, writing more tests just to increase code coverage just doesn't make sense.
For example, depending on your language/technology, testing the UI may not be practical or even feasible. A lot of tests will probably rely on what a user sees and can't be automated. How would you test that a method for generating a captcha produces an image that is readable by a human for example?
If a complete set of tests is going to take you three days to write, the likelihood of a bug being introduced in that component down the track is very low, and the function itself only takes half an hour to write, you should probably think hard about whether that time is worth it. Maybe just writing a basic sanity check for that function would provide value?
My general advice would be that you should test components fully where tests can be written relatively easily. However, if it's an area that's very hard to test, draw a line in the sand and write tests that will test the area at a higher level rather than test it fully.
In the previous captcha example, maybe write tests that check an image of the correct size and format is returned and that no exceptions are thrown. That gives you some level of assurance without going overboard.