At my alma mater we distinguished between organisers and coaches for the ACM contests. Organisers could easily be non-programmers and were responsible for communication with ACM /on-site organizers, team registration, travel bookings, etc.
Our coach(es), however, were usually recruited from past team members who were no longer eligible to compete themselves. They had a) strong algorithmic skills and b) experience with the contest minutiae. Their responsibilities included the following:
Organize a learning regimen for newcomers: We competed with multiple teams at the regional, and hence, our university focused on getting fresh blood in. Downside of that was that first year students have next to no idea about the required algorithms. So the coach prepared them with learning material, sample contest questions, etc. in a manner that quickly brought them up to speed on all basic algorithms.
Organize a work-out regimen for the participating teams: The participating teams were constantly practicing. It was the task of the coach to schedule the practice sessions, in particular with regard to various online contests. While it is possible to practice based on old ACM contests, one should also practice the real deal, i.e. against real opponents with a limited time setting and as a team.
Recap of practice sessions: The coach typically monitored the practice sessions (usually keeping himself busy by teaching newcomers as mentioned above, whilst the real teams were left to participate in their training contest on their own). During the practice, however, the coach monitored the team's performance and gave a recap in the end (or on the next day). For newer teams, the recap may involve algorithmic issues (f.ex. bad implementations, wrong algorithm choices, etc.). For technically more experienced teams, coaches usually had little to add in that respect, but instead added value based on their contest experience. They gave tips on how to deal with the team's overall performance (who solves which problems and when, whose turn is it at the computer, how to deal with debugging, etc etc).
Basically, our coach was accompanying newcomers into the world of algorithms, then showing them how a real contest is run, supported them in gaining more experience, and finally helped them form a real team that was able to compete at a high-level (most years we won the regional contest and often had 3 of our teams in the top 10).
That being said, contest rules have changed all the time and maybe there's now something like an on-site coach during regional. In that case, note that the above considers a coach as being located at your university and there was no need for a coach to be present during the regionals (sometimes they came along, sometimes they didn't).