What should every programmer know about web development?

Question

What things should a programmer implementing the technical details of a web application consider before making the site public? If Jeff Atwood can forget about HttpOnly cookies, sitemaps, and cross-site request forgeries all in the same site, what important thing could I be forgetting as well?

I'm thinking about this from a web developer's perspective, such that someone else is creating the actual design and content for the site. So while usability and content may be more important than the platform, you the programmer have little say in that. What you do need to worry about is that your implementation of the platform is stable, performs well, is secure, and meets any other business goals (like not cost too much, take too long to build, and rank as well with Google as the content supports).

Think of this from the perspective of a developer who's done some work for intranet-type applications in a fairly trusted environment, and is about to have his first shot and putting out a potentially popular site for the entire big bad world wide web.

Also, I'm looking for something more specific than just a vague "web standards" response. I mean, HTML, JavaScript, and CSS over HTTP are pretty much a given, especially when I've already specified that you're a professional web developer. So going beyond that, Which standards? In what circumstances, and why? Provide a link to the standard's specification.

Answer 1

The idea here is that most of us should already know most of what is on this list. But there just might be one or two items you haven't really looked into before, don't fully understand, or maybe never even heard of.

Interface and User Experience

• Be aware that browsers implement standards inconsistently and make sure your site works reasonably well across all major browsers. At a minimum test against a recent Gecko engine (Firefox), a WebKit engine (Safari and some mobile browsers), Chrome, your supported IE browsers (take advantage of the Application Compatibility VPC Images), and Opera. Also consider how browsers render your site in different operating systems.
• Consider how people might use the site other than from the major browsers: cell phones, screen readers and search engines, for example. — Some accessibility info: WAI and Section508, Mobile development: MobiForge.
• Staging: How to deploy updates without affecting your users. Ed Lucas's answer has some comments on this.
• Don't display unfriendly errors directly to the user.
• Don't put users' email addresses in plain text as they will get spammed to death.
• Add the attribute rel="nofollow" to user-generated links to avoid spam.
• Build well-considered limits into your site - This also belongs under Security.
• Learn how to do progressive enhancement.
• Redirect after a POST if that POST was successful, to prevent a refresh from submitting again.
• Don't forget to take accessibility into account. It's always a good idea and in certain circumstances it's a legal requirement. WAI-ARIA and WCAG 2 are good resources in this area.

Security

• It's a lot to digest but the OWASP development guide covers Web Site security from top to bottom.
• Know about SQL injection and how to prevent it.
• Never trust user input, nor anything else that comes in the request (which includes cookies and hidden form field values!).
• Hash passwords using salt to prevent rainbow attacks. Use a slow hashing algorithm, such as bcrypt (time tested) or scrypt (even stronger, but newer) (1, 2), for storing passwords. (How To Safely Store A Password). The NIST also approves of PBKDF2 to hash passwords", and it's FIPS approved in .NET (more info here). Avoid using MD5 or SHA family directly.
• Don't try to come up with your own fancy authentication system. It's such an easy thing to get wrong in subtle and untestable ways and you wouldn't even know it until after you're hacked.
• Know the rules for processing credit cards. (See this question as well)
• Use SSL/HTTPS for login and any pages where sensitive data is entered (like credit card info).
• Prevent session hijacking.
• Avoid cross site scripting (XSS).
• Avoid cross site request forgeries (XSRF).
• Keep your system(s) up to date with the latest patches.
• Make sure your database connection information is secured.
• Keep yourself informed about the latest attack techniques and vulnerabilities affecting your platform.
• Read The Google Browser Security Handbook.
• Read The Web Application Hacker's Handbook.

Performance

• Implement caching if necessary, understand and use HTTP caching properly as well as HTML5 Manifest.
• Optimize images - don't use a 20 KB image for a repeating background.
• Learn how to gzip/deflate content (deflate is better).
• Combine/concatenate multiple stylesheets or multiple script files to reduce number of browser connections and improve gzip ability to compress duplications between files.
• Take a look at the Yahoo Exceptional Performance site, lots of great guidelines, including improving front-end performance and their YSlow tool (requires Firebug). Also, Google page speed (use with browser extension) is another tool for performance profiling, and it optimizes your images too.
• Use CSS Image Sprites for small related images like toolbars (see the "minimize HTTP requests" point)
• Busy web sites should consider splitting components across domains. Specifically...
• Static content (i.e. images, CSS, JavaScript, and generally content that doesn't need access to cookies) should go in a separate domain that does not use cookies, because all cookies for a domain and its subdomains are sent with every request to the domain and its subdomains. One good option here is to use a Content Delivery Network (CDN).
• Minimize the total number of HTTP requests required for a browser to render the page.
• Utilize Google Closure Compiler for JavaScript and other minification tools.
• Make sure there’s a favicon.ico file in the root of the site, i.e. /favicon.ico. Browsers will automatically request it, even if the icon isn’t mentioned in the HTML at all. If you don’t have a /favicon.ico, this will result in a lot of 404s, draining your server’s bandwidth.

SEO (Search Engine Optimization)

• Use "search engine friendly" URLs, i.e. use example.com/pages/45-article-title instead of example.com/index.php?page=45
• When using # for dynamic content change the # to #! and then on the server $_REQUEST["_escaped_fragment_"] is what googlebot uses instead of #!. In other words, ./#!page=1 becomes ./?_escaped_fragments_=page=1. Also, for users that may be using FF.b4 or Chromium, history.pushState({"foo":"bar"}, "About", "./?page=1"); Is a great command. So even though the address bar has changed the page does not reload. This allows you to use ? instead of #! to keep dynamic content and also tell the server when you email the link that we are after this page, and the AJAX does not need to make another extra request.
• Don't use links that say "click here". You're wasting an SEO opportunity and it makes things harder for people with screen readers.
• Have an XML sitemap, preferably in the default location /sitemap.xml.
• Use <link rel="canonical" ... /> when you have multiple URLs that point to the same content, this issue can also be addressed from Google Webmaster Tools.
• Use Google Webmaster Tools and Bing Webmaster Tools.
• Install Google Analytics right at the start (or an open source analysis tool like Piwik).
• Know how robots.txt and search engine spiders work.
• Redirect requests (using 301 Moved Permanently) asking for www.example.com to example.com (or the other way round) to prevent splitting the google ranking between both sites.
• Know that there can be badly-behaved spiders out there.
• If you have non-text content look into Google's sitemap extensions for video etc. There is some good information about this in Tim Farley's answer.

Technology

• Understand HTTP and things like GET, POST, sessions, cookies, and what it means to be "stateless".
• Write your XHTML/HTML and CSS according to the W3C specifications and make sure they validate. The goal here is to avoid browser quirks modes and as a bonus make it much easier to work with non-standard browsers like screen readers and mobile devices.
• Understand how JavaScript is processed in the browser.
• Understand how JavaScript, style sheets, and other resources used by your page are loaded and consider their impact on perceived performance. It is now widely regarded as appropriate to move scripts to the bottom of your pages with exceptions typically being things like analytics apps or HTML5 shims.
• Understand how the JavaScript sandbox works, especially if you intend to use iframes.
• Be aware that JavaScript can and will be disabled, and that AJAX is therefore an extension, not a baseline. Even if most normal users leave it on now, remember that NoScript is becoming more popular, mobile devices may not work as expected, and Google won't run most of your JavaScript when indexing the site.
• Learn the difference between 301 and 302 redirects (this is also an SEO issue).
• Learn as much as you possibly can about your deployment platform.
• Consider using a Reset Style Sheet or normalize.css.
• Consider JavaScript frameworks (such as jQuery, MooTools, Prototype, Dojo or YUI 3), which will hide a lot of the browser differences when using JavaScript for DOM manipulation.
• Taking perceived performance and JS frameworks together, consider using a service such as the Google Libraries API to load frameworks so that a browser can use a copy of the framework it has already cached rather than downloading a duplicate copy from your site.
• Don't reinvent the wheel. Before doing ANYTHING search for a component or example on how to do it. There is a 99% chance that someone has done it and released an OSS version of the code.
• On the flipside of that, don't start with 20 libraries before you've even decided what your needs are. Particularly on the client-side web where it's almost always ultimately more important to keep things lightweight, fast, and flexible.

Bug fixing

• Understand you'll spend 20% of your time coding and 80% of it maintaining, so code accordingly.
• Set up a good error reporting solution.
• Have a system for people to contact you with suggestions and criticisms.
• Document how the application works for future support staff and people performing maintenance.
• Make frequent backups! (And make sure those backups are functional) Ed Lucas's answer has some advice. Have a restore strategy, not just a backup strategy.
• Use a version control system to store your files, such as Subversion, Mercurial or Git.
• Don't forget to do your Acceptance Testing. Frameworks like Selenium can help.
• Make sure you have sufficient logging in place using frameworks such as log4j, log4net or log4r. If something goes wrong on your live site, you'll need a way of finding out what.
• When logging make sure you're capture both handled exceptions, and unhandled exceptions. Report/analyse the log output, as it'll show you where the key issues are in your site.

Lots of stuff omitted not necessarily because they're not useful answers, but because they're either too detailed, out of scope, or go a bit too far for someone looking to get an overview of the things they should know. If you're one of those people you can read the rest of the answers to get more detailed information about the things mentioned in this list. If I get the time I'll add links to the various answers that contain the things mentioned in this list if the answers go into detail about these things. Please feel free to edit this as well, I probably missed some stuff or made some mistakes.

Answer 2

Rule number one of security:

Never trust user input.

This means:

1. Always encode user input that will be displayed publicly (Prevent XSS)
2. Never present user input directly to a database (Prevent SQL injection)
3. More generally, when you write a web page that accepts any input what-so-ever (forms, query string, AJAX, etc.), ask yourself, "What could I do with this if I were evil?"

Answer 3

• Never put email addresses in plain text because they will get spammed to death.
• Be ultra paranoid about security.
• Get it looking correct in Firefox first, then Internet Explorer.
• Avoid links that say "click here". It ruins a perfectly good SEO opportunity.
• Understand that you'll spend 20% of your time developing and 80% maintaining, so code accordingly.

Answer 4

• Web standards: it's cheaper to aim for standards than testing for every browser available (and in a public website you will see a lot of different browsers/version/OS combinations (30+))
• SEO-friendly URLs: changing URLs later in the game is quite painful for the developers and the site will most probably take a PageRank hit.
• Understand HTTP. If you have only worked with ASP.NET webforms, then you probably don't really understand HTTP. I know people that have worked with webforms for years and they don't know the difference between a GET and a POST, let alone cookies or how session works.
• HTTP Caching: Understand what to cache and what NOT to cache.
• Optimize image weights. It's not cool to have a 20 KB image for a repeating background...
• Read and understand Yahoo's best practices (http://developer.yahoo.com/performance/rules.html). Not every rule applies to every website.
• Use YSlow for guidance, but understand its limitations.
• Understand how JavaScript is processed on the browser. If you put tons of external scripts at the beginning of your page, it's going to take forever to load.
• Consider cell phone usability: some users will access your site using their native cell phone browser (I'm not talking about iPhones or Opera Mini). If your site is pure Ajax, they will probably be out of luck.
• Learn the difference between 301 and 302 redirects: it's not the same for search engines.
• Set up Google Analytics (or any other analytics package) right from the start.

Not specific to public websites, but useful nevertheless:

• Server caching: identify and exploit any caching opportunities, it makes a big performance difference. It's often overlooked on non-public websites.
• Set up a good error reporting solution, with as many details as possible. You will get a lot of errors when you launch, no matter how much you tested, so you better get all the details you can.
• Set up an Operation Database (see for example http://ayende.com/Blog/archive/2008/05/13/DevTeach-Home-Grown-Production-System-Monitoring-and-Reports.aspx) so you can quickly identify bottlenecks.
• Set up a good deployment strategy. You will probably deploy more often than non-public sites (we deploy daily).
• Realize that web applications are inherently concurrent, you will have lots of visitors (typically much more than in non-public websites), and threads are not unlimited.

Answer 5

Security

• Filter and validate incoming user input ('amount' does not need to accept alphabetical characters) and escape outgoing user input (a ' in user input, is NOT the same as an SQL ').
  Never trust any data given by the user.
• And the above will help with protecting against SQL injection.
• Understand SSL
• Keep your systems up to date with the latest patches.
• Protect yourself from cross site scripting
• How to resist session hijacking
• Find out about HTTPOnly cookies
• How to handle authentication/permissions
• Understand PKI (public keys)
• Keep up to date! This is the most important thing, make sure to follow all the latest information about possible security issues and vulnerabilities that affect your platform.

SEO

• Create SEO friendly URLs - example.com/articles/rampaging-bull-tramples-unicorn NOT example.com?article=45
• Use an XML sitemap so that site engines can crawl your site more intelligently
• Set up Google Analytics (or another analytics package) from the start
• Learn the difference between 301 and 302 redirects: it's not the same for search engines.
• Set up a robots.txt file

Performance

• How to cache
• What not to cache
• How to gzip
• Make regular backups. Don't just rely on your hosting provider - have another backup source in case something critical is destroyed (like a database table)
• Read Yahoo's best practices (http://developer.yahoo.com/performance/rules.html) for information on improving performance
• Set up an Operation Database (http://ayende.com/Blog/archive/2008/05/13/DevTeach-Home-Grown-Production-System-Monitoring-and-Reports.aspx) to quickly identify bottlenecks.
• Look into performance monitoring

Productivity

• Documentation!
• Code from the beginning with maintainability in mind
• Have a good deployment strategy - don't save it to the very end to figure this out.
• URLs designed with REST in mind could save you a headache in the future.
• Use patterns like MVC to separate your application flow from your database logic.
• Be aware of the many frameworks out there that will speed up your development
• Use staging and a version control system to deploy updates so that your users won't be affected
• Set up an error logging system. No matter how well coded your website will have errors when it is released. Don't wait for the user to let you know; be proactive in identifying errors and bugs
• Have a bug tracker
• Know your environment. Your OS, language, database. When you need to debug it will be important to understand how these things work at a basic level in the least.

User experience

• Be aware of accessibility. This is a legal requirement for some programmers in some jurisdictions. Even if it's not, you should bear it in mind.
• Never put email addresses in plain text, or they will be spammed to death.
• Have some method for users to submit their comments and suggestions
• Catch errors and don't display them to the user; display something they can understand instead
• Remember that cell phones and other mobile devices with browsers are becoming more common. Sometimes they have very poor JavaScript support. Will your site look okay on one of these?

Core Web technologies

• Understand HTTP, and things like GET, POST, cookies and sessions.
• How to work with absolute and relative paths
• Realize that web applications are inherently multi-threaded, you will have lots of visitors (typically much more than in non-public websites), and threads are not unlimited.

Answer 6

I, personally, avoid using extensions like .php in my URLs. For example:

http://www.example.com/contact

http://www.example.com/contact.php

Not only does the first URL look cleaner, but if I decided to switch languages, it would be less of an issue.

So how does one implement this? Here is the .htaccess code I found works best:

# If requested URL-path plus ".php" exists as a file
RewriteCond %{DOCUMENT_ROOT}/$1.php -f
# Rewrite to append ".php" to extensionless URL-path
RewriteRule ^(([^/]+/)*[^.]+)$ /$1.php [L]

Source: http://www.webmasterworld.com/apache/3609508.htm

Answer 7

I'll add one:

• how to do caching

Answer 8

Here are a couple of thoughts.

• First, staging:

For most simple sites developers overlook the idea of having one or more test or staging environments available to smoothly implement changes to architecture, code or sweeping content. Once the site is live, you must have a way to make changes in a controlled way so the production users aren't negatively affected. This is most effectively implemented in conjunction with the use of a version control system (CVS, Subversion, etc.) and an automated build mechansim (Ant, NAnt, etc.).

• Second, backups:

This is especially relevant if you have a database back-end serving content or transaction information. Never rely on the hosting provider's nightly tape backups to save you from catastrophe. Make triple-sure you have an appropriate backup and restore strategy mapped out just in case a critical production element gets destroyed (database table, configuration file, whatever).

Answer 9

In addition to caching

• know how to set up and gzip content.

Answer 10

1. Web standards
2. Awareness of browsers
3. Awareness of accessibility
4. Awareness of usability

Answer 11

The cruel, hard facts:

Users spend as much time on your website as an interviewer does reading your resume when submitted in a pile of thousands of others

• Users spend very little time on your website: Read, seconds.
• Users are lazy and they would rather be somewhere else
• If the user can't find what they are looking for within seconds, they leave
• If the user cannot identify what the website is all about, they leave
• If the website does not 'just work', they leave
• If the website annoys the user or does not appeal aesthetically to him, they leave

Everything about websites and website design revolves around these facts.

• Clear Navigation
• Conciseness
• Branding strategies
• Colors, schemes, aesthetics, text placement, text formatting
• Helpful, not hindering, Ajax/JavaScript
• Not reinventing the wheel when it comes to website use, navigation, etc.

This is just an outline on why it is so important to adhere to standards and read those website design books.

Answer 12

It might be a bit outside of the scope, but I'd say that knowing how robots.txt and search engine spiders work is a plus.

Answer 13

Aside from basic competence in the base language and the key technologies which might be assumed (although shouldn't be taken for granted):

• Platform - no good attempting to develop an ASP.NET application for a server that doesn't support .NET, no good attempting to provide a SQL Server database to be hosted on a MySQL Server... etc.
• Deadline/Budget - Does it need to be done by next week and therefore potentially has lots of quick hacks and workarounds vs. coding to strict standards and doing everything the right way.
• Content - who is providing it? Has it been vetted for quality and approved for publication? Have all applicable copyrights been checked? Etc.
• Team/Stakeholders - Who needs to be kept in the loop for development, who will the developer be working with, who do they need to keep happy? Etc. Will there be a designer or is the developer the designer too? Don't hire a top notch developer and assume their design skills are all that - most of them are not. I get by and can make something that looks reasonably professional, but I wouldn't consider myself a designer even by a long stretch of the imagination.
• Target Audience - savvy, not-savvy, intranet, Internet. Make no assumptions here, there's a great quote that goes "Programming today is a race between software engineers striving to build bigger and better idiot-proof programs, and the Universe trying to produce bigger and better idiots. So far, the Universe is winning."
• Hardware Base - how much performance has/have the host machine(s) got? Do we have to be concerned about limited memory/diskspace/resources? Obviously if it's only got a small amount of memory, then we need to make sure that minimal memory resources are used in the design of our application. Likewise for diskspace, etc.
• Platform - overall architectural/network topology
• Maintenance - who will be maintaining this product? If the maintenance crew all have a VB background and haven't the first clue about PHP or C#, don't write it in those languages!! If the maintenance crew is you, then code in whatever you're most comfortable in.

This is all before you even get to a web environment really. Once you get into a web environment you would really expect them to understand (in no particular order):

• Stateless interfaces
• Web protocols (HTTP/HTTPS/FTP) etc
• JavaScript and/or other relevant client-side coding techniques
• Various Persistence techniques - Cookies, Sessions, ViewState, ObjectState (and/or any others that relate to the APIs being used)
• At least a basic understanding of HTTP handlers and how they do their job
• Page Lifecycle
• Security in web environments - XSS, SQL Injection, Session hijacking, etc., etc.

After that:

• Competence in the language used to develop the site
• Knowledge of standards and best practices and an ability to apply them effectively
• A good understanding of Cross browser techniques and hacks
• CSS techniques and standards (if the developer is expected to design too)
• Understanding of various browsers and their idiosynchrosies and workarounds

And then - if your site is data driven

• An understanding of the database technologies to be used
• RDBMS design and performance tuning if you're asking them to design the underlying database. If you've got a DBA for that, then this is not such a major concern.

Answer 14

When to say "no" to the designer or client, and how to do so gracefully and diplomatically.

Answer 15

Well, everyone else has already mentioned most things I thought of - but one thing I always forget is a favicon. Sounds stupid, I know, but I think it's one of those little things that helps to emphasise your brand, and I never seem to remember it. Please check Scott Hanselman's post about how to use it carefully.

I agree with some of the rest too - I think it's important to know as much as possible about your chosen language, so that you can code it with best practices and maintainability in mind. I've come across functions and patterns that I wish I'd known about when I did my first few crappy, amateur projects, as it would have saved me writing some retarded WTF-ey workarounds!

Answer 16

If you have any influence on design, please read, "Don't Make Me Think" by Steve Krug. It is an easy read, and will almost certainly make you think...

Answer 17

You also have to:

• Keep your system up to date with the latest patches.
• Keep yourself up to date with knowledge of new vulnerabilities affecting your platform, and attack techniques in general.

I follow several security related blogs and podcasts.

• Bruce Schneier
• Security Now!
• The Register
• Dark Reading

In addition, I get email alerts from SANS https://portal.sans.org/. (you need to register, but it's a great source).

(I'm always interested in learing about other good sources, too).

Answer 18

How to work with absolute and relative paths.

Answer 19

• Valid (X)HTML - with the appropriate tags.
• No broken links (See above about relative links)

Answer 20

Security:

• Consider using an Application Firewall such as UrlScan that works by blocking specific HTTP requests, UrlScan helps prevent potentially harmful requests from being processed by web applications on the server.
• Disable Directory listing.
• Consider using a lower privilege identity.
• Don’t use Blacklists, but use Whitelists instead, teach your application what to accept, not what to avoid.
• If using ASP.NET then encrypt your connection strings using aspnet_regiis. This tool is so easy to use and requires simple steps to both encrypt and decrypt connection’s strings.
• Pages with sensitive data should not be cached: page content is easily accessed using the browser’s history.
• Validate user inputs in the application, promote the use of Regular Expressions (and be assured that they work the way they are meant to be)
• Avoid, at all costs, client side validation (e.g. using Ajax or all JavaScript related validation libraries). JavaScript can and will, be turned off and so your protections).
• Do NOT use GET for anything that changes the server state or contains sensitive information. GET requests are logged in the web server access logs. They are also shown in the browser history.
• DO use POST for every action that changes the server state and reject all non-POST methods. POST prevents unintentional actions, most search engines won’t crawl POST forms and it also helps prevent duplicate submissions.
• If using Cookies, mark them as HTTPOnly using System.Net.Cookie. Set the httpOnlyCookies attribute on the authentication cookie. Internet Explorer Service Pack 1 supports this attribute, which prevents client-side script from accessing the cookie from the document.cookie property.
• Robots.txt files are the first place hackers look at. Use access controls to protect them.
• When constructing SQL queries, use type safe SQL parameters. AKA use stored procedures and stored procedures only. Using stored procedures is always the best approach, from both technical and security point of views.
• If using SQL Server, use a least privileges user. Create a SQL Server login for the account. Map the login to a database user in the required database. Place the database user in a database role. Grant the database role limited permissions to only those stored procedures or table your application really needs. By using a database role, you avoid granting permissions directly to the database user. This isolate you from potential damage to the database.
• Use SSL where possible, this will encrypt and protect your data while on the wire. Using SSL doesn’t necessary means you are secure. It simply means your data is encrypted while on the go. If using SSL, restrict authentication tickets to HTTPS connections only.

Performance:

• Minimize HTTP Requests
• Add an Expires or a Cache-Control Header
• Gzip Components
• Put Stylesheets at the Top
• Put Scripts at the Bottom
• Minify JavaScript and CSS
• Optimize Images

Answer 21

I would think that knowing all you can about your deployment environment would rank up there.

IIS, MSSQL or Apache, MySQL, etc? ASP.NET, PHP, etc.?

Perhaps this is a no-brainer, but surely someone out there has written code that relies on [insert dependency] only to find out their client's server was missing [aforementioned dependency].

Answer 22

How to build a scalable design in the off chance that the site becomes really popular.

Answer 23

Good thread. Here are some areas I think no one's mentioned:

Accessibility (A11y), WAI-ARIA tags and so forth and since it's 2010, why not start adding some HTML5 into the mix also.

checkout Selenium for jUnit-izing client-side testing.

And lastly, Content Distribution Networks, don't host your static files if you can avoid them. e.g. Akamai or Google's instance of JQuery.

Answer 24

• Consider URLs, a URL design with REST in mind could make exposing APIs easier in the future. Definitely much easier to get your URLs right the first time then to change them in the future and deal with the SEO consequences.
• Read Josh Porter's book Designing for the Social Web.
• Have some way to accept criticism and suggestions.
• Know what progressive enhancement an graceful degradation are, JavaScript is NOT a requirement to operate the web and should be treated as such.

Answer 25

Know how to hinder Denial of Service (DoS) attacks on user login forms by keeping track of the number of failed logins over a given period of time. In the event you hit a certain threshold above the running average, increase the duration of all recurring login attempts by a particular amount (say 5 sec.).

Someone feel free to modify for clarity :)

Answer 26

Found a new one today:

• Reset Style sheets

They style sheets you included as a base-line when starting a project to give you more consistent behavior across different browsers. See this question:
http://stackoverflow.com/questions/167531/is-it-ok-to-use-a-css-reset-stylesheet

Answer 27

On a public site, make sure you are using an XML sitemap to help search engine crawlers crawl your content more intelligently.

If you have non-HTML content on your site, you should also look into Google's extensions of the sitemap protocol to make sure you are using whatever is appropriate. They have specific extensions for News, Video, Code, Mobile-specific content and Geospatial content.

One thing I learned that was not obvious in the Google help, is that each of these content-specific sitemaps should be a separate file and joined together at the root with a sitemap index file. For some reason Google doesn't like you to mix content in one sitemap. Also, when you use Google Webmaster tools to tell Google about your sitemaps, tell it about each of the special sitemaps you have separately and use the drop-down to specify the type. You would think the crawler could use the XML to auto-detect this stuff, but apparently not.

Answer 28

I don't have sufficient good karma to edit, my contribution: Looks like everyone here is from the US :)

i18n and l10n

• Use the correct character encoding for your web page (charset encoding)
• Read the Accept-Language to define the page rendering language. I have seen too many web sites that localise depending upon my IP address and ignore "Accept-Language" header information! Painful as I have no idea how to view the site in English anymore.
• Localise for users timezone. It's difficult to get this right, but users will appreciate the fact.
• Format numbers, currency, date, time, address and names as per users region. Default to IP address based localisation if you don't have users region information in profile.

Answer 29

Ensure that whatever framework/server-side scripting/web server/other you're using doesn't expose error messages directly to the user.

Checking that whatever has been put in place to facilitate the above during development is switched off or reversed. Obviously the preference is to have this stuff properly configured in first place - but it will still occur time and time again.

That's mainly written from a security standpoint, but very much related is the usability issue of ensuring that should errors occur, the user get something that makes sense to them and tries as best possible to get them back to what they were doing.

Answer 30

How to avoid Cross site request forgeries (XSRF) (this is not cross site scripting (XSS))

Now I'll probably be modded down for overuse of parentheses.