Testing - Localizations Achilles Heel

Testing is essential to software development. Testing your localized software is essential to the overall localization process.
Getting localization of technology right is important in ensuring that a high quality product can be shipped in a timely fashion at a reasonable cost. In order to strike the right balance, you may choose to outsource part or all of the testing process. Selecting those parts of testing that can be outsourced will depend on a number of factors:

Hardware restrictions. Your application may require hardware prototypes that are in short supply (new printers, laptops, or custom-built hardware). This can be a difficult obstacle to overcome.
Turn-around time. The pressure to release product quickly is relentless. Consolidating localization work can reduce time to market. For example, outsourcing the translation but doing localization testing in-house is likely to take longer than assigning both of these tasks to your localization vendor. The reason for this is that shipping files to and from a vendor adds overhead and delays to the overall process.
Security. Is it possible to extend your internal security processes to an external organization? All products we work on are of a highly sensitive nature, and thus Rubric has developed compatible security practices. Question your vendor before committing to a project with them.

Labour cost. In the last couple of years the cost of testing has dropped sharply. Based on cost arguments alone, it is increasingly difficult to justify keeping localization testing staff in-house. Be wary of the myth that because testing staff "are there anyway" that internal testing "cost nothing".

Time to transfer knowledge. Your application may be complex or highly configurable. The time taken to transfer product knowledge may be out of proportion to the amount of testing to be done. For example, one Rubric customer has a 4-week introductory course on how to use their application; however, the testing time required is a mere week. Training a vendor to test the localization of this product would be cost-prohibitive.

But other factors can affect your choice in this matter. This particular customer is expanding the number of languages in which their product is available, and accelerating the number of releases. Together this means their original decision to keep testing in-house is no longer as cost effective as before.

There is no single answer to the question of how much localization testing to outsource-it is highly dependent on your product and the surrounding factors.

Sadly, the scope of testing and the subsequent results can vary widely when you decide to outsource part of localization testing. Poorly communicated expectations are the typical culprit. Every localization vendor performs different levels of testing by default. Therefore it is crucial that you understand and document what you expect from your localization partner. This avoids unpleasant surprises from vendors who lack the technical capabilities to meet your needs, but who take on your project anyway.

Having decided on outsourcing, a number of factors determine which parts of testing you want to outsource and how extensive this testing should be:

Platform testing. If, for example, your product works on Windows ME, Windows 2000 and Windows XP, do you want to test every language version on every version of Windows, including various service pack levels? Your budget will quickly balloon if you choose to test every permutation.
Integration testing. The product may integrate with SQL server, Oracle and Sybase. Do you test the localized versions against all three? In order to contain costs, consider first determining the most popular databases in each language. Your Japanese customers may have a preference for Oracle, while German customers might predominately use SQL server.

Functional testing. Your source product has been stress-tested for use by 10-5,000 concurrent users. Do you need to load test the language versions too?

Character support and other internationalization issues. Will your application accept all German characters? What constitutes a German character? The EUR (euro) symbol probably needs to be included. Is the EUR (euro) symbol also to be included in the testing of the Japanese or Korean versions? The ° (degree) symbol is supported in the ANSI (Europe) and Shift-JIS (Japan) code pages, but not in the Korean code page.

User Interface testing. This is the type of testing most usually associated with localization. The issues you are hoping to eliminate are those of text clipping, hot-key conflicts, untranslated text, incorrect spelling, and use of gender.

Software localization testing problems can arise when you decide to split the testing work between yourself and a vendor, as each party may have different ideas of what falls into each category of testing.

Let us assume that your product works in conjunction with Microsoft PowerPoint and you decide to do the integration testing yourself. Does this mean the localization vendor can test the localized version of the application against the English version of MS PowerPoint?

Another example might be data sorting and searching functions. Does this fall within the area of UI testing or functional testing? Testing if non-domestic zip codes or telephone numbers work could equally well be classed as UI testing or functional testing.

The point herein is that it probably makes little difference to your vendor how you define the project, but you need to be clear as to what is expected and what is not. Under-defining the scope of their work can lead to either insufficient testing or excessive testing that cripples your budget.