Tag Archives: User testing

Due diligence in public documents – test before publishing

Plain language / Plain English, ,

Due diligence: action that is considered reasonable for people to be expected to take in order to keep themselves or others and their property safe. (Cambridge English Dictionary – my emphasis)

Businesses and government agencies write public documents to inform people about products, services, obligations and opportunities. People may be harmed if they do not fully understand fact sheets, letters, contracts, websites and the like. They may not receive the product or service they expect, they may act in a damaging way, or they miss out on something due to them.

The impact of misunderstanding may be serious or trivial. If the document is about a work process, misunderstanding may result in death or injury. If it is about a financial product, misunderstanding may lead to poverty and disadvantage.

The development process for most public documents is something like

  1. a junior person writes a first draft
  2. a manager reviews and edits the draft
  3. the marketing people provide input
  4. the legal department review, to make sure there is no risk to the organisation
  5. final graphic design and publish.

Organisations may perform these steps with skill and care, but that is not due diligence. It is merely people internal to the organisation talking to each other about the document. They make untested assumptions about how the target audience will understand and react to the document.

Document testing is both reasonable and necessary for public documents

Testing helps keep other people safe. Document testing checks the information can be read, understood and acted on before it is published.

Choosing not to test is reckless.Reckless writing: not caring about readers Simply passing a document around the organisation and throwing it out to the public is not reasonable or responsible. We would never allow a physical product to enter the public arena that way – why do we tolerate it with information products?

Document testing is not difficult, expensive or time consuming. You can find most usability problems by testing with just a handful of potential users.

You’ve written it with care, but will it be read, understood and acted on?

Uncategorized,

User testing is a must to increase the chance that a document is read, understood and acted on. (A document could be a printed process, fact sheet or report, or it could be web text.)

Effective business documents have an impact on the thinking, attitudes or behaviour of readers. If they don’t, the investment in researching, writing and reviewing is all wasted. Writing with care is vital; but care in crafting the document doesn’t guarantee success. Even well organised, plainly written documents can fail to achieve their purpose.

Testing documents with users before they are released is the best way to improve the likelihood of success. And it’s not difficult or expensive.

There are multiple ways to test a document, but one of the simplest is to design a set of questions to be used in a structured interview. Give the test subject the document or website to read. Then, ask them questions exploring what they understand and what they would do next.

You can get helpful insights with just a handful of tests.

Testing with real users, or people who represent real users, is far more powerful than passing the text around the office for your colleagues to review. Only real users have the perspectives and the constraints needed for a valid test.

There is never enough time or money to do it right, but there is always enough time and money to do it again.

Publishing a document without testing – is that always reckless?

Uncategorized,

Reckless writing: Preparing a document without a deliberate and considered concern for readers, or a writer failing to apply their mind to consider how a document will be understood.

Any document published by business or government is a ‘product’. A document is the result of creative effort, designed to meet a particular need.

Nearly all physical products are tested before being released to the market. We couldn’t imagine an untested vehicle or drug being offered for sale – the risk is too great. We would not allow people to risk their lives just because an engineer or scientist says “I’ve worked hard and done my best.”; we insist they test their products in some way.

Yet so many information products, documents, are published without being tested, or released after only a low form of testing. They are published whenever the writer says “that’s good enough”. In many cases this amounts to recklessness – an indifference to whether the information is understood, a carelessness about how information is acted on.

Product testing is related to risk. I’ll only do minimal testing on this blog because the risk, both likelihood and consequence, of you not understanding what I am saying is small. But that’s not true of many other types of documents.

Many internal and external documents should be tested rigorously with end users. To not do so is reckless. For example:

  • Financial documents – loan agreements, insurance documents, financial advice and the like. Not thoroughly understanding these types of documents puts users at significant financial risk.
  • Medical information – misunderstanding this information can lead to poor decision making and lifelong consequences.
  • Any document written by government or a regulator – if a user does not understand these documents there is a risk they may break the law, or not receive things they are entitled to.
  • Legal documents and agreements – people must fully understand what they are agreeing to and what they are compelled to do.
  • Procedures, whether for an internal or external audience. Misunderstanding or ignoring procedures (because they are hard to read) can have significant impact.

Has this insurance document been written recklessly?

Uncategorized,

Reckless writing: Preparing a document without a deliberate and considered concern for readers, or a writer failing to apply their mind to consider how a document will be understood.

A friend read this statement in his Schedule of Insurance:

This Policy also covers your legal liability in respect to loss or damage to third parties’ goods caused by your negligence and whilst such goods are in your physical or legal care, custody or control.

“Sweet”, he thought, “I won’t need that extra insurance on my hire car. My liability policy will cover me if I have an accident.” – a reasonable conclusion, and something a reasonable person would expect to be covered in a business liability policy.

But the insurance company denied the claim after a minor accident in the hire car.

Buried deeply in the policy (page 15 of 22) was an exclusion clause – a ‘gotcha’. The exclusions started on page 10 with

This Policy does not cover any liability;

Five pages later, after wading through a poorly organised and long list of excluded items, processes and events is this:

23. Vehicles
for Personal Injury or Property Damage arising out of the ownership, possession or use by the Insured of any Vehicle:
(i) which is registered or which is required under any legislation to be registered,

There it is in black and white – a registered vehicle, in this case a hire car, is not covered under the policy. The insurance company blamed the policy holder, the reader, for this misunderstanding. The insurance company would likely claim this was a case of reckless reading, but I reckon it’s more likely a case of reckless writing.

A foundational principle of plain language is that the writer, not the reader, takes prime responsibility for effective communication. So, it is never OK to blame the reader.

Why I think this document may have been written recklessly:

  • I doubt the document was tested in any way with potential readers. If you don’t test a product (in this case, a document) how can you have any idea how it will perform?
  • It is unlikely the writer applied their mind to how the document would be understood – the writer likely did not consider what the reader expected the policy to cover.
  • There is no evidence the writer has a deliberate and considered concern for the reader – the policy is primarily designed to limit the liability of the insurance company.
  • The clause in the shorter document (the schedule), the document more likely to be read, does not mention any exclusions.
  • The content of the policy, especially the exclusions section, does not seem to have a logical, reader-based structure.
  • Basic Plain English techniques (familiar words, active voice, verbs, short sentences, conversational style) have not been used consistently.
  • Readability statistics on the policy are poor: Flesch reading ease: 23.0, Flesch-Kincaid Grade Level: 17.9, Passive sentences: 41%. (Readability statistics do not give a definitive measure of how well a document serves readers’ needs, but are a helpful indicator)

Audit vs evaluation – the importance of asking the right questions

Uncategorised, ,

audit vs evaluation‘Audit’ answers ‘Did we do what we said we would do?’ For example, did we publish the document?, did it go out on time?, did it reach all our clients?

‘Evaluation’ answers ‘Did we achieve the outcome?’ For example, do our clients now understand our services?, are our procedures efficient?, how well did we predict that outcome?

Both types of questions are important and have a role in managing an organisation well. But often matters that should be evaluated are audited instead.

For example, suppose you recognise a communication need – you want your clients to better understand what you can do for them, or you want the public to take advantage of a new policy or initiative. The usual course of events goes something like this:

  1. Decide you need an information product (people usually think a document or website is the best way to deliver information)
  2. Get a project team together
  3. Develop a project timeline and determine a budget.
  4. Set tasks and assign responsibility
  5. Get the resources you think you will need
  6. Do the work – develop the product
  7. Deliver

How do we usually measure a project? Usually against the project plan. If the product is on time and on budget it is usually considered a job well done. We assume that if the document has been written and published then the job has been done.

But that’s audit, not evaluation. Only rarely are projects measured against purpose.

Evaluation questions are like: Did people see and read the document? Did they understand it, and understand the importance of the information? Have they changed their thinking or behaviour as a result of reading?

Financial disclosure – being clear, concise and effective

Uncategorized, ,

financial disclosureThe law says that financial disclosure documents must be worded and presented in a clear, concise and effective manner. This is a requirement of the Corporations Act 2001:

  • Financial Services Guides – Sect 942A (6a)
  • Product Disclosure Statements – Sect 1013C (3)
  • Statements of Advice – Sect 947C (6)

My observation is that many disclosure documents do not meet the standard of ‘clear, concise and effective’. Writers seem to focus on complying with other aspects of the law – having  the correct content. They assume that being complete and accurate is enough. That’s a very risky position to take, and it may break the law.

These standards of ‘clear, concise and effective’ are tough to meet because they can only be judged with reference to the reader, or user, of the document.

An effective document is one that achieves purpose. The general purpose of disclosure documents, as described in the law, is to help people make a decision about whether to buy financial services or products, or follow advice. To be effective, disclosure documents must work to achieve that purpose. That means you must know something about your readers and their decision making process, and use that knowledge to inform your writing. It may mean including additional content not required by the law, but that is helpful for your readers.

Being concise means not including unnecessary material, not saying the same thing multiple times (removing redundancy) and not using more words than is needed to express an idea. It means you should not blindly use a boiler-plate template that includes material that is irrelevant to some readers.

Being clear is a matter of

  1. organising content in a way that makes sense to your readers
  2. using words, sentence structure and a style that can be understood easily.

In short, all financial disclosure documents must be written using the principles of plain language, with a deliberate focus on the needs of users.

Reckless writing

Uncategorized, ,

reckless

Reckless writing is when writers just float a document ‘out there’ without any serious thought about their readers. Like reckless driving, it’s just doing what you think is OK without properly considering the consequences.

Even if your document is complete and accurate, not considering your readers is reckless. Not considering how they will understand and what they will do as a result of reading is both arrogant and careless. Not taking the time to structure your document in a way that makes sense, and not writing plainly in language readers can understand, is irresponsible.

It is never enough just to write what you think is good enough – the reader is the sole judge of effective communication.

Unfortunately, reckless writing is common. It’s unusual for authors to carefully articulate the purpose of the document and the needs of their users before they start writing. Proper user (reader) testing is rare. And document reviews don’t always fix the problem – they can end up being group recklessness.

But reckless writing is very risky behaviour for government agencies and businesses. It’s especially risky if you write documents that people use when making decisions.

Sometimes the risk is that the document is ignored. In that case, all that has been lost is

  • research effort in defining the content
  • writing effort, as authors struggle to find words to convey ideas
  • design effort when graphic elements are sourced to supplement the words
  • review effort as the document passes through various levels of approval
  • publishing cost, perhaps printing or placing it online
  • archival and governance effort to keep proper records of the document
  • the opportunity – the document was intended to achieve some outcome but didn’t.

Other times the risk is that users (readers) misunderstand the document and take an action that they shouldn’t, or that they would not have if they had understood it properly. When that happens, the cost to you and to them can be huge and may potentially involve legal action.

Consider your users when you write and what they need to do with the information. Test to make sure you are communicating effectively.

The curse of knowledge

Uncategorized,

curse of knowledgeSometimes poor writing is not the result of laziness or a desire to be obscure, it’s just because we know the content too well.

When you know something really well it is extremely difficult to imagine not knowing it. That makes it hard to share knowledge with others because you can’t easily put yourself in your readers’ shoes. It’s like trying to un-see or un-hear something.

The first principle of writing clearly is to understand your users or readers; to understand their needs, wants, interests and mind set. But that is really hard to do if you are an expert trying to communicate with non-experts.

Experts immerse themselves in the subject for years. They end up speaking and writing using abstract terms to summarise all the concrete data in their heads. They describe ‘replacing the bolt’ as ‘taking corrective measures’, or ‘wind and rain’ as ‘unfavourable weather’, or ‘stopping pollution’ as ’emissions reduction’. But novices don’t get it. They only hear vague phrases.

Experts forget how difficult it has been to acquire knowledge. They now see much of their knowledge as ‘obvious’ or ‘common sense’.

Elizabeth Newton illustrated the curse of knowledge in a simple game. A “tapper” was asked to tap out the rhythm of a well-known song, such as “Happy Birthday”. The listener’s job was to guess the song. Newton asked the tappers to predict the probability that listeners would guess correctly. They predicted 50%, but listeners actual success rate was 2.5%. The reason for such a difference? The tapper can’t avoid hearing the tune in their head playing with the taps, but all the listener can hear is a kind of Morse code.

Deliberately working to put yourself in the users’ shoes is a start, but doesn’t really solve the problem. The only solution is to test what you have written. Ask your user, or a surrogate user, what they understand from your writing.

(inspired by reading Steven Pinker’s book, The Sense of Style)