At exactly the time late last year when the Wikileaks saga was occupying seemingly endless media column centimetres, important amendments were implemented to the Commonwealth’s Freedom of Information regime. They flowed from a reform process implemented by Senator John Faulkner. Their commencement attracted almost no attention in the mainstream media. I wrote about the amendments the previous year and expressed the opinion that the reforms they contained were welcome if incomplete.
Specialist FOI blogger Peter Timmins writes about these issues all the time but otherwise FOI doesn’t get all that much attention in the populist media or the blogosphere, although Fairfax has a FOI editor in Matthew Moore and Murdoch has Michael McKinnon in a similar role. Yet effective official FOI mechanisms are much more critical to a functioning liberal democracy than strange individuals like Julian Assange or the random public sector leaks they publish.
The biggest omission from the Faulkner reforms in my opinion was that it failed to address the often very high cost to FOI applicants. The Act allows departments and agencies to charge applicants the cost of retrieving and perusing documents the subject of a FOI request, and that might amount to many thousands of dollars if the request isn’t carefully framed. Moreover, s 24 allows agencies to refuse to comply with a FOI request which involves an “unreasonable diversion” of the agency’s resources.
However, these problems could be relatively easily fixed, as my previous post argued:
Requiring all departments and agencies to invest in and maintain “state-of-the-art” document management systems would greatly reduce the cost and time involved in processing FOI applications, even ones where the application criteria are broad enough to encompass thousands of documents. If all government documents (inwards and outwards) were digitised and indexed by a Google-like searchbot, and also indexed on creation by humans against obvious basic non-disclosure criteria (created for dominant purpose of legal proceedings; Cabinet document; obtained in express situation of confidentiality etc.), then the time for retrieving and assessing even very large FOI applications, and therefore the costs charged to applicants, would be vastly reduced. Moreover, such systems would deliver numerous other efficiencies and cost savings to government entirely unrelated to FOI.
I’m interested in Troppo readers’ thoughts on such a reform. It seems to me that it would in many ways be what Nicholas Gruen refers to as “low-hanging fruit” of policy reform. I implemented an incomplete version of the digital “paperless” office in my own legal practice before I closed it down to enter academia, but I didn’t really get far enough to gain a clear idea of the obstacles and opportunities it might involve for large public sector organisations. No doubt that would involve a range of information security and document integrity issues that are much less problematic for a small private firm.
Ken – Most departments use a parliamentary workflow system (PaWS) for ministerial correspondence.
These systems can store digitised copies of incoming correspondence, track the progress of departmental responses and provide access to outgoing correspondence. They enable authorised users to search and retrieve documents.
An example is ExecCorro. This is the system that AGs use.
Thanks Don. Yes I know many if not most agencies have a document management system. But as I understand it, they aren’t used universally in many departments. I guess that’s inherent in your observation that they’re used for Ministerial correspondence. Moreover the fact that they CAN store digitised copies of inwards correspondence doesn’t mean they have general policies of digitising all inwards correspondence/documents.
And if it isn’t used universally then its value for FOI purposes is greatly reduced because you can’t then retrieve all documents meeting the FOI request by a single automated search. It seems to me that it needs to be a legislatively mandated requirement, perhaps implemented in stages over (say) 3 years.
I agree that it is low hanging fruit that should be implemented, but my experience was that even highly targeted (say, single document) FOI applications would still attract large fee quotes for consideration of the document (particularly if it was a report). Departments that do not want to let go of documents would find ways to get those fees up regardless of the actual costs they are likely to incur.
I think that open government is a great idea. I’m not sure a top down approach requiring certain systems to be implemented is the best way to go about it. While the systems may reduce the marginal cost of FOI requests, the overall costs may be greater than the costs of FOI requests without the system. It’s not just the centralised costs that are an issue; it’s also the overhead costs that get added on to the everyday work.
I’d rather see some standardisation of costs and time limits for FOI requests and leave it to the individual departments to figure out the best way to meet those requirements.
This reminds me why governments are so keen on contracting out.
Contractors can concentrate on delivering services. Public servants often have too much work to do to get around to doing that.
desipis
The big argument against complete standardisation of costs (by which I presume you mean fixed cost charges) is that it could encourage wasteful “fishing expeditions” seeking huge numbers of documents because there’s no disincentive to doing so. I think departmental CEOs exaggerate the extent of that risk but it does exist. I think the amendements tighten to some extent the oversight of the reasonableness of charges but there’s room for more tightening.
Contracting out raises another whole argument about whether contracted out service providers ought still to be subject to FOI and other public accountability measures e.g. it would be interesting what FOI requests might discover about the way SERCO operates our migration detention centres.
Ken – Judging by what’s coming out of estimates, it might be less than you’d think. Here’s Adam Brereton recent piece for New Matilda:
Sorry, yes I agree. You’re talking about contractors like Serco being subject to FOI.
Ken, that’s why I went for “standardised” rather than “fixed”. I’m thinking more of something that’s a function of the type, age and quantity of documents requested. There could also some sort of penalty rate after X number of documents have been requested in a given period.
Ultimately, I think the best model going forward would be a “default-to-publish” model where many things are published online without request. Obviously dealings with individuals would have privacy issues and be excepted, but surely any sort of aggregate statistics or report could be published without much of a problem.
“You’re talking about contractors like Serco being subject to FOI.
Yes exactly. In fact the UK goes even further and regards some completely privatised bodies as public for accountability purposes”(e.g. the Jockey Club or the Stock Exchange). It flows in part from obligations imposed on Britain by the European Convention on Human Rights.
But of course that raises the obvious problem of how you classify what is inherently “public” and “private”. For extreme libertarian conservatives nothing is inherently public, including police forces, prisons etc. Nevertheless, if government is paying private providers to deliver services that were previously provided by the public sector (e.g. SERCO, Job Network providers) then they should be subject to public accountability checks and balances like FOI, Ombudsman, judicial review etc., not just market forces which are largely mythical in such situations.
Sorry Ken,
Having been involved in the implementation of a few of these “state of the art” document management systems (or EDMS) I can say that I’ve never seen one work as advertised in the glossy brochure.
It’s much harder than you think to ensure that “all government documents [are] digitised and indexed … and also indexed on creation by humans against obvious basic non-disclosure criteria”.
Firstly: Start with email. The average government employee probably sends and receives a minimum of 20 emails per day. For even a medium-sized agency (say 500 employees), that’s 2.4 million documents a year, each of which authors would need to catalog, and ensure that at least one person catalogs and indexes incoming emails. And then for emails with attachments, multiply that time by X.
Secondly: The whole concept of a “document” is becoming increasingly slippery. If an agency runs a customer relationship management (CRM) system, they are likely storing tens of thousand of records with hundreds of updates every day, but in a multi-dimensional format that doesn’t neatly correlate to a “document store”. And once you add legacy and ad-hoc systems, the problem becomes much worse.
Thirdly: Mandating use of a particular system is never as easy as it looks. Even something as simple as emailing a document contained in an EDMS or saving a document from an email requires relatively complex import/export process. And if your EDMS gets too much in the way of people doing their work they’ll simply go off system and use thumb drives, GMail, personal laptops, whatever it takes.
The “traditional” recordkeeping model of putting your memos into the out-tray and having them processed and managed by dedicated recordkeeping staff is long gone. But the implicit replacement of having everyone do all their own electronic filing isn’t working either outside of well-defined processes. Busy staff cut corners; that’s the reality. My current project is to get a working implementation of recordkeeping from zero pieces of metadata captured from users for this very reason.
Are these problems insurmountable? Of course not. There are big pieces of technology you can throw at your systems and get them to index everything. But these are really just speeding the first part of a legal “discovery” process — there’s no shortcut to having to manually review the results and cull personal emails, redact non-related data and so on.
I would estimate to do the job properly you would need to mandate it as a core responsibility and factor in an additional 5% or so to all department and agency budgets. But is this level of expenditure seen as a priority in an era of continual efficiency dividends? Nope! And I can’t see that changing in a hurry.
Thanks Stephen. That was exactly the sort of feedback I was seeking. Can you see any workable combination of technology and organisational structure that would increase efficiency/speed of meeting FOI applications?
I certainly take the point that there probably isn’t any way of avoiding having a real human being peruse at least most of the documents that meet the request parameters to assess whether exemption from disclosure should be claimed i.e. you’re right that it’s just speeding up the first stage of a statutorily mandated “discovery” process called FOI. However most of the automatic exemption categories (Cabinet docs etc) are so obvious that it ought not to be difficult or time-consuming to catalogue them as such on initial storage so they don’t need to be looked at again when a FOI request is received.
Finally, while deferring to your “coalface” knowledge, I must say that my small firm experience of an EDMS over a decade ago (i.e. much more primitive than now) was that it was entirely feasible to implement and saved heaps of time in locating and cataloguing documents for a range of purposes. It integrated Outlook email as well as other Office docs fairly seamlessly and easily. It actually reduced staff workload rather than the reverse. However admittedly it was a very small office and we weren’t dealing with a wide range of file formats.
BTW One suggestion that a student came up with the other day was that charges ought to be abolished entirely as the default position, but with the (politically independent) Information Commissioner having power to impose financial penalties for requests assessed by him as excessively wide or unjustified “fishing expeditions”. That would put the general onus back on departments and agencies to find more efficient ways to process requests because they couldn’t be confident of being able to pass the costs on to the citizen applicant. I wonder whether they would then miraculously find an EDMS less problematic and more susceptible to making staff (overworked or otherwise) use it. However it’s the sort of reform that only a new government would ever venture, before they became house-trained by the Sir Humphrey types. It’s a shame Oakeshott or Windsor didn’t make it part of the price of agreeing to support the minority government
The other day, Ken, we spoke what “possible” means to a software engineer. What you’re asking for is definitely possible (and I think desirable, especially with a default-publish policy). But it will take a lot of time and a lot of money to bring the whole of government up to speed.
I would suggest a decade. At least. Because ultimately you want to digitise everything ever recorded by the government and that turns out to be … a lot of stuff.
And if you want it in structured form that can have useful queries run against it, you’re looking at another order of magnitude both in benefits and costs.
Aside from that, the trick is, as Stephen points out, that the system has to work invisibly vis-a-vis adding new documents. If it relies on human diligence it will inevitably fail. This is not a reflection on public servants, it is a reflection on human nature.
Hi Ken,
Notwithstanding that EDMS technology really hasn’t progressed very much in the past decade (it’s just more widespread), it’s not really the technology that is lacking but the personal and organisational issues around its use. EDMS is a perfect fit for a legal office but there are real, obvious, and tangible benefits to keeping everything to do with a particular topic or case in the one logical spot.
Similarly, wherever the concept of “case files” exist in the public service, records management is generally pretty good. The critical lack lies in tracking the conversational aspect of decision-making, which is most commonly done in email now. Maybe 95% of the time this would be of no interest to FoI applications, but it’s knowing how to adequately manage the other 5% which is the difficult bit.
You may recall in the Yes Minister books, the authors use the conceit that Sir Humphrey kept handwritten minutes of all his conversations with the Minister, the Cabinet Secretary and so on. The fictional discipline he applied to this process was sometimes quite incredible!
The solution is both simple and diabolical – we need a an electronic “Sir Humphrey”! If there were email tools could recognise and capture the intricate conversation trails that happen via email for the purposes of recordkeeping we would be a long way along to a solution. Outlook and Exchange (the de facto standard) have almost all the pieces in place but not the final result.
A typical scenario involves 4-5 departmental people and several rounds of emails. What we need is for any one of those people to recognise the importance of that conversation, and to select any email within that thread as needing to be saved in a file. Exchange would then search forward and backward in time for all emails related to that email and automatically create a record which could be vetted and closed after a certain period of inactivity (eg 3 days).
Technologically possible, but even then fraught with privacy/team implications (since the tool would have to capture “sidebar” conversations where some of the participants were excluded for various reasons). We would still see some avoidance but at least where there was a legitimate decision-making process going on (rather than just personal sniping) this would be much more like to be captured. It’s the best I’ve got to offer, anyway!
In my experience, the huge fee they wanted was not for finding documents, but for reading them to black out supposedly commercial in confidence or personal information.
Ken,
A welcome interesting discussion. A couple of points(in no particular order.)
Michael McKinnon left News Ltd in 2007 I think and has been with the Seven Network since. Sean Parnell is FOI Editor at The Australian.There are a few others scattered across the media landscape. The big change in the last 5 years or so is that FOI has been mainstreamed in many media rganisations- a tool to be used by any journalist in the course of work. In some the person with an FOI tag is an internal consultant in many respects.
Serco etc-one of the November 2010 changes is that some documents held by a contractor delivering services to the public on behalf of a government agency are taken to be held by the agency in the event of an FOI application. Contracts since 1 November are required to include a clause to this effect. Lots of definitional issues-and nothing in the law about what happens if the contract doesn’t include a provision or the contractor refuses to provide the documents.
We should be thinking nore broadly about whether FOI for the public sector only continues to make sense in an era where many ” public services” are privately provided, and some private sector bodies are too big to fail, or otherwise have major impacts on society. John Faulkner said in 2009 the Government would refer this issue to the ALRC that year. Nothing has happened since. When I raised this some months ago Minister O’Connor informed me he would give it some consideration.
On fees and charges Tasmania leads- its Right to Information Act retains a small application fee but abolished other charges. There is now no charge for a Commonwealth application not dealt with in time. The OAIC is to report on the fees and charges regime within 12 months of the reforms commencing.
Keep up the good work.
Many departments also use Slipstream for ministerial correspondence. There is no reasonable excuse for procrastination over most FOI requests given all documents are now scanned under the various categories such as VIPs (Minister to Minister, Minister to PM, Heads of State etc), Organisations and General Correspondence. Briefings, PPQs, QONs, Messages are also contained within the database. The only exceptions of course being classifed documents and Cabinet documents which are usually housed in separate and secure databases.
One way to reduce processing time would be to transfer all relevant documents relating to a decision to the departmental website. FOI could be viewed as a natural right to access information without the need to make an application. If all briefings, letters and other relevant documentation were published after a qualifying period similar to the new requirements regarding Disclosure Logs it would certainly reduce pressure on processing. Why wait for an FOI request why not work transparently as much as possible.
One could easily make the case that private contractors providing services on behalf of government are engaged in the business of government and would naturally fall under the umbrella of FOI.
A recent FOI request I submitted for a new FOI based website (under construction) contained documents with blacked out paragraphs deemed exempt under current FOI legislation. While decisions on release rest with departments much of the exemptions around Business Affairs, Personal Information (business related) or Contrary to Public Interest. The balance between public interest and contrary to public interest has in my experience been a fine line. Decisions are generally made on the business interest side rather than public interest often using the argument that the public interest is not served by revealing business affairs or relationships of some organisations with government ie. affecting the free and honest flow of information between government and organisations/lobby groups.
Lots of work to be done yet. Heartening to see more people are keeping FOI in the news.
[…] electronic document management may not be as egregious as it seems. When I wrote about this at Club Troppo last year, federal government IT consultant Stephen Bounds […]
I’ve read about this for a while now, especially after the Wikileaks saga, where information is power, even if this kind of information is misleading. Though I have not experienced of this on a grand scale, but have heard of people who have been given much information and not being to take care of this, regardless of its sensitivity. I was with a few business acquaintances in the self storage business in Sydney, and we were discussing about this when one of us shared what the power of information could affect the business. The web administrator has accidentally printed out incorrect information in the self storage FAQs, and was not taken well by one of the customers. The sales representative was not aware and got into an argument with the customer. It blew out of proportion that it almost cost the employment of the administrator. Gladly, it was nipped in the bud and everything went to normal. Though it is not of grand scale like it is mentioned in the blog, but information, whether it is free or costly, it should be carefully put out there, and not just taken lightly.