In his new book, The Black Box Society, Frank Pasquale describes the state of "the secret algorithms that control money and information" in as comprehensive terms as the considerable and mostly intact secrecy allows. He draws from the ample supply of anecdotes that have been exposed to liven the description of systems for reputation management (credit scores, and all that's now tied to them), search, and financial manipulation, detailing the risks and the costs to society and the implications for privacy, security and governance. From the penultimate chapter's description of "watching the watchers" and some suggestions for improving oversight, he concludes with his recommendations for moving "toward an intelligible society."
The topic of the book caught my interest without any particular knowledge of the author, but the deep sourcing for his research and reporting, his well better-than-average vocabulary and finally his particular attitude when coming to recommendations for change made me curious to see... that his back cover flap bio blurb is as dense as the book: Professor of Law at the University of Maryland, and Affiliate Fellow at Yale Law School's Information Society Project, and a member of the Council for Big Data, Ethics and Society.
While I enjoyed the use of a new copy borrowed via the Boise Library! system, this is a book worth owning, and the first that strikes me as better owned as an e-book: there are 84 pages of more than 900 notes to the introduction and 5 chapters, replete with hyperlinks that will be handier to click than to retype. Of course, there's plenty to try before you buy (or borrow): the Harvard University Press book page includes a stack of "related links" to American Public Media's Marketplace, the Christian Science Monitor's security and privacy blog, the L.A. and N.Y. Times, CBC Radio, Texas Public Radio, and more.
When Pasquale's book was still "forthcoming" last fall, he wrote an op-ed for the NYT on The Dark Market for Personal Data which is worth looking up for Sam Potts' amusing op-ed graphic, at least. It's a bite-sized introduction to the risks in one of the three dimensions of black boxes he covers in the book, reputation management. Lists of prospects (or "suckers," or whatever) being produced by mining big data with impenetrable "trade secret" algorithms, invisible to the public, and traded by information brokers for "a few cents per name,"
"don’t have to be particularly reliable to attract eager buyers — mostly marketers, but also, increasingly, financial institutions vetting customers to guard against fraud, and employers screening potential hires.
"There are three problems with these lists. First, they are often inaccurate. ...
"Second, even when the information is accurate, many of the lists have no business being in the hands of retailers, bosses or banks. Having a medical condition, or having been a victim of a crime, is simply not relevant to most employment or credit decisions.
"Third, people aren’t told they are on these lists, so they have no opportunity to correct bad information."
Just as the quality of the documentation (or validity) of an individual contract did not matter to bundlers of mortgage-backed securities, the market in this information "offers little incentive for accuracy." All a buyer needs is a certain "hit rate," which can be a tiny fraction of a huge list to provide a financial return. The individuals harmed by derogatory—and often false—reputation data are simply collateral damage.
As our political discussions devolve to simultaneously oversimplified and over-bundled black (or at best gray) boxes of legislation supposed to provide "regulation" or "deregulation" in turns, it's imperative to recognize the need to do more than solve "the problem," and be done with it. The "mutual influence and transformative 'circularity'" of the participants in business, legislatures and the alphabet soup of intelligence collecting agencies is an ever-metamorphosing system, requiring systemic approaches for disclosure and essential control. Threats will proliferate and take on new forms, new targets, new measures and counter-measures, and responses will have to do likewise.
"The stakes are too high for us to ignore this new reality: that politicians and bureaucrats will contravene only so far the interests of a business community they aspire to join or serve. ... [T]he state's immense powers of compulsion and enforcement can now be enlisted in support of the black box technologies of the search, reputation and finance sectors. Pundits overlook real dangers to indulge a peurile fixation on the obsolete polarity between 'state' and 'market' solutions. This is a recipe for paralysis and worse; it is a guaranteee that we will never achieve the societal ideals of security, fairness, and dignity that most of us desire, if not always in identical detail. It is time to take a fresh look at where we want to go from here, and at what gets in our way."
If you want to cut to the chase, go to page 208, following that paragraph, and consider "The Promise of Public Alternatives," and the rest of the concluding 11 pages "Toward an Intelligible Society."
This is a superb piece of difficult work, and highly recommended.
Tom von Alten, February 28, 2015