Written Testimony of Justin Gus Hurwitz Associate Professor of Law Director of the NU Governance and Technology Center Co-director of the Space Cyber Telecom Law Program University of Nebraska College of Law Director of Law Economics Programs International Center for Law Economics Before the United States House of Representatives Committee on Energy and Commerce Subcommittee on Consumer Protection HEARING ON AMERICANS AT RISK MANIPULATION AND DECEPTION IN THE DIGITAL AGE January 8 2020 Page 1 of 21 Written Testimony of Justin Gus Hurwitz1 January 8 2020 Introduction “Dark pattern” is a new term for an old practice using design to prompt desired if not necessarily desirable behavior 2 For instance a website may present terms of service or an upgrade offer in a window that is more difficult to cancel than it is to accept A website might possibly falsely report to a user that many other users have made a similar purchase recently or that only a limited number of units of a product remaining 3 A car salesperson may present addons or upgrades at the end of a high-pressure negotiation or a supermarket may stock a checkout aisle with high margin “impulse purchase” items 4 An employer might offer on-site amenities and perks that make employees happier but that also result in them spending more time on the job Subscription services – online and offline – may run customers through a “maze” of customer service representatives to cancel service A social-media platform may make it easy and rewarding to uncritically “share” posts facilitating the widespread dissemination of false information 5 The basic idea of dark patterns is straightforward humans are not perfectly rational decision-makers Rather we constantly use various heuristics to efficiently make decisions subject to imperfect information These heuristics can be turned against us however and used to some extent to “program” us for specific behavior 6 There are myriad common examples of these cognitive biases But this is a case where it may be easier to show than to tell the images at the top of the next page demonstrate simple “dark patterns” at work As these images demonstrate there are patterns in how we interact with information Designers study these patterns and can use them to present information in ways that influence 1 2 3 4 5 6 The author thanks the Committee for the opportunity to present this material In addition he thanks Kristian Stout and Justin McCully for extraordinary assistance in completing this testimony in a short period of time All views expressed are those of the author and do not necessarily represent any other individual or organization Any errors or omissions are the authors alone Given the short timeframe on which this material has been prepared errors are unfortunately likely – best efforts have been made to ensure the accuracy of this material DARK PATTERNS https www darkpatterns org last visited Jan 2 2020 Practices such as these are discussed in Arunesh Mathur et al Dark Patterns at Scale Findings from a Crawl of 11k Shopping Websites discussed infra But see Mario Miranda Determinants of Shoppers’ Checkout Behaviour at Supermarkst 16 J TARGETING MEASUREMENT ANALYSIS FOR MARKETING 312 2008 finding that “shoppers' purchases at grocery checkouts may not be spontaneous and unreflective … but demonstrative of conscious concern with making efficient use of their shopping time Not all purchases at checkouts can therefore be casually referred to as impulse purchase ” See Alex Kantrowitz The Man Who Built the Retweet “We handed a Loaded Weapon to 4-Year-Olds” BUZZFEED JULY 23 2019 https www buzzfeednews com article alexkantrowitz how-the-retweet-ruined-the-internet See also Soroush Vosoughi et al The spread of true and false news online 359 6380 SCIENCE 1146 2018 See Brett Frischmann and Evan Selinger RE-ENGINEERING HUMANITY Cambridge 2018 Page 2 of 21 how we respond to that information Designers may present information in a manner that follows the flow of how readers or users are likely to naturally process it or in a way that highlights details that may be easily missed or by “hiding” information despite it being plainly disclosed The first image7 takes advantage of how humans scan information in an image or on a page In this case design is being used to make the reader feel like they are being controlled by the image While this is presented in a somewhat jocular or didactic manner it may nonetheless leave some readers perplexed or even feeling manipulated The second image is somewhat more nefarious 8 even if innocuously so it contains errors that most readers’ brains will automatically correct and skip over as they are read and plays with the reader by calling attention to these overlooked errors Imagine if instead of minor typos or grammar errors this image had “tricked” the reader into accepting substantive errors such as the inclusion or omission of the work “not ” or an extra digit in the price of a product Patterns like these could be used to “trick” users into accepting terms or disclosing information ostensibly knowingly While there is nothing terribly new about merchants shaping the customer experience to their own advantage new attention has been paid in recent years to practices like these when used in the online environment First given the name “dark patterns” at the beginning of last decade concern about these practices has grown in the academic literature and popular press in 7 8 Zer0Effect AND YOU WILL READ THIS AT THE END 2019 https www reddit com r dankmemes comments apcf4f and_you_will_read_this_at_the_end MEMEPRO1 IF YOU DID IT GREAT 2018 https imgflip com i 225k37 Page 3 of 21 recent years 9 The phenomenon has also increasingly gained legislative attention 10 Recently attention has been driven in particular by concerns in the privacy community about the effectiveness of privacy disclosures and notice-and-consent requirements and concerns about mis- and dis-information 11 This testimony addresses dark patterns – what they are and the extent to which we should be concerned about them The first part contains a background discussion of the characteristics of dark patterns paying particular attention to how the concept may differ in the online context as compared to the offline context The second part then discusses the difficulties of design especially of software interfaces and argues that “patterns ” dark or otherwise are both inevitable and difficult to understand This discussion foreshadows part three which addresses the extent to which we should be worried about dark patterns and what if anything we should do to address these concerns I thank the Committee for the opportunity to share these thoughts This is an important topic at the forefront of a complex and dynamic area – it is important that the Committee be considering these issues In line with the complex and dynamic nature of this area I submit this material with the important proviso that any one perspective set of examples or expression of concerns or assurances can at most shine a small light on a large issue Dark Patterns What they are First coined in 2010 12 the term “dark patterns” was created to describe user interface design patterns that are “crafted with great attention to detail and a solid understanding of human psychology to trick users into do things they wouldn’t otherwise have done ” 13 The term is used primarily to describe user interface design choices intended to invoke particular behavior usually to the benefit of the designer and or the designer’s employer Many if not most examples have offline analogs But the arguably unique thing about dark patterns is 9 10 11 12 13 See e g Ryan Calo Digital Market Manipulation 82 GEO WASH L REV 995 2014 Paul Ohm Forthright Code 56 2 HOUS L REV 471 2018 Ari Ezra Waldman Power Process and Automated Decision-Making 88 2 Fordham L Rev 613 2019 Frischmann Selinger supra note 6 A WestLaw search of news sources for the term “dark patterns” yields 198 results from 2019 114 results from 2018 and an average of about 45 results per year for from 2013-2017 See e g Deceptive Experiences to Online Users Reduction Act DETOUR Act S 1084 116th Cong 2019 available at https www congress gov bill 116th-congress senate- bill 1084 text See Ari Ezra Waldman Power Process and Automated Decision-making 88 FORDHAM L REV 1 2019 Ari Ezra Waldman Privacy’s Law of Design 9 U C IRVINE L REV 1239 2019 Lindsey Barrett Confiding in Con Men U S Privacy Law the GDPR and Information Fiduciaries 42 SEATTLE U L REV 1057 2019 Neil M Richards Woodrow Hartzog The Pathologies of Digital Consent 96 WASH U L REV 1461 2019 Lior Jacob Strahilevitz Jamie Luguri Consumertarian Default Rules 82 L CONTEMP PROBS 139 2019 Ohm supra note 9 Lauren E Willis Why not Privacy by Default 29 BERKELEY TECH L J 61 2014 Frischmann Selinger supra note 6 Harry Brignull Dark Patters dirty tricks designers use to make people do stuff 90 PERCENT OF EVERYTHING Jul 8 2010 https www 90percentofeverything com 2010 07 08 dark-patterns-dirty-tricks-designers-use-to-make-peopledo-stuff Id Page 4 of 21 that software interfaces to online platforms are infinitely and instantly malleable there is practically no limit to design choices and those design choices can be changed tweaked updated and targeted with ease – including in real-time and in response to specific users or user actions This is different from more traditional sales channels For instance a supermarket checkout aisle needs to be roughly a constant size needs to target the average customer insofar as its impracticable to send customers to different aisles based on e g their buying history can only fit so many products on the shelves and can’t be easily changed outside of a set schedule Another unique aspect of dark patterns is that sometimes the underlying code is available So for instance if a webpage is targeting different interfaces to different users using browser-side techniques the underlying code can be inspected Similarly online interfaces are typically used from the relative comfort of one’s home or office or while out and about on one’s mobile device Both of these factors give users greater control over how they choose to interact with an interface than is possible in many offline settings Dark patterns take advantage of a few key behaviors of imperfectly rational humans First people are unwilling to devote a large amount of cognitive resources to relatively low value activities As such people skim when they read often missing some details – particularly those that may be designed in a way that makes them relatively easier to miss Our eyes follow common patters when reading text on a screen or page based upon how we have learned salient information is likely to be presented 14 Second if there is a cost to correct a mistake people may just accept the mistake if the cost in time or effort exceeds the cost of continuing on their present course Few people will take the time to return a product for a $2 00 refund even if that product was shipped to them and they were charged for it in error or fraud Third people are social creatures and we frequently rely on the behavior of others to guide out their own conduct Thus when presented with information such as “Bonnie in New Jersey recently purchased item X” or “12 other people are looking at this deal right now ” consumers will potentially feel an elevated sense of pressure to commit to a purchase This heuristic sometimes referred to as “social proof ” can be understood as entirely rational reflecting the wisdom of the crowd but it can also be taken advantage of to make a decision seem more desirable than it really is 15 There is no doubt that firms use dark patterns or that they may be effective One recent study analyzed 53 000 different product pages across 11 000 different online shopping sites and found 1 818 instances of dark pattern usage 16 In another study respondents presented with either a “moderate” or “aggressive” dark pattern designed to push them into purchasing credit monitoring services were 228%-371% more likely to purchase the offered services 17 14 15 16 17 A search on Amazon com for books on “eye tracking ” for instance yields dozens of results Indeed the term “social proof ” is generally traced to Robert Cialdini’s 1984 book INFLUENCE THE PSYCHOLOGY OF PERSUASION one of the seminal books on the psychology of persuasion and marketing Arunesh Mathur et al Dark Patterns at Scale Findings from a Crawl of 11k Shopping Websites Proceedings of the ACM on Human-Computer Interaction Nov 2019 https arxiv org pdf 1907 07032 pdf Jamie Luguri Lior Strahievitz Shining a Light on Dark Patterns 21-22 U of Chicago Pub Law Working Paper No 719 2019 available at https papers ssrn com sol3 papers cfm abstract_id 3431205 Page 5 of 21 At the same time and as discussed below design is simply put hard and not all “dark” patterns are intentional or malicious Some are benign or even beneficial Design decisions are necessary to any interface and negative effects may be inadvertent or practicably unavoidable For example one of the studies above used screen shots from the PlayStation live service and its promotion of a 12-month subscription over the 1-month option by using larger text for the former to demonstrate a deceptive dark pattern 18 But considering the large volume of gamers that use that service it may simply be the case that the annual savings and convenience of not having to subscribe month-to-month benefits one group of users even though it may be annoying or undesirable to a second set of users In other words using larger text sizes to make the option most desired by most users easier to find while leaving the alternate option available on the same page for users who prefer it may be the preferred design for most users Further designs intended to bring about certain effects may be ineffective and intended effects may be beneficial – for example reminding users of abandoned shopping carts and reminding users of necessary complementary products may confer a benefit on both the seller more sales and the buyer purchasing desired products It may be the case that the annoyance of being “pushed” to purchase items in a cart or to buy items related to those in a cart is relatively minor even spread across thousands of users to avoid a greater inconvenience for users who fail to click the final button to complete a purchase or who are about to purchase a product only to later discover that they needed to have purchased complementary goods to use it Dark patterns also are nothing new Indeed most have existed in one form or another in the offline world for a long time Stores keep candy near registers because it’s easier for parents to simply placate a whining child than to discipline them in a checkout aisle and tabloids to entertain customers and distract them from the feeling of impatience while waiting to pay When purchasing a car at a dealership the salesperson may “consult” with a hidden “manager” to make a customer feel he is getting a good deal and the customer then frequently needs to go through two or three layers of personnel for finalizing the deal each time being offered various “upgrades” to the vehicle being purchased Homeowners needing contractors for home remodeling fence installation or a major repair will frequently not be able to receive a price over the phone – even if pricing is relatively standard – because such companies prefer to send a salesperson to the premises who can talk the potential customer through objections These are all common “tricks” of the sales trade – they are patterns of doing business that allow firms to generate more revenue from customers In some cases these may be deceptive or harmful or at least have no positive social value as opposed to merely transferring wealth from customer to firm In other cases there may be real value to them A company may prefer to send contractors to visit customers’ homes because experience shows customers often don’t understand which product best suites their needs or have the wrong work done on their house to solve a given problem Sending the contractor to inspect the job site before giving a quote may allow for better quotes and performance and even more important avoid creating unhappy and complaining customers And in other cases these “tricks” may be a mechanism for price discrimination – sorting customers by their willingness to pay for a given product While controversial the economics of price discrimination are widely understood and it is generally 18 Id at 13-17 Page 6 of 21 legal The net effect of price discrimination in competitive markets generally doesn’t increase firm revenues significantly Rather by charging some customers more and keeping the average price the same firms are able to offer other customers lower prices which can allow them to offer their goods or services to consumers who may otherwise be priced out of the market Dark Patterns The Good The Bad and the Ambiguous 19 Although the literature on dark patterns is relatively new there are some readily identifiable patterns which deserve discussion What follows is a discussion of some of these common patterns and an attempt to differentiate them along other examples in terms of “good ” “bad” and “ambiguous” effects Bad-effect Design Websites may use design to trick consumers into undesired action This includes for instance employ things like “countdown timers” 20 indicating that a customer only has “X” amount of time remaining to complete a purchase If fraudulent information this may create a needless sense of urgency that compels a customer to make a purchase that they would not upon less pressured reflection Sites also employ a “limited-time message” “scarcity message” 21 indicating a particular deal will only exist for a short period of time or that the item is on the verge of selling out When fraudulent this is used to motivate a buyer without need Upsells are also common a design that introduces steps meant to encourage users to purchase an additional good or service e g insurance for a travel ticket When a design “confirmshames” 22 users it employs a choice interface “yes” or “no” in a way that manipulates a consumer’s emotions Thus instead of just allowing a “no” choice to decline optional insurance for a vacation the offered choice may be “No I don’t want to protect my valuables and loved ones during my trip ” Visual interference23 is used to put important text in obscured or otherwise difficult to see color and layout schemed 24 One way this manifests online is to offer users upgrade options in a window that offers them an obvious way to accept but obscures how to decline the offer The cognitive effect of this design is that it gives users inclined to decline the offer a few additional seconds to change their minds and because we have a natural predisposition to ideas that we have encountered recently may in fact make them marginally more likely to do so Even if the 19 20 21 22 23 24 Note these “bad ambiguous good” behavior headings are approximate meant to offer intuitive examples to demonstrate that design can be good or bad Mathuer et al supra note 16 at 12 Id at 16-17 Id at 17 Id At times this pattern goes beyond simple design choices in terms of font and color and moves into introducing wholly out of place elements clearly meant to confuse a user For example one shoe retailer placed a picture of a hair over top of their otherwise legitimate ad in an effort to trick users into swiping up Some users thinking they were ridding their screen of a hair actually ended up on the retailer’s web site See Katitlyn Tiffany This Instagram Story ad with a fake hair in it is sort of disturbing VERGE Dec 11 2017 available at https www theverge com tldr 2017 12 11 16763664 sneaker-ad-instagram-stories-swipe-up-trick note however that deceptively obtained consent is ineffective Page 7 of 21 conversion rate is small if offered immediately after a sale this mechanism only has upside revenue potential More traditionally supermarkets manage the length of lines to generate a sunk-cost bias Also as noted above impulse buy items are offered alongside the waiting shoppers to encourage them to add more to their order Sites sometimes use sneaking 25 cramming which automatically adds items to a shopper’s cart One of the most classic examples of off-line manipulative behavior is creating roadblocks or users to shape their behavior Typically this takes the form of making it difficult to cancel a service or return a product For example canceling cell phone service frequently requires transfers between multiple sales representatives and navigation of complex phone menus Canceling cable or Internet services often requires consumers to go through a similarly circuitous experience Ambiguous-effect Design There are a host of behaviors that arguably straddle the line between benign and unsavory Websites frequently employ automated messaging systems to periodically remind browsing customers of items they left in their carts Technically unsolicited messages such as these may be an annoyance but may also serve to remind users of purchases that they want to return to or even thought that they had completed Complementary product notices are similar To some users being offered complementary products may be an annoyance or even induce undesired purchases but for others they can provide important information and avoid substantial future costs For instance a site may suggest a customer who is buying a plumbing fixture also buy Teflon plumbing tape If the customer is unaware that Teflon plumbing tape is needed to properly install most fixtures this is valuable information that may save the consumer from having to make a subsequent purchase or even from improperly installing the fixture On the other hand if the customer already has such tape this may be a minor annoyance And if the suggested product is not actually needed this suggestion may be harmful to the customer Grocery stores use inconsistent labeling on the price stickers placed on goods – similar items may have their unit prices calculated using different units 26 This can be misleading making more expensive products appear less expensive or just irritating as it forces consumers to do their own comparisons and makes pure price competition among venders more difficult Some argue that this is a devious mechanism forcing consumers into buying more expensive products by making it harder for customers to identify which products have the best prices But it can also be way of promoting non-price competition where consumers are unlikely to compare the quality of products if their sole focus is price Indeed research suggests that consumers may over-rely on price comparisons as strong indictors of quality 27 25 26 27 Id Melanie Pinola How the Unit Pricing Labels in Stores Can Trick You into Spending More LIFEHACKER Oct 03 14 https lifehacker com how-the-unit-pricing-labels-in-stores-can-trick-you-int-1641793755 Dengfeng Yan Jaideep Sengupta Robert S Wyer Jr Package size and perceived quality The intervening role of unit price perception 24 1 J OF CONSUMER PSYCHOL 3 14 2014 finding that consumers use unit price as a proxy to determine quality when comparing similarly sized and different sized goods Page 8 of 21 Arguably even familiar and widely used user interface elements such as a “like” button or a “retweet” button represent a degree of user manipulation albeit with ambiguous effects Social networks are today defined to some extent on the degree of reach that individual users can affect Much of this reach is measured by user engagement which is in turn driven by activities such as liking and retweeting 28 These design features were explicit choices meant to encourage user interaction on the social networks and thus represents user manipulation to a degree The social value of these platforms is subject to important debate and scrutiny from their ability to serve as vectors for and amplifiers of mis- and dis-information and concerns about potentially addictive behavior patterns 29 Nonetheless social media has unquestionably been beneficial to many in society – often most to minority and other disadvantaged voices that have historically not had access to high-profile platforms and for which social media has served as a significant amplifier of their messages concerns and ideas – and the design elements that have allowed these platforms to succeed have allowed these user groups to benefit from them Or to return to an echo of the PlayStation example used above during its regular membership drive NPR strongly encourages listeners to become “sustaining members ” That is They want listeners to agree to small automatic monthly donations instead of larger one-time donations But why should NPR care if a listener gives $120 once in January or $10 month over a period of 12 months The answer is that this is a dark pattern 30 Getting listeners to sign up for the monthly subscription makes is more likely that they will continue paying long into the future – rather than hoping that each year they affirmatively choose to make a single large donation the psychological burden is shifted to the listener to discontinue making small regular donations which many are unlikely to do NPR of course is a good honest hardworking news organization with pure motives so would never be criticized for taking advantage of its listeners by tricking them into emptying their pocketbooks into public broadcasting’s coffers But when companies like Microsoft and Adobe use this same practice it is clearly deceptive 31 Good-effect Design Design choices can also be obviously aimed at good ends Apple and Amazon are two of the best examples of carefully considered design meant to drive positive user experiences The so-called “Apple tax ” the price premium that Apple is able to charge for its products compared to similar-quality products from other companies is a reflection of Apple’s reputation for producing well-designed products 32 Amazon likewise to an important degree made e28 29 30 31 32 Jeffrey Krans 7 Social Media Engagement Metrics for Tracking Followers and Growing Community B UFFER Sept 21 2015 https buffer com resources measure-social-media-engagement See Christian Montag et al Addictive Features of Social Media Messenger Platforms and Freemium Games against the Background of Psychological and Economic Theories 16 14 INT’L J ENVTL RES PUB HEALTH 2612 2623 2019 Hilary Anderson Social media apps are ‘deliberately addictive to users BBC NEWS Jul 8 2018 https www bbc com news technology-44640959 See Priscella Esser Getting Users’ Long-Term Commitment with a Monthly Charge INTERACTION DESIGN FOUNDATION 2018 https www interaction-design org literature article getting-users-long-term-commitment-with-amonthly-charge Id Lest the dripping irony be lost the effects of these practices in the cases of both NPR and commercial entities like Microsoft and Adobe are ambiguous with both positive and negative effects for different groups of users Kevin Downey Why are Apple products so friggin’ expensive KIMKOMANDO Mar 9 2019 https www komando com money-tips why-are-apple-products-so-friggin-expensive 549472 Page 9 of 21 commerce accepted and trusted through the great strides it made in both creating secure environments that customers could trust and removing as much of the friction in the shopping experience as possible Its famous “1-click” patent and the associated ease with which it designed its checkout experience was an important part of that innovation 33 Individual apps that cater to differ user lifestyles also introduce design choices – often using the same techniques derided as manipulative in the social media context – to encourage for example healthier lifestyles Apple’s watch has a built-in app that reminds users to breathe deeply periodically34 and an app that reminds users to stand up and walk around once an hour to combat the problems associated with modern work habits 35 Other apps help dieters remember when they are allowed to eat encourage them to make healthier choices and to drink enough water Design Patterns Design is difficult It is also necessary A car must have a mechanism for steering which must be located somewhere and be articulated in a certain manner Design choices will affect how easy it is to operate the car how responsive the car is to the driver and to road conditions and how safely the car can be operated Design decisions will affect the aesthetics of the car how comfortable the car is and the cost of manufacturing the car Indeed the decision of whether to invest significantly in R D relating to the car’s steering mechanisms will affect the cost quality and safety of the car And things just get more complicated from there If regulators want to ensure the safety of cars they need to design systems for measuring monitoring and enforcing safety metrics If for instance regulators use crash test dummies modelled after the typical male driver car manufacturers will design cars that are safe for typical male drivers – and possibly unsafe for female drivers 36 Design in other words is difficult … it’s Complicated In some systems including nearly all software-based systems design is more that just difficult it is “complicated ” Complex systems are systems with many interconnected parts in which changes to any one of those parts can affected other parts often in unexpected and hard to understand ways The measure of complexity in these systems is said to grow polynomially exponentially or even factorially in proportion to the total number of components in the system In other words doubling the number of components in a system from 5 to 10 may increase the 33 34 35 36 Why Amazon’s ‘1-Click’ Ordering Was a Game Changer KNOWLEDGE@WHARTON Sept 14 2017 https knowledge wharton upenn edu article amazons-1-click-goes-off-patent # Lucy Hattersley What is Breath for Apple Watch I How to use Apple Breathe app in watchOS3 M ACWORLD Oct 03 2016 https www macworld co uk feature iphone what-is-breathe-for-apple-watch-how-use-apple-breathe-app-inwatchos-3-3643692 Close your rings APPLE https www apple com watch close-your-rings last visited Jan 5 2020 This is a topic that has been discussed extensively in recent years For one example see Astrid Linder Mats Svensson Road Safety The Average Male as a Norm in Vehicle Occupant Crash Safety Assessment 44 INTERDISCIPLINARY SCI REVS 140 2019 Page 10 of 21 overall complexity – the possible number of interactions between those components – by a factor of over 30 000 One of the primary goals of “design” is to reduce complexity This is primarily done by reducing the number of possible interactions between the components of a system – and this in turn means reducing the overall functionality of the system The challenge is figuring out which functionality to excise and which to retain Sometimes reducing overall system complexity can even entail adding new components For instance a system can be designed with a “basic” or “default” mode in which users cannot change most settings but can also have an additional “advanced” mode in which the user has greater control This requires developing two separate interfaces and a way to switch between them – and to educate users on this multi-interface system Complexity abounds often with tragic results The Three Mile Island disaster is a classic example – perhaps the most famous As described by the Washington Post following the disaster “The Three Mile Island control room is a vision from science fiction It sits under the shadow of the 190-foot-high domed reactor containment building Inside a horseshoe-shaped panel stretches 40 feet along three walls lined with dials gauges and 1 200 warning lights color-coded red and green ”37 All of those dials gauges and warning lights were working well when the disaster occurred But they presented too much information to be useful and did so in a way that could not be useful in the event of a real-time emergency Subsequent investigation determined that the indicator light for the pump responsible for the chain of events that led to the eventual disaster communicated ambiguous information that misled the facility staff as they tried to figure out why the power plant was malfunctioning 38As Don Norman Emeritus Professor and Director of the University of California San Diego Design Lab explained it “the control room and computer interfaces at Three Mile Island could not have been more confusing if they had tried ”39 The August 21 2017 collision of the Navy destroyer John S McCain presents a more recent and more poignantly tragic example of the complexity and stakes of design decisions The NTSB’s report on that incident identifies “the design of the destroyer’s Integrated Bridge and Navigation System” as one of the factors contributing to the collision and finds that “The design of the John S McCain’s touch-screen steering and thrust control system increased the likelihood of the operator errors that led to the collision ” 40 Moreover it focuses extensively on issues relating to operational procedures and crew training that are directly related to the design of the IBNS 41 As documented in a subsequent ProPublica report the IBNS design failures eerily 37 38 39 40 41 The Washinton Post A Pump Failure and Claxon Alert THE WASHINTON POST 1979 http www washingtonpost com wp-srv national longterm tmi stories ch1 htm Pulkit Verma 3 button designs from 3 different decades that almost results in catastrophe UX COLLECTIVE Oct 18 2019 https uxdesign cc 3-button-designs-from-3-different-decades-that-almost-results-in-catastrophe-9ac65498c9c4 Id NAT’L TRANSP SAFETY BD MARINE ACCIDENT REPORT NTSB MAR-1901 COLLISION BETWEEN US NAVY DESTROYER JOHN S MCCAIN AND TANKER ALNIC MC SINGAPORE STRAIT 5 MILES NORTHEAST OF HORSBURGH LIGHTHOUSE 33 2019 https www ntsb gov investigations AccidentReports Reports MAR1901 pdf hereinafter “NTSB” Id Page 11 of 21 echo the design failures at Three Mile Island an easily-overlooked pop-up window indicated which station had steering and thrust control at any given time 42 In a more modern twist the use of touch-screens added additional complexity As noted by the NTSB report “the touch-screen throttle controls deprived the lee helmsman of tactile feedback when the throttles were unganged and mismatched ” which was likely another contributing factor to the incident 43 Both of these tragedies are examples of “normal accidents” – a term first coined by Charles Perrow 44 The core of Perrow’s insight into “normal accidents” is that they are an inevitable part of any sufficiently complex tightly coupled system Perrow specifically considered the potential for these accidents in systems with a high catastrophic potential – Three Mile Island was his motivating example – to make argue that as a society we must either accept the inevitable tragedies that accompany complex systems such as these or abandon them But his basic insight that complex systems will behave in unpredictable and at times undesirable ways and that we cannot design this characteristic out of them generalizes across any complex system Almost all software is a complex system subject to the analysis above Consider alone the challenges that websites face in standardizing their user interface across different web browsers and operating systems Although the problem is less severe now due to browsers relying on more standardized rendering engines for the first decade or two of the world wide web it was a common phenomenon for a website to only work well on one browser and one operating system typically Windows with Internet Explorer This was not the result of a nefarious plan on the part of web developers but was rather the result of developers making design decisions under imperfect conditions 45 The rendering engines of different browsers often made it difficult to perfectly render the same user interface in the same manner across every browser and OS combination 46 Thus websites frequently would have problems with certain sections not rendering correctly functionality missing or scripts not executing as expected With the entrance of mobile phones and tablets the problem has been made more complicated Designers now face the challenge of designing interfaces to run on multiple browsers running on multiple classes of devices with dramatically different user interfaces – both in terms of display and input – across desktops laptops tablets and phones Sometimes firms have the resources to customize their interfaces for many combinations of devices and browsers but this is often not the case Thus designers create interfaces that attempt to average out the differences across device and browser combinations or choose to focus on certain more popular or higher-value combination to the exclusion of others 42 43 44 45 46 T Christian Miller Megan Rose Robert Faturechi Agnes Chang Collision Course PROPUBLICA Dec 20 2019 https features propublica org navy-uss-mccain-crash navy-installed-touch-screen-steering-ten-sailors-paid-with-theirlives NTSB supra note 40 at 33 Charles Perrow NORMAL ACCIDENTS LIVING WITH HIGH-RISK TECHNOLOGIES 1984 Tom Warren Chrome is turning into the new Internet Explorer 6 VERGE Jan 4 2018 https www theverge com 2018 1 4 16805216 google-chrome-only-sites-internet-explorer-6-web-standards Marco Tabini Why some websites don’t work properly in your favorite browser MACWORLD Jan 10 2013 https www macworld com article 2023682 why-some-websites-dont-work-properly-in-your-favorite-browser html Page 12 of 21 These concerns are compounded when we add in different types of users – both in terms of soft characteristics like preferences and harder characteristics like age and disability It is nigh impossible to design an interface that accommodates any given set of user preferences and system requirements perfectly And the more variables that you try to accommodate the more complex the system becomes – with the result that the better a job a designer tries to do in delivering a satisfactory experience to all users the more likely it becomes that the system will fail catastrophically Of course the degree of catastrophe between Three Mile Island and a website recommending the wrong product to a shopper is not truly comparable But it is nonetheless the case that the underlying causes of many seemingly “dark patterns” may be as innocent and inevitable as the Three Mile Island accident This of course is not to excuse the truly and myriad inexcusable examples of deceptive dark patterns that many firms unquestionably use A firm that programs its system to provide false information to a user knowing that the user may act upon that information is not an example of a normal accident or the sort of design mishap that results from the complex nature of systems On the other hand this is a cautionary story about inserting regulators or regulation into the design process Such regulatory intervention increases complexity sometimes dramatically This is not a reason not to undertake design-related regulation – it is however a reason to do so cautiously and narrowly … it’s Competitive Product design is a key margin along which firms compete Consumers desire products that are “user friendly” and “easy to use ” Importantly “user friendly” and “easy to use” are defined in terms of the users not the product designers The story of Apple’s success is one tale that captures this Apple’s recent history and the role of design in it is reasonably well know 47 the iPod the iMac and the iPhone were all as revolutionary and successful as they were largely due to their design Apple took a streamlined and minimalist approach to design delivering products with simplified interfaces designed to operate smoothly and intuitively This approached served Apple and Apple’s customers well – but it is important to note that it does not serve all customers well But Apple’s history goes back to far before the iPod The introduction of the original Macintosh computer in 1984 was arguably even more revolutionary it marked a transition in computer design from computers that were designed for computer engineers to computers that were designed for ordinary users It could be used by anyone without specialized training it included basic applications that did most of the things that ordinary users wanted in ways that 47 For a recent account focusing on the recent departure of Apple’s longtime chief of design Johnny Ives see Chris Welch Jony Ive leaving Apple after nearly 30 years to start new design firm VERGE Jun 27 2019 https www theverge com 2019 6 27 18761736 jony-ive-apple-leave-iphone-chief-design-officer-lovefrom-companyquit Page 13 of 21 most of them understood Simple word processing simple graphics editing simple file management a simple graphical interface But this simplicity – both from the Macintosh era and the iPod era – comes at a cost Apple products are exceptionally good at doing what they are designed to do – but part of creating such products is “locking them down ” They can be relatively difficult to customize or to configure for applications unanticipated by Apple’s design The result is that some users rather dislike Apple products The competition for the personal computer in the 1980s was largely between locked-down architectures like Apple’s and open architectures like the IBM compatible PC The competition on mobile devices today is largely between the closed-platform iPhone and open-platform Android devices Neither of these approaches is necessarily better or worse than the other To the contrary these design elements define how the platforms compete Apple provides a more consistent uniform and in some ways limited set of product features and affords greater integration across its ecosystem of products Android is less consistent but supports a wider range of hardware and applications and generally requires more complicated tools for cross-device integration Different users prefer differently designed systems The fact that we have multiple different competing designs makes all users better off It is also important to consider the development process that is popular among technology producers Given the complexity of design the initial version of new products rarely support a full range of features platforms users c It is prohibitively expensive to develop fully-featured software in an initial release particularly given the high failure rate of most new products Rather firms develop an initial release targeting a specific cohort for entry – perhaps a hypothetical typical customer or perhaps a specific type of customer that the firm thinks is suitable to target for the product’s initial launch Once the product has achieved a minimum successful launch the design can be incrementally modified to support wider or more specific user bases This model of software design has distinct benefits it enables rapid delivery of new goods and services to market and it enables competition from smaller firms Introducing requirements that a design must be “complete” before release – however that is determined – would make entry difficult or impossible for a large amount of potential entrepreneurs Further even the products of medium and large firms would be negatively affected by requiring completed designs The rapid prototyping process works the same for both small and large firms In the context of dark patterns these observations urge two types of caution First what may appear to be a “dark pattern” may merely be a design artifact A product may have been designed for one user cohort or for one interface and may now be being used by other users or on other devices The default settings for an initial user base may not be the same as we would expect for the expanded user base and it may appear that the platform is designed to push users into disadvantageous decisions Or an interface that was designed for instance to run on desktop or laptop computers may be awkward to use on a mobile device in ways that again seem to be intentionally-designed dark patterns On the other side of this coin requiring firms to “completely” design systems prior to launching them is at best a burden that is detrimental to Page 14 of 21 competition and at worse impossible Such a requirement would dramatically increase the cost of developing new products and bringing them to market disproportionately hampering smaller competitors And it would make these firms liable for unanticipated uses of their products A better approach to addressing concerns like this is to rely on competition Customers are generally keenly aware of design issues – there is little better way to drive customers away from a product than for it to have an awkward cumbersome of “unfriendly” interface Where firms are able to compete and especially where there is evidence that firms to compete regulation over design elements or design decisions is likely undesirable except in the rarest cases of overtly intentional or exceptionally harmful design patterns Regulating Patterns None of the discussion above is meant to argue that dark patters may not be used in problematic ways – or that they are in fact being used in problematic ways There is without a doubt plenty of bad conduct happening both online and off Industry behavior in this regard is frequently disappointing The question becomes what should be do about it particularly given the sometimes-difficulties of distinguishing between good and bad design practices the potential for competitive pressures to address some of these concerns and the danger that regulating poorly may exacerbate already-difficult design challenges This is made even harder in the online setting where so many parts of the ecosystem continue to change to the extent industry standards and self-regulation presents viable solutions to these concerns such mechanisms are yet in their infancy Given time such mechanisms may address many of the concerns of dark patterns – or they may not In other words the point of the above is that we need to be careful in how and why we regulate these practices including understanding when and whether we should at all In some cases regulatory efforts may be better focused on other areas in some cases it may make more sense to allow the underlying technology and markets to continue to improve before stepping in with regulatory intervention and in other cases still beneficial regulatory intervention may simply not be possible Assessing the problem There is yet little empirical evidence about the extent of the dark patterns as a problem – meaning both the incidence of use of dark patterns the effectiveness of those patterns and ultimately the extent to which use of these patterns actually harms consumers The literature cited above such as recent studies showing that various dark patterns are being used on shopping web sites and that these patterns can be effective at increasing the likelihood of consumers taking actions that they otherwise would not are compelling evidence that there is reason to be concerned 48 On the other hand behavioral psychology literature studying the effects of disclosure rules in high-stakes transactions such as home mortgages have found that regulation of 48 See discussion supra at notes 16-17 Page 15 of 21 disclosures – effectively the design of how and what information is presented to consumer borrowers – have little to no effect on borrowing behavior 49 This raises questions about whether regulation of dark patterns is justified If their effect is only limited to low-value transactions the impact on consumers may not be sufficient to justify regulation that may or may not prove effective Indeed if the concern is that firms use dark patterns to extract small additional revenue from a large number of consumers that may be particularly at-risk of exploitation caution may be particularly warranted increasing regulatory compliance costs on these firms could result in them leaving markets entirely leaving those consumers entirely unserved rather than incurring compliance costs and facing potential enforcement actions if they do not comply correctly We live in an imperfect world and need to be careful to judge proposed regulations by their likely real-world effects not against a world of costless and perfectly effective regulation 50 It also unclear how much of this behavior is fraudulent or deceptive and how much of it is simply advertising by another name Calling a shopper’s attention to a complementary product during a checkout flow could be called trickery but it’s not clear how it is materially different than showing the user an advertising they need to dismiss when they land on the site’s home page On the other hand practices like cramming slamming and “sneak into cart” are much more likely to be harmful – the transaction costs of returning or cancelling unwanted items may exceed the value that the firm extracts from the consumer leading the consumer to move on with her day and take the loss Research on the effects of dark patterns on consumers is still in its infancy There probably is not enough today to justify any broad regulatory undertakings that would not incur substantial risk of unintended consequences In all likelihood the best regulatory approach – to the extent that one proves to be justified – will be one that is tailored to specific types of pattern Such regulation could for instance make specific design practices e g providing fraudulent information to consumers at or near the time of purchase illegal or could alternatively task or empower an agency such as the Federal Trade Commission to identify specific practices as violative of the FTC Act The marketplace is working to address these problems Even as some firms take advantage of dark patterns other firms are voluntarily working to protect consumers from them Google to take one example banned advertisers from its network that used pop-under ads which it viewed as a poor design pattern providing a bad user experience 51 Most major browsers now allow users to automatically block pop-up windows – another design practice designed to draw users attention similar to windows that cannot easily be closed Malware and spyware frequently attempted to takeover a user’s web browsing experience 49 50 51 See e g Michael S Barr Sendhill Mullainathan Eldar Shafir Behaviorally Informed Home Mortgage Credit Regulation Joint Center for Housing Studies of Harv U Working Paper UCC08-12 2008 available at https papers ssrn com sol3 papers cfm abstract_id 1121199 Harold Demsetz Information and Efficiency Another Viewpoint 12 J OF L ECON 1 1969 elaborating the “Nirvana Fallacy” comparing the ideal scenario as more efficient than the real choices presented Sarah Perez Google bans its ads on sites that use those annoying ‘pop-unders’ TECHCRUCH Jul 11 2017 https techcrunch com 2017 07 11 google-bans-its-ads-on-sites-that-use-those-annoying-pop-unders Page 16 of 21 via browser hijacking – the installation of a software add-on that would permit third parties to interfere with and observe the web browsing of a user As of Windows 10 Microsoft disabled the key behavior of web browsers that facilitated browser hi-jacking Those are all examples of platform-level efforts to combat these practices by disabling features needed to implement designs that are particularly likely to be harmful to users There is also effort among industry professionals to combat the use by designers of dark pattern techniques The figure below shows the first three pages of results for a Google search of the term “dark patterns ” It shows that 27 of the top 30 results marked in red boxes for the search term “dark patterns” demonstrate a widespread understanding and condemnation of using dark patterns to trick users These search results show that designers are warning peers not to use these and similar tactics and where the practice may have value they offer alternative design tools The remaining three search results link to more general discussions of dark patterns – these discussions all also describe use of them approach as problematic Page 17 of 21 Given the complexity of design there is reason to prefer to rely on the marketplace to address the concerns raised by dark patterns – particularly given that this market-based approach appears to be working Some patterns that seem to be or even in fact are being used in ways that are problematic may also have good uses For instance pop-up windows are generally problematic but some websites make good use of them Rather than prohibit them entirely modern web browsers indicate to users when a website has tried to use a pop-up window and allow users to allow them on a case-by-case basis for specific websites or generally This is a more nuanced approach than regulation is likely to implement Moreover this change was phased in over a period of time and across a range of browser platforms allowing for industry to experiment and gather data on how best to implement this feature And it is also notable that this feature was implemented at the browser platform level Regulation of design features can be undertaken at any number of levels in the software stack – from the operating system to the protocols and programming languages use to send content to web browsers to the programmers who write the code that controls the design of the website to the browsers that render that code To whom should regulation of design patterns apply How does this choice affect the overall complexity of the design ecosystem Indeed even aside from this problem there is a great deal of value in maintaining stable interfaces even where those interfaces may contain some poor design Frequent design change is itself a dark pattern Consumers are more likely to make mistakes – or to be tricked into doing things they would not otherwise do – if they are unfamiliar with a design or an interface 52 Regulatory intervention into design could force widespread redesign of interfaces especially if undertaken regularly or in a way that lacks the precision of changes that industry itself may be able to make This in turn could have widespread adverse effects on consumers Again this is not to say that regulation is unwarranted or not possible – only that it must be undertaken with care and with due consideration to alternatives such as industry standardization which would increase stability both over time and across websites and self-regulation The sufficiency of existing law Existing law is sufficient to address many possibly most of the concerns raised by dark patterns Most of the egregious dark patterns should fall within the ambit of the FTC’s consumer protection authority To the extent that they are harmful most of these patterns involve making representations or engaging in practices that are designed to deceive consumers Such conduct is covered by Section 5 of the FTC Act’s prohibition against unfair and deceptive acts and practices 53 In order to make out such a claim the FTC Act and the FTC’s subsequently adopted Policy Statement on Deception 54 the Commission must establish that the practice is likely to mislead the ordinary reasonable consumer in a way that is material to injury to that consumer 55 The Commission may presume that express claims are material 56 Thus the Commission need 52 53 54 55 56 See e g Miranda supra note 4 15 U S C § 45 a FEDERAL TRADE COMM’N FTC POLICY STATEMENT ON DECEPTION 1983 https www ftc gov system files documents public_statements 410531 831014deceptionstmt pdf Id Id at note 48 Page 18 of 21 only demonstrate injury – i e that some consumer did in fact make purchases that they otherwise would not have – to take action against firms employing design practices dark patterns such as falsely asserting that a certain number of people have recently purchased a product or that a specific limited number of unit remain available for sale Other practices such as obscuring how to close a window may require that a more substantial evidentiary burden by the Commission be met Should the FTC decide to take action against firms making use of dark patterns there are several approaches that it could take In general like most regulatory agencies the FTC has both adjudicative and rulemaking authorities – though its rulemaking authority is more involved than the traditional APA rulemaking procedures 57 In general the Commission may bring an administrative enforcement action to enjoin any conduct that the Commission determines after an investigation and administrative hearing 58 it may also seek damages for such action in federal court for conduct that “a reasonable man would have known under the circumstances was dishonest or fraudulent ”59 It may also issue rules that “define with specificity acts or practices which are unfair or deceptive ”60 Once enacted it can enforce such rules through administrative action or directly in federal court seeking both injunctive relief or damages 61 In recent decades the FTC has been reluctant to engage in rulemaking proceedings due largely to misunderstandings of both the FTC Act and general administrative law dating back to important judicial losses in the 1980s – but this does not mean that it lacks such authority 62 Given the broad and generally unexplored depth of the FTC’s authority directly relevant to the practice of dark patterns it would be preferable for the FTC to take the lead in developing rules relating to dark patterns It only makes sense for legislative approaches to be explored should the FTC’s authority prove insufficient to the task It also bears note that in addition to authority that the FTC has it is established law that consent obtained through deception isn’t valid 63 Many dark patterns exploit the boundaries of consent But this issue is broader than the issue of dark patterns relating for instance to contracts of adhesion the process of contract formation in the online setting and the enforceability of contracts that are generally known to go unread These are topics of significant and ongoing arguably endless discussion – to the extent that legislative attention should be given to this issue it should focus on the validity of consent not on the sub-issue of dark patterns 57 58 59 60 61 62 63 See 15 U S C § 57a These procedures were amended in 1975 by the Magnuson-Moss Warranty—Federal Trade Commission Improvement Act of 1975 Pub L 93-637 to facilitate heightened Congressional oversight of FTC rules 15 U S C § 45 a 2 15 U S C § 57b a 2 15 U S C § 57a a 1 B 15 U S C § 45 a 2 15 U S C § 57b a 1 See generally Justin Hurwitz Chevron and the Limits of Administrative Antitrust 76 U PITT L REV 209 2014 See also FTC Commissioner Rohit Chopra Comment of Federal Trade Commissioner Rohit Chopra Hearing #1 on Competition and Consumer Protection in the 21st Century 2018 available at https www ftc gov system files documents public_statements 1408196 chopra_-_comment_to_hearing_1_9-6-18 pdf See RESTATEMENT SECOND OF TORTS § 892B CONSENT UNDER MISTAKE MISREPRESENTATION OR DURESS AM LAW INST 1979 RESTATEMENT SECOND OF CONTRACTS § 163 WHEN A MISREPRESENTATION PREVENTS FORMATION OF A CONTRACT AM LAW INST 1981 Page 19 of 21 The need for new law or regulation To the extent that existing legal rules are insufficient to address harms from dark patterns it is likely either because the conduct is not clearly harmful or it may at times be beneficial If such is the case the conduct likely should not be prohibited Nonetheless this is a reasonable area of legislative concern where regulation either today or in the future may be warranted Should regulation be desired a few ideas to keep in mind when approaching regulation in this area are discussed below This is an area well suited to industry self-regulation where standardized industry practices are given some presumption of being inoffensive but entities deviating from those practices bear a burden of demonstrating that their design choices are in the interest of consumers As discussed above industry is and has consistently been working to improve the status quo and deter the use of pernicious dark patterns 64 The most viable approach would likely be to allow firms to use contemporaneous documentation that is documentation supporting design decisions at the time those decisions were made to demonstrate that design decisions were made with the interest of consumers and users in mind Such a factor could be influential both for the development of standardized industry practices as well as for firms that deviate from those practices by placing an expressly consumer-focused R D element at the heart of the design practice Such documentation would tend to suggest that pro-consumer justifications exist for design decisions Moreover to the extent that designers are not concerned with consumer experience today such as if they are focused more narrowly on designs that are appealing on technological or aesthetic grounds but that may in fact be detrimental to the user experience of products it would create an strong incentive for designers and industry groups to focus expressly on the effects of design decisions on consumers This may also be an area well-suited to the development of an expedited review and rulemaking process such as that developed in the DMCA for the review of circumvention technologies 65 For instance the FTC could periodically report to Congress on practices that it is seeing that have the potential to harm consumers but fall outside of its existing statutory authority Ideas such as these would bolster the FTC’s authority in this area without need for the enactment of a substantial new regulatory regime or enactment of ossifying laws In general the FTC should be encouraged to explore the limits of its authority to address these concerns including through narrow legislative interventions such as discussed above or through FTCgenerated reports on these issues before implementing new Congressionally-crafted regulatory regimes Importantly administrative remedies should be limited to injunctions with civil penalties only available through the federal courts And except in case of clearly intentional fraudulent behavior – such as would already be covered under existing Section 5 authority – the 64 65 See supra note 47 and nearby text See 17 U S C 1201 a 1 C Page 20 of 21 preferred initial remedy should be for firms to forego the problematic conduct with the purpose of improving the overall standard of conduct of the industry in a non-adversarial manner To the extent the law proscribes certain designs it must do so carefully including thinking about what alternative designs may be adopted – both legitimate and illegitimate ones As discussed above design is hard66 – these are complex systems – and any regulation puts regulators in the shoes of the designers What’s more it ossifies design Finally given that many dark patterns are used both online and offline – and more generally that the concerns created by dark patterns are not unique to the online setting Congress should consider whether the scope of its interest in this area should be limited to the online setting For instance many firms engage in practices that make it difficult to cancel service or return products To the extent that concern is justified about analogous online practices it does not make sense to cabin that concern – or any exploration of it through reports or regulation – to the online setting If new rules are adopted regulators should consider whether any proscribed practices should be limited to online actors or whether they should be rules of more general applicability 66 See supra notes 33-46 Page 21 of 21