# The city chatbot that told businesses to break the law > New York City's MyCity chatbot gave businesses incorrect and illegal guidance. The city added disclaimers, then shut it down in 2026. Source: https://conversationalaidesign.com/case-studies/nyc-mycity-chatbot.html. Author: Nelson Abacherli Kawakami. Reviewed October 27, 2026. ## Key facts - Investigation: The Markup and THE CITY, published March 29, 2024 - Examples reported: Said employers could take a portion of workers' tips and that businesses could refuse cash; also suggested landlords could discriminate against Section 8 tenants - City response in 2024: Kept it online and added disclaimers - Outcome: The city took it down in February 2026; its page said the beta test had ended - Cost reported: About half a million dollars, per the mayor; the foundations reportedly cost nearly $600,000 ## What happened New York City launched the MyCity chatbot to help small-business owners navigate city rules. In March 2024, journalists at The Markup and THE CITY tested it and reported that it often gave incorrect, and in some cases illegal, guidance with confidence. Reported examples included telling employers they could take a portion of their workers' tips, and telling businesses they could refuse to accept cash, both contrary to city rules. Other reporting described it suggesting landlords could discriminate against tenants using housing vouchers. After the investigation, the city kept the chatbot online and strengthened its disclaimers, which now told users to double-check answers and not to treat them as legal or professional advice. The Markup's follow-up testing found that the chatbot still produced problematic answers and, when asked, was unaware of its own disclaimer. In January 2026, New York's new mayor called the chatbot "functionally unusable" and said it was costing about half a million dollars. In February 2026 the city took it down; the page said the beta test had ended and sent visitors to the city's main website. ## Why it happened: a design reading The reporting tells us what happened, not the internal architecture. From the outside, the design problems are clear: - **Wide scope on high-stakes law.** A free-form assistant covering the whole body of business regulation is a very hard problem. Experts told reporters that other cities typically confined chatbots to narrower inputs. - **Confident answers on shaky ground.** The assistant gave definitive replies instead of showing the official text or admitting uncertainty. - **Inconsistency.** The same questions could produce different answers, which makes testing and trust difficult. - **Disclaimers as the safeguard.** The disclaimer sat on a web page while the bot itself didn't know about it, so the safeguard wasn't part of the conversation. - **No visible plan for correction.** The city said it was fixing problems, yet journalists could still reproduce them, which points to weak testing and monitoring. ## Design lessons 1. **Narrow the scope where law is involved.** Start with the top few questions, answer from official text with citations, and expand only with evidence (module 9, module 22). 2. **Make the not-found path real.** If the sources don't contain the answer, say so and send people to the official page or a person (module 6). 3. **Put safeguards in the system, not in a footer.** Instruct the assistant, check its outputs, and refuse unsafe categories. 4. **Treat outside testers as free red-teamers.** Build an evaluation set from the journalists' questions and run it before every release (module 14). 5. **Plan the lifecycle.** Define success metrics, owners, review cadence, and criteria for fixing or sunsetting the service. ## How to build it better A responsible version might answer a small set of well-understood questions, and for everything else show the relevant official page. **Staying inside what it can verify** > **User:** Can I take a share of my employees' tips? > **Bot:** I can't give legal advice. The city's guidance on tips and labor rules is here: [Official guidance: Tips and gratuities]. In general, rules about tips are strict, so it's worth checking the official page or contacting the agency before making a decision. The key moves: cite the official source, avoid confident legal conclusions, and send users to authoritative help. ## Try it Pick a regulated topic and write ten "should I?" questions where a wrong answer causes harm. Decide for each whether your assistant should answer from a source, decline, or hand off, then write the test. ## Sources - The Markup: Malfunctioning NYC AI chatbot still active despite widespread evidence it's encouraging illegal behavior: https://themarkup.org/news/2024/04/02/malfunctioning-nyc-ai-chatbot-still-active-despite-widespread-evidence-its-encouraging-illegal-behavior - The Markup: Mamdani to kill the NYC AI chatbot we caught telling businesses to break the law: https://themarkup.org/artificial-intelligence/2026/01/30/mamdani-to-kill-the-nyc-ai-chatbot-we-caught-telling-businesses-to-break-the-law - TechRadar: Mamdani is set to kill off New York's functionally unusable business chatbot: https://www.techradar.com/pro/zohran-mamdani-is-set-to-kill-off-new-yorks-functionally-unusable-business-chatbot-which-often-gave-out-illegal-advice - Route Fifty: Mamdani to kill NYC AI chatbot: https://www.route-fifty.com/artificial-intelligence/2026/02/mamdani-kill-nyc-ai-chatbot-was-caught-telling-businesses-break-law/411116