LLM Prompt Surveillance: The Dark Side of Conversational AI

Why am I the only one worried about what happens to our LLM prompts?

Low on time? Listen to a 9m podcast discussion about this topic:

Note: This audio is AI-generated and misgenders me multiple times, which is an example of bias in AI. Haha

1. Introduction

Conversational AI products such as OpenAI’s GPT-4 have seemingly revolutionized everything from email writing to strategic planning–overnight! LLM stands for Large Language Model, which is an advanced AI system trained on vast amounts of text data to understand and generate human-like language. It’s pretty cool stuff.

Yet, as we embrace this Machine Learning technology, we must confront a growing concern: the risks of "prompt surveillance"–a term I’m coining here to describe the hidden and somewhat amorphous dangers posed by the collection and analysis of user inputs to LLMs.

In this report, I will explore why this issue deserves broader recognition.

Note: I use the terms “LLM provider” and “AI company” interchangeably. All LLM providers are AI companies, but not all AI companies are LLM providers.

2. What Are LLM Prompts?

At their core, LLM prompts are the inputs we provide to AI models to elicit a response. These can range from simple questions ("What's the weather like today?") to complex instructions ("Draft a business plan for a sustainable energy startup").

Technically, when you enter a prompt, it's converted into a series of numerical representations (tokens) that the AI can process. The model then uses sophisticated algorithms to generate a response, token by token.

This process, while fascinating, is less important for our discussion than understanding the types of information prompts can contain:

• Personal details and experiences

• Business strategies and trade secrets

• Legal queries and case details

• Creative ideas and intellectual property

• Health concerns and symptoms

• Financial information and investment strategies

• Political opinions and ideological leanings

In essence, prompts serve as the interface between human intent and the LLMs vast knowledge representation, allowing users to leverage the model's capabilities for various tasks.

Prompts entered into LLMs can contain a wide range of information, including personal details, business secrets, legal queries, and creative ideas. For example, a lawyer might use an LLM to draft a contract, or a business owner might ask it for strategic advice. In these cases, the prompt itself may contain sensitive or confidential information. When this data is logged by the operators of the model, it becomes vulnerable to misuse, either by internal actors (like employees) or external attackers (like hackers).

3. Why Are LLM Prompts Valuable?

The value of prompts extends far beyond their immediate use in generating responses. For users, prompts are a gateway to accessing the LLMs capabilities, whether for work, creativity, or personal assistance.

But for corporations and bad actors, prompts are a goldmine of information.

Prompts also hold value for companies beyond improving their models. The data can be analyzed to understand user behavior, preferences, and trends, which may then be monetized for targeted advertising or sold to third parties. Over time, a pattern of user prompts can reveal intimate details about a person's habits, beliefs, or even mental health, making the potential for misuse highly concerning.

4. What Is LLM Prompt Surveillance?

This practice represents a potential invasion of privacy, a threat to intellectual property, and a powerful tool for manipulation and control at population-scale.

Prompt Surveillance is the systematic collection, analysis, and potential exploitation of user inputs (prompts) in Large Language Models (LLMs), encompassing both the intentional gathering of data for model improvement and the incidental accumulation of user information.

This practice involves the retention, processing, and potential misuse of prompts, which may contain sensitive personal, professional, or proprietary information, creating risks to privacy, intellectual property, and individual autonomy.

5. Etiology

Centralized LLMs, such as OpenAI’s GPT-4, require vast computational resources. Because of this, they are typically hosted on cloud infrastructure controlled by major technology companies, such as those owned by Amazon, Google, and Microsoft. Centralization allows LLMs to process a multitude of user queries simultaneously, leveraging powerful cloud infrastructure to handle computation-intensive tasks. Users interact with these models through APIs or web interfaces by sending their prompts to those remote servers for processing. While centralization allows for efficient scaling to millions of users, it also creates inherent vulnerabilities in the collection, storage, and analysis of prompts. This architecture introduces a series of cascading risks at each LLM provider, where vast amounts of user data are concentrated.

It is also possible that some AI companies have already realized the vast datasets scraped to build their models were just the beginning.

The next, potentially more valuable datasets they develop could be our own intimate expressions, freely given while interacting with their models under a broad presumption of privacy. This user-generated content, comprising our thoughts, questions, and ideas, may become the new frontier of data harvesting—all collected under the guise of model interaction.

No one’s talking about this yet! (Except me, hehe)

6. Downstream Risks

The practice of prompt surveillance could open users up to significant risks, which I list here in no particular order:

6a. Data Persistence and Leakage

Prompts are often stored for performance enhancement or fine-tuning the LLMs. Even when anonymization techniques are applied, the risk of re-identifying sensitive personal or proprietary information persists.

Prompt data can leak due to several scenarios:

• Insider Threats: System administrators or engineers with access to backend systems might misuse sensitive prompt data.

• Third-Party Exploits: Cloud infrastructure hosting LLMs is vulnerable to hacking, leading to unauthorized exposure of stored prompts.

• Inadequate Encryption: Insufficient security measures could result in the interception of unencrypted prompt data, heightening the risk of leaks during transmission.

Scenario: Dr. Raúl Mendoza, a biochemist working on a breakthrough cancer treatment, uses an LLM to help analyze complex protein interactions. He inputs detailed research data, including unpublished findings. Months later, a rival pharmaceutical company announces a suspiciously similar treatment. Investigation reveals that a state-sponsored hacking group had breached the LLM provider's servers, stealing years of accumulated research data from various scientists, including Dr. Mendoza’s work.

6b. Inference Attacks and Model Inversion

Centralized LLMs process and store vast amounts of user data, which can become vulnerable to sophisticated attacks:

• Model Inversion: In this type of attack, adversaries can use model outputs to reconstruct the inputs, potentially revealing sensitive information such as legal strategies or proprietary business plans.

• Membership Inference: Attackers can infer whether specific data points were used to train the LLM, violating privacy expectations and exposing individuals’ data.

Scenario: Chen Wei, a human rights activist in a politically oppressive regime, carefully uses an LLM to draft coded messages for organizing peaceful protests. Unknown to Chen, the government has coerced the AI company to grant access to their systems. Using advanced inference techniques, they decode Chen's prompts, leading to the arrest of Chen and fellow activists before the protest can take place.

6c. Data Aggregation and Profiling

Over time, prompt data can be aggregated across millions of users, allowing service providers to build detailed profiles:

• Behavioral Profiling: LLM operators can analyze user prompts to infer preferences, habits, or even personal struggles, which could be sold to advertisers or third parties.

• Surveillance and Censorship: Governments or corporations may use this aggregated data to monitor politically sensitive behavior, stifling free speech or curtailing privacy rights.

Scenario: Jamal, a teenager struggling with his sexual identity, frequently uses an LLM for advice on LGBTQ+ issues, mental health, and dealing with conservative parents. The AI company sells this data to a data aggregator, which resells it to a political consulting firm. Months later, Jamal's deeply religious parents start receiving targeted ads for conversion therapy and anti-LGBTQ+ content, creating a hostile environment at home before Jamal is ready to come out.

6d. Predictive Behavioral Modeling

One of the most concerning aspects of prompt surveillance is the potential for organizations to develop predictive behavioral models based on patterns found in user prompts.

By analyzing a user’s language and the frequency or tone of their queries, organizations can predict future actions or decisions, using this data for several purposes:

• Targeted Advertising: Predictive models could be used to tailor advertisements more effectively, reaching users at moments of perceived vulnerability or desire.

• Opinion Shaping: These models might influence public opinion by understanding an individual’s cognitive patterns and beliefs, enabling the delivery of tailored information that subtly shifts their viewpoints.

• High-Risk Identification: In more dystopian scenarios, predictive modeling could be used to flag individuals as "high-risk" based on their thought patterns or queries. For example, those who express financial stress or political dissent could be targeted for closer surveillance or intervention.

Scenario: Sofia, a talented software engineer with bipolar disorder, uses an LLM regularly for work and personal queries. The AI company's predictive models identify patterns suggestive of her condition. This information is sold to an employment screening service. When Sofia applies for her dream job, she's rejected without explanation, unaware that the company's AI-driven hiring tool flagged her as a "high-risk" employee based on her LLM usage patterns.

6e. Privacy Breaches

Users often inadvertently reveal personal details in their prompts, ranging from medical conditions to financial issues.

For instance, someone querying about health symptoms may unwittingly disclose their medical status, which could be exploited by insurance companies or employers, leading to discrimination or unfair treatment.

Scenario: Alejandro, an investigative journalist in a country with a history of press suppression, uses an LLM to research government corruption. His prompts, detailing sources and evidence, are quietly logged and accessed by intelligence agencies. Alejandro's sources start disappearing, and he faces trumped-up criminal charges, effectively silencing his investigation.

6f. Intellectual Property Theft

Prompts frequently contain proprietary business information, creative ideas, or sensitive intellectual property.

A company developing a new product might ask the LLM for assistance, potentially exposing trade secrets through their prompts, leaving them vulnerable to theft or misuse.

Scenario: Yuki, a small-scale inventor in rural Japan, uses an LLM to refine her designs for a revolutionary sustainable energy device. Unbeknownst to her, the AI company has a partnership with a major tech corporation. Yuki's innovative ideas, extracted from her prompts, are quietly passed to the corporation's R&D department. Months later, the corporation patents a similar device, leaving Yuki with no legal recourse.

6g. Behavioral Control and Manipulation

The insights gained from prompt surveillance could enable targeted manipulation and control:

• Targeted Advertising: Analyzed prompts reveal users’ vulnerabilities. For example, someone asking for financial advice might be targeted with high-interest loan offers.

• Opinion Shaping and Propaganda: Knowing a user's beliefs through their prompts allows political entities to tailor propaganda, subtly influencing individuals’ opinions.

Scenario: Marcus, a recovering gambling addict, uses an LLM for financial advice and emotional support. The AI company sells this behavioral data to online casinos. Marcus starts receiving highly personalized ads for "investment opportunities" that are thinly veiled gambling platforms, carefully timed to coincide with his paydays and moments of emotional vulnerability, threatening his hard-won recovery.

6h. Discrimination and Bias Amplification

Algorithmic bias inherent in LLMs can be further exacerbated by prompt analysis:

• Hiring Discrimination: An LLM-based recruitment tool might inadvertently discriminate against certain demographic groups due to biased language patterns in their prompts.

• Reinforcement of Prejudices: Recommendations systems informed by prompt data could foster echo chambers, reinforcing societal biases or divisions.

Scenario: Aisha uses an LLM to prepare for job interviews in the tech industry. Unknown to her, the LLMs training data is biased, and it subtly alters her language to sound "less foreign." When she uses this advice in interviews, she feels she's compromising her identity. Companies that use AI-driven interview analysis consistently rate her poorly for "cultural fit," despite her excellent qualifications.

7. Psychological Inferences

Because centralized LLM providers collect vast amounts of user data, I believe they are uniquely positioned to infer far more about individual psychology than most users realize. Through the patterns in the prompts submitted, these systems can extract detailed insights into a person’s cognitive processes, emotional states, values, and even their vulnerabilities.

Here are several mechanisms through which these inferences could be made (again, these are listed in no particular order):

7a. Revealing Cognitive Patterns and Thought Processes

High-usage or prolific users inevitably submit a large volume of prompts over time. These inputs are not random; they follow consistent cognitive patterns reflective of an individual's thought processes, problem-solving approaches, and emotional tendencies.

By analyzing the structure, tone, and content of the prompts, AI companies can detect underlying cognitive tendencies such as:

• Analytical vs. Intuitive Thinking: How users frame questions and problems can reveal whether they lean more toward logical analysis or intuitive leaps in their thinking. Frequent prompts that focus on technical precision may suggest a more analytical mindset, while creative, open-ended prompts may indicate more intuitive, divergent thinking.

• Decision-Making Styles: Users who frequently return to the LLM for advice on life decisions—ranging from career choices to personal dilemmas—reveal much about their decision-making frameworks. AI companies can infer whether a user is risk-averse, decisive, or prone to second-guessing themselves.

7b. Emotional and Psychological Profiling

Prompts are often highly reflective of a user's emotional and psychological state at the time of their submission. Whether consciously or unconsciously, users may express anxiety, frustration, curiosity, or even excitement in the wording of their queries.

By tracking these emotional markers over time, LLM providers can build emotional profiles that include:

• Mood Patterns: Repeated use of phrases like "I’m feeling stressed," or "I don’t know what to do" can reveal patterns of anxiety or depression. On the flip side, frequent excitement or motivation-based queries may point to high energy periods. AI companies could use these insights to track mood swings, predict periods of emotional vulnerability, or even detect mental health conditions like anxiety or depression.

• Emotional Triggers: By analyzing which topics elicit certain emotional tones, AI companies can infer what triggers emotional responses. For example, a user who frequently expresses anxiety around money management or relationships is inadvertently providing a window into their emotional sensitivities.

7c. Behavioral Intent and Habitual Tendencies

High-usage users inevitably form habitual interaction patterns with LLMs. From these, providers can infer not only behaviors but also predict future actions based on recurring queries.

Behavioral intent becomes more transparent when users seek repetitive advice in specific areas:

• Reinforcement of Goals: If a user regularly asks questions about self-improvement, fitness routines, or learning new skills, the AI company can infer long-term behavioral goals. This understanding allows them to predict what the user may aim for in the future—whether it’s a career change, a lifestyle shift, or even deeper personal transformations.

• Habitual Tendencies: Behavioral cues can also reveal a user’s habits and routines. For example, a person who consistently asks about managing time may indicate a chronic struggle with productivity or time management. Repetitive requests about quitting bad habits like smoking or procrastination can show self-perceived weaknesses that AI companies can analyze and use for targeted content or advertising.

7d. Personal Values and Beliefs

Language and content can often provide a clear window into a user’s value system. High-usage LLM users—especially those who engage in philosophical or moral inquiries—leave behind clues about their core beliefs, principles, and worldview.

These inferences are made possible through:

• Moral Dilemmas and Value Judgments: Questions that involve ethics or personal dilemmas can point to what values a user prioritizes, such as fairness, freedom, loyalty, or authority. These values may be inferred from how they phrase moral questions or from the topics they consistently grapple with.

• Worldview and Ideological Leanings: A user’s prompts around politics, social issues, or cultural questions reveal deeper ideological leanings. For instance, recurring questions about social justice topics or political ideology can provide an understanding of where a user lies on the political spectrum, their social priorities, and how strongly they feel about certain issues.

7e. Vulnerabilities and Exploitable Traits

The ability to analyze user input over time allows centralized LLM providers to not only understand users' strengths but also to detect psychological vulnerabilities that could be exploited:

• Financial Struggles: A user who repeatedly queries the LLM for advice on financial difficulties, debt management, or budgeting is exposing a core vulnerability—financial stress. AI companies can then predict this person's sensitivity to offers of financial products like loans or budgeting tools, creating an opportunity for targeted (and potentially manipulative) advertising.

• Social or Emotional Isolation: A user frequently asking questions about loneliness, relationships, or self-esteem may reveal social or emotional struggles. This opens a door for manipulative content that appeals to their need for connection, whether through social media engagement or emotionally charged advertising.

• Cognitive Dissonance: Users who ask questions that reveal a gap between their beliefs and their actions (for example, asking for advice on resisting temptations they know are harmful) show a form of cognitive dissonance. AI companies can exploit this by offering tailored content that either resolves this dissonance or exacerbates it for profit or influence.

7f. Predicting Future Behavior

Through predictive behavioral modeling, LLM providers can make highly accurate predictions about future actions. A prolific user’s query history can be a rich dataset for modeling future intent.

For example:

• Career Moves: If a user repeatedly asks questions about new career paths, updating résumés, or learning skills relevant to a different industry, the AI company may predict that the user is preparing to change jobs. In fact, with enough data, corporate analytics can anticipate the timeframe for this decision.

• Life Events: A user frequently querying about wedding planning, home buying, or parenting is signaling significant upcoming life events. AI companies can anticipate these transitions and target the user with content, ads, or services that align with these life changes—whether it’s wedding services, mortgage offers, or parenting advice.

8. A Growing Threat to Privacy

As LLMs become more integrated into everyday life, the potential for abuse through prompt surveillance grows exponentially. The risks range from privacy violations to intellectual property theft, manipulation, and discrimination. The centralized nature of these systems provides a fertile ground for actors—whether corporations, governments, or malicious attackers—to exploit the vast troves of data users submit through their prompts.

While existing privacy laws provide some protection, they often fall short when applied to the novel challenges posed by prompt surveillance. The landscape of AI providers is diverse and complex, with practices varying widely across the industry. Many companies retain prompt data indefinitely for model improvement, and have no strict deletion policies. Transparency about data practices also differs significantly among AI companies, with some offering insights into their processes, and others remaining opaque. User control over personal data is also inconsistent; certain LLM providers allow history deletion or data collection opt-outs, while most others do not. The commercial use of insights gained from prompt analysis is another area of divergence, with some companies strictly prohibiting such practices and others incorporating them directly into their business models and Terms of Service. Moreover, the robustness of security measures protecting stored prompt data can vary dramatically between AI companies.

Perhaps most concerningly, average users lack the technical expertise to effectively evaluate these multifaceted aspects of a corporate LLM provider’s data handling processes, or claims about the same. Open-source LLMs generally offer more transparency regarding training data and processes compared to their proprietary counterparts, but only for experts who are trained to evaluate them.

Consequently, individuals are left to navigate this complex terrain alone, and forced to make judgment calls about what information they're willing to entrust to centralized, non-locally-hosted LLMs without fully understanding the potential ramifications of their choices.

I propose that the implications of LLM prompt surveillance may long transcend immediate privacy concerns.

Once submitted, prompts create a permanent, irrevocable record of an individual's thoughts and behaviors. These accumulated interactions paint an increasingly detailed picture of a person's life, remaining vulnerable to exploitation indefinitely. As AI capabilities advance, this historical data becomes subject to ever more sophisticated analysis, potentially revealing new insights about individuals long after their initial interactions.

This evolving risk persists throughout a person's lifetime, with the potential for exploitation growing as technology progresses and advanced capabilities become more accessible to a wider range of actors.

9. Conclusion

The evolution of large language models presents incredible opportunities, but it also introduces significant threats to privacy, autonomy, and societal well-being. Prompt surveillance is a practice which goes beyond mass data collection and poses emergent risks which remain largely unexplored.

As LLMs become increasingly integrated into our daily lives, it is crucial to develop comprehensive safeguards—encompassing technical, ethical, and regulatory measures—to protect individuals and society from the far-reaching consequences of unchecked data exploitation.

By proactively understanding and addressing these risks, we can assertively work to ensure that AI's benefits enhance, rather than compromise, individual human rights and societal integrity.

If you found this original research valuable, please consider subscribing.