DRAFT 

Deepfake and Identity Spoofing

• Risk: Generative AI can create highly realistic fake audio, video, or images, enabling attackers to bypass biometric authentication or impersonate legitimate users. It would be a significant challenge for online(video) authentication. The rapid advancement of AI-driven deepfake technologies, can undermine biometric authentication mechanisms such as facial recognition.
• Solution: Implement deepfake detection tools, multi-factor authentication (MFA), and robust identity verification processes to reduce reliance on single biometric factors. Also update continuously the forgery detection algorithms.

Prompt Injection and Policy Manipulation 

• Risk: Since policies are enforced through cryptographic protocols rather than natural language interpretation, making prompt injection harder. But if the wallet uses AI-driven assistants or automated decision-making (e.g., for verifying credentials or guiding users), attackers can craft malicious prompts to manipulate the AI’s logic.  

  Even if the wallet itself is secure, any connected AI-based helpdesk or verification service could be exploited via prompt injection.

• Solution: Apply prompt hardening techniques, context isolation, and strict input validation. Use allowlists/denylists and sandbox testing for untrusted inputs.

Data Leakage and Membership Inference

• Risk: If identity-related data is used to train AI models, attackers may infer sensitive attributes or reconstruct original data through model inversion or membership inference attacks. 

  If the wallet uses AI services (e.g., for fraud detection, identity verification, or UX personalization), sensitive identity data might be exposed during model training or inference. If wallet operations involve external AI APIs, data could leak through logs or model updates. Even if raw data isn’t shared, patterns in queries or metadata could allow attackers to infer user attributes. e.g your AI tools on your mobile phone have access to your ID wallet!!

  
  

  If AI is integrated for convenience (e.g., chatbots or automated KYC), and those models access identity data, leakage and inference risks reappear. Metadata (timestamps, transaction patterns) can still be exploited for inference even if credentials are protected.

• Solution: Enforce data minimization, segregate sensitive datasets, adopt privacy-preserving training methods (e.g., differential privacy), and secure the entire data lifecycle.

Misinformation and Social Engineering

• Risk: AI-generated content can be used to create convincing phishing messages or fake instructions, tricking users into revealing recovery phrases or credentials for identity wallets. 

  Digital identity wallets rely on users to manage credentials and recovery phrases. Attackers can still use AI-generated phishing emails, fake instructions, or fraudulent websites to trick users into revealing sensitive information. Social engineering campaigns can impersonate official wallet support or government identity services, convincing users to share credentials or approve malicious transactions.

• Solution: Deploy misinformation detection systems, educate users on security best practices, and implement strict content moderation and auditing for AI outputs. [1]

Synthetic Identity Fraud (looks repeated)

• Risk: The emergence of synthetic identities created by combining real and fabricated data, which can bypass traditional identity verification systems. If such identities are stored or validated within digital identity wallets, they can compromise the overall trust model.
• Solutions:

  • AI-driven behavioral and pattern analysis

  • Enhanced fraud detection mechanisms

  • Verification based on multiple trusted sources rather than a single authority

Scalability and Accuracy Limitations of Existing Systems (could it be a correct risk for wallets?)

• Risk: Many current digital identity security systems lack the scalability and accuracy required to handle large volumes of users and increasingly sophisticated AI-based attacks. This limitation poses a significant challenge for identity wallets operating at national or cross-border scale.
• Solutions:

  • Deployment of scalable and resilient system architectures

  • Use of AI to automate threat detection and response

  • Continuous improvement of algorithmic accuracy under high-load conditions [2]

GenAI: here we mean using GenAI outside the wallet (AI-as-a-Service)

Implicit data leakage (even without “sending data”)

Even if you think you’re only sending:

• policies

• capability lists

• proof requests

…the structure, timing, and combinations of requests can leak:

• user attributes

• behavior patterns

• service usage profiles

This is called inference leakage. Over time, the AI provider can reconstruct who you are and what you’re doing — without seeing raw identity data.

Loss of user sovereignty

When AI runs outside the wallet:

• decision logic lives elsewhere

• prompt logic evolves without the user’s control

• model updates silently change behavior

Result: The wallet becomes a UI, not an agent.

This quietly breaks self-sovereign identity principles.

Policy manipulation & dark negotiation

External AI can:

• bias disclosure decisions

• “optimize” for platform goals

• subtly over-disclose to reduce friction

Even without malice:

• optimization objectives ≠ user interests

This is algorithmic coercion, not a bug.

Prompt and context retention

Most AI services:

• log prompts

• retain context

• reuse data for tuning or monitoring

Even anonymized logs can:

• correlate identities across services

• deanonymize users through linkage attacks

Once logged: You can’t revoke it.

Correlation across wallets and services

A single AI provider serving many wallets can:

• correlate request fingerprints

• identify the same user across devices or contexts

• create a shadow identity graph

This recreates centralized identity — without consent.

Regulatory and jurisdictional drift

External AI services may:

• run in foreign jurisdictions

• be subject to subpoenas

• fall under surveillance regimes

This creates:

• unclear data residency

• legal exposure for wallet providers

• compliance contradictions (GDPR, eIDAS, etc.)

Model hallucination becomes a security risk

Inside a wallet:

• AI mistakes are bounded
  Outside:
  

• hallucinated policy interpretations

• incorrect legal assumptions

• wrong proof selection

These can cause:

• over-disclosure

• invalid consent

• irreversible identity actions

Hallucination here is not UX noise — it’s identity damage.

1. A. Golda et al., "Privacy and Security Concerns in Generative AI: A Comprehensive Survey," in IEEE Access, vol. 12, pp. 48126-48144, 2024, doi: 10.1109/ACCESS.2024.3381611.  →  https://ieeexplore.ieee.org/document/10478883
2. 'THE EVOLUTION OF IDENTITY SECURITY IN THE AGE OF AI: CHALLENGES AND SOLUTIONS ', International Journal of Computer Engineering and Technology (IJCET)  Volume 16, Issue 1, Jan-Feb 2025, pp. 2305-2319, Article ID: IJCET_16_01_165 Available online at https://iaeme.com/Home/issue/IJCET?Volume=16&Issue=1 ISSN Print: 0976-6367; ISSN Online: 0976-6375; Journal ID: 5751-5249 Impact Factor (2025): 18.59 (Based on Google Scholar Citation) DOI: https://doi.org/10.34218/IJCET_16_01_165