DRAFT 

Actors: Holder- Issuer- Verifier- Wallet(Agent)- Governing Authority

1. General Risks

1.1. Deepfake and Identity Spoofing

  • Risk: Generative AI can create highly realistic fake audio, video, or images, enabling attackers to bypass biometric authentication or impersonate legitimate users. It would be a significant challenge for online(video) authentication. The rapid advancement of AI-driven deepfake technologies, can undermine biometric authentication mechanisms such as facial recognition.
  • use cases:

    • Using video authentication to log in to the wallet could create a vulnerability.

    • Facial authentication is required to ensure that the verifiable credential (VC) cannot be misused by someone else.

    • Biometric authentication is necessary whenever an action must be performed specifically by the VC owner and not by anyone else who may have access to the wallet. e.x. to grant power of attorney

       
  • Solution: Implement deepfake detection tools, multi-factor authentication (MFA), and robust identity verification processes to reduce reliance on single biometric factors. Also update continuously the forgery detection algorithms.
  • Complexity of Solution
  • Obstacles
  • Affected Group: End user/ 
  • Active Actor of mitigation

1.2. Prompt Injection and Policy Manipulation 

  • Risk: Since policies are enforced through cryptographic protocols rather than natural language interpretation, making prompt injection harder. But if the wallet uses AI-driven assistants or automated decision-making (e.g., for verifying credentials or guiding users), attackers can craft malicious prompts to manipulate the AI’s logic.  

    Even if the wallet itself is secure, any connected AI-based helpdesk or verification service could be exploited via prompt injection.

  • Solution: Apply prompt hardening techniques, context isolation, and strict input validation. Use allowlists/denylists and sandbox testing for untrusted inputs.

1.3. Data Leakage and Membership Inference*

  • Risk: If identity-related data is used to train AI models, attackers may infer sensitive attributes or reconstruct original data through model inversion or membership inference attacks. 

    If the wallet uses AI services (e.g., for fraud detection, identity verification, or UX personalization), sensitive identity data might be exposed during model training or inference. If wallet operations involve external AI APIs, data could leak through logs or model updates. Even if raw data isn’t shared, patterns in queries or metadata could allow attackers to infer user attributes. e.g your AI tools on your mobile phone have access to your ID wallet!!


    If AI is integrated for convenience (e.g., chatbots or automated KYC), and those models access identity data, leakage and inference risks reappear. Metadata (timestamps, transaction patterns) can still be exploited for inference even if credentials are protected.

  • Solution:
  • Enforce data minimization→ wallet selective dislousure is already there to solve ths problem but by using AI data can reveal werden. (is it really a solution?)
  • segregate sensitive datasets
  • adopt privacy-preserving training methods* (e.g., differential privacy), and secure the entire data lifecycle.

1.4. Misinformation and Social Engineering

  • Risk: AI-generated content can be used to create convincing phishing messages or fake instructions, tricking users into revealing recovery phrases or credentials for identity wallets. 

    Digital identity wallets rely on users to manage credentials and recovery phrases. Attackers can still use AI-generated phishing emails, fake instructions, or fraudulent websites to trick users into revealing sensitive information. Social engineering campaigns can impersonate official wallet support or government identity services, convincing users to share credentials or approve malicious transactions.

  • Solution: Deploy misinformation detection systems, educate users on security best practices, and implement strict content moderation and auditing for AI outputs. [1]

1.5. Synthetic Identity Fraud (looks repeated)

  • Risk: The emergence of synthetic identities created by combining real and fabricated data, which can bypass traditional identity verification systems. If such identities are stored or validated within digital identity wallets, they can compromise the overall trust model.
  • Solutions:

    • AI-driven behavioral and pattern analysis

    • Enhanced fraud detection mechanisms

    • Verification based on multiple trusted sources rather than a single authority

1.6. Scalability and Accuracy Limitations of Existing Systems (question) (could it be a correct risk for wallets?)

  • Risk: Many current digital identity security systems lack the scalability and accuracy required to handle large volumes of users and increasingly sophisticated AI-based attacks. This limitation poses a significant challenge for identity wallets operating at national or cross-border scale.
  • Solutions:

    • Deployment of scalable and resilient system architectures

    • Use of AI to automate threat detection and response

    • Continuous improvement of algorithmic accuracy under high-load conditions [2]


2. Risks as AI-as-a-Service

GenAI: here we mean using GenAI outside the wallet (AI-as-a-Service)

2.1.1. Implicit data leakage (even without “sending data”)

Even if you think you’re only sending:

  • policies

  • capability lists

  • proof requests

…the structure, timing, and combinations of requests can leak:

  • user attributes

  • behavior patterns

  • service usage profiles

This is called inference leakage. Over time, the AI provider can reconstruct who you are and what you’re doing — without seeing raw identity data.

2.1.2. Loss of user sovereignty

When AI runs outside the wallet:

  • decision logic lives elsewhere

  • prompt logic evolves without the user’s control

  • model updates silently change behavior

Result: The wallet becomes a UI, not an agent.

This quietly breaks self-sovereign identity principles.

2.1.3. Policy manipulation & dark negotiation

External AI can:

  • bias disclosure decisions

  • “optimize” for platform goals

  • subtly over-disclose to reduce friction

Even without malice:

  • optimization objectives ≠ user interests

This is algorithmic coercion, not a bug. + Explainable Generative AI*

2.1.4. Prompt and context retention

Most AI services:

  • log prompts

  • retain context

  • reuse data for tuning or monitoring

Even anonymized logs can:

  • correlate identities across services

  • deanonymize users through linkage attacks

Once logged: You can’t revoke it.

2.1.5. Correlation across wallets and services

A single AI provider serving many wallets can:

  • correlate request fingerprints

  • identify the same user across devices or contexts

  • create a shadow identity graph

This recreates centralized identity — without consent.

In addition, if the AIaaS provider experiences an outage, millions of users could be locked out of essential services simultaneously.

2.1.6. Regulatory and jurisdictional drift (question) 

External AI services may:

  • run in foreign jurisdictions

  • be subject to subpoenas

  • fall under surveillance regimes

This creates:

  • unclear data residency

  • legal exposure for wallet providers

  • compliance contradictions (GDPR, eIDAS, etc.)

Data may be stored in servers under foreign laws, complicating compliance with national regulations like GDPR and creating legal uncertainty. (question) 

2.1.7. Model hallucination becomes a security risk

Inside a wallet:

  • AI mistakes are bounded
    Outside:

  • hallucinated policy interpretations

  • incorrect legal assumptions

  • wrong proof selection

These can cause:

  • over-disclosure

  • invalid consent

  • irreversible identity actions

Hallucination here is not UX noise — it’s identity damage.


2.1.8. "Black Box" Opacity

Complex AI models are not transparent. Users can be denied access (e.g., false non-match) without a clear, explainable reason or recourse.

2.1.9. Algorithmic Bias & Discrimination

AI models can inherit biases, leading to unfair denials of access for specific demographic groups. The system is also vulnerable to adversarial attacks designed to fool it.

*

Membership Inference (or Membership Inference Attack, often shortened to MIA) is a type of privacy attack against machine‑learning models. In this attack, someone (an attacker) tries to figure out whether a specific data sample was part of the model’s training data. In simple words: The attacker wants to know “Was this person’s data used to train the model?” If the attacker can guess this correctly, they can learn private information about individuals.

Privacy-preserving training methods →  Privacy‑preserving training methods are techniques used in machine learning and AI to ensure that sensitive information from the training data cannot be reconstructed, identified, or leaked, while still allowing the model to learn useful patterns.like :

  • Homomorphic Encryption: Allows computations directly on encrypted data.
  • Secure Multi-Party Computation - MPC/SMPC: Multiple parties collaborate to train a model without seeing each other's data.
  • Differential Privacy - DP: Adds mathematically controlled noise during training so the model cannot reveal information about any specific individual.
  • Federated Learning (FL): The model is trained across many devices or servers, and raw data never leaves the device. Each device trains locally and only model updates (not data) are sent to a central server, updates are aggregated securely
  • Trusted Execution Environments (TEE): Training happens in a secure, hardware‑isolated environment. like Microsoft Azure confidential computing
  • Synthetic Data Generation: Instead of using real private data, a model (e.g., a generative model) produces synthetic data that mimics the statistical properties of the original dataset.

Explainable Generative AI → Explainable Generative AI (XGAI) refers to methods and systems that make generative AI models (such as GPT, diffusion models, image generators, code‑gen models, etc.) understandable, transparent, and interpretable to humans.


  1. A. Golda et al., "Privacy and Security Concerns in Generative AI: A Comprehensive Survey," in IEEE Access, vol. 12, pp. 48126-48144, 2024, doi: 10.1109/ACCESS.2024.3381611.  →  https://ieeexplore.ieee.org/document/10478883
  2. 'THE EVOLUTION OF IDENTITY SECURITY IN THE AGE OF AI: CHALLENGES AND SOLUTIONS ', International Journal of Computer Engineering and Technology (IJCET)  Volume 16, Issue 1, Jan-Feb 2025, pp. 2305-2319, Article ID: IJCET_16_01_165 Available online at https://iaeme.com/Home/issue/IJCET?Volume=16&Issue=1 ISSN Print: 0976-6367; ISSN Online: 0976-6375; Journal ID: 5751-5249 Impact Factor (2025): 18.59 (Based on Google Scholar Citation) DOI: https://doi.org/10.34218/IJCET_16_01_165 


  • No labels