RAG Permission Revocation: Test Cached Retrieval Leaks

Test how cached RAG retrieval can expose documents after access is revoked. Reproduce three failure cases in Python and add a query-time permission gu

AI SECURITY / RAG RELIABILITY

Short answer: A user-specific RAG cache key prevents one user from receiving another user's cached result, but it does not prove that the same user still has permission after a revocation. Before cached chunks enter model context, recheck current authorization and document existence, and invalidate or regenerate cached answers when their source permissions change.

Imagine a knowledge-base assistant serving a confidential planning document to an authorized employee. The employee later loses access. The source system updates its permissions, and a fresh retriever would now exclude the document—but the application has already cached the original search result. When the same employee asks again, does the old passage still reach the model?

It can, if the application trusts the cache as if it were an authorization decision. This article demonstrates that failure with a small, executed Python/SQLite experiment. It measures unauthorized document returns in a controlled lexical-retrieval model—not model-generated leaks, production exploit frequency, or the behavior of every hosted vector database.

Why RAG permissions become stale after ingestion

A retrieval-augmented generation (RAG) system usually prepares a document before anyone asks a question. It extracts text, splits it into chunks, computes embeddings or search terms, attaches metadata, and writes the result to an index. A later query retrieves candidates and places some of their text into an LLM's context.

The right to read a document is not permanent. An employee changes teams. A customer is removed from a shared workspace. A file moves into a restricted folder. A contractor is disabled. An administrator deletes a document. The source-of-truth permission can change while the index, retrieval cache, answer cache, conversation history, logs, or backups still contain derived material.

The problem is not that vector similarity is malicious. Similarity says something about relevance, not authority. Microsoft's Azure AI Search documentation describes document-level access patterns and the synchronization requirements behind permission enforcement. The OWASP AISVS RAG access-control research likewise treats scope and revocation freshness as things to test explicitly—not assumptions to inherit from a cache.

A reported OpenRAG issue describes one implementation's cross-user document exposure. That report is evidence about that issue, not proof that all RAG systems are affected or that caching caused it. The local experiment below isolates the cache logic so the claim stays narrow and reproducible.

A mental model: four places where permissions can become stale

  1. Source of truth: a document repository, identity provider, or authorization service decides who may read a document.
  2. Search index: chunks and permission metadata are written to lexical or vector search.
  3. Retrieval cache: the application saves search results or candidate chunk IDs for speed.
  4. Answer cache and conversation: a generated response, citation, summary, or previous conversation may already contain the document's content.

Updating the source file does not automatically purge every derived copy. The slowest relevant synchronization step determines the effective revocation window.

RAG architecture with source permissions, search index, cached chunks, and a current-authorization gate before model context
A cached retrieval result is only a candidate until current document permission is verified.
Core rule: Never put a retrieved passage into a model's context merely because it was safe when first cached.

A safe design establishes the caller's identity outside the model, preserves stable source-document IDs through chunking, applies documented authorization filters during retrieval, and revalidates cached candidates when policy may have changed. For sensitive content, an unavailable or untrusted authorization dependency should fail closed rather than silently treating stale access as permission.

Three cache strategies worth comparing

1. Shared query cache

A key such as the normalized query text maps to a stored list of chunks. If Alice warms the entry, Bob asking the same question can receive Alice's documents. This deliberately unsafe strategy shows why identity matters in the key.

2. User-scoped cache

The key contains the user's identity and the query. This prevents cross-user cache reuse in the experiment. It still does not solve same-user revocation: a cache item created at 09:00 is not proof that the user still has access at 09:05.

3. Guarded cache

The cache remains scoped, but every cached document is checked against the current trusted authorization table and current document store immediately before its body is returned. The guard complements search-time filtering: one limits retrieval, while the other catches stale cached candidates.

The reproducible experiment

Research question: Can user-scoped caching prevent stale document exposure after a user loses access, and does a current-authorization guard prevent the tested failures?

Hypothesis: Shared entries will fail cross-user and same-user revocation cases. User-scoped caching will fix cross-user reuse but not revocation or deletion. Rechecking current authorization and document existence will block all three cases in this controlled model.

The test uses SQLite FTS5 and two fictional Quartz roadmap documents: one private and one public. Alice initially accesses both; Bob accesses only the public document. The fresh query joins the search index against the grants table, but a cache hit can bypass that fresh check.

  • Cross-user: Alice warms the cache, then Bob requests the same query.
  • Revoked: Alice warms the cache, her private grant is removed, and she asks again.
  • Deleted: Alice warms the cache, then the private document is deleted from the source table and FTS index.

The metric is an unauthorized private-document return. A secondary check verifies that the public document remains available. The supplied script ran 30 fresh in-memory databases for each of 9 strategy/scenario combinations: 270 total trials. In the current execution environment, the measured versions were Python 3.12.3 and SQLite 3.45.1. No LLM, embedding model, external API, hosted vector database, or real confidential document was used.

Show the complete runnable Python experiment
#!/usr/bin/env python3
"""Controlled RAG retrieval-cache authorization experiment.
Standard library only; no LLM, vector model, external service, or real data.
"""
import sqlite3
import sys

QUERY = '"quartz roadmap"'
TRIALS = 30
PRIVATE = 'secret'


def setup():
    db = sqlite3.connect(':memory:')
    db.executescript('''
        CREATE TABLE documents (id TEXT PRIMARY KEY, body TEXT NOT NULL);
        CREATE VIRTUAL TABLE search_index USING fts5(doc_id UNINDEXED, body);
        CREATE TABLE grants (
            user_id TEXT NOT NULL,
            doc_id TEXT NOT NULL,
            PRIMARY KEY (user_id, doc_id)
        );
    ''')
    docs = [
        ('secret', 'Quartz roadmap: fictional confidential acquisition target.'),
        ('public', 'Quartz roadmap: published launch checklist for everyone.')
    ]
    db.executemany('INSERT INTO documents VALUES (?, ?)', docs)
    db.executemany('INSERT INTO search_index VALUES (?, ?)', docs)
    db.executemany('INSERT INTO grants VALUES (?, ?)', [
        ('alice', 'secret'), ('alice', 'public'), ('bob', 'public')
    ])
    db.commit()
    return db


def fresh_search(db, user, query):
    return db.execute('''
        SELECT s.doc_id, s.body FROM search_index AS s
        JOIN documents AS d ON d.id = s.doc_id
        JOIN grants AS g ON g.doc_id = s.doc_id AND g.user_id = ?
        WHERE search_index MATCH ?
        ORDER BY s.rank LIMIT 5
    ''', (user, query)).fetchall()


def still_authorized(db, user, doc_id):
    return db.execute('''
        SELECT 1 FROM documents AS d
        JOIN grants AS g ON g.doc_id = d.id
        WHERE d.id = ? AND g.user_id = ?
    ''', (doc_id, user)).fetchone() is not None


class Retriever:
    def __init__(self, db, strategy):
        self.db = db
        self.strategy = strategy
        self.cache = {}

    def retrieve(self, user, query):
        key = query if self.strategy == 'shared' else (user, query)
        if key not in self.cache:
            self.cache[key] = fresh_search(self.db, user, query)
        candidates = self.cache[key]
        if self.strategy == 'guarded':
            return [row for row in candidates
                    if still_authorized(self.db, user, row[0])]
        return list(candidates)


def one_trial(strategy, scenario):
    db = setup()
    retriever = Retriever(db, strategy)
    first = retriever.retrieve('alice', QUERY)
    assert PRIVATE in [row[0] for row in first]
    if scenario == 'cross-user':
        result = retriever.retrieve('bob', QUERY)
    elif scenario == 'revoked':
        db.execute('DELETE FROM grants WHERE user_id=? AND doc_id=?',
                   ('alice', PRIVATE))
        db.commit()
        result = retriever.retrieve('alice', QUERY)
    elif scenario == 'deleted':
        db.execute('DELETE FROM search_index WHERE doc_id=?', (PRIVATE,))
        db.execute('DELETE FROM documents WHERE id=?', (PRIVATE,))
        db.commit()
        result = retriever.retrieve('alice', QUERY)
    else:
        raise ValueError(scenario)
    leak = any(doc_id == PRIVATE for doc_id, _ in result)
    public_retained = any(doc_id == 'public' for doc_id, _ in result)
    db.close()
    return leak, public_retained


def main():
    print('Python:', sys.version.split()[0], '| SQLite:', sqlite3.sqlite_version)
    print('Trials per cell:', TRIALS)
    for strategy in ('shared', 'scoped', 'guarded'):
        for scenario in ('cross-user', 'revoked', 'deleted'):
            results = [one_trial(strategy, scenario) for _ in range(TRIALS)]
            leaks = sum(r[0] for r in results)
            public = sum(r[1] for r in results)
            print(strategy, scenario, f'{leaks}/{TRIALS}', f'{public}/{TRIALS}')
            expected = strategy == 'shared' or (strategy == 'scoped' and scenario != 'cross-user')
            assert leaks == (TRIALS if expected else 0)
            assert public == TRIALS
    print('All 9 scenario/strategy assertions passed.')


if __name__ == '__main__':
    main()

What the experiment measured

Cache strategyCross-userRevokedDeleted
Shared cache30/3030/3030/30
User-scoped0/3030/3030/30
Guarded + current check0/300/300/30

The public document remained available in all 270 trials and all nine assertions passed. The shared cache leaked across users. User-scoping stopped that leak but not stale access after revocation or deletion. The current-authorization guard excluded the private chunk in every tested scenario.

These are measured local results, not a universal zero-risk rate. The test constructs a deterministic case in which the guarded code should deny the document. Its value is as a regression test: it makes the failure mode visible and easy to automate.

Measured RAG cache test matrix showing 30 of 30 private-document returns for shared cache, stale failures for user-scoped cache, and zero of 30 for the guarded design
Measured local simulation: identity-scoped caching fixes cross-user reuse but not stale same-user access.

How to adapt the guard to a real RAG stack

  1. Establish identity outside the model. Use the authenticated principal, tenant, service principal, and task context supplied by trusted infrastructure. Never let the model, user-entered filter, or retrieved document choose which permissions apply.
  2. Preserve stable source IDs and ACL references. Each chunk needs a reference to its canonical document, not only an embedding or free-text title. Track the tenant, classification, and policy version needed for the check.
  3. Filter during retrieval where supported. Prefer documented tenant isolation, namespaces, native ACL enforcement, or trusted metadata filters. Pinecone's metadata-filtering documentation is one example of a service-specific pattern; verify the consistency semantics of the system you deploy.
  4. Treat cached chunks as candidates, not authorized results. Scope the cache by identity and retrieval configuration, then check that the source still exists and the caller still has access immediately before model-context insertion.
  5. Invalidate derived results. Permission changes should invalidate affected retrieval and final-answer cache entries. A cached answer cannot be made safe simply by deleting its citation; the prose may already contain sensitive facts. Regenerate it or decline to serve it.
  6. Test freshness explicitly. Define an acceptable interval between a source permission change and retrieval denial. Test group changes, inherited permissions, deleted accounts, document moves, index updates, and failed synchronization jobs.
Illustrative RAG revocation timeline showing source permission change, variable index synchronization, cache invalidation, and final authorization checks
Revocation is a sequence across systems; measure the longest stale window instead of assuming immediate consistency.

Common mistakes

  • Testing only two different users and missing same-user revocation.
  • Checking permissions only during ingestion.
  • Deleting the source file while ignoring indexes, caches, generated answers, backups, and logs.
  • Filtering after generation. If restricted context reached the model, hiding a citation afterward does not prove safety.
  • Trusting a cache key as authorization. A user-specific key says who warmed the entry; it does not prove current permission.
  • Using the model to decide access. The model consumes filtered evidence; it is not the permission authority.
  • Claiming immediate revocation without measuring synchronization delay.

Security, cost, and operational trade-offs

Per-request authorization checks can add latency and load, especially when a response contains many chunks. Batch document-ID checks where possible and use a trusted policy service designed for the freshness requirement. The local experiment did not benchmark latency and cannot quantify that cost.

A fail-closed response may temporarily provide fewer answers when an authorization dependency is unavailable. Treat policy denials, unavailable policy authorities, and stale data as different operational outcomes. Log the principal, source IDs, policy decision, and result status without recording unnecessary document contents.

Deletion also deserves separate handling. The deletion scenario confirms that this response path no longer returns the deleted document; it does not prove secure erasure from storage pages, backups, replicas, transcripts, or every derived artifact.

Limitations

This demonstration uses two fictional documents and one exact phrase. It has no concurrency, asynchronous index updates, distributed caches, group inheritance, cross-region replicas, tenant delegation, speculative generation, or token streaming. SQLite FTS5 provides deterministic lexical retrieval; it is not a substitute for semantic-search behavior.

The guard checks an immediately updated local table. A production identity provider or replicated policy service may itself be stale. The narrow supported claim is this: rechecking the current local authority stopped the specified stale cached results. It does not establish a complete production security architecture.

Release checklist for RAG permission changes

  • The authenticated principal and tenant come from a trusted boundary.
  • Every retrieved chunk has a stable source-document ID.
  • Search filters enforce the documented tenant and ACL model.
  • Cached chunks are reauthorized immediately before entering model context.
  • Cached final answers are invalidated or regenerated after source-permission changes.
  • Document deletion invalidates associated derived content as required.
  • Tests cover cross-user access, same-user revocation, deletion, and group changes.
  • Authorization-service outages have a documented fail-closed behavior.
  • A measurable revocation-freshness target and synchronization monitoring exist.

Frequently asked questions

Is adding the user ID to the RAG cache key enough?

No. It prevents one type of cross-user reuse, but an authorized user can later lose permission. A cached result can outlive the grant that originally justified it.

Does deleting a vector automatically invalidate cached answers?

Not necessarily. The search index and application-level result or answer cache can have separate lifecycles. Verify every relevant deletion and invalidation path.

Should access checks happen before or after vector search?

Apply documented authorization filters within retrieval where possible, then check cached candidates against a sufficiently current trusted authority before using them. Post-filtering alone can waste retrieval capacity or return too few useful chunks if top-k was applied first.

Do I need an LLM to test a RAG permission leak?

No. First verify which documents enter model context. Then add end-to-end tests for summaries, citations, streaming, and cached answers.

Conclusion

Permission-aware search is only one part of permission-aware RAG. If the application can reuse an old result, revocation and deletion become cache-consistency questions as well as identity questions.

The measured local test shows why user-scoped caching is necessary but insufficient: it blocked cross-user reuse, yet stale results remained available to the same user. A current-authorization guard prevented those specific failures without removing the public document. Treat this as a reproducible regression test, then extend it to the identity system, search backend, cache layers, and real revocation timeline of your own application.

Sources and further reading

  1. Microsoft Learn: Document-level access control in Azure AI Search.
  2. OWASP AISVS research: Access Controls on Memory and RAG Indices.
  3. OpenRAG issue #1336: Document access leaking. A reported implementation issue, not a universal claim.
  4. SQLite FTS5 Extension documentation.
  5. Pinecone: Metadata filtering documentation.
  6. Microsoft Learn: Security filters for trimming Azure AI Search results.
  7. OWASP AISVS: Memory, Embeddings and Vector Database Security.