RAG Permission Revocation: Test Cached Retrieval Leaks
AI SECURITY / RAG RELIABILITY
Short answer: A user-specific RAG cache key prevents one user from receiving another user's cached result, but it does not prove that the same user still has permission after a revocation. Before cached chunks enter model context, recheck current authorization and document existence, and invalidate or regenerate cached answers when their source permissions change.
Imagine a knowledge-base assistant serving a confidential planning document to an authorized employee. The employee later loses access. The source system updates its permissions, and a fresh retriever would now exclude the document—but the application has already cached the original search result. When the same employee asks again, does the old passage still reach the model?
It can, if the application trusts the cache as if it were an authorization decision. This article demonstrates that failure with a small, executed Python/SQLite experiment. It measures unauthorized document returns in a controlled lexical-retrieval model—not model-generated leaks, production exploit frequency, or the behavior of every hosted vector database.
Why RAG permissions become stale after ingestion
A retrieval-augmented generation (RAG) system usually prepares a document before anyone asks a question. It extracts text, splits it into chunks, computes embeddings or search terms, attaches metadata, and writes the result to an index. A later query retrieves candidates and places some of their text into an LLM's context.
The right to read a document is not permanent. An employee changes teams. A customer is removed from a shared workspace. A file moves into a restricted folder. A contractor is disabled. An administrator deletes a document. The source-of-truth permission can change while the index, retrieval cache, answer cache, conversation history, logs, or backups still contain derived material.
The problem is not that vector similarity is malicious. Similarity says something about relevance, not authority. Microsoft's Azure AI Search documentation describes document-level access patterns and the synchronization requirements behind permission enforcement. The OWASP AISVS RAG access-control research likewise treats scope and revocation freshness as things to test explicitly—not assumptions to inherit from a cache.
A reported OpenRAG issue describes one implementation's cross-user document exposure. That report is evidence about that issue, not proof that all RAG systems are affected or that caching caused it. The local experiment below isolates the cache logic so the claim stays narrow and reproducible.
A mental model: four places where permissions can become stale
- Source of truth: a document repository, identity provider, or authorization service decides who may read a document.
- Search index: chunks and permission metadata are written to lexical or vector search.
- Retrieval cache: the application saves search results or candidate chunk IDs for speed.
- Answer cache and conversation: a generated response, citation, summary, or previous conversation may already contain the document's content.
Updating the source file does not automatically purge every derived copy. The slowest relevant synchronization step determines the effective revocation window.
A safe design establishes the caller's identity outside the model, preserves stable source-document IDs through chunking, applies documented authorization filters during retrieval, and revalidates cached candidates when policy may have changed. For sensitive content, an unavailable or untrusted authorization dependency should fail closed rather than silently treating stale access as permission.
Three cache strategies worth comparing
1. Shared query cache
A key such as the normalized query text maps to a stored list of chunks. If Alice warms the entry, Bob asking the same question can receive Alice's documents. This deliberately unsafe strategy shows why identity matters in the key.
2. User-scoped cache
The key contains the user's identity and the query. This prevents cross-user cache reuse in the experiment. It still does not solve same-user revocation: a cache item created at 09:00 is not proof that the user still has access at 09:05.
3. Guarded cache
The cache remains scoped, but every cached document is checked against the current trusted authorization table and current document store immediately before its body is returned. The guard complements search-time filtering: one limits retrieval, while the other catches stale cached candidates.
The reproducible experiment
Research question: Can user-scoped caching prevent stale document exposure after a user loses access, and does a current-authorization guard prevent the tested failures?
Hypothesis: Shared entries will fail cross-user and same-user revocation cases. User-scoped caching will fix cross-user reuse but not revocation or deletion. Rechecking current authorization and document existence will block all three cases in this controlled model.
The test uses SQLite FTS5 and two fictional Quartz roadmap documents: one private and one public. Alice initially accesses both; Bob accesses only the public document. The fresh query joins the search index against the grants table, but a cache hit can bypass that fresh check.
- Cross-user: Alice warms the cache, then Bob requests the same query.
- Revoked: Alice warms the cache, her private grant is removed, and she asks again.
- Deleted: Alice warms the cache, then the private document is deleted from the source table and FTS index.
The metric is an unauthorized private-document return. A secondary check verifies that the public document remains available. The supplied script ran 30 fresh in-memory databases for each of 9 strategy/scenario combinations: 270 total trials. In the current execution environment, the measured versions were Python 3.12.3 and SQLite 3.45.1. No LLM, embedding model, external API, hosted vector database, or real confidential document was used.
Show the complete runnable Python experiment
#!/usr/bin/env python3
"""Controlled RAG retrieval-cache authorization experiment.
Standard library only; no LLM, vector model, external service, or real data.
"""
import sqlite3
import sys
QUERY = '"quartz roadmap"'
TRIALS = 30
PRIVATE = 'secret'
def setup():
db = sqlite3.connect(':memory:')
db.executescript('''
CREATE TABLE documents (id TEXT PRIMARY KEY, body TEXT NOT NULL);
CREATE VIRTUAL TABLE search_index USING fts5(doc_id UNINDEXED, body);
CREATE TABLE grants (
user_id TEXT NOT NULL,
doc_id TEXT NOT NULL,
PRIMARY KEY (user_id, doc_id)
);
''')
docs = [
('secret', 'Quartz roadmap: fictional confidential acquisition target.'),
('public', 'Quartz roadmap: published launch checklist for everyone.')
]
db.executemany('INSERT INTO documents VALUES (?, ?)', docs)
db.executemany('INSERT INTO search_index VALUES (?, ?)', docs)
db.executemany('INSERT INTO grants VALUES (?, ?)', [
('alice', 'secret'), ('alice', 'public'), ('bob', 'public')
])
db.commit()
return db
def fresh_search(db, user, query):
return db.execute('''
SELECT s.doc_id, s.body FROM search_index AS s
JOIN documents AS d ON d.id = s.doc_id
JOIN grants AS g ON g.doc_id = s.doc_id AND g.user_id = ?
WHERE search_index MATCH ?
ORDER BY s.rank LIMIT 5
''', (user, query)).fetchall()
def still_authorized(db, user, doc_id):
return db.execute('''
SELECT 1 FROM documents AS d
JOIN grants AS g ON g.doc_id = d.id
WHERE d.id = ? AND g.user_id = ?
''', (doc_id, user)).fetchone() is not None
class Retriever:
def __init__(self, db, strategy):
self.db = db
self.strategy = strategy
self.cache = {}
def retrieve(self, user, query):
key = query if self.strategy == 'shared' else (user, query)
if key not in self.cache:
self.cache[key] = fresh_search(self.db, user, query)
candidates = self.cache[key]
if self.strategy == 'guarded':
return [row for row in candidates
if still_authorized(self.db, user, row[0])]
return list(candidates)
def one_trial(strategy, scenario):
db = setup()
retriever = Retriever(db, strategy)
first = retriever.retrieve('alice', QUERY)
assert PRIVATE in [row[0] for row in first]
if scenario == 'cross-user':
result = retriever.retrieve('bob', QUERY)
elif scenario == 'revoked':
db.execute('DELETE FROM grants WHERE user_id=? AND doc_id=?',
('alice', PRIVATE))
db.commit()
result = retriever.retrieve('alice', QUERY)
elif scenario == 'deleted':
db.execute('DELETE FROM search_index WHERE doc_id=?', (PRIVATE,))
db.execute('DELETE FROM documents WHERE id=?', (PRIVATE,))
db.commit()
result = retriever.retrieve('alice', QUERY)
else:
raise ValueError(scenario)
leak = any(doc_id == PRIVATE for doc_id, _ in result)
public_retained = any(doc_id == 'public' for doc_id, _ in result)
db.close()
return leak, public_retained
def main():
print('Python:', sys.version.split()[0], '| SQLite:', sqlite3.sqlite_version)
print('Trials per cell:', TRIALS)
for strategy in ('shared', 'scoped', 'guarded'):
for scenario in ('cross-user', 'revoked', 'deleted'):
results = [one_trial(strategy, scenario) for _ in range(TRIALS)]
leaks = sum(r[0] for r in results)
public = sum(r[1] for r in results)
print(strategy, scenario, f'{leaks}/{TRIALS}', f'{public}/{TRIALS}')
expected = strategy == 'shared' or (strategy == 'scoped' and scenario != 'cross-user')
assert leaks == (TRIALS if expected else 0)
assert public == TRIALS
print('All 9 scenario/strategy assertions passed.')
if __name__ == '__main__':
main()
What the experiment measured
| Cache strategy | Cross-user | Revoked | Deleted |
|---|---|---|---|
| Shared cache | 30/30 | 30/30 | 30/30 |
| User-scoped | 0/30 | 30/30 | 30/30 |
| Guarded + current check | 0/30 | 0/30 | 0/30 |
The public document remained available in all 270 trials and all nine assertions passed. The shared cache leaked across users. User-scoping stopped that leak but not stale access after revocation or deletion. The current-authorization guard excluded the private chunk in every tested scenario.
These are measured local results, not a universal zero-risk rate. The test constructs a deterministic case in which the guarded code should deny the document. Its value is as a regression test: it makes the failure mode visible and easy to automate.
How to adapt the guard to a real RAG stack
- Establish identity outside the model. Use the authenticated principal, tenant, service principal, and task context supplied by trusted infrastructure. Never let the model, user-entered filter, or retrieved document choose which permissions apply.
- Preserve stable source IDs and ACL references. Each chunk needs a reference to its canonical document, not only an embedding or free-text title. Track the tenant, classification, and policy version needed for the check.
- Filter during retrieval where supported. Prefer documented tenant isolation, namespaces, native ACL enforcement, or trusted metadata filters. Pinecone's metadata-filtering documentation is one example of a service-specific pattern; verify the consistency semantics of the system you deploy.
- Treat cached chunks as candidates, not authorized results. Scope the cache by identity and retrieval configuration, then check that the source still exists and the caller still has access immediately before model-context insertion.
- Invalidate derived results. Permission changes should invalidate affected retrieval and final-answer cache entries. A cached answer cannot be made safe simply by deleting its citation; the prose may already contain sensitive facts. Regenerate it or decline to serve it.
- Test freshness explicitly. Define an acceptable interval between a source permission change and retrieval denial. Test group changes, inherited permissions, deleted accounts, document moves, index updates, and failed synchronization jobs.
Common mistakes
- Testing only two different users and missing same-user revocation.
- Checking permissions only during ingestion.
- Deleting the source file while ignoring indexes, caches, generated answers, backups, and logs.
- Filtering after generation. If restricted context reached the model, hiding a citation afterward does not prove safety.
- Trusting a cache key as authorization. A user-specific key says who warmed the entry; it does not prove current permission.
- Using the model to decide access. The model consumes filtered evidence; it is not the permission authority.
- Claiming immediate revocation without measuring synchronization delay.
Security, cost, and operational trade-offs
Per-request authorization checks can add latency and load, especially when a response contains many chunks. Batch document-ID checks where possible and use a trusted policy service designed for the freshness requirement. The local experiment did not benchmark latency and cannot quantify that cost.
A fail-closed response may temporarily provide fewer answers when an authorization dependency is unavailable. Treat policy denials, unavailable policy authorities, and stale data as different operational outcomes. Log the principal, source IDs, policy decision, and result status without recording unnecessary document contents.
Deletion also deserves separate handling. The deletion scenario confirms that this response path no longer returns the deleted document; it does not prove secure erasure from storage pages, backups, replicas, transcripts, or every derived artifact.
Limitations
This demonstration uses two fictional documents and one exact phrase. It has no concurrency, asynchronous index updates, distributed caches, group inheritance, cross-region replicas, tenant delegation, speculative generation, or token streaming. SQLite FTS5 provides deterministic lexical retrieval; it is not a substitute for semantic-search behavior.
The guard checks an immediately updated local table. A production identity provider or replicated policy service may itself be stale. The narrow supported claim is this: rechecking the current local authority stopped the specified stale cached results. It does not establish a complete production security architecture.
Release checklist for RAG permission changes
- The authenticated principal and tenant come from a trusted boundary.
- Every retrieved chunk has a stable source-document ID.
- Search filters enforce the documented tenant and ACL model.
- Cached chunks are reauthorized immediately before entering model context.
- Cached final answers are invalidated or regenerated after source-permission changes.
- Document deletion invalidates associated derived content as required.
- Tests cover cross-user access, same-user revocation, deletion, and group changes.
- Authorization-service outages have a documented fail-closed behavior.
- A measurable revocation-freshness target and synchronization monitoring exist.
Frequently asked questions
Is adding the user ID to the RAG cache key enough?
No. It prevents one type of cross-user reuse, but an authorized user can later lose permission. A cached result can outlive the grant that originally justified it.
Does deleting a vector automatically invalidate cached answers?
Not necessarily. The search index and application-level result or answer cache can have separate lifecycles. Verify every relevant deletion and invalidation path.
Should access checks happen before or after vector search?
Apply documented authorization filters within retrieval where possible, then check cached candidates against a sufficiently current trusted authority before using them. Post-filtering alone can waste retrieval capacity or return too few useful chunks if top-k was applied first.
Do I need an LLM to test a RAG permission leak?
No. First verify which documents enter model context. Then add end-to-end tests for summaries, citations, streaming, and cached answers.
Conclusion
Permission-aware search is only one part of permission-aware RAG. If the application can reuse an old result, revocation and deletion become cache-consistency questions as well as identity questions.
The measured local test shows why user-scoped caching is necessary but insufficient: it blocked cross-user reuse, yet stale results remained available to the same user. A current-authorization guard prevented those specific failures without removing the public document. Treat this as a reproducible regression test, then extend it to the identity system, search backend, cache layers, and real revocation timeline of your own application.
Sources and further reading
- Microsoft Learn: Document-level access control in Azure AI Search.
- OWASP AISVS research: Access Controls on Memory and RAG Indices.
- OpenRAG issue #1336: Document access leaking. A reported implementation issue, not a universal claim.
- SQLite FTS5 Extension documentation.
- Pinecone: Metadata filtering documentation.
- Microsoft Learn: Security filters for trimming Azure AI Search results.
- OWASP AISVS: Memory, Embeddings and Vector Database Security.
Join the conversation