Structuring a Team Prompt Library

Why you need a prompt library

Early in a project, you might have a handful of prompts. You keep them in a code comment, a Google Doc, or scattered across notebooks. This works fine for one person or a small team.

But as your team grows, prompts become a shared asset:

• Different people want to reuse the same prompt. Instead of asking "has anyone written a summarization prompt?", they should be able to search your library.
• Prompts decay. A prompt that worked well last quarter might be outdated now (model changed, use case refined, or it's been proven inferior via A/B testing). Without a review process, old prompts stay in circulation.
• Reimplementation waste. Two people write similar extraction prompts independently, unaware of each other's work. Wasted effort, inconsistent approach.
• Audit and compliance. If a prompt is causing problems (hallucination, bias, data leakage), you need to know which systems use it and update them.

A structured prompt library solves these problems. It's a shared, discoverable, versioned, and maintained collection of prompts.

Designing a folder structure

Organize your library by use case or product domain, not by model or technique. This is how a developer thinks: "I need a summarization prompt" not "I need a GPT-4 few-shot prompt."

Example structure

prompts/
  ├── summarization/
  │   ├── v1/
  │   │   ├── abstractive_summary.md
  │   │   ├── bullet_point_summary.md
  │   │   └── tldr.md
  │   └── v2/
  │       ├── abstractive_summary.md
  │       └── ...
  ├── extraction/
  │   ├── v1/
  │   │   ├── entity_extraction.md
  │   │   ├── structured_data.md
  │   │   └── key_value_extraction.md
  │   └── v2/
  │       └── ...
  ├── classification/
  │   ├── sentiment_analysis.md
  │   ├── intent_detection.md
  │   └── content_moderation.md
  ├── generation/
  │   ├── email_drafting.md
  │   ├── product_description.md
  │   └── ...
  └── customer_service/
      ├── issue_resolution.md
      ├── faq_bot.md
      └── ...

Why organize by use case?

• Developers think in terms of tasks, not techniques.
• Easier to discover related prompts (all summarization prompts in one place).
• Simpler to deprecate old versions (move v1 to v1-deprecated, upgrade to v2).

Versioning at the folder level lets you maintain backward compatibility. Existing systems using v1 continue to work while new projects can use v2.

Required metadata per prompt

Each prompt file should include structured metadata. Use YAML frontmatter (same as the lesson files):

---
title: "Extractive Summarization"
slug: extractive-summary-v1
use_case: summarization
owner: [email protected]
status: active # or deprecated, testing, archived
created_date: 2026-01-15
last_reviewed: 2026-07-01
next_review_due: 2026-10-01

# Performance baselines
baseline_metric: coherence_score
baseline_value: 8.2/10
test_set: internal_docs_q1_2026

# Dependencies
model_compatible:
  - claude-3-sonnet
  - gpt-4
context_limit: 12000 tokens # approximate, depends on input
estimated_cost_per_use: $0.02

# Versioning and change log
version: 1.0
previous_version: null
breaking_changes: null
changelog: "Initial version"

# Guardrails
guardrails:
  - "Do not include personally identifiable information"
  - "Flag unverifiable claims"

tags:
  - extraction
  - structured-output
  - low-latency

related_prompts:
  - abstractive-summary-v1
  - extractive-summary-v2
---

## Prompt

[The actual prompt text here]

## Usage Examples

```python
client = anthropic.Anthropic()
message = client.messages.create(
    model="claude-3-sonnet-20240229",
    max_tokens=1024,
    messages=[
        {"role": "user", "content": prompt}
    ]
)

Known Limitations

• Works best with English text
• May struggle with scientific abstracts
• Hallucination rate ~5% on diverse domains

Evaluation Notes

Tested on 500 news articles, 50 academic papers, 100 product reviews. Coherence score averaged 8.2. Tested 2026-07-01 by [email protected].

### Metadata breakdown

| Field | Purpose |
|-------|---------|
| `title` | Human-readable prompt name |
| `slug` | Machine-readable ID (used in code to import) |
| `use_case` | Broad category (summarization, extraction, etc.) |
| `owner` | Who maintains this prompt |
| `status` | active, deprecated, testing, archived |
| `last_reviewed` / `next_review_due` | When was it last validated? When should it be checked again? |
| `baseline_metric` / `baseline_value` | What metric proves it works? (e.g., accuracy, latency, coherence) |
| `model_compatible` | Which models is this tested on? |
| `version` | Semantic versioning (1.0, 1.1, 2.0) |
| `breaking_changes` | What changed from the previous version? Did behavior change? |
| `guardrails` | What constraints apply? (safety, compliance, domain rules) |
| `tags` | Searchable keywords |
| `related_prompts` | Links to similar prompts |

---

## Making the library discoverable

Your developers need to find and use prompts. Options:

### Option 1: Git repository + documentation

Store prompts in a Git repo with a README. Developers clone, browse, and copy-paste.

**Pros:** Simple, version-controlled, free.
**Cons:** Manual discovery, copy-paste errors, hard to track usage.

### Option 2: Web-based UI (Prompt Registry)

Build or use a tool (LangChain Prompt Hub, Promptable, LlamaIndex Prompt Manager) that shows:
- Searchable list of all prompts
- View metadata and examples
- Rate and comment on prompts
- Version history

**Pros:** Centralized, searchable, tracked usage.
**Cons:** Extra tooling to maintain.

### Option 3: Internal SDK

Wrap your Git repo in a Python/Node package. Developers import prompts programmatically:

```python
from prompts_library import get_prompt

prompt = get_prompt("summarization", "extractive_summary_v1")
# Returns the prompt text and metadata

Pros: Enforces consistent usage, easy to update all references. Cons: Requires SDK maintenance.

A deprecation process

Prompts become outdated. Maybe the model improved, or you found a better approach in an A/B test. You need a process to retire old prompts without breaking systems that depend on them.

Deprecation workflow

1. Mark as deprecated: Change status to "deprecated" and set a sunset date.

status: deprecated
sunset_date: 2026-12-31
deprecation_reason: "Replaced by extractive_summary_v2, which scores 9.1 vs. 8.2"
migration_guide: "See extractive_summary_v2 for drop-in replacement"

2. Notify users: Automated email to all teams using the deprecated prompt.

Subject: Prompt deprecated: summarization/extractive_summary_v1

The prompt "Extractive Summarization v1" is being deprecated on 2026-12-31.

Reason: Version 2.0 improved coherence score from 8.2 to 9.1.

Migration: See extractive_summary_v2. It's a drop-in replacement.

Questions? Contact [email protected].

3. Track migration: Monitor which systems are still using the deprecated prompt.

4. Archive: After sunset date, move the prompt to an "archived" folder. Keep it for reference, but don't show in the active library.

Worked example: building a prompt library for a customer service team

Scenario: Your customer service team uses three prompts:

• Intent detection (for classifying customer issues)
• Response drafting (for suggesting replies)
• Escalation criteria (for flagging serious issues)

Step 1: Organize by use case

customer_service/
├── intent_detection/
│   ├── v1/
│   │   └── classifier.md
│   └── v2/
│       └── classifier.md
├── response_drafting/
│   └── v1/
│       └── email_responder.md
└── escalation_criteria/
    └── v1/
        └── escalation_detector.md

Step 2: Add metadata to each

For customer_service/intent_detection/v2/classifier.md:

---
title: "Intent Detection - Improved v2"
slug: intent-detection-v2
use_case: customer_service
owner: [email protected]
status: active
baseline_metric: accuracy
baseline_value: "92% (tested on 500 historical tickets)"
version: 2.0
previous_version: intent-detection-v1
breaking_changes: "Returns intent with confidence score (0-1). v1 returned intent only."
changelog: "Added confidence scoring and new intent type: 'product-defect'"
tags:
  - intent-detection
  - classification
  - customer-service
related_prompts:
  - intent-detection-v1
  - escalation-criteria-v1
---

## Prompt

You are a customer service triage system. Classify the incoming customer message into one of these intents:
...

Step 3: Set up a review schedule

Add to a shared calendar: quarterly review of all prompts.

Schedule for Q4 2026: Bob reviews intent_detection. Carol reviews response_drafting. Diana reviews escalation_criteria.

Step 4: Deprecate v1 when v2 is proven

After 2 weeks of A/B testing shows v2 is better:

---
title: "Intent Detection - v1 (Deprecated)"
slug: intent-detection-v1
status: deprecated
sunset_date: 2026-10-31
deprecation_reason: "v2 achieves 92% accuracy vs. 88%. v2 is drop-in replacement."
migration_guide: "Change slug from 'intent-detection-v1' to 'intent-detection-v2' in code."
---

Notify the team. Update documentation. After 2026-10-31, move v1 to archive.

Governance and ownership

Clear ownership

Each prompt should have an owner (typically the person who created it or the team that uses it). The owner is responsible for:

• Keeping documentation updated
• Responding to questions/issues
• Scheduling reviews
• Proposing deprecation when needed

Review cadence

Quarterly or semi-annual reviews. For each prompt:

• Is it still being used?
• Has the model changed? Does the prompt still work?
• Are there known issues or complaints?
• Should it be deprecated or updated?

Document findings in the metadata:

last_reviewed: 2026-07-01
review_notes: "Used by 3 teams. A/B test shows 1% improvement from v1. No issues."
next_review_due: 2026-10-01

Access control

For sensitive domains (healthcare, finance, legal), control who can edit prompts:

• Only the owner or a security team can modify.
• Changes require code review or approval.
• Changelog is mandatory.

Common mistakes

Mistake 1: No metadata, just prompt text

Bad:

# Summarization Prompt

Please summarize the following text...

Nobody knows when it was last tested, which model it's for, or who to ask about it.

Better: Include full metadata. Make it searchable and maintainable.

Mistake 2: Allowing prompts to drift out of sync with reality

Bad: A prompt says it works on "GPT-3.5" but it was never tested on GPT-4. Someone uses it on GPT-4, gets poor results, and blames the prompt.

Better: Update metadata after testing. Mark model_compatible and last_reviewed date. Enforce review cycles.

Mistake 3: No deprecation process

Bad: You improve a prompt and create v2. But v1 stays in the library labeled "active," so new developers don't know which to use.

Better: Mark v1 as deprecated, provide a migration guide, and set a sunset date. Eventually archive it.

Mistake 4: Over-versioning

Bad: You have v1, v1.1, v1.1.1, v1.1.1-hotfix, v2-beta, v2, v2-testing, v3-draft. Too many versions. Confusion.

Better: Use semantic versioning: v1, v2, v3. If you need to patch, use v1.1, v1.2 (not v1-hotfix). Keep it simple.

Mistake 5: Not tracking usage

Bad: You deprecate a prompt without knowing which systems use it. One system breaks because it still references the old version.

Better: Log every prompt usage. When deprecating, query the log to find all dependent systems. Notify them before the sunset date.

Integration with CI/CD

When a prompt is updated in the library, you might want to:

1. Re-evaluate automatically: Run the prompt against your benchmark test set. If performance degrades below baseline, block the merge.

2. Notify downstream systems: If a system imports from the library, notify it that a new version is available. Let teams opt into upgrades.

3. Enforce review: Prompt changes require approval from the owner or a designated reviewer before merging.

Example GitHub Actions workflow:

name: Validate Prompt Changes
on: [pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Check prompt baseline
        run: python scripts/evaluate_prompts.py
      - name: Verify metadata
        run: python scripts/validate_metadata.py
      - name: Notify systems
        if: success()
        run: python scripts/notify_consumers.py

Search and discovery strategies

How should developers find prompts? Options have trade-offs:

| Discovery Method | Pros | Cons | Best For | |------------------|------|------|----------| | Git + README | Simple, version-controlled, free | Manual browsing, copy-paste errors, low discoverability | Small teams (<10 engineers), low velocity | | Web UI (Prompt Registry) | Searchable, rateable, centralized, usage tracked | Additional tooling, hosting costs, maintenance | Medium teams (10-100), mission-critical prompts | | Python/Node SDK | Enforces consistency, easy updates, programmatic access | Requires maintenance, version compatibility issues | Any team using code-based pipelines | | Slack bot | Quick access, integrated workflow, notification-friendly | Limited search capability, context loss | Teams using Slack heavily; brainstorming/exploration | | Hybrid (Git + SDK) | Flexibility, version control + programmatic access | More maintenance | Large teams (100+), mixed workflows |

Real case study: Scaling a prompt library from 5 to 50+ prompts

A data science team at a mid-size company grew their prompt collection:

Month 1-2 (5 prompts):

• Stored in a shared Google Doc
• "Works great for now" mentality
• No versioning, no owner tracking
• Problem: Developers couldn't find prompts; reimplementation happened

Month 3-4 (15 prompts):

• Moved to a Git repo with folder structure
• Added basic metadata (title, description, owner)
• Problem: No clear versioning; prompts were edited in-place; unclear which version to use

Month 5-6 (30 prompts, team growing to 12 engineers):

• Implemented semantic versioning (v1.0, v1.1, v2.0)
• Added full metadata template (YAML frontmatter)
• Set up quarterly review process
• Problem: No discoverability tool; developers had to browse GitHub; high onboarding friction

Month 7-9 (50+ prompts, team now 20 engineers):

• Built a simple web UI (Prompt Registry) with search
• Integrated usage analytics (track which systems use which prompts)
• Automated CI/CD checks (metadata validation, baseline re-evaluation)
• Implemented clear deprecation workflow
• Added Slack notifications when prompts are updated
• Result: Discoverability improved, deprecation became manageable, team could identify impact of changes

Scaling insights (illustrative estimate):

• Time spent on prompt discovery: ~2 hours/week at month 3 → ~15 minutes/week by month 9 (7-8x improvement)
• Prompt reuse rate: ~30% at month 3 → ~65% by month 9 (indicating better discoverability and trust)
• Time to onboard a new engineer on prompts: ~4 hours → ~1 hour (clear structure + searchable library)

Building a simple search-based discovery system

If you're starting small, here's a minimal API:

import json
from pathlib import Path
from typing import Optional, List

class PromptLibrary:
    """Simple search-based prompt library."""

    def __init__(self, library_dir: str):
        self.library_dir = Path(library_dir)

    def list_prompts(self, use_case: Optional[str] = None, status: Optional[str] = None) -> List[dict]:
        """List all prompts, optionally filtered."""
        prompts = []

        for prompt_file in self.library_dir.rglob("*.md"):
            with open(prompt_file, 'r') as f:
                content = f.read()

            # Parse YAML frontmatter (simplified; real implementation would use yaml lib)
            if content.startswith("---"):
                end_marker = content[3:].find("---")
                # In practice, parse YAML here
                metadata = {"file": str(prompt_file)}
                # (This is pseudo-code; real parsing needed)
                prompts.append(metadata)

        # Filter
        if use_case:
            prompts = [p for p in prompts if p.get("use_case") == use_case]
        if status:
            prompts = [p for p in prompts if p.get("status") == status]

        return prompts

    def search(self, query: str) -> List[dict]:
        """Search prompts by title or tags."""
        results = []
        query_lower = query.lower()

        for prompt in self.list_prompts():
            title = prompt.get("title", "").lower()
            tags = [t.lower() for t in prompt.get("tags", [])]

            if query_lower in title or any(query_lower in t for t in tags):
                results.append(prompt)

        return results

    def get_prompt(self, slug: str) -> Optional[dict]:
        """Retrieve a prompt by slug."""
        for prompt_file in self.library_dir.rglob("*.md"):
            if prompt_file.stem == slug:
                with open(prompt_file, 'r') as f:
                    return {"slug": slug, "content": f.read()}

        return None

# Usage:
library = PromptLibrary("./prompts")

# Search
results = library.search("summarization")
for r in results:
    print(f"Found: {r['title']} (version {r['version']})")

# Retrieve
prompt = library.get_prompt("extractive_summary_v2")
if prompt:
    print(prompt["content"])

Comparison: Approaches to handling prompt versions

| Approach | Example | Pros | Cons | |----------|---------|------|------| | One file per version | extractive_summary_v1.md, extractive_summary_v2.md | Clear, version-separated | Duplication; hard to see diffs | | Git branches per version | main, v1-deprecated, v2-next | Full version control; easy diffs | Branch management overhead | | Folder per major version | v1/, v2/, with minor versions inside | Organized; scaling-friendly | Requires more structure | | Single file with version history in comments | One file with changelog at bottom | Compact | Poor UX; hard to navigate | | Database-backed (Prompt Registry) | Web UI with version dropdown | Best discoverability; supports metadata queries | Higher complexity, hosting cost |

Recommendation: For teams <20: folder-per-major-version + Git. For teams 20+: add a web UI or database backend.

Measuring library health

To keep your prompt library healthy, track these metrics:

def measure_library_health(library_dir: str) -> dict:
    """Audit your prompt library."""
    stats = {
        "total_prompts": 0,
        "active": 0,
        "deprecated": 0,
        "testing": 0,
        "overdue_review": 0,
        "missing_metadata": 0,
    }

    for prompt_file in Path(library_dir).rglob("*.md"):
        stats["total_prompts"] += 1

        with open(prompt_file, 'r') as f:
            content = f.read()

        # Check status
        if "status: deprecated" in content:
            stats["deprecated"] += 1
        elif "status: testing" in content:
            stats["testing"] += 1
        else:
            stats["active"] += 1

        # Check if review is overdue
        from datetime import datetime, timedelta
        if "next_review_due:" in content:
            # Extract date (simplified)
            # if date < today: stats["overdue_review"] += 1
            pass

        # Check metadata completeness
        required_fields = ["title", "owner", "baseline_metric", "version"]
        if not all(f in content for f in required_fields):
            stats["missing_metadata"] += 1

    return stats

# Example output:
# {
#     "total_prompts": 47,
#     "active": 42,
#     "deprecated": 3,
#     "testing": 2,
#     "overdue_review": 5,  # Alert! Need to schedule reviews
#     "missing_metadata": 1   # Alert! Fix metadata
# }

Summary

A team prompt library is:

1. Organized by use case, not by model or technique.
2. Rich metadata: title, owner, status, baseline metric, version, next review date, guardrails, tags.
3. Discoverable: Via Git + docs (small teams), web UI (medium teams), or SDK (any size).
4. Versioned: Multiple versions co-exist (v1 deprecated, v2 active, v3 testing).
5. Governed: Clear ownership, review cycles, deprecation process, usage tracking.
6. Maintained: Regular reviews and health checks prevent prompt rot.

A well-structured library scales your team's prompt engineering efforts. Instead of everyone writing their own prompts, you leverage shared, tested, documented assets. As your team grows (5 → 50 → 500 engineers), this becomes essential. Start simple (Git + metadata) and add tooling (web UI, CI/CD) as you scale.