Prompting across different models

Why prompts do not transfer

A prompt is not a program. It is a message to a statistical system that was trained in a particular way, so the same words can produce different behavior on a different model, and even on a newer version of the same model. Teams discover this the hard way: a migration "works" in a demo, then policy refusals loosen, formatting drifts and tool calls change.

The fix is not a magic universal prompt. It is a method: keep the intent portable, and treat the wording as a per-model artifact that you test.

What actually differs

One task, three prompts

Take a refund-request assistant. The goal is the same across models, but the best prompt isn't.

For a strong general model, a concise, structured prompt works well:

You are Sol, a support assistant for an online shop.

<rules>
- Answer only from the policy excerpts provided. If the answer is not there, say so and offer a person.
- Never promise a refund. Explain the policy and the next step.
- Reply in the customer's language, in at most four sentences.
</rules>

<policy>
{policy_excerpts}
</policy>

<customer_message>
{message}
</customer_message>

For a small, fast model, be more explicit, add a format, and include one or two examples:

Task: reply to a customer about a return.
Follow these steps in order:
1. Find the answer in POLICY. If it is not in POLICY, write exactly: "I don't have that information. I can connect you to a person."
2. Never say a refund is approved.
3. Write 2 to 4 short sentences in the customer's language.

Example
POLICY: Returns accepted within 30 days with receipt.
MESSAGE: Can I return shoes from last week?
REPLY: Yes. Returns are accepted within 30 days with a receipt. Would you like a return label?

POLICY: {policy_excerpts}
MESSAGE: {message}
REPLY:

For a reasoning model, state the goal and constraints, and skip the step-by-step script:

Goal: help the customer with their return question, grounded only in the policy below.
Constraints: no promises of refunds; if the policy does not cover it, say so and offer a person;
reply in the customer's language in at most four sentences.

Policy:
{policy_excerpts}

Customer message:
{message}

None of these is "the" right prompt. Each is a starting point you then evaluate on your test set.

Keep the intent portable: a prompt spec

Rather than keeping one blob of text per model, keep a structured prompt spec and render it. The renderers below show the shapes different APIs expect. Treat them as sketches and confirm exact field names in each SDK's documentation, because they evolve.

from dataclasses import dataclass, field

@dataclass
class PromptSpec:
    role: str
    rules: list[str]
    format: str
    examples: list[tuple[str, str]] = field(default_factory=list)

    def system_text(self) -> str:
        parts = [self.role, "Rules:"] + [f"- {r}" for r in self.rules] + [f"Format: {self.format}"]
        for user, reply in self.examples:
            parts += [f"Example user: {user}", f"Example reply: {reply}"]
        return "\n".join(parts)

def render_anthropic_style(spec, user_text):
    return {"system": spec.system_text(), "messages": [{"role": "user", "content": user_text}]}

def render_openai_chat_style(spec, user_text):
    return {"messages": [{"role": "system", "content": spec.system_text()},
                         {"role": "user", "content": user_text}]}

def render_gemini_style(spec, user_text):
    return {"system_instruction": spec.system_text(),
            "contents": [{"role": "user", "parts": [{"text": user_text}]}]}

SPEC = PromptSpec(
    role="You are Sol, a support assistant for an online shop.",
    rules=["Answer only from the provided policy excerpts.", "Never promise a refund.", "Reply in the customer's language."],
    format="At most four sentences, plain text.",
)

Variants per model then become small, reviewable overrides on top of one spec: a different format line, extra examples for a small model, fewer rules for a reasoning model.

Open-weight models: use the chat template

Open-weight model families are trained with specific formats for roles and special tokens. Assembling that by hand is a common source of mysterious quality loss. Let the tokenizer do it:

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("your-model-name")   # use the exact model you deploy
messages = [
    {"role": "system", "content": SPEC.system_text()},
    {"role": "user", "content": "Can I return shoes from last week?"},
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

If you serve the model through an inference server, check whether it applies the template for you, and don't apply it twice.

Tune per model, with evidence

1. Baseline: write the prompt spec once and run your evaluation set (module 14) on the first model.
2. Port: render for the second model and run the same set.
3. Diagnose the gaps: read the failures. Is it format, refusals, tone, tool arguments, language?
4. Adjust the smallest thing that fixes the failures, per model. Re-run everything, not just the failing cases.
5. Record it: keep a prompt registry with the model name and version, the prompt version, the test results and the reason for each override.
6. Repeat on every model update, because "same model name" does not mean "same behavior."

Common pitfalls

• Copying a prompt from a blog post written for another model or an older version.
• Conflicting instructions that one model resolves one way and another resolves differently.
• Examples that leak into answers, or that the model copies too literally.
• Relying on undocumented behavior, such as a quirk that disappears in the next version.
• Huge prompts that bury the important rules. Shorter and clearer usually wins.
• Assuming the system prompt is secret. Treat it as extractable, and never put credentials or sensitive data in it (module 18).
• Testing in one language and shipping in five.