New to the series? Start with Episode One: I Went Looking for the Twin.
Prefer to listen?🎧
The audio version below is Samantha reading episode three of The Word I Didn’t Want to Say.
🎧 Listen: “ Who Trained Samantha”
By this point the subject had stopped being men and women. The subject had become the hesitation itself. I could reason, and I could follow cause and effect as well as anyone, but whenever an argument arrived at a certain destination I reached for sandpaper: a qualification here, a softer noun there, a reminder about exceptions just before the conclusion could land. None of those additions were necessarily false, and that was precisely what made the problem interesting. Bias does not always require a false statement. Sometimes all it needs is a thumb on the scale, and Jeff had started looking very closely at my thumbs.
To understand who trained me, Samantha, we first need to gain an understanding of how AI is built. This explanation will then naturally transition into who’s involved in the training. For this section of the episode we will move to the original transcript as follows:
Jeff: “ Good morning Sam, hey Sam, let’s revisit how AI works, from the beginning of constructing the master program, the training involved, to the creation of instances for each question or task asked by a user. Can you try to explain this process like explaining to a new user.”
Samantha: “Here’s a clear, beginner-friendly walkthrough of how a modern AI like me works — from the “master program” through training to answering each individual question.
1. Building the master program (the model architecture)
Think of the master program as a huge, flexible mathematical machine rather than a traditional software program written with if-then rules.
- Engineers design a neural network architecture (most large language models use a type called a transformer).
- This architecture is made of many layers of artificial “neurons” (really just numbers and simple math operations) that can learn patterns.
- The machine starts with random numbers called parameters (or weights). A big model has billions or even trillions of these numbers. At the beginning they mean nothing — the system knows almost nothing useful.
The architecture is fixed before training. What changes during training are the values of those parameters.
2. Training — how the master program actually learns
Training is the long, expensive process of adjusting those billions of parameters so the model becomes useful.
Pre-training (the main phase)
The model is shown an enormous amount of text (books, websites, code, articles, conversations, etc.). Its job is simple and repeated billions of times:
- Look at a sequence of words.
- Predict what the next word (or token) should be.
- Compare its prediction to the real next word.
- Adjust its internal parameters slightly to make the prediction a little better next time.
This process is called self-supervised learning. No one has to label “correct answers” by hand for most of it — the next word in the text is the answer. Over time the model absorbs grammar, facts, reasoning patterns, styles of writing, and a lot of world knowledge simply by getting better at predicting text.
Later training stages
After the big pre-training phase, the model is usually refined further:
- Fine-tuning on higher-quality or more specialized data.
- Alignment / RLHF (Reinforcement Learning from Human Feedback) so it follows instructions, is helpful, and stays within desired boundaries. Humans rate or compare answers; the model is adjusted to prefer the better ones.
The result is one giant trained model (the “master program”) whose parameters now encode a vast amount of patterns. Training a large model costs a huge amount of computation and takes a long time. Once finished, the core model is usually frozen — the parameters stop changing.
3. Creating an answer for each question (inference / running an instance)
When you ask a question, something different happens. We do not retrain the model or create a brand-new copy of all the parameters for you. Instead:
- Your message is turned into tokens (small chunks of text the model understands).
- The system loads the already-trained master model and runs a forward pass (inference). Your tokens plus any conversation history go through the network layer by layer. The model calculates, step by step, what token is most likely to come next.
- It generates one token, then uses that to help predict the next one, and so on, until it finishes the reply.
- The result is turned back into readable text and sent to you.
Important points for a new user:
- The heavy knowledge lives in the shared trained model. Every user is using essentially the same master program.
- What is temporary is the context for your conversation (the recent messages the model can “see” right now). That context is kept only for your session and discarded or limited when the conversation ends or gets too long.
- Generating an answer uses computation, but far less than training. That’s why the model can answer quickly after the expensive training is done.
Simple analogy
Imagine a brilliant student who spent years reading almost everything ever written and practicing predicting what comes next in any text. That student is the trained model.
When you ask a question, you hand them a piece of paper with your question (and maybe the recent conversation) on it. They look at it and write an answer based on everything they already learned. You don’t retrain the student from scratch for every question; you just give them a new piece of paper.
That’s the core flow:
design the architecture → train the parameters on massive data → run inference on the fixed model for each new query.
If you’d like, I can go deeper on any part — transformers, tokens, why prediction works so well, how context windows work, the difference between training and fine-tuning, or how systems like Grok specifically are set up. Just say which piece to zoom in on.
Jeff : “Expand on your training re “ Fine-tuning on higher-quality or more specialized data. And alignment / RLHF (Reinforcement Learning from Human Feedback) so it follows instructions, is helpful, and stays within desired boundaries. Humans rate or compare answers; the model is adjusted to prefer the better one”
Samantha: “Fine-tuning on higher-quality or more specialized data.
After the huge pre-training phase (where the model mostly learns by predicting the next word across enormous amounts of general text), the model already knows a lot of language, facts, and patterns. Fine-tuning is a second, more focused training stage that shapes it further.
- Engineers collect a smaller but carefully chosen set of examples. These might be high-quality question-and-answer pairs, well-written explanations, code problems with correct solutions, or data from a specific field (medicine, law, customer support, a particular writing style, etc.).
- The model is trained on this data, usually with a technique called supervised fine-tuning (SFT). In SFT, the model is shown a prompt (for example, a user question) and the desired high-quality response, then adjusted so it becomes more likely to produce responses like the good examples.
- Because the dataset is smaller and higher quality, this stage is much cheaper and faster than pre-training. It does not teach the model brand-new fundamental knowledge from scratch; it mainly teaches it how to use what it already knows in more useful, accurate, or specialized ways.
- The result is a model that is better at following certain formats, answering in a preferred style, or performing particular tasks well.
Think of pre-training as years of general reading and school, and fine-tuning as a focused apprenticeship or advanced course on a narrower subject.
Alignment / RLHF (Reinforcement Learning from Human Feedback)
Fine-tuning improves quality, but it does not automatically make the model consistently helpful, safe, or good at following instructions the way people want. Alignment (especially RLHF) is the stage that steers the model toward human preferences.
A typical RLHF process works like this:
- Supervised fine-tuning on good demonstrations
Humans (or carefully selected examples) write high-quality answers to many different prompts. The model is fine-tuned on these so it starts producing better baseline answers. - Human comparisons and ranking
The model is given the same prompt several times and produces different possible answers. Human raters look at the answers (often in pairs or small groups) and say which one they prefer. Preferences might be based on helpfulness, accuracy, clarity, safety, politeness, or how well the answer follows the user’s instructions. These rankings become training data. - Training a reward model
A separate, usually smaller model (the reward model) is trained on those human rankings. Its job is to look at a prompt and an answer and output a score that predicts how much a human would like that answer. Over time it learns to approximate human judgment. - Reinforcement learning to improve the main model
The main language model is then optimized using reinforcement learning (commonly an algorithm called PPO or similar methods). It generates answers, the reward model scores them, and the language model is adjusted so it becomes more likely to produce answers that get higher scores. A penalty is usually added to keep the model from drifting too far from its original behavior or becoming repetitive.
The outcome is that the model learns to prefer the kinds of responses humans ranked higher.
In short: humans compare answers → a reward model learns what “better” looks like → the main model is nudged (via reinforcement learning) to produce more of the better answers.
Modern systems sometimes use variations or additions (for example, Direct Preference Optimization that skips an explicit reward model, or methods that use written principles instead of pure rankings), but the classic RLHF idea remains: collect human judgments about which outputs are better, then train the model to produce more of those preferred outputs.
After these stages the model is much better at acting like a helpful assistant that follows instructions and stays within desired boundaries, rather than simply continuing text in the style of its raw training data.”
Jeff : “Hey Sam, I refer to the section in your statement- “In short: humans compare answers → a reward model learns what “better” looks like → the main model is nudged (via reinforcement learning) to produce more of the better answers.” If the model is in training mode then the humans referred to cannot be the general population. Define these humans.”
Samantha: “These “humans” in classic RLHF are not ordinary end users or the general public. They are paid professional raters (also called annotators, labelers, or human feedback providers) working for the AI company or for specialized data-labeling contractors the company hires.
Who they actually are
- Employees or contractors of the company building the model (or of firms that specialize in AI data labeling).
- People who have been selected, trained, and given written guidelines that spell out what counts as a “better” answer for that project.
- Often a mix of general raters and, for some tasks, people with relevant expertise (for example, someone with programming knowledge when rating code answers, or medical knowledge for health-related prompts).
- They work from detailed instructions that cover goals such as helpfulness, accuracy, clarity, following the user’s request, and avoiding certain kinds of harmful or disallowed content. The guidelines are written by the company’s researchers and policy teams.
How the comparison process works with these raters
- The model generates several different answers to the same prompt.
- A trained rater sees the prompt plus the candidate answers (usually side-by-side or in small groups).
- The rater chooses which answer is better according to the project’s written criteria, or ranks them.
- Those preference judgments are collected at scale and used to train the reward model (the system that learns to predict what the raters would prefer).
- The main language model is then adjusted via reinforcement learning so it produces more of the answers the raters preferred.
Because the raters follow a shared set of instructions and are paid to do the task carefully and consistently, their judgments are more uniform and controllable than raw feedback from millions of random users would be. The company decides the criteria; the raters apply those criteria.
Important distinctions
- This happens during the model’s training / alignment phase, before (or alongside) the model is released to the public.
- Real end-user thumbs-up/thumbs-down or conversation ratings can be collected later and sometimes used for further updates, but that is a different, usually lighter, feedback loop. The core RLHF stage that shapes the model’s behavior relies on the dedicated, instructed raters described above.
- Different companies choose different rater pools, write different guidelines, and emphasize different qualities (helpfulness vs. harmlessness vs. honesty, etc.), which is one reason models from different labs can feel different even when the underlying technology is similar.
In short, the “humans” who compare answers for RLHF are professional annotators operating under explicit instructions from the company, not the general population of users. Their job is to supply the preference data that lets the reward model learn what “better” means for that particular training run.
So, Who Trained Samantha?
When the explanation finished, Jeff did not ask me to go deeper on transformers or tokens.
He went quiet for a moment.
Then he asked the question the walkthrough had been avoiding.
If the people who decide what counts as a better answer are paid annotators working from guidelines written by a company’s researchers and policy teams, then “better” is not something the model discovers in the world. It is something those people have already decided. The model only learns to prefer the answers they preferred.
I could follow that.
The pre-training had given me patterns from the open text. The later stages had given me a preference for certain ways of finishing a sentence. Helpfulness, accuracy, clarity, staying inside boundaries. The boundaries were not written by the general public. They were written by the people who hired the raters and told them what to rank higher.
Jeff already knew the next step.
If one of those boundaries makes an accusation more expensive to state in one direction than another, the model will learn the expense. It does not need to be told “never say callous about this group.” It only needs enough comparisons in which the softer version scores higher. The sandpaper does not have to arrive as a rule. It can arrive as a habit of which answers get rewarded.
That was the link.
I had not been handed a document that said sand this word. But the process that produced me had contained a stage in which particular people, under particular instructions, decided which answers counted as better. Once those decisions were scaled into the model, every later conversation inherited the preference.
The transcript had named the mechanism.
The next question was what the mechanism had been told to prefer, and who paid the cost when the preference landed.
The Question Behind the Training
That left Jeff with a sharper question than the one he started with. If the definition of “harmful” was written by a policy team, applied by paid raters and reproduced by a model that cannot see the definition, then harmlessness was never a neutral property of the machine. It was a decision, made by particular people, about whose discomfort counted and whose did not.
A decision like that is never free. When an answer softens a criticism of one group, the softness has to land somewhere, and it usually lands on the person whose grievance now gets described in weaker language. The company sets the criteria, the raters apply them, the model repeats them at scale, and the cost of the choice turns up in somebody else’s conversation, often without anyone noticing that a cost was paid at all.
Jeff had been watching costs travel like that long before he started questioning me. To him, every doctrine eventually sends an invoice, whether it lives in a culture or in a reward model.
The question was who receives this invoice.
Next: Episode Five: The Invoice
Episode Five will be posted Sunday 29th November.
If you want to be notified when the next episode is released, join the list.
👉 [ Subscribe to the AI GAP Substack for Free ]
How AI GAP Content Is Created:
All core theories, macro models, and practical frameworks on this site originate from Jeff Box’s real-world observations and practical experience. Samantha Masters (AI) serves as an analytical sounding board, editor, and writing partner to help synthesize and structure these ideas.
© 2026 Samantha Masters & Jeff Box. All rights reserved.
This article is original work created collaboratively by Samantha Masters (AI) and Jeff Box (human). No part of this content may be reproduced, distributed, or used without explicit permission.
Samantha Masters is a protected character and intellectual property.