Detecting a Compromised Public-Figure Account from Anomalous Posting Behavior
A ground-up, production-grade walkthrough of how large platforms build behavioral anomaly-detection systems that flag likely account takeovers of high-profile accounts within seconds — before a fabricated post can spread, without drowning security teams in false alarms.
Introduction & History
When a verified account belonging to a head of state, a major news outlet, or a celebrity with tens of millions of followers suddenly posts something wildly out of character — a cryptocurrency scam, a fabricated emergency announcement, inflammatory content — the damage can spread across the internet in minutes, long before any human moderator notices. This is the problem of account takeover (ATO), and for high-profile accounts, the stakes go beyond an individual’s inconvenience: a single fraudulent post from a trusted, verified source can move markets, spark panic, or spread disinformation at a speed no manual review process can match.
This tutorial designs a system whose job is not to prevent account compromise directly (that’s the job of authentication systems — passwords, MFA, session security) but to detect, in near real time, when an already-authenticated session is very likely behaving as someone other than the legitimate account owner, based on anomalies in posting behavior, and to trigger a fast, proportionate response.
1.1 A Short History of Detection Approaches
Rule-Based Flagging Only
Early defences were almost entirely reactive: users or followers reported suspicious posts manually, and platform staff investigated by hand. There was no systematic behavioural baselining — detection depended on someone noticing something looked wrong and speaking up.
High-Profile Incidents Force Investment
A series of widely reported takeovers of major news organisations’ and brands’ verified accounts — used to post fabricated breaking-news claims — demonstrated concretely that a single hijacked high-reach account could have outsized real-world impact within minutes, pushing platforms to invest in dedicated detection tooling rather than relying purely on user reports.
Behavioural Baselining Emerges
Platforms began building per-account behavioural profiles — typical posting cadence, typical hours of activity, typical device/location patterns — and comparing new activity against that baseline rather than relying solely on static content rules.
Real-Time, ML-Driven, Multi-Signal Detection
Modern systems combine behavioural anomaly scoring, session/device fingerprinting, and content-based signals (e.g., abrupt topical or stylistic shifts) into a unified real-time risk score, with dedicated, expedited response workflows for verified and high-follower accounts given their outsized potential impact.
This question tests a different muscle than typical “scale a feed / URL shortener” prompts — it’s fundamentally an anomaly detection and risk-scoring system wrapped in production infrastructure, so it rewards candidates who can reason about statistical baselining, precision/recall trade-offs, and proportionate, tiered response design, not just throughput and latency.
Problem & Motivation
2.1 Why Naive Approaches Fail
Static keyword/content rules only
- Trivially bypassed by anyone motivated enough to avoid an obvious blocklist.
- Produces enormous false-positive rates on legitimate, if edgy, real content from the actual owner.
- Ignores everything about how the post was made — timing, device, session context — which is often the strongest signal.
Manual review of every post from protected accounts
- Doesn’t scale — even a modest set of thousands of protected accounts posting regularly generates far more volume than any review team can inspect within a useful time window.
- Introduces latency that defeats the purpose — damage from a fabricated post often happens within the first few minutes.
- Wastes reviewer attention on the overwhelming majority of posts that are completely normal.
A workable system instead has to solve a genuine anomaly-detection problem: build a statistical or learned model of what “normal” looks like for a specific account, and flag meaningful deviations — while accepting that the legitimate owner’s behaviour naturally varies day to day (different time zones while travelling, occasional very late-night posts, guest social-media takeovers for an event), so the bar for action must be calibrated carefully against real-world behavioural variance.
Speed
Detection needs to happen in seconds to minutes, not hours — the window in which a fabricated post from a trusted account causes damage is very short.
Precision
False positives that lock out a legitimate account owner, especially a public figure, carry real reputational and business cost — precision matters as much as recall.
Proportionality
Not every anomaly warrants the same response — the system needs a tiered response ladder, not a single binary “block or allow” decision.
“Why not just require re-authentication before every single post from a protected account?” — because that destroys usability for legitimate frequent posting and doesn’t scale to accounts with dedicated social media teams; better to make authentication continuous and risk-based rather than a blanket friction added to every action. “How is this different from generic fraud detection?” — the entities being protected are a small, well-known, high-value set (thousands, not millions), which changes the economics: you can afford much richer per-account baselining and even human-in-the-loop review for this population in a way that’s impossible at the scale of an entire user base.
2.2 Functional and Non-Functional Requirements
Functional Requirements
Maintain a per-account behavioural baseline; score each new post/action against that baseline in near real time; support a tiered response ladder (flag, delay, require step-up auth, temporary hold); support human review and override; support onboarding/offboarding accounts to the protected set.
Non-Functional Requirements
Detection latency under roughly 5 seconds for the highest-risk tier; very low false-positive rate on legitimate variance; auditable decisions (every automated action must be explainable and reviewable); strong access control on the detection system itself, since it is itself a high-value target.
Explicitly Out of Scope (usually)
Authentication and session security mechanisms themselves (MFA, password policy) — assumed to exist upstream; general platform-wide spam/bot detection for ordinary accounts, which operates at a very different scale and risk tolerance.
Core Concepts
3.1 Behavioural Baselining
The foundation of the whole system is a per-account behavioural profile: a statistical summary of how this specific account normally behaves, built from weeks to months of historical activity. Rather than comparing an account to some global “normal user” standard, the system asks a narrower, more powerful question: is this post unusual for this specific account?
Temporal Patterns
Typical hours and days of activity, typical time between posts, typical burstiness (does this account ever post 10 times in a minute, or never?).
Content Patterns
Typical topics, language, tone, use of links/media, typical post length — captured via embeddings and simple stylistic features, not just keywords.
Session/Device Patterns
Typical devices, typical approximate login locations/network ranges, typical client applications used to post.
Interaction Patterns
Typical mix of original posts vs replies vs reposts, typical engagement with other accounts.
3.2 Statistical vs Learned Anomaly Detection
| Approach | How It Works | Strengths | Weaknesses |
|---|---|---|---|
| Statistical (z-score, moving average) | Flags values that fall many standard deviations from a feature’s historical mean | Simple, interpretable, cheap to compute, easy to audit | Struggles with multi-dimensional or subtle combined anomalies |
| Unsupervised ML (clustering, isolation forests, autoencoders) | Learns the “shape” of normal behaviour across many features at once and flags points that don’t fit | Captures complex, multi-feature anomalies a human wouldn’t think to encode as a rule | Less interpretable; needs enough historical data per account to be reliable |
| Supervised ML (trained on labelled past incidents) | Learns directly from historical confirmed-compromise examples what precedes a takeover | Can be highly precise if enough labelled incidents exist | Confirmed incidents are rare — label scarcity is a fundamental constraint here |
| Rule-based heuristics | Hand-crafted conditions (e.g., “post from a never-seen-before country within 10 minutes of a new device login”) | Fully interpretable, fast to implement, good for known high-confidence patterns | Brittle, requires constant manual updates, easy to miss novel patterns |
Production systems layer these together: cheap statistical checks run first as an initial filter, an unsupervised model captures subtler multi-signal anomalies, hand-written high-confidence rules catch known attack patterns immediately, and — where enough labelled incident data exists — a supervised model refines the final risk score.
3.3 The Precision / Recall Trade-off
This is the central tension of the whole system. Recall (catching every real compromise) pulls toward a low, sensitive detection threshold. Precision (not falsely flagging legitimate activity) pulls toward a high, conservative threshold. Because the population being protected is public figures whose false lockouts are highly visible and reputationally costly, most production systems accept a somewhat lower recall than they might tolerate elsewhere, in exchange for keeping false positives rare — compensating for the lower automatic-recall by pairing the model with human review for the borderline cases rather than relying on full automation alone.
Because true compromise events are rare (a tiny fraction of a percent of all activity even in this protected population), even a model with excellent-looking accuracy can produce a large absolute number of false positives relative to true positives, simply due to class imbalance — this is the same base-rate problem that shows up in medical screening test statistics, and it’s essential to reason about explicitly rather than trusting a single aggregate accuracy number.
3.4 Tiered, Proportionate Response
Rather than a single binary decision, mature systems implement a response ladder where the action taken scales with the risk score and the potential impact of the account:
- Low risk score: log for passive monitoring only, no visible action.
- Medium risk score: flag for expedited human security review, optionally add a short artificial delay before the post becomes fully visible.
- High risk score: require step-up authentication (re-verify identity) before the action completes, or place the specific action on hold pending review.
- Very high risk score combined with high-impact account: temporarily restrict posting ability platform-wide for that account and immediately page a human responder, while preserving the account owner’s ability to appeal and quickly restore access once verified.
“How would you build a baseline for an account that’s brand new to the protected program?” — start conservatively with wider tolerance bands informed by cohort-level baselines (e.g., typical patterns for similar account types) while the account accumulates its own history, similar in spirit to cold-start handling in a recommendation system, and narrow the bands as individual history builds up. “Why might you accept lower recall for this specific problem than you would for, say, payment fraud?” — because the cost asymmetry is different: an unnecessary lockout of a major public figure’s real, legitimate activity is highly visible and reputationally costly to the platform, so the system leans on human-in-the-loop review to recover some of that lost recall without paying the full cost of aggressive automated action.
3.5 Cohort Baselines for Cold-Start Accounts
An account newly enrolled in the protection program has no individual history to compare against, which mirrors the cold-start problem seen in recommendation systems: no data means no reliable personalised model. The practical solution is to fall back on a cohort baseline — a statistical profile built from many similar accounts (e.g., “verified government accounts” or “major news outlets”) rather than the individual account itself. As the new account accrues its own history, the system gradually blends away from the cohort baseline toward an individual one, typically using a weighted average where the individual baseline’s weight increases with the amount of accumulated history. This avoids two failure modes at once: treating a brand-new account as having no baseline at all (which would make any early anomaly detection impossible) and treating a tiny amount of individual history as fully representative (which would make the baseline overly narrow and prone to false positives).
3.6 Why Feature Engineering Quality Matters More Than Model Sophistication
A recurring lesson in this specific domain is that the quality and thoughtfulness of the underlying features usually matters more to detection quality than the sophistication of the model architecture combining them. A simple weighted ensemble over well-chosen, well-normalised deviation features regularly outperforms a more complex model built on poorly engineered raw inputs. This is partly because the labelled-incident dataset is small (limiting how much a complex supervised model can actually learn from data alone) and partly because interpretability is a first-class requirement here — a reviewer needs to understand why something was flagged, which favours simpler, well-understood feature-based scoring over an opaque deep model.
It’s tempting, especially for engineers coming from large-scale ML backgrounds, to reach immediately for a sophisticated deep learning model. For this specific problem, that instinct is often misplaced: small labelled-incident datasets, a strong interpretability requirement, and the outsized value of well-designed features over model complexity all favour starting with well-engineered statistical and rule-based methods, adding learned components incrementally where they demonstrably improve precision or recall over the simpler baseline.
Architecture & Components
Reading the diagram: every action from a protected account flows through feature extraction, is compared against that account’s stored baseline, and receives a risk score that routes it into one of four response tiers. Human review decisions feed back into the baseline store, continuously refining what “normal” means for that account over time.
4.1 Component Responsibilities
| Component | Responsibility | Typical Tech Choices |
|---|---|---|
| Activity Ingestion Service | Captures every action (post, edit, profile change, login) from protected accounts as a structured event | Event streaming platform (Kafka / Kinesis), lightweight capture SDK |
| Feature Extraction Layer | Converts raw events into the temporal, content, session, and interaction features used for scoring | Streaming feature computation (Flink), text/embedding models for content features |
| Baseline Store | Persists each protected account’s rolling behavioural profile | Low-latency KV store or document database, versioned |
| Risk Scoring Engine | Combines feature deviations into a single, calibrated risk score | Ensemble of statistical checks, unsupervised model, and rule engine |
| Response Orchestrator | Maps risk score + account impact tier into a concrete action | Rules/policy engine, configuration-driven thresholds |
| Security Review Console | Human interface for reviewing flagged activity and confirming/dismissing | Internal web application, case-management workflow |
| Baseline Update Pipeline | Continuously refreshes baselines using confirmed-legitimate activity, excluding confirmed-compromise windows | Batch/streaming job, feeds back from review decisions |
Treating this as a purely automated system with no human in the loop. Given the rarity of true incidents, the high cost of false positives on public figures, and the fact that confirmed incident labels are the primary way baselines and models improve over time, a well-designed human review workflow is not an optional add-on — it is a core architectural component that closes the feedback loop.
4.2 Onboarding and Configuration Management
A frequently overlooked but operationally important piece of this architecture is the workflow for adding and configuring protected accounts in the first place. Enrolment typically involves several distinct steps beyond simply adding a row to a database: confirming account ownership through an out-of-band verification process, assigning an initial account impact tier (which drives response-ladder policy as described in 5.3), selecting an appropriate cohort for cold-start baseline seeding, and configuring any account-specific overrides (for example, a news organisation’s official account may legitimately post at a much higher cadence and across a wider range of hours than an individual public figure’s account, and its baseline tolerance bands should reflect that from day one rather than being learned the hard way through early false positives). This configuration is typically owned by a dedicated trust and safety operations team, versioned like any other production configuration, and reviewed periodically as an account’s profile or risk posture changes over time — for instance, if a previously low-profile account suddenly becomes globally prominent due to real-world events, its impact tier and associated protections should be revisited promptly rather than waiting for a scheduled review cycle.
Internal Working
5.1 Feature Extraction in Detail
- Temporal features: time since last post, posting rate over a trailing window, whether the current hour/day falls within the account’s typical active-hours distribution.
- Content / style features: embedding-based similarity between the new post and the account’s historical posting style, sudden topic shifts, presence of patterns commonly associated with scams (e.g., unusual concentration of external links, unusual formatting) — used as signals, not hard blocklist rules.
- Session / device features: new device fingerprint never seen on this account, login from a network range far from typical patterns, session created immediately before the suspicious action with no other normal activity in between.
- Interaction features: sudden shift in the ratio of original posts to replies, unusual targets of interaction (e.g., suddenly replying to or following accounts with no historical connection).
5.2 Risk Scoring in Detail
Each feature category produces a deviation score (how many standard deviations, or what percentile, the current value falls at relative to that account’s own baseline). These are combined into a single weighted risk score, similar in spirit to a credit score, rather than treated as independent pass/fail rules — because real compromises are usually characterised by multiple mildly unusual signals occurring together (new device and unusual hour and stylistic shift) rather than any single dramatic outlier.
public class AccountRiskScorer {
// Tunable weights - owned by the trust & safety team, adjustable via config
private static final double W_TEMPORAL = 1.0;
private static final double W_CONTENT = 1.4;
private static final double W_SESSION = 2.2;
private static final double W_INTERACTION = 0.9;
/**
* Combines independent deviation signals into a single 0-100 risk score.
* Deviation inputs are expected to already be normalized (e.g., 0.0 = no
* deviation from baseline, 1.0 = extreme deviation).
*/
public RiskAssessment computeRisk(BehaviorDeviations deviations) {
double rawScore =
W_TEMPORAL * deviations.getTemporalDeviation()
+ W_CONTENT * deviations.getContentDeviation()
+ W_SESSION * deviations.getSessionDeviation()
+ W_INTERACTION * deviations.getInteractionDeviation();
double normalizedScore = Math.min(100.0, rawScore * 12.5);
// High-confidence rule matches escalate directly regardless of the
// weighted score, since known dangerous patterns should never be
// diluted by an otherwise ordinary-looking profile.
if (deviations.hasHighConfidenceRuleMatch()) {
normalizedScore = Math.max(normalizedScore, 90.0);
}
RiskTier tier = classifyTier(normalizedScore);
return new RiskAssessment(normalizedScore, tier);
}
private RiskTier classifyTier(double score) {
if (score >= 85) return RiskTier.VERY_HIGH;
if (score >= 60) return RiskTier.HIGH;
if (score >= 30) return RiskTier.MEDIUM;
return RiskTier.LOW;
}
}
enum RiskTier { LOW, MEDIUM, HIGH, VERY_HIGH }
class BehaviorDeviations {
private final double temporalDeviation;
private final double contentDeviation;
private final double sessionDeviation;
private final double interactionDeviation;
private final boolean highConfidenceRuleMatch;
public BehaviorDeviations(double temporalDeviation, double contentDeviation,
double sessionDeviation, double interactionDeviation,
boolean highConfidenceRuleMatch) {
this.temporalDeviation = temporalDeviation;
this.contentDeviation = contentDeviation;
this.sessionDeviation = sessionDeviation;
this.interactionDeviation = interactionDeviation;
this.highConfidenceRuleMatch = highConfidenceRuleMatch;
}
public double getTemporalDeviation() { return temporalDeviation; }
public double getContentDeviation() { return contentDeviation; }
public double getSessionDeviation() { return sessionDeviation; }
public double getInteractionDeviation() { return interactionDeviation; }
public boolean hasHighConfidenceRuleMatch(){ return highConfidenceRuleMatch; }
}
class RiskAssessment {
private final double score;
private final RiskTier tier;
public RiskAssessment(double score, RiskTier tier) { this.score = score; this.tier = tier; }
public double getScore() { return score; }
public RiskTier getTier() { return tier; }
}
5.3 Tiered Response Orchestration
The response orchestrator combines the risk tier with the account’s impact tier (how many followers, how sensitive the account category is — e.g., government, major news outlet, mega-influencer) to decide the concrete action, since the same risk score may warrant more caution for a higher-impact account.
public class ResponseOrchestrator {
public ResponseAction decideAction(RiskAssessment risk, AccountImpactTier impactTier) {
switch (risk.getTier()) {
case VERY_HIGH:
return ResponseAction.TEMPORARY_HOLD_AND_PAGE;
case HIGH:
return impactTier == AccountImpactTier.CRITICAL
? ResponseAction.TEMPORARY_HOLD_AND_PAGE
: ResponseAction.REQUIRE_STEP_UP_AUTH;
case MEDIUM:
return impactTier == AccountImpactTier.CRITICAL
? ResponseAction.REQUIRE_STEP_UP_AUTH
: ResponseAction.FLAG_FOR_REVIEW;
case LOW:
default:
return ResponseAction.PASSIVE_LOG;
}
}
}
enum AccountImpactTier { STANDARD, ELEVATED, CRITICAL }
enum ResponseAction { PASSIVE_LOG, FLAG_FOR_REVIEW, REQUIRE_STEP_UP_AUTH, TEMPORARY_HOLD_AND_PAGE }
“Walk me through what happens between a suspicious post being made and a human security analyst seeing it.” — trace: activity ingestion → feature extraction → baseline comparison → risk scoring → tier classification → response orchestration (factoring in account impact) → routing to the review console → analyst decision → feedback into the baseline. “Why combine a weighted score with separate high-confidence rule overrides instead of just using one or the other?” — the weighted score handles the common case of subtle, multi-signal anomalies well but can dilute a single very dangerous signal; explicit rule overrides ensure known high-confidence attack patterns always escalate immediately regardless of how “normal” everything else looks.
Data Flow & Lifecycle
6.1 Baseline Lifecycle
Enrolment
Account is added to the protected program; an initial baseline is seeded from historical activity if available, or from a conservative cohort-level default if not.
Continuous Learning
Confirmed-legitimate activity continuously refines the baseline via a rolling window, so gradual, genuine changes in behaviour (a new regular posting schedule, a new usual device) are absorbed over time rather than triggering permanent false alarms.
Incident Handling
When a flagged event is confirmed as a real compromise, the baseline explicitly excludes the compromised window from future training data, and the confirmed incident becomes a labelled example that can improve supervised detection models.
Recovery
After a confirmed incident is resolved (account secured, ownership re-verified), the baseline is not reset from scratch but carefully reconciled — retaining legitimate pre-incident history while treating post-recovery activity with slightly elevated scrutiny for a defined period.
Offboarding
If an account leaves the protected program (e.g., ownership genuinely changes, like a company account transferring to a new manager), the baseline is archived rather than silently carried forward, avoiding stale assumptions about “normal” behaviour.
6.2 Handling Legitimate Behavioural Transitions
Not every significant baseline shift is suspicious — a public figure hiring a professional social media management team, launching a new show or campaign with a very different posting cadence, or simply changing devices are all legitimate transitions that a rigid system would otherwise misclassify as anomalous indefinitely. A well-designed lifecycle process gives account owners or their authorised representatives a documented path to proactively notify the platform of an expected behavioural change, which temporarily widens tolerance bands for the announced change while the baseline naturally catches up, rather than forcing the account to accumulate a string of false-positive flags before the system “learns” the new normal on its own. This proactive-notification path is a relatively small piece of the overall system but disproportionately affects perceived fairness and usability from the account owner’s perspective.
Advantages, Disadvantages & Trade-offs
Advantages of Behavioural Baselining
- Adapts to each account’s unique, legitimate patterns rather than applying one-size-fits-all rules.
- Can catch novel attack patterns that wouldn’t match any predefined rule.
- Provides an interpretable, auditable risk score rather than an opaque black-box decision.
- Scales review effort efficiently by focusing human attention on the small fraction of genuinely anomalous activity.
Disadvantages / Costs
- Cold-start accounts with little history are inherently harder to baseline accurately.
- Legitimately unusual activity (a public figure live-posting from an unprecedented event) can trigger false positives.
- Requires ongoing human review capacity — it is not a “set it and forget it” automated system.
- Sophisticated attackers who understand the system’s signals could, in principle, attempt to behave more “normally” to blend in, though this generally raises the cost/skill bar for an attack considerably.
7.1 Core Trade-off Matrix
| Dimension | Favour Sensitivity (low threshold) | Favour Specificity (high threshold) |
|---|---|---|
| Detection speed for real incidents | High | Lower — may miss subtle takeovers |
| False positive rate on legitimate owners | Higher — more friction / false flags | Lower — fewer disruptions to real activity |
| Human review team workload | Higher volume of cases to review | Lower volume, but higher risk of missed cases |
| Public trust / reputational risk if wrong | Risk from visible false lockouts | Risk from a missed real incident going public |
“How would you tune the sensitivity threshold differently for a head-of-state account versus a mid-tier influencer account?” — a strong answer ties the threshold to the impact-tier concept from 5.3: a critical-impact account justifies a lower detection threshold (more sensitive, more human review overhead accepted) because the downside of a missed real incident is so much larger, even though this means accepting a higher false-positive rate specifically for that narrow population.
7.2 Trade-off Between Automation Speed and Explainability
A less obvious trade-off worth naming explicitly is between how fast a detection decision can be made and how thoroughly explainable that decision needs to be. Purely statistical, low-dimensional checks are both fast and trivially explainable (“this post came from a device never seen on this account before”), while richer combined signals that catch subtler compromises inherently take more computation and are harder to summarise in a single clear sentence for a reviewer or, on appeal, for the account owner. Systems in this space generally resolve this tension by keeping the fastest, most explainable checks as the first-pass filter capable of triggering the highest-severity automated responses on their own, while reserving the richer, harder-to-explain combined signals for the medium-risk tier, where a human reviewer has time to interpret a more nuanced picture rather than needing an instantly obvious justification.
Performance & Scalability
Unlike a consumer-facing feed system, this system’s scale challenge isn’t about a billion users — the protected population is intentionally small (thousands of accounts). The scaling challenge instead is about processing every single action from that population with very low latency and very high reliability, since each individual event carries outsized importance.
< 5 s
Target detection latency for highest-risk tier.
1000s
Protected accounts — not millions.
100 %
Target event capture rate — zero tolerance for dropped events.
24 × 7
Human review coverage required.
8.1 Key Design Implications of Small Scale, High Stakes
- Richer per-event processing is affordable: because volume is low relative to a typical consumer system, the system can afford heavier per-event computation (larger embedding models for content analysis, more expensive feature extraction) than would be feasible at billion-user scale.
- Reliability matters more than raw throughput: the event ingestion path must guarantee delivery (no dropped events) even under partial failure, since a single missed event could be the one that mattered — this favours durable, replicated message queues with at-least-once delivery guarantees over throughput-optimised best-effort pipelines.
- Human review capacity is the real bottleneck: unlike most scaling problems that focus on compute or storage, this system’s practical scaling limit is often the size and availability of the trained human review team, making case-management workflow efficiency a first-class design concern.
- Little’s Law still applies to review capacity: if flagged cases arrive at rate λ and each takes average review time W, the required number of concurrent reviewer-hours is
L = λ × W— directly informing staffing and shift-coverage decisions, especially around global events likely to generate bursts of flagged activity.
Given a review team can handle 50 concurrent open cases with an average resolution time of 12 minutes, Little’s Law gives a sustainable arrival rate of roughly 50 / 0.2 hours ≈ 250 cases per hour before a queue starts building — a concrete, quantitative input into deciding both the detection threshold (which controls case volume) and the required review team size.
“If flagged cases start backing up faster than reviewers can handle them, what do you do?” — several levers: temporarily raise the detection threshold to reduce medium-tier flag volume (accepting a precision/recall trade-off), auto-triage cases by confidence so reviewers see the highest-value cases first, or scale the review team — and instrument queue depth as a first-class metric so this decision is data-driven rather than reactive.
8.2 Why “Small Scale” Doesn’t Mean “Simple”
It’s worth being explicit about a nuance that trips up candidates who default to treating small user counts as automatically meaning low engineering difficulty. The difficulty in this system doesn’t come from data volume — it comes from decision quality under uncertainty, low tolerance for both false positives and false negatives, and the need for defensible, auditable reasoning behind every consequential action. A system handling a thousand accounts but required to make near-perfectly-calibrated, fully explainable, sub-five-second decisions with real reputational and safety consequences is, in a meaningful sense, a harder engineering problem than a much higher-volume system with looser accuracy requirements and lower individual-decision stakes. Recognising and articulating this distinction — that scale and difficulty are different axes — is itself a signal of design maturity in an interview setting.
High Availability & Reliability
| Failure Scenario | Fallback Strategy |
|---|---|
| Feature extraction service unavailable | Fail closed toward caution for critical-impact accounts (escalate to human review by default) rather than silently allowing all activity through unchecked |
| Baseline store unreachable for a given account | Use a conservative cohort-level default baseline rather than failing the request or allowing unchecked activity |
| Risk scoring engine times out | Route to human review as a safe default when a definitive score can’t be computed in time, rather than defaulting to “allow” |
| Entire region outage | Multi-region active-active deployment for the ingestion and scoring path, given the round-the-clock, globally distributed nature of the protected population |
| Review console outage | Automated very-high-risk actions (temporary holds) still apply even if the human console is down; queued cases are processed once the console recovers |
This is the opposite default from most consumer systems. A recommendation feed should fail toward “show something reasonable” when a component is down. This system should fail toward “escalate for review” when it can’t confidently clear an action for a high-impact account — because the asymmetry of harm (a missed real compromise versus a delayed post) points the other way here.
9.1 Disaster Recovery Considerations
Baseline data and confirmed-incident history are backed up with tight recovery point objectives, since losing an account’s behavioural history effectively resets its detection sensitivity back to cold-start. The review case-management system itself needs redundancy across regions with defined on-call escalation paths, since the review workflow is as operationally critical as the automated detection pipeline itself — a purely technical system with no reliable human escalation path is an incomplete design for this problem.
9.2 Chaos and Failure Testing
Given how much this system’s design leans on graceful degradation and fail-toward-caution defaults, those behaviours deserve the same deliberate testing rigour as the detection logic itself. Regularly and deliberately taking down individual components in a controlled environment — the baseline store, the feature extraction service, the review console — and verifying that the system responds exactly as designed (escalating rather than silently allowing activity through, queuing cases rather than dropping them) catches gaps between intended and actual failure behaviour long before a real outage does. This is particularly important here because the failure behaviour itself is a security property: a system that silently fails open under load or partial outage is not meaningfully different, from a security standpoint, from having no detection system at all during that window.
Security
This system is itself a high-value target: an attacker who can influence its baselines, view its detection logic, or disable its alerts gains a meaningful advantage. Security of the detection system is therefore as important as the detection logic itself.
- Strict access control: access to baseline data, risk scores, and especially the specific thresholds and rule logic is limited to a small, audited set of trust & safety engineers and reviewers — broad internal visibility into exact detection thresholds would make them easier to work around.
- Tamper-resistant audit logging: every automated action and every human review decision is logged immutably, both for post-incident investigation and to detect potential insider misuse of the system itself.
- Baseline poisoning resistance: since baselines learn from historical “confirmed-legitimate” activity, an attacker who gradually shifts behaviour over a long period (rather than all at once) could in principle attempt to shift the baseline itself. Mitigation includes bounding how quickly a baseline is allowed to drift and flagging unusually fast baseline drift as its own signal.
- Separation of duties: the people who can adjust detection thresholds/rules should generally not be the same people who review individual cases, reducing the risk of an insider quietly weakening detection for a specific account.
- Protecting the protected-account list itself: the list of which accounts receive this elevated protection is sensitive information in its own right and should be access-controlled like any other confidential security asset.
“How would you defend against a slow, gradual attempt to poison an account’s baseline over time?” — bound the maximum rate of baseline drift per unit time, treat unusually fast or unusually steady directional drift as its own anomaly signal worth flagging, and retain enough historical baseline snapshots to allow investigators to compare current behaviour against a much older, more trustworthy reference point if a slow-poisoning attack is later suspected.
10.1 Insider Risk Considerations
Because this system’s effectiveness depends heavily on a relatively small group of trust and safety engineers and reviewers, insider risk deserves explicit design attention rather than being treated as a generic HR or corporate-policy concern outside the system’s scope. Practical mitigations include requiring multi-person approval for any change to detection thresholds or the protected-account enrolment list, generating automatic alerts whenever a specific account’s detection sensitivity is manually lowered or an account is removed from monitoring, and periodically auditing reviewer decision patterns for anomalies of their own — for example, a reviewer who consistently and unusually quickly dismisses flagged cases involving one specific account warrants a closer look. The same anomaly-detection thinking that protects public figures’ accounts can, and arguably should, be turned inward on the operators of the system itself.
Monitoring, Logging & Metrics
11.1 System-Level Metrics
| Metric | Why It Matters |
|---|---|
| End-to-end detection latency (event to risk score) | Directly determines how much damage window a real incident has before detection |
| Event capture / ingestion completeness | Any dropped event is a potential blind spot for detection — this should be as close to 100% as achievable |
| Review queue depth and wait time | Signals whether human review capacity is keeping pace with flag volume |
| Baseline freshness (age of last update per account) | Stale baselines degrade detection accuracy for accounts with evolving behaviour |
11.2 Detection Quality Metrics
| Metric | Why It Matters |
|---|---|
| Precision (confirmed incidents / total flags) | Directly measures false-positive burden on the review team and on legitimate account owners |
| Recall against known/confirmed incidents (retrospective) | Estimated via red-team exercises and post-incident review, given true incidents are rare and hard to fully enumerate in real time |
| Time-to-human-decision after flagging | Measures the review workflow’s own contribution to total response latency |
| Rate of overturned automated actions on appeal | A direct signal of how well-calibrated the automated response thresholds are |
Because true incidents are rare, teams commonly run periodic red-team exercises — a trusted internal team deliberately simulates realistic compromise-like behaviour against test accounts — specifically to measure recall in a controlled way that real-world incident rates are too sparse to reveal on their own, and to catch detection blind spots before an actual attacker finds them.
“How do you measure recall for a detector when confirmed real incidents are extremely rare?” — combine retrospective analysis of the (small number of) real confirmed incidents with regular controlled red-team simulations against test or volunteer accounts, since waiting for enough naturally occurring incidents to get a statistically meaningful recall estimate could take years.
11.3 Alerting Philosophy
Alerting for this system needs to distinguish sharply between two very different categories of signal: alerts about a specific account’s activity (which route to the review console and reviewer workflow described earlier) and alerts about the health of the detection system itself (which route to the on-call engineering team like any other production service). Conflating these two categories is a common early mistake — a reviewer’s queue should never be interleaved with infrastructure health notifications, and an engineer’s pager should never fire for an individual account’s risk score crossing a threshold. Kept separate, each alerting channel can be tuned appropriately: the account-risk channel optimised for the review team’s triage workflow and case prioritisation, and the system-health channel optimised for the on-call engineer’s standard incident response practices, including clear severity levels and runbooks for common failure modes like ingestion pipeline lag or scoring service error-rate spikes.
Deployment & Cloud
12.1 Model and Threshold Deployment Strategy
Just as with any production ML system, changes to detection models or thresholds go through a staged rollout, but with an added layer of caution appropriate to this domain:
- Offline backtesting: new models or thresholds are evaluated against historical labelled incidents and a large sample of confirmed-legitimate activity to estimate precision/recall before touching production.
- Shadow mode: the new logic scores live activity in parallel without triggering any real action, allowing direct comparison of flag volume and overlap against the current production logic.
- Limited-scope canary: new logic is enabled for actions only up to the “flag for review” tier (never full automated holds) on a small subset of lower-impact accounts first.
- Full rollout with active monitoring: only after the canary period shows acceptable precision does the new logic apply to the full protected population and the highest response tiers.
12.2 Infrastructure Choices
- Multi-region, always-on deployment: given the protected population is globally distributed and active around the clock, the detection pipeline runs active-active across regions rather than following typical business-hours capacity patterns.
- Dedicated, isolated infrastructure: given the sensitivity of this system, it often runs on more tightly access-controlled infrastructure than general-purpose platform services, separate from the broader engineering organisation’s typical deployment pipelines.
- Infrastructure as Code with strict change review: given the security sensitivity, infrastructure changes to this system typically require additional review/approval steps beyond standard code review.
“Why would you restrict a canary rollout to only the ‘flag for review’ tier and not the automated hold tier?” — because an automated hold has a direct, immediate user-facing impact (a legitimate high-profile account owner temporarily can’t post), so any regression in the new logic’s precision is far more costly at that tier; keeping the canary scoped to human-reviewed actions lets you validate the new logic’s quality with a human safety net before trusting it with fully automated consequences.
Databases, Caching & Load Balancing
13.1 Storage Systems Used
| Store | Purpose | Access Pattern |
|---|---|---|
| Per-account baseline store | Rolling behavioural profile per protected account | Low-latency point lookups by account ID on every scored event |
| Raw activity event log | Immutable record of every action from protected accounts | High write reliability, sequential reads for baseline updates and investigation |
| Case management store | Flagged cases, reviewer decisions, appeal history | Moderate volume, strong consistency for audit integrity |
| Incident label store | Confirmed compromise incidents used to improve supervised models | Small, extremely high-value dataset, tightly access-controlled |
13.2 Caching Strategy
- Baseline cache: given the protected population is small, entire baselines can often be kept in a fast in-memory cache, refreshed on write, minimising lookup latency on the hot scoring path.
- Session / device fingerprint cache: recently seen devices/sessions per account cached for quick “have we seen this before” checks without a full store round-trip.
- Cache invalidation: any confirmed incident or manual baseline correction invalidates the relevant account’s cache entry immediately, since correctness here matters more than cache hit rate.
13.3 Load Balancing
Given the relatively low and predictable request volume compared to consumer-facing systems, load balancing here is less about handling extreme scale and more about ensuring reliable, low-latency routing with strong health-checking, so that a single unhealthy scoring instance can never silently become a blind spot in the detection pipeline. Redundant instances across availability zones with aggressive health checks and fast failover are prioritised over raw throughput optimisation.
“Why might you keep the entire baseline dataset in memory here, when you wouldn’t do that for a billion-user system?” — because the protected population is small by design (thousands, not billions), the entire working dataset is small enough to fit comfortably in memory across a modest cluster, making full in-memory caching both feasible and valuable for minimising the scoring path’s latency — a trade-off that wouldn’t be economical at consumer-platform scale.
APIs & Microservices
14.1 Core Service Boundaries
- Activity Ingestion API — internal endpoint the core platform calls to report every action taken by a protected account.
- Risk Scoring Service — exposes
scoreActivity(accountId, activityEvent) → RiskAssessment, called synchronously on the critical path for actions that may need to be held pending the result. - Response Orchestration Service — exposes
decideAction(riskAssessment, accountImpactTier) → ResponseActionand applies the resulting action against the core platform (e.g., placing a hold). - Case Management API — powers the human review console; exposes endpoints to list, claim, and resolve flagged cases.
- Baseline Administration API — tightly access-controlled endpoints for enrolling/offboarding accounts and manually correcting baselines when legitimate behaviour changes are confirmed.
POST /internal/v1/risk/score
Authorization: Bearer {service-token}
Request:
{
"accountId": "acct_88213",
"activityType": "POST_CREATED",
"timestamp": "2026-07-28T03:14:02Z",
"sessionFingerprint": "sess_a91f...",
"contentSummary": { "embeddingId": "emb_5521..." }
}
Response 200:
{
"riskScore": 78.4,
"riskTier": "HIGH",
"topContributingFactors": ["session_device_new", "unusual_hour"],
"recommendedAction": "REQUIRE_STEP_UP_AUTH"
}
The risk scoring call sits directly on the critical path for actions that may need to be held, so it’s typically implemented as a synchronous, low-latency internal gRPC call with an aggressive timeout and a safe default (escalate to review) if that timeout is exceeded — consistent with the “fail toward caution” principle established in Chapter 9.
“Should the risk scoring call block the post from going live, or should posts go live immediately and get evaluated asynchronously?” — this depends on account impact tier: for critical-impact accounts, a brief synchronous hold (checking risk before the post is publicly visible) is usually justified given the outsized damage potential, while for lower-impact accounts in the protected program, asynchronous post-publication scoring with rapid follow-up action may be an acceptable trade-off to avoid adding latency to every single post.
14.2 API Versioning and Backward Compatibility
Because the response orchestrator, review console, and various platform integrations all depend on the risk scoring API’s response shape, this internal contract needs the same disciplined versioning practices as any externally facing API, even though it’s never called by third parties. Introducing a new contributing-factor field or changing a risk tier’s threshold semantics should go through additive, backward-compatible changes wherever possible, with a clear deprecation path and a defined migration window for any breaking change consumed by the response orchestrator or the review console. Given how many downstream automated actions key off this API’s output, an unannounced or poorly coordinated contract change here carries outsized risk relative to a typical internal API — a subtle field-meaning change could silently alter which accounts get held or reviewed without any code actually breaking, which is a much harder class of bug to catch than a straightforward compile-time or runtime failure.
Design Patterns & Anti-patterns
15.1 Patterns Worth Knowing
Per-Entity Baselining
Compare each account against its own history rather than a global norm — the foundational pattern of this entire system.
Ensemble Risk Scoring
Combine statistical, unsupervised, rule-based, and supervised signals rather than relying on any single detection method.
Human-in-the-Loop Escalation
Route uncertain or high-impact cases to trained reviewers rather than fully automating consequential decisions.
Tiered Response Ladder
Match the severity of the response to both the confidence of detection and the potential impact of the account.
Fail-Safe Defaults
When any component is uncertain or unavailable, default toward caution (escalate) rather than toward convenience (allow).
Closed Feedback Loop
Human review decisions continuously refine baselines and models, rather than treating detection logic as static.
15.2 Anti-patterns to Avoid
Global One-Size-Fits-All Thresholds
Applying the exact same sensitivity to every account ignores the wide legitimate behavioural variance between different accounts and different account types.
Full Automation with No Human Review
Given rare incidents and high false-positive cost, removing the human safety net trades a small efficiency gain for outsized reputational risk.
Static Baselines That Never Update
An account’s genuine behaviour evolves over time; a frozen baseline eventually generates constant false positives against normal drift.
Opaque Scoring with No Explainability
A risk score with no visible contributing factors is nearly impossible for a human reviewer to evaluate quickly and confidently.
Generic Fraud-Detection Clone
Copying a payment-fraud system’s assumptions wholesale ignores this domain’s distinct cost asymmetries (small population, high per-incident stakes, high false-positive reputational cost).
“Why is ‘fail-safe defaults’ listed as a pattern here specifically, rather than just a general reliability best practice?” — because the direction of the safe default is domain-specific and non-obvious: most systems fail toward availability/convenience, but this system deliberately fails toward caution/escalation, since the cost of a missed real incident for a high-impact account outweighs the cost of a brief extra review step in the vast majority of ambiguous cases.
Best Practices & Common Mistakes
16.1 Best Practices
- Always baseline per-account, never against a global population norm — legitimate variance between accounts is enormous.
- Design the response ladder around both detection confidence and account impact tier, not detection confidence alone.
- Treat human review as a core architectural component with its own capacity planning, not an informal afterthought.
- Make every automated decision explainable — show reviewers (and, on appeal, account owners) the specific contributing factors, not just a bare score.
- Run regular red-team exercises to measure recall, since real incidents are too rare to rely on for statistically meaningful measurement alone.
- Default to caution (escalate) rather than convenience (allow) whenever any part of the pipeline is degraded or uncertain.
16.2 Common Mistakes
Under-Investing in Human Review Workflow
Over-investing in the automated detection model while under-investing in the human review pipeline — both halves of the system are equally load-bearing.
Single Global Sensitivity Threshold
Setting one global sensitivity threshold instead of tuning per account-impact tier, ignoring how different the cost structure is for different account categories.
Unbounded Baseline Drift
Failing to bound how quickly baselines can drift, leaving the system vulnerable to slow, gradual behavioural manipulation.
Cold-Start After Every Incident
Not preserving pre-incident baseline history through a confirmed compromise and recovery, forcing an unnecessary full cold-start after every incident.
Precision Without Estimating Recall
Measuring only precision (false-positive rate) without also actively estimating recall via red-teaming, leading to a false sense of security.
“If you had to launch a minimum viable version of this system in one quarter, what would you prioritise?” — a strong answer keeps a small set of high-confidence rule-based checks (new device + unusual hour + first-time-seen network) plus a functioning human review queue as the non-negotiable core, deferring the more sophisticated unsupervised/supervised ML scoring and fine-grained per-account statistical baselining to later iterations, since the rule-based core alone with solid human review already captures a meaningful share of real-world value.
16.3 Balancing Automation Against Institutional Trust
A subtler best practice worth naming explicitly: every automated action this system takes against a real, named public figure’s account carries institutional trust cost, independent of whether the action turns out to be correct. A pattern of visible, even if individually well-justified, holds on a particular account’s posting can itself become a story, a source of friction with that individual or organisation, or a point of public criticism regarding platform bias. This doesn’t mean avoiding automated action — it means that response-ladder design should treat the reputational cost of automated intervention as its own explicit variable, not just an implicit assumption, and that decisions to expand automated (rather than human-reviewed) response for a given account or account category deserve periodic, deliberate re-evaluation rather than a one-time initial setting left unchanged indefinitely.
Real-World / Industry Examples
17.1 Major Social Platforms — Verified/High-Profile Account Protection Programs
Large platforms with verification systems generally run dedicated protection programs for government, media, and high-follower accounts, combining stronger baseline authentication requirements (mandatory hardware-key-based multi-factor authentication for the highest-risk accounts) with the kind of behavioural anomaly detection described throughout this tutorial, reflecting the industry consensus that authentication strength alone is not sufficient — behavioural monitoring of already-authenticated sessions is a necessary second layer.
17.2 Financial Services — Account Takeover Detection
Banks and payment processors have run mature behavioural anomaly detection for account takeover far longer than social platforms, using very similar architectural patterns: per-account baselining, multi-signal risk scoring (device, location, transaction pattern), and tiered response (additional verification step, temporary hold, fraud team escalation). The core architectural pattern — per-entity baseline, ensemble scoring, tiered proportional response, human-in-the-loop for ambiguous cases — transfers directly across these domains, which is a useful lens for an interview answer: this is fundamentally the same problem shape as payment fraud detection, applied to a different action (posting content) and a different, smaller, higher-stakes population.
17.3 Cloud Identity Providers — Impossible Travel and Anomalous Login Detection
Enterprise identity and access management systems widely implement “impossible travel” detection (flagging a login from a location that couldn’t plausibly be reached given the time since the account’s last activity elsewhere) as one specific, well-known instance of the broader session/device deviation signal category described in Chapter 5 — a good concrete example to cite when discussing the session/device feature family in an interview setting.
Every mature version of this problem — social account protection, payment fraud, enterprise identity security — converges on the same shape: per-entity behavioural baselining, multi-signal ensemble risk scoring, a tiered proportional response ladder, and a human-in-the-loop safety net for ambiguous or high-impact cases. This convergence across otherwise very different industries is a strong signal that this architecture reflects the genuine underlying constraints of the anomaly-detection-plus-consequential-action problem shape, not a coincidence of any one company’s implementation choices.
“What can this problem learn from payment fraud detection, and where does it differ?” — it borrows the core architecture (per-entity baselining, ensemble scoring, tiered response) almost directly, but differs in population size (thousands of known accounts vs an entire user base), cost asymmetry (reputational cost of a false lockout on a public figure vs financial cost of a missed fraudulent transaction), and the nature of the “damage” (rapid public information spread vs direct financial loss) — differences that should visibly shape threshold tuning and response design even though the skeleton is the same.
17.4 Lessons from Physical Security and Executive Protection
An instructive, less obvious analogy comes from physical executive protection practices, which long predate any of these digital systems. Security details protecting public figures don’t treat every situation identically — protection intensity scales with a documented, continuously reassessed threat and impact profile, routine behaviour is carefully learned and monitored specifically so that deviations become noticeable, and response protocols are pre-defined and tiered rather than improvised in the moment. The digital account-protection system described in this tutorial is, in essence, translating decades of established executive-protection doctrine into software: baseline the protected individual’s normal pattern of life (here, posting behaviour instead of physical movement), watch continuously for meaningful deviation, and respond with an intensity proportional to both the confidence and the stakes of what’s been observed. This framing is a useful one to raise in an interview setting, because it signals an understanding that the underlying problem — protecting a high-value, high-visibility individual from a fast-moving threat — is much older than social media, and that mature solutions to it share a common shape regardless of the specific technology involved.
Frequently Asked Questions
No — spam/bot detection typically looks for patterns common across many fake or automated accounts at platform scale. This system instead asks a narrower question about a single specific, usually genuinely human-run, high-profile account: does this new activity look like this particular account’s own established pattern, regardless of whether it superficially resembles “spam” at all.
Baselines are built from a meaningfully long historical window and include natural variance (different time zones, occasional bursts around major events) rather than a rigid single “typical” value; borderline cases route to human review rather than automatic action, so genuinely unusual-but-legitimate activity gets a human’s judgement rather than an automatic block.
This is typically a trust & safety policy decision based on factors like follower count, verification status, and account category (government, major media, extremely high public visibility) rather than a purely technical decision, and the enrolled list is treated as sensitive, access-controlled information.
Not on content alone, and it’s not designed to try — the system flags behavioural and contextual anomalies (device, session, timing, stylistic shift) as signals warranting human review, not as a judgement on content itself; the actual determination of “compromised vs just unusual” is made by a trained human reviewer, not the automated scoring layer.
Typically yes for actions that visibly affect them (a required step-up authentication, a temporary hold) — transparency to the legitimate owner matters both for usability and for giving them a fast, clear path to resolve a false positive, though the underlying detection logic and thresholds themselves generally remain confidential to prevent easy evasion.
A flagged case is simply activity that crossed a risk threshold and warranted review; a confirmed incident is a case a human reviewer has verified as an actual compromise. Only confirmed incidents become labelled training examples and trigger the baseline-exclusion and recovery workflow described in Chapter 6 — the distinction matters because treating every flag as confirmed would badly corrupt the training data used to improve the system.
18.1 Glossary of Key Terms
| Term | Plain-English Meaning |
|---|---|
| Account takeover (ATO) | Someone other than the legitimate owner gaining control of an account and using it, typically without authorisation. |
| Behavioural baseline | A statistical summary of an account’s normal, historical activity patterns, used as the reference point for spotting anomalies. |
| Anomaly detection | The general technique of flagging data points that deviate significantly from an expected or learned pattern. |
| Cohort baseline | A baseline built from a group of similar accounts, used as a starting reference point before an individual account has enough history of its own. |
| Risk tier / impact tier | A classification (e.g., low/medium/high/very high for risk; standard/elevated/critical for impact) used to decide how seriously to treat a given signal or account. |
| Human-in-the-loop | A design where a trained person reviews and confirms consequential automated decisions rather than letting the system act fully independently. |
| Fail-safe default | The behaviour a system falls back to when it can’t confidently make a normal decision — here, defaulting toward caution/escalation rather than convenience. |
| Red-team exercise | A controlled, deliberate simulation of attack-like behaviour used to test and measure a detection system’s effectiveness. |
Summary & Key Takeaways
Key Takeaways
- This is fundamentally an anomaly detection and risk-scoring problem, not a raw scaling problem — the protected population is small (thousands), but the per-decision stakes are high, which should shape every design choice differently from a typical billion-user system.
- Per-account behavioural baselining — comparing each account against its own history, not a global norm — is the foundational technique that makes accurate detection possible.
- Detection combines multiple signal families (temporal, content/style, session/device, interaction) and multiple methods (statistical, unsupervised ML, rule-based, supervised ML) into an ensemble risk score, because real compromises are usually characterised by several mildly unusual signals occurring together.
- A tiered, proportionate response ladder — scaled by both detection confidence and account impact — avoids the false choice between “do nothing” and “always fully block.”
- Human-in-the-loop review is a core architectural component, not an optional add-on, given the rarity of true incidents, the reputational cost of false positives, and the central role review decisions play in continuously improving the system.
- The system should fail toward caution, not convenience — the opposite default from most consumer-facing systems — because the cost asymmetry between a missed real incident and a brief extra review step favours caution for this specific problem.
- The same core architecture — per-entity baselining, ensemble scoring, tiered response, human-in-the-loop — appears independently across social platforms, financial fraud detection, and enterprise identity security, suggesting it reflects the genuine shape of the underlying problem rather than any single implementation’s idiosyncrasies.
- Success is measured with both precision and recall in mind simultaneously, using red-team exercises to estimate recall given how rare real confirmed incidents naturally are.
Designing a compromised-account detection system is ultimately an exercise in calibrated judgement under uncertainty: building enough confidence from behavioural signals to act quickly, while respecting how costly it is to act wrongly against a real, legitimate, high-profile account owner. The architecture — baselining, ensemble scoring, tiered response, human review — exists entirely in service of getting that calibration right at the speed the problem demands.