PhishingNet

PhishingNet: privacy policy and project documentation

View the Project on GitHub dubaiplayer/TIS

CLAUDE.md — Phishing Email Analysis Tool

Project context for Claude Code. Read this first.

What we’re building

A “Grammarly for phishing”: read raw email text and return a structured, explainable breakdown of manipulation/phishing attributes — each with a score and highlighted evidence spans — plus an overall risk score/verdict. Priority is interpretability (every score traceable to evidence), not a black-box classifier.

Current status (2026-07-07)

Real-world hardening (2026-09-25) — READ BEFORE TUNING DETECTION

Two failures that the held-out test set (F1 0.98) could not see, both confirmed against the live tool and both now fixed and regression-tested:

  1. False positives on modern transactional mail. A genuine Google Payments receipt scored PHISHING 0.66; a genuine Atlassian subscription notice PHISHING 0.60 with the classifier at 0.15. Causes: (a) lift-only lexicon validation kept everyday business words (“payment”, “invoice”, “usd”, “deactivated”, “sign in”) as full-weight evidence — invisible on a corpus whose legit class is Enron internal mail; (b) noisy-OR is linear in the classifier probability, so an out-of-distribution email scoring an unsure 0.52 still cast a 0.50 vote.
  2. Plain-wording fraud waved through. The textbook CEO gift-card scam scored bec=0.89 and was reported LEGITIMATE at risk 0.068, because every real-world detector was report_only. This is the “spam walks past the TF-IDF model because it reads like a normal email” case — and it does, because there is no phishing vocabulary in it to find.

The four fixes, each with its measurement in the file that carries it:

Cost, measured honestly: in-distribution F1 on a 3,000-email test sample goes 0.9930 -> 0.9883 (recall 0.9930 -> 0.9844). That is the price of the real-world result, and it is the right trade — but do not report the old F1 alongside the new behaviour.

Before changing any of this, run BOTH suites. pytest -q and python -m evaluate_realworld. The second is the one that catches these. tests/fixtures/realworld_cases.py is where new cases go; see PILOT.md for how real corrections become fixtures.

Brand impersonation (2026-10-02)

Aimed at fluent, AI-written lures, where the wording gives nothing away. It ignores wording entirely: claimed brand vs provable sender.

Team pilot telemetry

server/telemetry.py + /stats, /export, /feedback, /dashboard. Off unless TELEMETRY_ENABLED=1 AND the extension has a team code. Records verdicts, signals and sender domains; never subject, body, links or addresses. The boundary is enforced by the privacy audit in scripts/seed_demo_telemetry.py — run it after any change to the telemetry schema. Full rollout guidance in PILOT.md.

Environment

Dataset

Location: C:\Users\15DGupta\Downloads\archive (4). We use 6 per-source files, merged by column name (column order differs between files — never merge by index).

Source Rows Labels Casing In merge
Enron_raw.csv (NEW) 15,800 0 only (legit) preserved ✅
CEAS_08.csv 39,154 0 + 1 preserved ✅
Ling.csv 2,859 0 + 1 flattened (lowercased) ✅
Nazario.csv 1,565 1 only preserved ✅
Nigerian_Fraud.csv 3,332 1 only preserved ✅
SpamAssasin.csv 5,809 0 + 1 preserved ✅
Enron.csv (old) 29,767 0+1 flattened ❌ excluded
phishing_email.csv 82,486 0+1 flattened ❌ excluded

Labels: 0 = legitimate, 1 = phishing. Text column is body (+ subject); some files also have sender/receiver/date/urls. urls is a has-URL 0/1 flag, NOT a label.

Why Enron was re-sourced (key decision)

The originally packaged Enron.csv was pre-lowercased and punctuation-stripped, which flattened all casing/tone/urgency signal for the legit class and confounded those features with source+label. We re-downloaded the raw Kaggle Enron dump (emails.csv, wcukierski), parsed headers→body, sampled 15,800 (seed 42), labeled 0, and saved Enron_raw.csv with schema subject,body,label.

Verified fix (scripts/casing_check.py, mean uppercase-letter ratio): old Enron 0.00% → Enron_raw 11.78% — now in the band of the other casing-preserving sources (CEAS 8.84%, Nazario 9.18%, SpamAssassin 8.94%).

Residual dataset caveats (keep in the data card)

Two-track corpus (core methodology)

Planned repo layout

TIS/
  requirements.txt, README.md, DATA_CARD.md
  data/processed/                  # versioned cleaned outputs (parquet + csv) + manifest.json
  scripts/
    build_enron_raw.py             # DONE — builds Enron_raw.csv
    casing_check.py                # DONE — per-source casing/punctuation report
  phishing_analyzer/
    data_prep.py                   # merge (by name) + two-track split + save
    explore.py                     # Step-1 checkpoint report
    text_clean.py                  # HTML strip / header-remnant strip / whitespace (preserve casing+punct)
    lexicons/                      # seed lexicons, data-validated by frequency lift
    attributes/                    # one module per attribute (below), each ->
                                   #   AttributeResult{score, label, explanation, evidence_spans}
    classifier.py                  # TF-IDF + LogisticRegression
    risk.py + weights.yaml         # explicit, adjustable score combination
    schema.py                      # pydantic JSON output models
    cli.py                         # demo: raw email -> JSON + highlighted terminal view
  evaluate.py                      # metrics + attribute validation
  tests/                           # pytest per attribute

Attribute schema (Step 2 — finalized after Step-1 checkpoint)

Each attribute is an independently testable module returning AttributeResult{score∈[0,1], label, explanation, evidence_spans:[{text,start,end}]}. evidence_spans are character offsets into the original text (drive the highlight UI).

Attribute Method Track
Urgency / time pressure Lexicon + regex (deadlines, “act now”) A phrases, B caps intensity
Emotional manipulation (fear/threat/reward/curiosity) Sub-lexicons per emotion A
Authority impersonation Lexicon (bank/IT/gov/exec) + sender cross-check A + sender
Financial request Lexicon (wire, gift card, invoice) + IBAN/BTC regex A
Credential/data harvesting Lexicon (verify/login/SSN/password) + URL/form cues A + links
Generic vs personalized greeting Regex (“Dear customer” vs a name) A
Sender-domain mismatch / spoofing Parse display-name vs domain; freemail-as-brand; lookalike (tldextract + edit distance) sender-present sources only
Suspicious links Shorteners, odd TLDs, IP-literal, raw-URL count; anchor/href only if HTML present A
Grammar/spelling anomalies pyspellchecker ratio + heuristics B (casing matters)
Caps/punctuation tone CAPS ratio, !/? bursts B only
Content classifier TF-IDF + LogisticRegression A

Lexicons: seed by hand, then validate against real phishing samples (per-term phishing-vs-legit frequency lift on Track A); drop no-lift terms, surface missed high-lift terms. No blind keyword guessing.

Risk combination (Step 3)

Decision: the TF-IDF + LogReg classifier is a CO-EQUAL signal with the rule-based attributes in the final risk score (not optional, not merely a backstop). Explicit, adjustable weighted aggregation of attribute scores + classifier probability, weights in weights.yaml. Two modes: (a) transparent hand-set weights; (b) optional meta-LogisticRegression over attribute scores + classifier prob (still interpretable via coefficients). Always output contributing signals + the classifier’s top contributing n-grams — never a fully opaque number.

Leakage guardrails (because ML now carries real weight): source correlates hard with label here (Enron=100% legit), so the classifier could inflate accuracy by learning “corporate Enron style = safe”. Mitigations: (1) report per-source metrics in Step 4 and specifically inspect Enron false-negative behavior; (2) prefer char+word n-grams and inspect top coefficients for source-identifying tokens (names, “enron”, signature artifacts) — down-weight/blocklist if found; (3) keep the classifier weight adjustable in weights.yaml so it can be dialed back if evaluation shows it’s riding source rather than phishing signal.

Evaluation (Step 4)

Output format (Step 5)

{
  "overall": {"risk_score": 0.0, "verdict": "phishing|suspicious|legitimate",
              "top_signals": ["urgency", "financial_request"]},
  "attributes": [
    {"name": "urgency", "score": 0.0, "label": "...", "explanation": "...",
     "evidence_spans": [{"text": "act now", "start": 42, "end": 49}]}
  ],
  "meta": {"sender_analyzed": true, "notes": ["no HTML -> anchor/href check skipped"]}
}

CLI demo (cli.py): raw email from --file/stdin → JSON + highlighted terminal view.

Conventions / gotchas for anyone editing

How to run (so far)

python scripts/build_enron_raw.py   # -> archive (4)/Enron_raw.csv  (seed 42, 15,800 rows)
python scripts/casing_check.py      # per-source casing/punctuation report