Bits of .NET

Accidentally committed API keys or passwords? Deleting the file in a new commit doesn’t remove it from history. Here’s how to purge it completely.

⚠️ The Problem:

# You committed config.json with API key
git add config.json
git commit -m "Add config"
git push

# Then realized mistake and deleted it
git rm config.json
git commit -m "Remove sensitive data"
git push

# ❌ API key still in history!
# Anyone can:
git checkout abc1234~1  # Go to commit before deletion
cat config.json         # See your API key!

The Nuclear Option – BFG Repo-Cleaner:

# Install BFG
# Mac: brew install bfg
# Windows: Download from https://rtyley.github.io/bfg-repo-cleaner/

# Clone fresh copy
git clone --mirror https://github.com/yourname/yourrepo.git

# Remove file from entire history
bfg --delete-files config.json yourrepo.git

# Or remove text pattern (API keys, passwords)
bfg --replace-text passwords.txt yourrepo.git

# passwords.txt contains:
# API_KEY=abc123  ==> API_KEY=***REMOVED***
# PASSWORD=secret ==> PASSWORD=***REMOVED***

# Clean up
cd yourrepo.git
git reflog expire --expire=now --all
git gc --prune=now --aggressive

# Force push (rewrites history!)
git push --force

# ⚠️ Everyone must re-clone repository after this

Manual Method – Git Filter-Branch:

# Remove file from all commits
git filter-branch --force --index-filter \
  "git rm --cached --ignore-unmatch config.json" \
  --prune-empty --tag-name-filter cat -- --all

# Explanation:
# --index-filter: Run on staging area (faster than --tree-filter)
# git rm --cached: Remove from index
# --ignore-unmatch: Don't fail if file doesn't exist in that commit
# --prune-empty: Remove commits that become empty
# --all: Apply to all branches

# Clean up
git reflog expire --expire=now --all
git gc --prune=now --aggressive

# Force push
git push origin --force --all
git push origin --force --tags

Remove Specific Text Pattern:

# Replace API key in all files throughout history
git filter-branch --tree-filter \
  "find . -type f -exec sed -i 's/sk-abc123xyz456/***REDACTED***/g' {} +" \
  HEAD

# This finds ALL occurrences of API key and replaces with ***REDACTED***

Modern Alternative – Git Filter-Repo:

# Install
pip install git-filter-repo

# Remove file
git filter-repo --path config.json --invert-paths

# Remove text pattern
git filter-repo --replace-text <(echo 'API_KEY=abc123==>API_KEY=***REMOVED***')

# Much faster than filter-branch!

Check What Will Be Removed:

# See all commits that touched the file
git log --all --full-history -- config.json

# See file size throughout history
git rev-list --objects --all | grep config.json

# List largest files in history
git rev-list --objects --all | \
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | \
  sed -n 's/^blob //p' | \
  sort -k2 -n -r | \
  head -20

# This shows what's eating repository space

After Purging – Team Communication:

# Email team:
"Repository history has been rewritten to remove sensitive data.

Everyone must:
1. Commit and push any local changes
2. Delete local repository
3. Clone fresh copy:
   git clone https://github.com/yourname/yourrepo.git

Do NOT merge or pull - you'll bring back the removed data!"

# Team members run:
cd ~/projects
rm -rf yourrepo
git clone https://github.com/yourname/yourrepo.git

Verify Removal:

# Search entire history for sensitive string
git log --all --source -S 'API_KEY=abc123'

# If returns nothing = successfully removed

# Or check specific file doesn't exist anywhere
git log --all --full-history -- config.json

# Should return nothing if file removed

Prevention – .gitignore:

# Add to .gitignore BEFORE committing
echo "config.json" >> .gitignore
echo ".env" >> .gitignore
echo "*.key" >> .gitignore
echo "secrets/" >> .gitignore

git add .gitignore
git commit -m "Add .gitignore"

# If you already staged sensitive file:
git rm --cached config.json  # Remove from index, keep file locally

Template .gitignore for Secrets:

# Secrets
.env
.env.local
*.key
*.pem
config.json
secrets.yml
credentials.txt

# Cloud provider
.aws/credentials
.gcloud/key.json
azure.json

# IDEs
.idea/
.vscode/settings.json

# OS
.DS_Store
Thumbs.db

Emergency: Invalidate Leaked Secrets Immediately:

If API key was exposed:
1. Rotate/delete API key FIRST (before cleaning history)
2. Then clean Git history
3. Generate new API key
4. Update production secrets

Even after removing from Git, assume key is compromised!
Attackers may have already cloned repository.