← Emberlern/The Data Diaries With Dee
How to Be a Top-Notch Data Analyst / Module 04

Cleaning data, and knowing when to trust it

Five lessons, about 50 minutes. Sahara Mart's stock-adjustment log is genuinely messy — this module gives you the discipline to find what's wrong before it corrupts a headline number.

Module 4 practice workbook A real branch stock-adjustment export with nine planted data-quality problems already inside it — the exact kind this module names. Find them, log each one, and calculate the corrected headline number. One .xlsx, four tabs.
Lesson 4.110 minutes

Profiling a dataset in ten minutes

Before cleaning anything, spend ten minutes just looking at the shape of a new dataset. This is not optional polish — it's how you find most of the nine kinds of problem this module names, before you've built a single formula on top of broken data.

A ten-minute profiling pass
  1. Row count. Does it match what you'd expect given the date range and typical volume?
  2. Column-by-column: nulls, ranges, distinct values. A quantity column with a maximum a hundred times the median is worth a second look before you trust it.
  3. Distinct values in any "category" style column. If Branch has fourteen distinct values but Sahara Mart has nine branches, five of them are almost certainly the same branch spelled differently.
  4. Duplicate rows. Compare total row count against distinct-row count across every column — a gap means something is duplicated.

Ten minutes of profiling before you start is nearly always cheaper than discovering the same problems one at a time, an hour into building on top of them.

Keep this
  • Profile row counts, ranges, nulls and distinct values before writing a single formula against new data.
  • A category column with more distinct values than the real-world count it represents almost always means inconsistent spelling, not real new categories.
Lesson 4.211 minutes

Duplicates that aren't, and vice versa

"Remove duplicates" sounds like a mechanical step. In practice, it's a judgement call every time, and getting it backwards in either direction quietly changes a headline number.

Two rows that look identical, aren't the same thing
A real duplicate

Same branch, same SKU, same date, same quantity, same reason, same person entering it. This is one event, entered twice — almost certainly a real double-entry. Drop one.

Not a duplicate, despite looking like one

Same branch, SKU, date and quantity — but the reason text describes two genuinely separate incidents that happened to involve the same product on the same day. Keep both.

The only reliable way to tell them apart is reading the row, not just comparing columns mechanically. A "remove duplicates" button applied blindly to every matching column will get this wrong in both directions across a large enough dataset — sometimes deleting a real second event, sometimes leaving a genuine double-entry standing because one field happened to differ by a typo.

The inverse trap: near-duplicates that ARE the same event

"Lekki" and "lekki" describing the same transaction on the same day are the same real event, recorded with inconsistent casing — this is a duplicate that superficial column-matching would miss entirely, because the text doesn't match exactly.

Keep this
  • Two identical-looking rows are not automatically duplicates — read the row before deciding.
  • Two differently-formatted rows can still be the same real event — standardise text (branch names, casing) before deduplicating, not after.
Lesson 4.310 minutes

Dates, time zones, and the silent type error

Two failure modes recur constantly in Sahara Mart's data: a date stored in a different text format than the rest of the column (which will sort and filter incorrectly the moment it's mixed with the rest), and a number stored as text (which most aggregate functions silently skip rather than error on).

Why the type error is worse than it sounds

A quantity column where one value is the text "3" instead of the number 3 looks completely normal displayed on screen. A SUM formula run over that column simply excludes it — no error, no warning, just a total that's quietly three units short. This is far more dangerous than a value that's obviously wrong, precisely because nothing about it looks wrong.

The fix is a habit, not a one-time cleanup: after importing any new data source, explicitly check and set the type of every date and numeric column, rather than trusting whatever type the import happened to guess.

Keep this
  • A number stored as text is silently excluded from most aggregates — check column types explicitly after every import, don't trust auto-detection.
  • A date in a different text format from the rest of its column will sort and filter wrong the moment it's mixed with the rest.
Lesson 4.410 minutes

Deciding what to drop, what to fix, and what to flag

Every problem you find sorts into one of three bins, and picking the wrong bin is its own kind of error even when you've correctly spotted the problem.

DropAn exact duplicate row. There is no information lost by removing it — the second copy adds nothing.
FixA value with an unambiguous correct form — a date in the wrong text format, a number stored as text, trailing whitespace, inconsistent branch-name casing. The correction is mechanical once identified.
FlagAnything where the "correct" value isn't actually knowable from the data alone — a missing SKU, an outlier that might be real or might be an error, a sign that doesn't match its stated reason. These need a human decision, not a formula.

The temptation with a "flag" problem is to quietly guess a fix so the workbook looks finished. Resist it — a wrong guess presented as a clean number is worse than an honest gap that's visibly marked as unresolved.

Keep this
  • Drop only what's truly redundant; fix only what has one unambiguous correct value; flag everything else for a human decision.
  • A confidently wrong guess is worse than a visibly flagged gap — never silently resolve genuine uncertainty.
Lesson 4.59 minutes

Writing down your assumptions so the number can be defended later

Six months from now, someone — quite possibly you — will ask why August's stock-adjustment total is what it is. If every cleaning decision you made lives only in your memory, that question has no good answer. A short, plain-language cleaning log fixes this permanently.

A cleaning log entry, done properly

"Row 29 (Garki, qty -60,000): treated as a data entry error, corrected to -6 based on comparison with the branch's typical adjustment range for that SKU. Original value preserved in a separate column. — analyst, 24 Aug."

That single entry does everything a defensible correction needs: states what was changed, states the reasoning, and preserves the original so the decision can be reviewed or reversed later. A headline number with no cleaning log behind it is a number nobody downstream can actually trust, whether or not it happens to be correct.

Keep this
  • Log every real correction: what changed, why, and what the original value was — not just the final "clean" number.
  • A number without a defensible cleaning log behind it can't really be trusted, however correct it happens to be.
Before the drill

Have a spreadsheet tool open — this drill is hands-on with a real file, not a written exercise.

Module 4 drillabout 60 minutes

Nine planted problems, one headline number

The practice workbook's "Raw Data" tab contains a real branch stock-adjustment export with nine planted problems, covering every category this module named: an exact duplicate, a near-duplicate that isn't, a near-duplicate that is, a wrong date format, an outlier, a type error, a sign mismatch, a missing key field, and an obvious data-entry mistake.

  1. Find and log all nine on the Findings Log tab — what's wrong, and what you'd do about it (drop, fix, or flag).
  2. Calculate the total quantity adjusted, uncorrected, straight from the raw data.
  3. Calculate it again after your corrections, and note how much it moved and in which direction.
  4. Write one paragraph you'd send Chidi explaining the corrected number and naming anything you flagged rather than resolved.
What this drill is really testing

Whether you can tell the difference between the categories from lessons 4.2 and 4.4 under real conditions — a duplicate that isn't, a fix that's actually a flag in disguise — rather than mechanically "cleaning" everything the same way. There is a worked answer key on the fourth tab; do the drill first.

Before module 5

Keep your corrected total and your reasoning — module 5 asks whether a change like this is actually statistically meaningful, or well within normal week-to-week noise.