← Emberlern/The Data Diaries With Dee
How to Be a Top-Notch Data Analyst / Module 07

Python where it beats the alternatives

Seven lessons, about an hour and 15 minutes. You'll take module 2's manual Excel report and turn it into a script that runs itself.

Python starter kit A real 900-row Sahara Mart export, a working starter script that loads and cleans it with pandas, and a README walking through exactly what to extend. Actually run it — this is code, not a reading exercise. One .zip: raw_sales.csv, clean_and_report.py, requirements.txt, README.
Lesson 7.19 minutes

When Python is the right tool, and when a spreadsheet wins

Python is not a strictly better version of Excel — it's a different tool, right for different jobs. Reaching for it out of habit, on a task Excel handles fine, wastes time relearning syntax for no real benefit.

Python wins whenThe same cleaning steps need to run on a new file every week (module 2's Power Query problem, but for cases Power Query can't reach); the data is too large to comfortably open in Excel; the logic is genuinely complex — multi-step joins, custom rules — and needs to live somewhere reviewable and testable.
Excel wins whenChidi or a branch manager needs to open the result themselves and poke at it; the task is a one-off; the audience for the output isn't technical.

The question isn't "which tool is more powerful." It's "which tool gets this specific job done and handed off cleanly." Often that's still Excel.

Keep this
  • Reach for Python when a task repeats, scales beyond comfortable spreadsheet size, or needs logic too complex to trust in formulas.
  • Reach for Excel when the output needs to be opened and manipulated directly by someone non-technical.
Lesson 7.213 minutes

pandas: load, filter, group, and the shape of a dataframe

A pandas DataFrame is the Python equivalent of an Excel Table from module 2 — rows and named columns, and every operation you already know from SQL and Excel has a close pandas equivalent. Open the starter kit's clean_and_report.py alongside this lesson.

The core four operations
Loaddf = pd.read_csv('raw_sales.csv', parse_dates=['date']) — parse_dates matters, the same way lesson 4.3's type-checking mattered in Excel.
Filterdf[df['transaction_type'] == 'SALE'] — the pandas equivalent of a WHERE clause.
Groupdf.groupby('branch')['revenue'].sum() — the pandas equivalent of SQL's GROUP BY, or Excel's pivot table.
Inspect the shapedf.shape, df.head(), df.dtypes — module 4's profiling habit, in three lines.

Notice how much of this module is really module 1 through 6's habits, expressed in a different syntax — know the grain, filter honestly, check types, aggregate deliberately. The tool changes far more than the thinking does.

Keep this
  • Every core pandas operation has a direct equivalent you already know from Excel or SQL — map the syntax onto concepts you already have.
  • Check df.dtypes after loading, the same discipline as module 4's explicit type-checking.
Lesson 7.312 minutes

Joining and reshaping without losing rows

Lesson 3.3's fan-out join trap applies exactly as much in pandas as in SQL. pd.merge(sales, products, on='sku') can silently multiply rows if products has duplicate SKUs — the fix is the same habit: check len(sales) before and after the merge.

A safer merge, checked

before = len(sales)
merged = pd.merge(sales, products, on='sku', how='left')
assert len(merged) == before, "merge changed row count — check for duplicate SKUs"

The assert isn't decoration — it stops the script outright, loudly, the moment a fan-out happens, rather than letting a quietly-inflated total flow into the rest of the report.

how='left' matters too: it keeps every row from sales even when a SKU has no match in products, rather than silently dropping transactions the way an inner join would — the pandas equivalent of choosing carefully between an INNER and LEFT JOIN in SQL.

Keep this
  • Check row counts before and after every merge — the fan-out trap from SQL applies identically here.
  • Choose how='left' deliberately when you need to keep every row from your main table, even ones with no match.
Lesson 7.410 minutes

A cleaning script you can rerun next month

The starter kit's load_and_clean() function is deliberately separated from weekly_summary() — this is the single habit that makes a script reusable rather than a one-off notebook of commands run in whatever order you happened to type them.

A function with a clear name, a defined input, and a defined output can be rerun on next week's file by changing exactly one line — the file path — rather than re-deriving the whole sequence of steps from memory. This is Power Query's "record once, replay on refresh" idea, written as code instead of clicked as steps.

The test that separates a real script from a scratch notebook

Could someone else — or you, in three months — run this on a brand-new file without reading every line first? If yes, you've built something reusable. If the answer is "they'd need to understand my whole thought process first," it's still a scratch notebook wearing a .py extension.

Keep this
  • Separate cleaning logic into a named function with a clear input and output, not inline steps run top to bottom.
  • A reusable script survives being run by someone else, cold, on new data — that's the actual bar.
Lesson 7.59 minutes

Quick charts for exploring, not for presenting

A one-line df.groupby('date')['revenue'].sum().plot() is for you, right now, spotting a pattern while you're still exploring — not for Chidi. Module 9's lesson on charts that argue is about deliberate, polished presentation; this is the opposite instinct, deliberately rough and fast.

Don't spend ten minutes styling a chart you're going to look at once and delete. If a quick matplotlib plot reveals something worth showing Chidi, that finding gets rebuilt properly — in Power BI, or as a deliberately designed chart — using module 9's discipline, not shipped as-is from an exploratory script.

Keep this
  • Exploratory charts are for speed, not polish — don't over-invest in something meant to be looked at once.
  • A finding worth sharing gets rebuilt deliberately for its actual audience, not shipped straight from exploration.
Lesson 7.611 minutes

Automating a recurring report end to end

Module 2's manual Monday-morning cleanup — nine branch CSVs, cleaned by hand every week — is exactly the kind of recurring task lesson 7.4's reusable function is built for. Once load_and_clean() and weekly_summary() exist as functions, wiring them into a fully automated report is mostly plumbing: a scheduled task that runs the script and emails or saves the output, with no manual step in between.

What "end to end" actually means here
  1. The script reads directly from wherever the branch exports land, not from a file you manually download first.
  2. It handles a missing or malformed file gracefully — logging a clear error, not crashing silently at 6am with nobody watching.
  3. The output lands somewhere Chidi can find it without asking you — a shared folder, an email, a dashboard refresh.
Keep this
  • "Automated" means no manual step anywhere in the chain, including a graceful, visible failure when something upstream breaks.
  • Build toward the output landing somewhere the recipient finds on their own, not toward you personally running and sending it.
Lesson 7.711 minutes

Version control and file discipline, the minimum that matters

You don't need to be a software engineer to benefit from git — you need three habits, consistently, and they cover most of what actually goes wrong with analyst code in practice.

  • Commit working states, with a real message. "Fixed branch name cleaning for Ibadan variants" tells a future reader what changed and why; "updates" tells them nothing.
  • Never commit raw data with real customer or financial detail into a public or shared repo. A .gitignore excluding your data files is the single most important line in the whole setup.
  • Commit before a risky change, not after. A working version you can return to is worth more than a clever uncommitted rewrite that might not pan out.
Keep this
  • Commit working states with messages that explain what and why, not just "updates."
  • Keep real data out of version control with a .gitignore — this is a habit worth building from your very first script.
Before the drill

Have Python and pandas installed (pip install pandas) — this drill runs real code against the starter kit's CSV.

Module 7 drillabout 75 minutes

Turn the manual report into a one-command script

Starting from the starter kit's clean_and_report.py, extend it to fully replace module 2's manual Monday-morning process.

  1. Add a category breakdown: merge in a small products lookup you build yourself (sku, category), checked for row-count safety per lesson 7.3.
  2. Add a week-over-week comparison, assuming a second week's file will exist next Monday — write the code so it works on any file path passed as an argument, not hard-coded.
  3. Add basic error handling: what happens if the CSV is missing a column, or a branch name doesn't match any known variant?
  4. Add one exploratory chart per lesson 7.5 — daily revenue trend — kept genuinely quick and rough.
  5. Write the three-line README you'd hand a teammate to run this cold, next Monday, without you.
What this drill is really testing

Whether you can extend real working code without breaking it — adding a join safely, handling a missing file gracefully, and leaving it in a state someone else could actually run. That is the entire point of module 7: not learning Python syntax in the abstract, but replacing a real manual process with something that runs itself.

Before module 8

Module 8 turns to working with AI tools directly — reviewing AI-drafted SQL and formulas, including ones an AI might suggest for a script exactly like the one you just built.