Looking at Your Data Before Touching It
The mistake beginners make in the first five minutes
You load a new dataset. There is a target column, there are some numbers, scikit-learn is right there. The temptation is to skip straight to df.head() and then straight into training a model. Do not. Half of the mistakes that ruin a model, a column that is secretly text pretending to be a number, a "yes/no" column that is actually "yes/No/Y/N", a rent value that is actually in lakhs instead of rupees, get caught in the two minutes you spend actually looking at the data. Catch them after training and you have wasted hours debugging a model that was never the problem.
This lesson is the checklist: the handful of things you run on any new dataset before you do anything else with it.
Step 1: shape, and what each row and column actually is
1import pandas as pd
2
3df = pd.read_csv("https://cod-e.vercel.app/data/pune-flats.csv")
4df.shape(21, 6)
Before reading a single value, ask two questions out loud: what does one row mean, and what does each column mean? Here, one row is one flat that was listed for rent, and the columns describe that flat and what it rented for. It sounds obvious with 21 rows in front of you, but with 40 columns and a vague name like "sales_data.csv", this is the step that stops you from misreading the whole dataset.
Step 2: what type is each column, really
dtypes tells you what pandas thinks each column is. This matters because a column can look like a number to your eyes and still be stored as text, which silently breaks any maths you try to do on it.
1df.dtypesarea object bhk int64 size_sqft int64 floor int64 furnished object rent int64 dtype: object
object means text, here area and furnished. int64 means whole numbers. If bhk had shown up as object instead of int64, that would be your first sign something is wrong, maybe a stray flat had "2 BHK" typed as text somewhere in the column instead of the number 2.
Step 3: is anything missing
isna() marks every missing cell as True, and summing it counts them per column.
1df.isna().sum()area 0 bhk 0 size_sqft 0 floor 0 furnished 0 rent 0 dtype: int64
This particular dataset is clean, zero missing values everywhere, which is exactly why the next lesson uses a deliberately messier copy of it. In a real dataset, do not expect to see all zeros. A column full of non-zero counts here tells you exactly which columns need attention before you go any further, which is the whole subject of lesson 3.3.
Step 4: any exact duplicate rows
1df.duplicated().sum()0
duplicated() flags a row as True if an earlier row is identical to it in every column. Zero here means every flat is distinct. A logging bug or a copy-paste mistake in a spreadsheet can easily submit the same row twice, and a duplicated row quietly tells your model that pattern matters twice as much as it should.
Step 5: the shape of your numbers
describe() runs count, mean, standard deviation and the quartiles on every numeric column at once. Read it as a sanity check, not as analysis: does anything look impossible?
1df.describe()bhk size_sqft floor rent count 21.000000 21.000000 21.000000 21.000000 mean 2.000000 960.714286 6.333333 26690.476190 std 0.774597 323.223165 3.291403 10248.751376 min 1.000000 525.000000 1.000000 13500.000000 25% 1.000000 675.000000 4.000000 20000.000000 50% 2.000000 925.000000 7.000000 26500.000000 75% 3.000000 1175.000000 9.000000 30000.000000 max 3.000000 1575.000000 12.000000 54000.000000
A few things worth reading here. min for bhk is 1 and max is 3, that is believable for flats. min rent is ₹13,500 and max is ₹54,000, a wide range but nothing that smells like a typo, like a rent of ₹5 or ₹50,00,000. std is how spread out the values are (you met this properly in lesson 2.7); a rent std of about ₹10,249 next to a mean of about ₹26,690 tells you rent varies a lot across these flats, which you would expect once you notice they span six different areas and one to three bedrooms.
Tip:The quickest bug describe() catches: a min that is negative where it should never be, like a negative size_sqft or a negative rent. That is almost always a data entry mistake, not a real flat.
Step 6: what values does a text column actually hold
describe() only looks at numbers. For text columns, check what values actually show up, because a column you expect to have two options sometimes has five, due to inconsistent typing.
1df["furnished"].unique()array(['no', 'yes'], dtype=object)
Exactly two options, spelled consistently. In a messier real dataset this same check might return array(['yes', 'no', 'Yes', 'NO', 'y']), five different spellings of two real options, and you would need to clean that before any encoding in the next lessons makes sense. value_counts() gives you the same list plus how often each one shows up:
1df["area"].value_counts()area Baner 4 Kothrud 4 Wakad 4 Hadapsar 3 Hinjewadi 3 Viman Nagar 3 Name: count, dtype: int64
Six areas, none wildly overrepresented. If one area had 18 of the 21 rows, you would know before training that the model will barely have seen the other five, and its predictions there deserve extra suspicion.
Where people get confused
Key Points
- "describe() covers everything." It only touches numeric columns by default. Text columns need unique() and value_counts(), separately.
- "Zero missing values means the data is clean." isna() only catches actual empty cells. A rent typed as 0 or -1 to mean "unknown" sails straight past it and needs a human eye, which is exactly what step 5 is for.
- "I only need to check this once." Check it again after every cleaning step in this chapter. Filling in missing values or merging tables can quietly introduce new problems.
Quick check
You load a dataset and bhk shows up as dtype object instead of int64. What does that tell you, before you look at a single value?
Something in that column is not a clean number, most likely stray text mixed into a column you expected to be purely numeric.
Why check unique() on a text column instead of trusting describe()?
describe() only summarises numeric columns. A text column can hide inconsistent spellings or unexpected extra categories that only unique() or value_counts() will show you.
What does a duplicated row do to a model, if you leave it in?
It makes that row, and whatever pattern it represents, count twice as much during training as it should, quietly biasing the model toward it.
Recap
Key Points
- Before touching values, know what a row and a column mean, and check df.dtypes to see how pandas is reading each column.
- df.isna().sum() counts missing values per column. df.duplicated().sum() counts exact duplicate rows.
- df.describe() sanity-checks numeric columns: does the min, max and mean look believable?
- unique() and value_counts() do the same job for text columns, and catch inconsistent spellings describe() cannot see.
- This is a checklist to run on every new dataset, and to run again after every cleaning step.
This lesson's data had zero missing values, which almost never happens outside a course. Next, a messier copy of the same flats, and every real way to deal with what is missing.
Theia
Can you explain this concept?
Practise Looking at Your Data Before Touching It with realistic interview questions. Choose spoken or typed answers, get feedback on your reasoning, then return to the material that needs work.
Sign in to practise this conceptThe session stays focused on this lesson. Your selected interview level sets the bar.