6 Essential Methods to Filter Rows in Pandas for Faster, Smarter Data Analysis
Pandas is the go-to Python library for working with tabular data. Its DataFrame and Series objects provide a host of methods to load, manipulate, analyze and visualize data. One of the most fundamental data preparation tasks you‘ll need to perform is filtering rows from a DataFrame.
Filtering allows you to extract only the data you‘re interested in, whether it‘s to explore a subset of records, remove invalid values, or hone in on specific segments. Pandas provides a variety of techniques to filter rows, from basic boolean indexing to advanced query methods.
In this guide, we‘ll walk through 6 key approaches to filter rows in pandas:
- Boolean indexing to filter rows by column values
- Logical operators to filter rows by multiple conditions
- Integer indexing to filter rows by position
- The query() method for fast, flexible filtering
- Handling missing data by filtering out null values
- Filtering string data using pandas string methods
For each method, we‘ll break down how it works and when to use it, with clear examples. By the end, you‘ll be equipped with a solid understanding of how to filter DataFrames in pandas to speed up your data analysis workflow. Let‘s dive in!
1. Boolean Indexing: Filter Rows by Column Values
The most basic and commonly used technique to filter rows is boolean indexing. This approach allows you to specify a logical condition that is checked for each row. Rows that match the condition are returned in the filtered DataFrame.
Here‘s an example. Suppose we have a DataFrame with exam scores for a class of students:
import pandas as pd
data = {
‘name‘: [‘Amy‘, ‘Bob‘, ‘Charlie‘, ‘David‘, ‘Emily‘],
‘grade‘: [11, 12, 11, 12, 11],
‘score‘: [85, 92, 76, 88, 94]
}
df = pd.DataFrame(data)
print(df)
Output:
name grade score
0 Amy 11 85
1 Bob 12 92
2 Charlie 11 76
3 David 12 88
4 Emily 11 94
To filter this DataFrame to only 12th graders, we can use boolean indexing like this:
df[df[‘grade‘] == 12]
Output:
name grade score 1 Bob 12 92 3 David 12 88
Here‘s how it works:
- df[‘grade‘] == 12 checks each value in the ‘grade‘ column and returns a Series of True/False values based on whether the grade equals 12
- Putting this inside df[] then filters the DataFrame to only the rows corresponding to True
This syntax is concise and easy to read. You can apply it to filter rows based on any column and value.
2. Logical Operators: Filter Rows by Multiple Conditions
What if you need to filter a DataFrame based on multiple criteria? Pandas allows you to combine conditions using logical operators like & (and), | (or) and ~ (not).
Let‘s filter our exam score dataset to only 11th graders who scored above 80:
df[(df[‘grade‘] == 11) & (df[‘score‘] > 80)]
Output:
name grade score
0 Amy 11 85
4 Emily 11 94
The key steps:
- (df[‘grade‘] == 11) checks for rows where grade is 11
- (df[‘score‘] > 80) checks for rows where score exceeds 80
- & combines the two conditions, returning rows where both are True
The | operator works similarly but returns rows where either condition is met. The ~ operator negates a condition, returning rows where it‘s False.
For complex filtering with many conditions, logical operators can quickly become hard to read. In those cases, the query() method (which we‘ll cover later) is often a better choice. But for simple filters involving a few conditions, logical operators are a handy tool to have in your pandas toolbox.
3. Integer Indexing: Filter Rows by Position
Sometimes you want to extract rows based on their position rather than their content. That‘s where integer indexing with .iloc comes in handy.
With .iloc, you filter rows by integer position. It takes two arguments:
- A start and end index for rows
- A start and end index for columns
Let‘s use it to get the first 3 rows and all columns of our exam score data:
df.iloc[0:3, :]
Output:
name grade score
0 Amy 11 85
1 Bob 12 92
2 Charlie 11 76
Here‘s the breakdown:
- 0:3 specifies the rows from index 0 up to but not including index 3
- : by itself specifies all columns
- df.iloc[0:3, :] thus returns the first 3 rows and all columns
You can also use negative numbers to count back from the end, skip rows/columns by specifying a step size, and more. This makes .iloc very flexible for grabbing different slices of your DataFrame.
4. The query() Method: Fast, Flexible Filtering
The query() method is a powerful way to filter rows using a string expression. It‘s often faster than standard boolean indexing, especially for large datasets.
Suppose we want to filter our exam score data for students whose name ends with ‘y‘. Here‘s how we could do it with query():
df.query("name.str.endswith(‘y‘)")
Output:
name grade score
0 Amy 11 85
4 Emily 11 94
This is equivalent to the boolean indexing expression:
df[df[‘name‘].str.endswith(‘y‘)]
But the query() version is more concise. The real strength of query() is that you can refer to columns directly by name (e.g. ‘name‘ instead of df[‘name‘]). This makes complex queries with multiple conditions much easier to write and read.
For example, to get 12th graders with scores over 90:
df.query("grade == 12 and score > 90")
Output:
name grade score 1 Bob 12 92
Pandas query() uses a subset of Python syntax, so you can express a wide range of conditions. It‘s a valuable method to master for fast, flexible filtering.
5. Handling Missing Data: Filter Out Null Values
In real-world datasets, missing values are common. Pandas represents these as NaN (not a number). Rows with NaN values can cause issues in your analysis, so it‘s often necessary to filter them out.
We can use the isna() and notna() methods to find rows with missing data. Suppose we have a DataFrame with some NaN values:
data = {
‘name‘: [‘Amy‘, ‘Bob‘, ‘Charlie‘, ‘David‘, ‘Emily‘],
‘grade‘: [11, 12, 11, 12, 11],
‘score‘: [85, 92, None, 88, 94]
}
df_missing = pd.DataFrame(data)
To filter out rows with missing scores:
df_missing[df_missing[‘score‘].notna()]
Output:
name grade score
0 Amy 11 85.0
1 Bob 12 92.0
3 David 12 88.0
4 Emily 11 94.0
notna() returns a boolean mask indicating which rows have non-null values. We can then use this mask to index the DataFrame and filter out the rows with NaNs.
The isna() method works similarly but returns True for NaN values. So to get only rows with missing scores:
df_missing[df_missing[‘score‘].isna()]
Output:
name grade score
2 Charlie 11 NaN
These methods are essential for data cleaning and ensuring your analyses run smoothly.
6. String Methods: Filter String Data
Pandas has a suite of string methods that make it easy to filter text data. These are accessed via the str attribute on a Series.
Common string methods for filtering include:
- contains(): Filter rows where a string pattern appears
- startswith() / endswith(): Filter rows that start or end with a pattern
- match(): Filter rows that match a regular expression
For example, let‘s filter students whose names start with ‘C‘:
df[df[‘name‘].str.startswith(‘C‘)]
Output:
name grade score
2 Charlie 11 76
The steps:
- df[‘name‘] accesses the ‘name‘ column Series
- .str makes string methods available on the Series
- .startswith(‘C‘) checks each name for the starting letter ‘C‘
We could also use contains() to find names with a substring like ‘ar‘:
df[df[‘name‘].str.contains(‘ar‘)]
Output:
name grade score
2 Charlie 11 76
These methods are case-sensitive by default, but you can make them case-insensitive by setting the case parameter to False.
To filter using a regular expression, we can use match(). For example, to get names starting with a vowel:
df[df[‘name‘].str.match(‘^[AEIOU]‘)]
Output:
name grade score 0 Amy 11 85 4 Emily 11 94
Pandas string methods give you a powerful toolkit for working with text fields and unlocking insights from unstructured data.
Summary and Takeaways
Filtering is a core operation in data analysis that enables you to narrow down a dataset to just the rows you need. Pandas provides several methods for filtering, each with its own strengths:
- Boolean indexing is the go-to choice for simple filters
- Logical operators are useful for combining a few conditions
- Integer indexing with .iloc is handy for selecting rows by position
- The query() method is fast and flexible for more complex queries
- isna() and notna() let you handle missing values
- String methods unlock filtering options for text data
As you work with pandas more, you‘ll get a sense of which filtering techniques work best for different scenarios. The key is to practice on a variety of datasets to build your pandas skills.
Filtering is just one step in the data pipeline. By learning how to efficiently filter DataFrames, you‘ll be able to streamline your analyses, ask more targeted questions, and uncover valuable insights faster.
Here are some resources to continue your pandas journey:
- Official pandas documentation on indexing and selection
- Real Python‘s tutorial on pandas string methods
- DataCamp‘s pandas cheat sheet
Happy filtering!
![What Is an API Gateway & How Does It Work? [+Best Service Providers]](https://greenlitcontent.com/wp-content/uploads/2026/04/20260423192459-69ea720b90451.jpg)
