How fast is Polars for realistic data processing jobs?¶

The dataframe library Pandas has long been a workhorse for data scientists, ML engineers, and other data professionals.
It has a broad, convenient, user-friendly API for transforming and summarizing data, which allows us to quickly discern insights from our data and engineer data pipelines for production. But how quickly, exactly? A new dataframe library Polars claims to be much faster at processing data than Pandas and other common tools.

I was curious about how much faster Polars might be not only on benchmarks but also as part of realistic data workflows. So, I created some data to be processed by Pandas or Polars. Since file-reading is typically among the slower parts of a data pipeline, I stored the data in either the older, ubiquitous comma-separated-values (CSV) format or the newer, column-based Parquet format. Additionally, Polars can be used from either Python or Rust, so comparing the workflow in those two languages would be useful. Finally, Polars jobs can be processed eagerly or lazily, so we can compare both execution approaches in our workflow.

To summarize, we’ll compare:

• Pandas vs. Polars
• Python Polars vs. Rust Polars
• CSV vs. Parquet
• Polars Eager vs. Polars Lazy

For each comparison pair in the list above, the bolded variation is the one I expected to be faster. That is, I expected Polars to be faster than Pandas, Rust to be faster than Python, Parquet to be faster than CSV, and Lazy to be faster than Eager.

Creating the Data¶

I created a data set of 100 tables, each with 100,000 rows and 20 columns (lines 88-90). The 100 CSV files occupied 2.0 GB of space on my hard drive, while the 100 Parquet files occupied 1.0 GB. The first 5 columns of each table were composed of randomly selected characters; the next 5 columns, of random integers; and the final 10 columns, of random floats. An example is shown below.

All 100 tables like this example above have the same column names and the same data types for their corresponding columns. After loading all these data tables, the first data processing step will be to vertically concatenate all these tables into a single, 10-million-row table (i.e., 100 tables x 100,000 rows = 10,000,000 rows).

I created an additional key table with 26 rows and 4 columns (lines 50-72). The first key column contains every letter of the alphabet, and the other 3 columns contain randomly selected integers.

The key table is shown above. It is used in the second data processing step, when it is joined on its key column to a column of characters in the 10-million-row table.

Data Processing Steps¶

Only 3 columns from each of the 100 tables are used from the CSV or Parquet files: a column of characters, a column of integers, and a column of floats. An example is shown below.

We’ve already mentioned the first two data processing steps after reading the tables: vertically concatenating the 100 tables and then joining the key table to the resulting, concatenated table. We then drop the key column, because it is no longer needed after the join. These 3 steps are shown below.

The vertically concatenated table is shown below.

The joined table is shown below.

The joined table without the key column is shown below.

The next step is to group the rows of the 10-million-row table by the column of characters and calculate the means of the remaining 5 columns (2 columns from the 100 tables and 3 columns from the key table) for each group. The resulting table is shown below.

Finally, the means of the groups are themselves averaged for each column, and the results are printed.

In summary, our steps are:

1. Read 3 columns from each of the 100 tables
2. Read the "key" table
3. Vertically concatenate the 100 tables
4. Join the "key" table to the concatenated table
5. Drop the "key" column
6. Group by the column of characters and calculate the group means
7. Calculate the means of the group means for each column 
8. Print the final set of means

These steps provide us with a set of tasks that we might see in a typical data pipeline. While the data values are randomly selected and thus not meaningful, they do offer a mix of characters/strings, integers, and floats, much like what we might find in a lot of data sets.

These steps are implemented for each of the processing approaches that we want to compare.

Data Processing Variations¶

As a reminder, we want to compare:

• Pandas vs. Polars
• Python Polars vs. Rust Polars
• CSV vs. Parquet
• Polars Eager vs. Polars Lazy

To do so, we need to implement the data processing steps for each variation that we want to compare: Python Pandas loading CSV files, Python Polars loading Parquet files, and so on.

Here is a list of all the variations, which are laid out in the directory here:

Data Processing Timing¶

To measure how long each data processing variation required, I ran each variation 50 times (line 54) and used the lowest recorded time. Specifically, each measured time was the sum of the reported system time and the user time. During timing runs, my machine wasn’t running any web browsers, multimedia players, or other processing-intense applications, and I watched the system monitor to verify that no unexpected processes were using substantial system resources.

Timing Results¶

Pandas vs. Polars¶

First we’ll compare processing the data in Pandas dataframes versus in Polars dataframes.

The barplot above shows the processing times for Pandas (blue) and Polars (orange). The gray bars are for Rust, which doesn’t use Pandas, so they aren’t relevant for our comparison.

When we compare “like” to “like”, i.e., when we compare the Python-Pandas-CSV blue bar to the two Python-Polars-CSV orange bars (for Eager and Lazy), it is clear that Polars processing times are lower than Python times. Likewise, the corresponding comparison for the Parquet bars shows that Polars is faster than Pandas. This confirms what we anticipated from those benchmarks.

Python Polars vs. Rust Polars¶

Our next comparison is between Python-Polars and Rust-Polars.

Python-Polars (blue) is faster than Rust-Polars (orange). Since Rust is a compiled language and has a reputation for speed, this is quite surprising.

First, did I do something wrong? That’s entirely possible, since Rust and Polars are fairly new to me. I ensured that Rust compilation times were excluded from the measurement by first running the Rust program a few times manually after compilation to verify that it was optimized and speedy, and then by calling Cargo with the “release” and “frozen” parameters during measurement runs (line 51).

Are they fair comparisons? My purpose was not to write the Python and Rust versions so that they did the exact same lower-level computations. After all, Python and Rust are different languages: I expect that they will do things somewhat differently “under the hood”, even if I write very similar-looking code. My purpose was to write hopefully-straightforward Pythonic and Rustacean code that implemented the same algorithmic ideas without awkward gymnastics to make them exactly the “same”. I think my implementations did that reasonably well. Plus, most of the heavy computational lifting is done by the same library Polars, so that may limit the impact of my particular Pythonic or Rustacean choices. So, I think the comparisons are fair, though someone else might produce fair comparisons that are rather different from mine.

Why is Rust-Polars slower than Python-Polars? Let’s look at the timing measurements in more detail.

Notice that the sum of the user and system times exceeds the elapsed times for both Python-Polars and Rust-Polars. This suggests that Polars is parallelizing its workload (“embarrassingly parallel“). The ratios for user+system:elapsed are sometimes higher for Rust-Polars than for Python-Polars — specifically for the Lazy variations. Does Rust-Polars incur more overhead for parallelization operations than Python-Polars even while the workload isn’t heavy enough to justify this extra cost? That’s an interesting hypothesis. But the differences in the ratios aren’t that large, and small fluctuations in the small elapsed denominators could alter the ratios substantially. Additionally, the ratios for the Eager cases either favor Python or are virtually identical, so we would need a better explanation that might account for both Eager and Lazy comparisons between the languages.

A deeper investigation might involve line-profiling the Python and Rust variations of each comparison to locate the most time-consuming operations. Absent that closer look, I would recommend not just assuming (as I did) that one language would be faster than another for a particular use case: if optimization for speed is that important, it’s worth testing the alternatives to know for sure.

CSV vs. Parquet¶

Next we compare reading data from CSV files to reading from Parquet files.

This comparison turned out as expected: loading data from Parquet files (orange) is faster than reading CSV files (blue). And that was true for every comparison pair: Python-Pandas, Python-Polars-Eager, Python-Polars-Lazy, Rust-Polars-Eager, and Rust-Polars-Lazy.

One feature of Parquet files that surely makes them faster in this case is that they store data as columns, meaning that values that are adjacent in a column are also adjacent in the file structure or in memory when they are read. By contrast, CSV files store data in rows. When only a subset of columns are loaded from a file, read times from Parquet are often fast because of this column-based storage.

Polars Eager vs. Polars Lazy¶

Finally we compare eager versus lazy execution in Polars.

There’s no discernible difference between eager (blue) and lazy (orange) Polars execution with the Python code. This would make me question whether I completely botched coding the Lazy implementation, except that for Rust, Lazy is indeed faster than Eager for both CSV and Parquet. That’s reassuring; I must’ve done something right.

Why is Lazy faster than Eager for Rust but not for Python? Perhaps Rust’s compiler allows optimizations between Polars’s Lazy execution mode and other Rust code that the Python interpreter can’t do. A similar improvement in Python might require a lot of manual investigation and optimization effort.

Conclusion¶

For the most part, the results were as I expected: Polars is faster than Pandas, Parquet is faster than CSV, and Lazy is faster than Eager.

The major exception was that Python-Polars was faster than Rust-Polars when I expected that Rust would be faster than Python. On the one hand, a computer scientist might scoff at the difference because it’s less than an order of magnitude; in fact, Rust is more than 2-times slower than Python in only one case, and even that is only 2.5-times slower:

On the other hand, a factor of 2 or 3 can provide important time savings in real-world applications, so optimizing for such a difference might be worthwhile.

Further investigation should focus on whether some minor adjustments to the Rust code would greatly improve its performance or whether Python has a general advantage for this type of data processing with Polars.