Haskell Finally Gets a Native Parquet Writer
Writing Parquet files using Haskell
DataHaskell's Dataframe library now includes a Parquet writer, letting Haskell users store data in the efficient columnar format used across the data science ecosystem. The writeParquet function works with sane defaults, while writeParquetWithOptions exposes tuning knobs like page size, row group size, compression codec, and writer strategy. The implementation uses pinned mutable byte arrays and an effectful fold over the dataframe to handle memory growth and produce well-structured Parquet files.
For Haskell to interoperate with the data ecosystem, it must be able to understand the standard formats in use by that ecosystem.
- mugul
Looks like a pretty cool community building great tools with care. Using a functional language for data transforms sounds like a sane idea, haven't played around with it yet but it's definitely on my list now.
However they claim using Haskell for data science is "fast", which doesn't really mean anything until you have numbers to show. A little benchmark with pandas and polars wouldn't hurt I guess.
- jmaker
I still love Haskell, had dedicated several years to it while at the university. Was impossible to land a job with Haskell. Did some Scala but it’s in demise. F# and OCaml are extremely niche. Ever fewer opportunities and engineering management is convinced it’s impossible to hire functional programmers.
In this new AI-driven world, is there still place for such a luxury as functional programming?
I mean few people still code by hand, few read the generated code, models aren’t trained on functional languages, it’s inefficient token wise to use functional languages - while a lot become self-proclaimed software engineers overnight by just prompting LLMs.
- Quothling
I'd personally rather see Haskell become part of the options for https://arrow.apache.org/, but this is still a cool project.