Running Billion-Scale Graph Algorithms on a Laptop with Apache DataFusion
Algorithms on billion-scale graph using 10GB RAM: I love DataFusion
I implemented graph map-reduce algorithms using Apache DataFusion to process billion-scale datasets on a standard laptop. By offloading data to disk and leveraging bulk scans, I successfully computed PageRank and identified weakly connected components with only 5 to 10 GB of RAM. This approach proves that massive graph analytics no longer require heavy clusters like Apache Spark, making powerful computation accessible on modest hardware.
Previously, I thought you needed Apache Spark and GraphFrames for billion-scale graph analytics. Now, however, I think all you need is a laptop.
- chrisweekly
> "I can compute PageRank on a directed graph with one billion edges (graph500-26 from the Graphalytics dataset) using 5 GB of memory. Alternatively, I can identify all the weakly connected components in a graph with two billion edges (twitter_mpi from the same dataset collection) using 10 GB of memory. Neither NetworkX nor Igraph can do this; most existing graph algorithms require the graph to fit into memory. Previously, I thought you needed Apache Spark and GraphFrames for billion-scale graph analytics. Now, however, I think all you need is a laptop. I have completely changed my old opinion about using Apache DataFusion for graph analytics."
Impressive!
- nylonstrung
Datafusion is undoubtedly one of the best open source projects of all time, it's so incredibly powerful and well designed.
The extensibility is insane, you can create your own query language that compiles to logical plans.
- cpdomina
cool! you might be interested in graphchi (2012), also designed to do large scale graph operations on a single machine
- adsharma
The idea of graph algorithms on Apache arrow at scale originated here. 100+ graph algorithms running on columnar memory.
https://github.com/Ladybug-Memory/icebug
Out of core with datafusion is the main innovation here in graphframes-rs. But it has only 2 algorithms so far.
Icebug and LadybugDB can be tightly integrated to efficiently move tables encoded as compressed sparse row (CSR) into arrow memory.
Jupyter notebooks available.
- yadgire7
Hello, I am new to hacker news and finding it really resourceful. I found this article interesting (having learnt KG and Map Reduce (spark) as part of my masters' course), appreciate the effort to post this.
I am here to seek guidance from the community. I want to refresh my memory on knowledge graphs and algorithms for Big Data Mining and Processing.
I believe KG can solve problems on Agent attacks (LLM agency) in real-time - so want to build knowledge around the topic.
Interested to join any interest/ discussion groups if any.
Thanks!