pyspark

the portable Python dataframe library

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

State of the Art Natural Language Processing

Apache Linkis builds a computation middleware layer to facilitate connection, governance and orchestration between the upper applications and the underlying data engines.

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Implementing best practices for PySpark ETL jobs and applications.

A curated list of awesome Apache Spark packages and resources.

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

SQL data analysis & visualization projects using MySQL, PostgreSQL, SQLite, Tableau, Apache Spark and pySpark.

Lightweight and extensible compatibility layer between dataframe libraries!

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

🚚 Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

Jupyter magics and kernels for working with remote Spark clusters

Hopsworks - Data-Intensive AI platform with a Feature Store

PySpark-Tutorial provides basic algorithms using PySpark

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

MapReduce, Spark, Java, and Scala for Data Algorithms Book

Sparkling Water provides H2O functionality inside Spark cluster

pyspark🍒🥭 is delicious，just eat it!😋😋

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.