rhetenor/awesome-awesomeness
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
Files
402629a3301c0f8e8bcd0e9ffc03d8139253e404
awesome-awesomeness/html/mlops.html
Jonas Zeunert 2d63fe63cd Restructure with Makefile + add html output
2024-04-20 19:22:54 +02:00

931 lines
46 KiB
HTML
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
<h1 id="awesome-mlops-awesome">Awesome MLOps <a
href="https://github.com/sindresorhus/awesome"><img
src="https://cdn.rawgit.com/sindresorhus/awesome/d7305f38d29fed78fa85652e3a63e154dd8e8829/media/badge.svg"
alt="Awesome" /></a></h1>
<p>A curated list of awesome MLOps tools.</p>
<p>Inspired by <a
href="https://github.com/vinta/awesome-python">awesome-python</a>.</p>
<ul>
<li><a href="#awesome-mlops">Awesome MLOps</a>
<ul>
<li><a href="#automl">AutoML</a></li>
<li><a href="#cicd-for-machine-learning">CI/CD for Machine
Learning</a></li>
<li><a href="#cron-job-monitoring">Cron Job Monitoring</a></li>
<li><a href="#data-catalog">Data Catalog</a></li>
<li><a href="#data-enrichment">Data Enrichment</a></li>
<li><a href="#data-exploration">Data Exploration</a></li>
<li><a href="#data-management">Data Management</a></li>
<li><a href="#data-processing">Data Processing</a></li>
<li><a href="#data-validation">Data Validation</a></li>
<li><a href="#data-visualization">Data Visualization</a></li>
<li><a href="#drift-detection">Drift Detection</a></li>
<li><a href="#feature-engineering">Feature Engineering</a></li>
<li><a href="#feature-store">Feature Store</a></li>
<li><a href="#hyperparameter-tuning">Hyperparameter Tuning</a></li>
<li><a href="#knowledge-sharing">Knowledge Sharing</a></li>
<li><a href="#machine-learning-platform">Machine Learning
Platform</a></li>
<li><a href="#model-fairness-and-privacy">Model Fairness and
Privacy</a></li>
<li><a href="#model-interpretability">Model Interpretability</a></li>
<li><a href="#model-lifecycle">Model Lifecycle</a></li>
<li><a href="#model-serving">Model Serving</a></li>
<li><a href="#model-testing--validation">Model Testing &amp;
Validation</a></li>
<li><a href="#optimization-tools">Optimization Tools</a></li>
<li><a href="#simplification-tools">Simplification Tools</a></li>
<li><a href="#visual-analysis-and-debugging">Visual Analysis and
Debugging</a></li>
<li><a href="#workflow-tools">Workflow Tools</a></li>
</ul></li>
<li><a href="#resources">Resources</a>
<ul>
<li><a href="#articles">Articles</a></li>
<li><a href="#books">Books</a></li>
<li><a href="#events">Events</a></li>
<li><a href="#other-lists">Other Lists</a></li>
<li><a href="#podcasts">Podcasts</a></li>
<li><a href="#slack">Slack</a></li>
<li><a href="#websites">Websites</a></li>
</ul></li>
<li><a href="#contributing">Contributing</a></li>
</ul>
<hr />
<h2 id="automl">AutoML</h2>
<p><em>Tools for performing AutoML.</em></p>
<ul>
<li><a href="https://github.com/awslabs/autogluon">AutoGluon</a> -
Automated machine learning for image, text, tabular, time-series, and
multi-modal data.</li>
<li><a href="https://github.com/keras-team/autokeras">AutoKeras</a> -
AutoKeras goal is to make machine learning accessible for everyone.</li>
<li><a href="https://github.com/automl/Auto-PyTorch">AutoPyTorch</a> -
Automatic architecture search and hyperparameter optimization for
PyTorch.</li>
<li><a href="https://github.com/automl/auto-sklearn">AutoSKLearn</a> -
Automated machine learning toolkit and a drop-in replacement for a
scikit-learn estimator.</li>
<li><a href="https://github.com/alteryx/evalml">EvalML</a> - A library
that builds, optimizes, and evaluates ML pipelines using domain-specific
functions.</li>
<li><a href="https://github.com/microsoft/FLAML">FLAML</a> - Finds
accurate ML models automatically, efficiently and economically.</li>
<li><a href="https://h2o.ai/platform/h2o-automl">H2O AutoML</a> -
Automates ML workflow, which includes automatic training and tuning of
models.</li>
<li><a href="https://github.com/mindsdb/mindsdb">MindsDB</a> - AI layer
for databases that allows you to effortlessly develop, train and deploy
ML models.</li>
<li><a href="https://github.com/AxeldeRomblay/MLBox">MLBox</a> - MLBox
is a powerful Automated Machine Learning python library.</li>
<li><a href="https://github.com/google/model_search">Model Search</a> -
Framework that implements AutoML algorithms for model architecture
search at scale.</li>
<li><a href="https://github.com/microsoft/nni">NNI</a> - An open source
AutoML toolkit for automate machine learning lifecycle.</li>
</ul>
<h2 id="cicd-for-machine-learning">CI/CD for Machine Learning</h2>
<p><em>Tools for performing CI/CD for Machine Learning.</em></p>
<ul>
<li><a href="https://github.com/allegroai/clearml">ClearML</a> -
Auto-Magical CI/CD to streamline your ML workflow.</li>
<li><a href="https://github.com/iterative/cml">CML</a> - Open-source
library for implementing CI/CD in machine learning projects.</li>
</ul>
<h2 id="cron-job-monitoring">Cron Job Monitoring</h2>
<p><em>Tools for monitoring cron jobs (recurring jobs).</em></p>
<ul>
<li><a href="https://cronitor.io/cron-job-monitoring">Cronitor</a> -
Monitor any cron job or scheduled task.</li>
<li><a href="https://healthchecks.io/">HealthchecksIO</a> - Simple and
effective cron job monitoring.</li>
</ul>
<h2 id="data-catalog">Data Catalog</h2>
<p><em>Tools for data cataloging.</em></p>
<ul>
<li><a href="https://www.amundsen.io/">Amundsen</a> - Data discovery and
metadata engine for improving the productivity when interacting with
data.</li>
<li><a href="https://atlas.apache.org">Apache Atlas</a> - Provides open
metadata management and governance capabilities to build a data
catalog.</li>
<li><a href="https://github.com/ckan/ckan">CKAN</a> - Open-source DMS
(data management system) for powering data hubs and data portals.</li>
<li><a href="https://github.com/linkedin/datahub">DataHub</a> -
LinkedIn’s generalized metadata search &amp; discovery tool.</li>
<li><a href="https://github.com/magda-io/magda">Magda</a> - A federated,
open-source data catalog for all your big data and small data.</li>
<li><a href="https://github.com/Netflix/metacat">Metacat</a> - Unified
metadata exploration API service for Hive, RDS, Teradata, Redshift, S3
and Cassandra.</li>
<li><a href="https://open-metadata.org/">OpenMetadata</a> - A Single
place to discover, collaborate and get your data right.</li>
</ul>
<h2 id="data-enrichment">Data Enrichment</h2>
<p><em>Tools and libraries for data enrichment.</em></p>
<ul>
<li><a href="https://github.com/snorkel-team/snorkel">Snorkel</a> - A
system for quickly generating training data with weak supervision.</li>
<li><a href="https://github.com/upgini/upgini">Upgini</a> - Enriches
training datasets with features from public and community shared data
sources.</li>
</ul>
<h2 id="data-exploration">Data Exploration</h2>
<p><em>Tools for performing data exploration.</em></p>
<ul>
<li><a href="https://zeppelin.apache.org/">Apache Zeppelin</a> - Enables
data-driven, interactive data analytics and collaborative
documents.</li>
<li><a href="https://github.com/tkrabel/bamboolib">BambooLib</a> - An
intuitive GUI for Pandas DataFrames.</li>
<li><a href="https://github.com/sfu-db/dataprep">DataPrep</a> - Collect,
clean and visualize your data in Python.</li>
<li><a href="https://colab.research.google.com">Google Colab</a> -
Hosted Jupyter notebook service that requires no setup to use.</li>
<li><a href="https://jupyter.org/">Jupyter Notebook</a> - Web-based
notebook environment for interactive computing.</li>
<li><a href="https://jupyterlab.readthedocs.io">JupyterLab</a> - The
next-generation user interface for Project Jupyter.</li>
<li><a href="https://github.com/mwouts/jupytext">Jupytext</a> - Jupyter
Notebooks as Markdown Documents, Julia, Python or R scripts.</li>
<li><a href="https://github.com/ydataai/pandas-profiling">Pandas
Profiling</a> - Create HTML profiling reports from pandas DataFrame
objects.</li>
<li><a href="https://polynote.org/">Polynote</a> - The polyglot notebook
with first-class Scala support.</li>
</ul>
<h2 id="data-management">Data Management</h2>
<p><em>Tools for performing data management.</em></p>
<ul>
<li><a href="https://www.arrikto.com/">Arrikto</a> - Dead simple, ultra
fast storage for the hybrid Kubernetes world.</li>
<li><a href="https://github.com/BlazingDB/blazingsql">BlazingSQL</a> - A
lightweight, GPU accelerated, SQL engine for Python. Built on RAPIDS
cuDF.</li>
<li><a href="https://github.com/delta-io/delta">Delta Lake</a> - Storage
layer that brings scalable, ACID transactions to Apache Spark and other
engines.</li>
<li><a href="https://github.com/dolthub/dolt">Dolt</a> - SQL database
that you can fork, clone, branch, merge, push and pull just like a git
repository.</li>
<li><a href="https://github.com/kevin-hanselman/dud">Dud</a> - A
lightweight CLI tool for versioning data alongside source code and
building data pipelines.</li>
<li><a href="https://dvc.org/">DVC</a> - Management and versioning of
datasets and machine learning models.</li>
<li><a href="https://git-lfs.github.com">Git LFS</a> - An open source
Git extension for versioning large files.</li>
<li><a href="https://github.com/activeloopai/Hub">Hub</a> - A dataset
format for creating, storing, and collaborating on AI datasets of any
size.</li>
<li><a href="https://github.com/intake/intake">Intake</a> - A
lightweight set of tools for loading and sharing data in data science
projects.</li>
<li><a href="https://github.com/treeverse/lakeFS">lakeFS</a> -
Repeatable, atomic and versioned data lake on top of object
storage.</li>
<li><a href="https://github.com/MarquezProject/marquez">Marquez</a> -
Collect, aggregate, and visualize a data ecosystem’s metadata.</li>
<li><a href="https://github.com/milvus-io/milvus/">Milvus</a> - An open
source embedding vector similarity search engine powered by Faiss,
NMSLIB and Annoy.</li>
<li><a href="https://www.pinecone.io">Pinecone</a> - Managed and
distributed vector similarity search used with a lightweight SDK.</li>
<li><a href="https://github.com/qdrant/qdrant">Qdrant</a> - An open
source vector similarity search engine with extended filtering
support.</li>
<li><a href="https://github.com/quiltdata/quilt">Quilt</a> - A
self-organizing data hub with S3 support.</li>
</ul>
<h2 id="data-processing">Data Processing</h2>
<p><em>Tools related to data processing and data pipelines.</em></p>
<ul>
<li><a href="https://airflow.apache.org/">Airflow</a> - Platform to
programmatically author, schedule, and monitor workflows.</li>
<li><a href="https://github.com/azkaban/azkaban">Azkaban</a> - Batch
workflow job scheduler created at LinkedIn to run Hadoop jobs.</li>
<li><a href="https://github.com/dagster-io/dagster">Dagster</a> - A data
orchestrator for machine learning, analytics, and ETL.</li>
<li><a href="https://hadoop.apache.org/">Hadoop</a> - Framework that
allows for the distributed processing of large data sets across
clusters.</li>
<li><a href="https://github.com/OpenRefine/OpenRefine">OpenRefine</a> -
Power tool for working with messy data and improving it.</li>
<li><a href="https://spark.apache.org/">Spark</a> - Unified analytics
engine for large-scale data processing.</li>
</ul>
<h2 id="data-validation">Data Validation</h2>
<p><em>Tools related to data validation.</em></p>
<ul>
<li><a href="https://github.com/pyeve/cerberus">Cerberus</a> -
Lightweight, extensible data validation library for Python.</li>
<li><a href="https://github.com/cleanlab/cleanlab">Cleanlab</a> - Python
library for data-centric AI and machine learning with messy, real-world
data and labels.</li>
<li><a href="https://greatexpectations.io">Great Expectations</a> - A
Python data validation framework that allows to test your data against
datasets.</li>
<li><a href="https://json-schema.org/">JSON Schema</a> - A vocabulary
that allows you to annotate and validate JSON documents.</li>
<li><a href="https://github.com/tensorflow/data-validation">TFDV</a> -
An library for exploring and validating machine learning data.</li>
</ul>
<h2 id="data-visualization">Data Visualization</h2>
<p><em>Tools for data visualization, reports and dashboards.</em></p>
<ul>
<li><a href="https://count.co">Count</a> - SQL/drag-and-drop querying
and visualisation tool based on notebooks.</li>
<li><a href="https://github.com/plotly/dash">Dash</a> - Analytical Web
Apps for Python, R, Julia, and Jupyter.</li>
<li><a href="https://datastudio.google.com">Data Studio</a> - Reporting
solution for power users who want to go beyond the data and dashboards
of GA.</li>
<li><a href="https://github.com/PAIR-code/facets">Facets</a> -
Visualizations for understanding and analyzing machine learning
datasets.</li>
<li><a href="https://grafana.com/grafana/">Grafana</a> - Multi-platform
open source analytics and interactive visualization web
application.</li>
<li><a href="https://github.com/lux-org/lux">Lux</a> - Fast and easy
data exploration by automating the visualization and data analysis
process.</li>
<li><a href="https://www.metabase.com/">Metabase</a> - The simplest,
fastest way to get business intelligence and analytics to everyone.</li>
<li><a href="https://redash.io/">Redash</a> - Connect to any data
source, easily visualize, dashboard and share your data.</li>
<li><a href="https://github.com/CloudOrc/SolidUI">SolidUI</a> -
AI-generated visualization prototyping and editing platform, support 2D
and 3D models.</li>
<li><a href="https://superset.incubator.apache.org/">Superset</a> -
Modern, enterprise-ready business intelligence web application.</li>
<li><a href="https://www.tableau.com">Tableau</a> - Powerful and fastest
growing data visualization tool used in the business intelligence
industry.</li>
</ul>
<h2 id="drift-detection">Drift Detection</h2>
<p><em>Tools and libraries related to drift detection.</em></p>
<ul>
<li><a href="https://github.com/SeldonIO/alibi-detect">Alibi Detect</a>
- An open source Python library focused on outlier, adversarial and
drift detection.</li>
<li><a href="https://github.com/IFCA/frouros">Frouros</a> - An open
source Python library for drift detection in machine learning
systems.</li>
<li><a href="https://github.com/torchdrift/torchdrift/">TorchDrift</a> -
A data and concept drift library for PyTorch.</li>
</ul>
<h2 id="feature-engineering">Feature Engineering</h2>
<p><em>Tools and libraries related to feature engineering.</em></p>
<ul>
<li><a href="https://github.com/feature-engine/feature_engine">Feature
Engine</a> - Feature engineering package with SKlearn like
functionality.</li>
<li><a href="https://github.com/alteryx/featuretools">Featuretools</a> -
Python library for automated feature engineering.</li>
<li><a href="https://github.com/blue-yonder/tsfresh">TSFresh</a> -
Python library for automatic extraction of relevant features from time
series.</li>
</ul>
<h2 id="feature-store">Feature Store</h2>
<p><em>Feature store tools for data serving.</em></p>
<ul>
<li><a href="https://github.com/quintoandar/butterfree">Butterfree</a> -
A tool for building feature stores. Transform your raw data into
beautiful features.</li>
<li><a href="https://github.com/bytehub-ai/bytehub">ByteHub</a> - An
easy-to-use feature store. Optimized for time-series data.</li>
<li><a href="https://feast.dev/">Feast</a> - End-to-end open source
feature store for machine learning.</li>
<li><a href="https://github.com/linkedin/feathr">Feathr</a> - An
enterprise-grade, high performance feature store.</li>
<li><a href="https://github.com/featureform/featureform">Featureform</a>
- A Virtual Feature Store. Turn your existing data infrastructure into a
feature store.</li>
<li><a href="https://www.tecton.ai/">Tecton</a> - A fully-managed
feature platform built to orchestrate the complete lifecycle of
features.</li>
</ul>
<h2 id="hyperparameter-tuning">Hyperparameter Tuning</h2>
<p><em>Tools and libraries to perform hyperparameter tuning.</em></p>
<ul>
<li><a href="https://github.com/tobegit3hub/advisor">Advisor</a> -
Open-source implementation of Google Vizier for hyper parameters
tuning.</li>
<li><a href="https://github.com/maxpumperla/hyperas">Hyperas</a> - A
very simple wrapper for convenient hyperparameter optimization.</li>
<li><a href="https://github.com/hyperopt/hyperopt">Hyperopt</a> -
Distributed Asynchronous Hyperparameter Optimization in Python.</li>
<li><a href="https://github.com/kubeflow/katib">Katib</a> -
Kubernetes-based system for hyperparameter tuning and neural
architecture search.</li>
<li><a href="https://github.com/keras-team/keras-tuner">KerasTuner</a> -
Easy-to-use, scalable hyperparameter optimization framework.</li>
<li><a href="https://optuna.org/">Optuna</a> - Open source
hyperparameter optimization framework to automate hyperparameter
search.</li>
<li><a href="https://github.com/scikit-optimize/scikit-optimize">Scikit
Optimize</a> - Simple and efficient library to minimize expensive and
noisy black-box functions.</li>
<li><a href="https://github.com/autonomio/talos">Talos</a> -
Hyperparameter Optimization for TensorFlow, Keras and PyTorch.</li>
<li><a href="https://docs.ray.io/en/latest/tune.html">Tune</a> - Python
library for experiment execution and hyperparameter tuning at any
scale.</li>
</ul>
<h2 id="knowledge-sharing">Knowledge Sharing</h2>
<p><em>Tools for sharing knowledge to the entire team/company.</em></p>
<ul>
<li><a href="https://github.com/airbnb/knowledge-repo">Knowledge
Repo</a> - Knowledge sharing platform for data scientists and other
technical professions.</li>
<li><a href="https://kyso.io/">Kyso</a> - One place for data insights so
your entire team can learn from your data.</li>
</ul>
<h2 id="machine-learning-platform">Machine Learning Platform</h2>
<p><em>Complete machine learning platform solutions.</em></p>
<ul>
<li><a href="https://www.veritone.com/aiware/aiware-os/">aiWARE</a> -
aiWARE helps MLOps teams evaluate, deploy, integrate, scale &amp;
monitor ML models.</li>
<li><a href="https://algorithmia.com/">Algorithmia</a> - Securely govern
your machine learning operations with a healthy ML lifecycle.</li>
<li><a href="https://allegro.ai/">Allegro AI</a> - Transform ML/DL
research into products. Faster.</li>
<li><a href="https://bodywork.readthedocs.io/en/latest/">Bodywork</a> -
Deploys machine learning projects developed in Python, to
Kubernetes.</li>
<li><a href="https://cnvrg.io/">CNVRG</a> - An end-to-end machine
learning platform to build and deploy AI models at scale.</li>
<li><a href="https://dagshub.com/">DAGsHub</a> - A platform built on
open source tools for data, model and pipeline management.</li>
<li><a href="https://www.dataiku.com/">Dataiku</a> - Platform
democratizing access to data and enabling enterprises to build their own
path to AI.</li>
<li><a href="https://www.datarobot.com/">DataRobot</a> - AI platform
that democratizes data science and automates the end-to-end ML at
scale.</li>
<li><a href="https://www.dominodatalab.com/">Domino</a> - One place for
your data science tools, apps, results, models, and knowledge.</li>
<li><a href="https://edgeimpulse.com/">Edge Impulse</a> - Platform for
creating, optimizing, and deploying AI/ML algorithms for edge
devices.</li>
<li><a href="https://github.com/tensorchord/envd">envd</a> - Machine
learning development environment for data science and AI/ML engineering
teams.</li>
<li><a href="https://fedml.ai/">FedML</a> - Simplifies the workflow of
federated learning anywhere at any scale.</li>
<li><a href="https://gradient.paperspace.com/">Gradient</a> - Multicloud
CI/CD and MLOps platform for machine learning teams.</li>
<li><a href="https://www.h2o.ai/">H2O</a> - Open source leader in AI
with a mission to democratize AI for everyone.</li>
<li><a href="https://www.hopsworks.ai/">Hopsworks</a> - Open-source
platform for developing and operating machine learning models at
scale.</li>
<li><a href="https://www.iguazio.com/">Iguazio</a> - Data science
platform that automates MLOps with end-to-end machine learning
pipelines.</li>
<li><a href="https://katonic.ai/">Katonic</a> - Automate your cycle of
intelligence with Katonic MLOps Platform.</li>
<li><a href="https://www.knime.com/">Knime</a> - Create and
productionize data science using one easy and intuitive
environment.</li>
<li><a href="https://www.kubeflow.org/">Kubeflow</a> - Making
deployments of ML workflows on Kubernetes simple, portable and
scalable.</li>
<li><a href="https://lynxkite.com/">LynxKite</a> - A complete graph data
science platform for very large graphs and other datasets.</li>
<li><a href="https://github.com/ml-tooling/ml-workspace">ML
Workspace</a> - All-in-one web-based IDE specialized for machine
learning and data science.</li>
<li><a href="https://github.com/MLReef/mlreef">MLReef</a> - Open source
MLOps platform that helps you collaborate, reproduce and share your ML
work.</li>
<li><a href="https://www.modzy.com/">Modzy</a> - Deploy, connect, run,
and monitor machine learning (ML) models in the enterprise and at the
edge.</li>
<li><a href="https://neu.ro">Neu.ro</a> - MLOps platform that integrates
open-source and proprietary tools into client-oriented systems.</li>
<li><a href="https://www.omniml.ai">Omnimizer</a> - Simplifies and
accelerates MLOps by bridging the gap between ML models and edge
hardware.</li>
<li><a href="https://www.pachyderm.com/">Pachyderm</a> - Combines data
lineage with end-to-end pipelines on Kubernetes, engineered for the
enterprise.</li>
<li><a href="https://www.github.com/polyaxon/polyaxon/">Polyaxon</a> - A
platform for reproducible and scalable machine learning and deep
learning on kubernetes.</li>
<li><a href="https://aws.amazon.com/sagemaker/">Sagemaker</a> - Fully
managed service that provides the ability to build, train, and deploy ML
models quickly.</li>
<li><a href="https://www.sas.com/en_us/software/viya.html">SAS Viya</a>
- Cloud native AI, analytic and data management platform that supports
the analytics life cycle.</li>
<li><a href="https://sematic.dev">Sematic</a> - An open-source
end-to-end pipelining tool to go from laptop prototype to cloud in no
time.</li>
<li><a href="https://sigopt.com/">SigOpt</a> - A platform that makes it
easy to track runs, visualize training, and scale hyperparameter
tuning.</li>
<li><a href="https://www.truefoundry.com">TrueFoundry</a> - A
Cloud-native MLOps Platform over Kubernetes to simplify training and
serving of ML Models.</li>
<li><a href="https://valohai.com/">Valohai</a> - Takes you from POC to
production while managing the whole model lifecycle.</li>
</ul>
<h2 id="model-fairness-and-privacy">Model Fairness and Privacy</h2>
<p><em>Tools for performing model fairness and privacy in
production.</em></p>
<ul>
<li><a href="https://github.com/Trusted-AI/AIF360">AIF360</a> - A
comprehensive set of fairness metrics for datasets and machine learning
models.</li>
<li><a href="https://github.com/fairlearn/fairlearn">Fairlearn</a> - A
Python package to assess and improve fairness of machine learning
models.</li>
<li><a href="https://github.com/pytorch/opacus">Opacus</a> - A library
that enables training PyTorch models with differential privacy.</li>
<li><a href="https://github.com/tensorflow/privacy">TensorFlow
Privacy</a> - Library for training machine learning models with privacy
for training data.</li>
</ul>
<h2 id="model-interpretability">Model Interpretability</h2>
<p><em>Tools for performing model
interpretability/explainability.</em></p>
<ul>
<li><a href="https://github.com/SeldonIO/alibi">Alibi</a> - Open-source
Python library enabling ML model inspection and interpretation.</li>
<li><a href="https://github.com/pytorch/captum">Captum</a> - Model
interpretability and understanding library for PyTorch.</li>
<li><a href="https://github.com/eli5-org/eli5">ELI5</a> - Python package
which helps to debug machine learning classifiers and explain their
predictions.</li>
<li><a href="https://github.com/interpretml/interpret">InterpretML</a> -
A toolkit to help understand models and enable responsible machine
learning.</li>
<li><a href="https://github.com/marcotcr/lime">LIME</a> - Explaining the
predictions of any machine learning classifier.</li>
<li><a href="https://github.com/tensorflow/lucid">Lucid</a> - Collection
of infrastructure and tools for research in neural network
interpretability.</li>
<li><a href="https://github.com/iancovert/sage">SAGE</a> - For
calculating global feature importance using Shapley values.</li>
<li><a href="https://github.com/slundberg/shap">SHAP</a> - A game
theoretic approach to explain the output of any machine learning
model.</li>
</ul>
<h2 id="model-lifecycle">Model Lifecycle</h2>
<p><em>Tools for managing model lifecycle (tracking experiments,
parameters and metrics).</em></p>
<ul>
<li><a href="https://github.com/aimhubio/aim">Aim</a> - A super-easy way
to record, search and compare 1000s of ML training runs.</li>
<li><a href="https://github.com/Oxid15/cascade">Cascade</a> - Library of
ML-Engineering tools for rapid prototyping and experiment
management.</li>
<li><a href="https://github.com/comet-ml">Comet</a> - Track your
datasets, code changes, experimentation history, and models.</li>
<li><a href="https://guild.ai/">Guild AI</a> - Open source experiment
tracking, pipeline automation, and hyperparameter tuning.</li>
<li><a href="https://github.com/replicate/keepsake">Keepsake</a> -
Version control for machine learning with support to Amazon S3 and
Google Cloud Storage.</li>
<li><a href="https://losswise.com">Losswise</a> - Makes it easy to track
the progress of a machine learning project.</li>
<li><a href="https://mlflow.org/">Mlflow</a> - Open source platform for
the machine learning lifecycle.</li>
<li><a href="https://github.com/VertaAI/modeldb/">ModelDB</a> - Open
source ML model versioning, metadata, and experiment management.</li>
<li><a href="https://neptune.ai/">Neptune AI</a> - The most lightweight
experiment management tool that fits any workflow.</li>
<li><a href="https://github.com/IDSIA/sacred">Sacred</a> - A tool to
help you configure, organize, log and reproduce experiments.</li>
<li><a href="https://github.com/wandb/client">Weights and Biases</a> - A
tool for visualizing and tracking your machine learning
experiments.</li>
</ul>
<h2 id="model-serving">Model Serving</h2>
<p><em>Tools for serving models in production.</em></p>
<ul>
<li><a href="https://banana.dev">Banana</a> - Host your ML inference
code on serverless GPUs and integrate it into your app with one line of
code.</li>
<li><a href="https://beam.cloud">Beam</a> - Develop on serverless GPUs,
deploy highly performant APIs, and rapidly prototype ML models.</li>
<li><a href="https://github.com/bentoml/BentoML">BentoML</a> -
Open-source platform for high-performance ML model serving.</li>
<li><a href="https://github.com/ebhy/budgetml">BudgetML</a> - Deploy a
ML inference service on a budget in less than 10 lines of code.</li>
<li><a href="https://github.com/replicate/cog">Cog</a> - Open-source
tool that lets you package ML models in a standard, production-ready
container.</li>
<li><a href="https://www.cortex.dev/">Cortex</a> - Machine learning
model serving infrastructure.</li>
<li><a href="https://docs.geniusrise.ai">Geniusrise</a> - Host inference
APIs, bulk inference and fine tune text, vision, audio and multi-modal
models.</li>
<li><a href="https://github.com/gradio-app/gradio">Gradio</a> - Create
customizable UI components around your models.</li>
<li><a href="https://oracle.github.io/graphpipe">GraphPipe</a> - Machine
learning model deployment made simple.</li>
<li><a
href="https://github.com/Hydrospheredata/hydro-serving">Hydrosphere</a>
- Platform for deploying your Machine Learning to production.</li>
<li><a href="https://github.com/kubeflow/kfserving">KFServing</a> -
Kubernetes custom resource definition for serving ML models on arbitrary
frameworks.</li>
<li><a href="https://github.com/mudler/LocalAI">LocalAI</a> - Drop-in
replacement REST API that’s compatible with OpenAI API specifications
for inferencing.</li>
<li><a href="https://github.com/gojek/merlin">Merlin</a> - A platform
for deploying and serving machine learning models.</li>
<li><a href="https://github.com/iterative/mlem">MLEM</a> - Version and
deploy your ML models following GitOps principles.</li>
<li><a href="https://github.com/ml-tooling/opyrator">Opyrator</a> -
Turns your ML code into microservices with web API, interactive GUI, and
more.</li>
<li><a href="https://github.com/apache/predictionio">PredictionIO</a> -
Event collection, deployment of algorithms, evaluation, querying
predictive results via APIs.</li>
<li><a href="https://quix.io">Quix</a> - Serverless platform for
processing data streams in real-time with machine learning models.</li>
<li><a href="https://github.com/hotg-ai/rune">Rune</a> - Provides
containers to encapsulate and deploy EdgeML pipelines and
applications.</li>
<li><a href="https://www.seldon.io/">Seldon</a> - Take your ML projects
from POC to production with maximum efficiency and minimal risk.</li>
<li><a href="https://github.com/streamlit/streamlit">Streamlit</a> -
Lets you create apps for your ML projects with deceptively simple Python
scripts.</li>
<li><a href="https://www.tensorflow.org/tfx/guide/serving">TensorFlow
Serving</a> - Flexible, high-performance serving system for ML models,
designed for production.</li>
<li><a href="https://github.com/pytorch/serve">TorchServe</a> - A
flexible and easy to use tool for serving PyTorch models.</li>
<li><a href="https://github.com/triton-inference-server/server">Triton
Inference Server</a> - Provides an optimized cloud and edge inferencing
solution.</li>
<li><a href="https://github.com/vespa-engine/vespa">Vespa</a> - Store,
search, organize and make machine-learned inferences over big data at
serving time.</li>
</ul>
<h2 id="model-testing-validation">Model Testing &amp; Validation</h2>
<p><em>Tools for testing and validating models.</em></p>
<ul>
<li><a href="https://github.com/deepchecks/deepchecks">Deepchecks</a> -
Open-source package for validating ML models &amp; data, with various
checks and suites.</li>
<li><a href="https://github.com/star-whale/starwhale">Starwhale</a> - An
MLOps/LLMOps platform for model building, evaluation, and
fine-tuning.</li>
<li><a href="https://github.com/trubrics/trubrics-sdk">Trubrics</a> -
Validate machine learning with data science and domain expert
feedback.</li>
</ul>
<h2 id="optimization-tools">Optimization Tools</h2>
<p><em>Optimization tools related to model scalability in
production.</em></p>
<ul>
<li><a href="https://github.com/huggingface/accelerate">Accelerate</a> -
A simple way to train and use PyTorch models with multi-GPU, TPU,
mixed-precision.</li>
<li><a href="https://dask.org/">Dask</a> - Provides advanced parallelism
for analytics, enabling performance at scale for the tools you
love.</li>
<li><a href="https://github.com/microsoft/DeepSpeed">DeepSpeed</a> -
Deep learning optimization library that makes distributed training easy,
efficient, and effective.</li>
<li><a href="https://uber.github.io/fiber/">Fiber</a> - Python
distributed computing library for modern computer clusters.</li>
<li><a href="https://github.com/horovod/horovod">Horovod</a> -
Distributed deep learning training framework for TensorFlow, Keras,
PyTorch, and Apache MXNet.</li>
<li><a href="https://mahout.apache.org/">Mahout</a> - Distributed linear
algebra framework and mathematically expressive Scala DSL.</li>
<li><a href="https://spark.apache.org/mllib/">MLlib</a> - Apache Spark’s
scalable machine learning library.</li>
<li><a href="https://github.com/modin-project/modin">Modin</a> - Speed
up your Pandas workflows by changing a single line of code.</li>
<li><a href="https://github.com/nebuly-ai/nebullvm">Nebullvm</a> -
Easy-to-use library to boost AI inference.</li>
<li><a href="https://github.com/nebuly-ai/nos">Nos</a> - Open-source
module for running AI workloads on Kubernetes in an optimized way.</li>
<li><a href="https://github.com/uber/petastorm">Petastorm</a> - Enables
single machine or distributed training and evaluation of deep learning
models.</li>
<li><a href="https://rapids.ai/index.html">Rapids</a> - Gives the
ability to execute end-to-end data science and analytics pipelines
entirely on GPUs.</li>
<li><a href="https://github.com/ray-project/ray">Ray</a> - Fast and
simple framework for building and running distributed applications.</li>
<li><a href="http://singa.apache.org/en/index.html">Singa</a> - Apache
top level project, focusing on distributed training of DL and ML
models.</li>
<li><a href="https://github.com/EpistasisLab/tpot">Tpot</a> - Automated
ML tool that optimizes machine learning pipelines using genetic
programming.</li>
</ul>
<h2 id="simplification-tools">Simplification Tools</h2>
<p><em>Tools related to machine learning simplification and
standardization.</em></p>
<ul>
<li><a href="https://chassisml.io">Chassis</a> - Turns models into
ML-friendly containers that run just about anywhere.</li>
<li><a href="https://github.com/a3data/hermione">Hermione</a> - Help
Data Scientists on setting up more organized codes, in a quicker and
simpler way.</li>
<li><a href="https://github.com/facebookresearch/hydra">Hydra</a> - A
framework for elegantly configuring complex applications.</li>
<li><a href="https://github.com/databricks/koalas">Koalas</a> - Pandas
API on Apache Spark. Makes data scientists more productive when
interacting with big data.</li>
<li><a href="https://github.com/uber/ludwig">Ludwig</a> - Allows users
to train and test deep learning models without the need to write
code.</li>
<li><a href="https://github.com/aporia-ai/mlnotify">MLNotify</a> - No
need to keep checking your training, just one import line and you’ll
know the second it’s done.</li>
<li><a href="https://pycaret.org/">PyCaret</a> - Open source, low-code
machine learning library in Python.</li>
<li><a href="https://github.com/Kenza-AI/sagify">Sagify</a> - A CLI
utility to train and deploy ML/DL models on AWS SageMaker.</li>
<li><a href="https://github.com/ploomber/soopervisor">Soopervisor</a> -
Export ML projects to Kubernetes (Argo workflows), Airflow, AWS Batch,
and SLURM.</li>
<li><a href="https://github.com/ploomber/soorgeon">Soorgeon</a> -
Convert monolithic Jupyter notebooks into maintainable pipelines.</li>
<li><a
href="https://github.com/jrieke/traingenerator">TrainGenerator</a> - A
web app to generate template code for machine learning.</li>
<li><a href="https://github.com/apple/turicreate">Turi Create</a> -
Simplifies the development of custom machine learning models.</li>
</ul>
<h2 id="visual-analysis-and-debugging">Visual Analysis and
Debugging</h2>
<p><em>Tools for performing visual analysis and debugging of ML/DL
models.</em></p>
<ul>
<li><a href="https://www.aporia.com/">Aporia</a> - Observability with
customized monitoring and explainability for ML models.</li>
<li><a href="https://www.arize.com/">Arize</a> - A free end-to-end ML
observability and model monitoring platform.</li>
<li><a href="https://github.com/comet-ml/comet-llm">CometLLM</a> -
Track, visualize, and evaluate your LLM prompts and chains in one
easy-to-use UI.</li>
<li><a href="https://github.com/evidentlyai/evidently">Evidently</a> -
Interactive reports to analyze ML models during validation or production
monitoring.</li>
<li><a href="https://www.fiddler.ai/">Fiddler</a> - Monitor, explain,
and analyze your AI in production.</li>
<li><a href="https://github.com/uber/manifold">Manifold</a> - A
model-agnostic visual debugging tool for machine learning.</li>
<li><a href="https://github.com/NannyML/nannyml">NannyML</a> - Algorithm
capable of fully capturing the impact of data drift on performance.</li>
<li><a href="https://github.com/lutzroeder/netron">Netron</a> -
Visualizer for neural network, deep learning, and machine learning
models.</li>
<li><a href="https://phoenix.arize.com">Phoenix</a> - MLOps in a
Notebook for troubleshooting and fine-tuning generative LLM, CV, and
tabular models.</li>
<li><a href="https://www.superwise.ai">Superwise</a> - Fully automated,
enterprise-grade model observability in a self-service SaaS
platform.</li>
<li><a href="https://github.com/whylabs/whylogs">Whylogs</a> - The open
source standard for data logging. Enables ML monitoring and
observability.</li>
<li><a
href="https://github.com/DistrictDataLabs/yellowbrick">Yellowbrick</a> -
Visual analysis and diagnostic tools to facilitate machine learning
model selection.</li>
</ul>
<h2 id="workflow-tools">Workflow Tools</h2>
<p><em>Tools and frameworks to create workflows or pipelines in the
machine learning context.</em></p>
<ul>
<li><a href="https://github.com/argoproj/argo">Argo</a> - Open source
container-native workflow engine for orchestrating parallel jobs on
Kubernetes.</li>
<li><a
href="https://www.veritone.com/applications/automate-studio/">Automate
Studio</a> - Rapidly build &amp; deploy AI-powered workflows.</li>
<li><a href="https://github.com/couler-proj/couler">Couler</a> - Unified
interface for constructing and managing workflows on different workflow
engines.</li>
<li><a href="https://github.com/dstackai/dstack">dstack</a> - An
open-core tool to automate data and training workflows.</li>
<li><a href="https://flyte.org/">Flyte</a> - Easy to create concurrent,
scalable, and maintainable workflows for machine learning.</li>
<li><a href="https://github.com/dagworks-inc/hamilton">Hamilton</a> - A
scalable general purpose micro-framework for defining dataflows.</li>
<li><a href="https://github.com/kubeflow-kale/kale">Kale</a> - Aims at
simplifying the Data Science experience of deploying Kubeflow Pipelines
workflows.</li>
<li><a href="https://github.com/quantumblacklabs/kedro">Kedro</a> -
Library that implements software engineering best-practice for data and
ML pipelines.</li>
<li><a href="https://github.com/spotify/luigi">Luigi</a> - Python module
that helps you build complex pipelines of batch jobs.</li>
<li><a href="https://metaflow.org/">Metaflow</a> - Human-friendly lib
that helps scientists and engineers build and manage data science
projects.</li>
<li><a href="https://github.com/mlrun/mlrun">MLRun</a> - Generic
mechanism for data scientists to build, run, and monitor ML tasks and
pipelines.</li>
<li><a href="https://github.com/orchest/orchest/">Orchest</a> - Visual
pipeline editor and workflow orchestrator with an easy to use UI and
based on Kubernetes.</li>
<li><a href="https://github.com/ploomber/ploomber">Ploomber</a> - Write
maintainable, production-ready pipelines. Develop locally, deploy to the
cloud.</li>
<li><a href="https://docs.prefect.io/">Prefect</a> - A workflow
management system, designed for modern infrastructure.</li>
<li><a href="https://github.com/instill-ai/vdp">VDP</a> - An open-source
tool to seamlessly integrate AI for unstructured data into the modern
data stack.</li>
<li><a href="https://github.com/maiot-io/zenml">ZenML</a> - An
extensible open-source MLOps framework to create reproducible
pipelines.</li>
</ul>
<hr />
<h1 id="resources">Resources</h1>
<p>Where to discover new tools and discuss about existing ones.</p>
<h2 id="articles">Articles</h2>
<ul>
<li><a
href="https://databaseline.tech/a-tour-of-end-to-end-ml-platforms/">A
Tour of End-to-End Machine Learning Platforms</a> (Databaseline)</li>
<li><a href="https://martinfowler.com/articles/cd4ml.html">Continuous
Delivery for Machine Learning</a> (Martin Fowler)</li>
<li><a
href="https://azure.microsoft.com/mediahandler/files/resourcefiles/gigaom-Delivering-on-the-Vision-of-MLOps/Delivering%20on%20the%20Vision%20of%20MLOps.pdf">Delivering
on the Vision of MLOps: A maturity-based approach</a> (GigaOm)</li>
<li><a href="https://arxiv.org/abs/2205.02302">Machine Learning
Operations (MLOps): Overview, Definition, and Architecture</a>
(arXiv)</li>
<li><a
href="https://cloud.google.com/solutions/machine-learning/mlops-continuous-delivery-and-automation-pipelines-in-machine-learning">MLOps:
Continuous delivery and automation pipelines in machine learning</a>
(Google)</li>
<li><a
href="https://towardsdatascience.com/ml-ops-machine-learning-as-an-engineering-discipline-b86ca4874a3f">MLOps:
Machine Learning as an Engineering Discipline</a> (Medium)</li>
<li><a
href="https://developers.google.com/machine-learning/guides/rules-of-ml">Rules
of Machine Learning: Best Practices for ML Engineering</a> (Google)</li>
<li><a
href="https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/aad9f93b86b7addfea4c419b9100c6cdd26cacea.pdf">The
ML Test Score: A Rubric for ML Production Readiness and Technical Debt
Reduction</a> (Google)</li>
<li><a
href="https://blogs.nvidia.com/blog/2020/09/03/what-is-mlops/">What Is
MLOps?</a> (NVIDIA)</li>
</ul>
<h2 id="books">Books</h2>
<ul>
<li><a
href="https://www.amazon.com/Beginning-MLOps-MLFlow-SageMaker-Microsoft/dp/1484265483">Beginning
MLOps with MLFlow</a> (Apress)</li>
<li><a
href="https://www.oreilly.com/library/view/building-machine-learning/9781492053187">Building
Machine Learning Pipelines</a> (O’Reilly)</li>
<li><a
href="https://www.oreilly.com/library/view/building-machine-learning/9781492045106">Building
Machine Learning Powered Applications</a> (O’Reilly)</li>
<li><a href="https://www.amazon.com/gp/product/6180033773">Deep Learning
in Production</a> (AI Summer)</li>
<li><a
href="https://www.oreilly.com/library/view/designing-machine-learning/9781098107956">Designing
Machine Learning Systems</a> (O’Reilly)</li>
<li><a
href="https://www.packtpub.com/product/engineering-mlops/9781800562882">Engineering
MLOps</a> (Packt)</li>
<li><a
href="https://www.oreilly.com/library/view/implementing-mlops-in/9781098136574">Implementing
MLOps in the Enterprise</a> (O’Reilly)</li>
<li><a
href="https://www.oreilly.com/library/view/introducing-mlops/9781492083283">Introducing
MLOps</a> (O’Reilly)</li>
<li><a
href="https://www.oreilly.com/library/view/kubeflow-for-machine/9781492050117">Kubeflow
for Machine Learning</a> (O’Reilly)</li>
<li><a
href="https://www.oreilly.com/library/view/kubeflow-operations-guide/9781492053262">Kubeflow
Operations Guide</a> (O’Reilly)</li>
<li><a
href="https://www.oreilly.com/library/view/machine-learning-design/9781098115777">Machine
Learning Design Patterns</a> (O’Reilly)</li>
<li><a
href="https://www.manning.com/books/machine-learning-engineering-in-action">Machine
Learning Engineering in Action</a> (Manning)</li>
<li><a
href="https://www.oreilly.com/library/view/ml-ops-operationalizing/9781492074663">ML
Ops: Operationalizing Data Science</a> (O’Reilly)</li>
<li><a
href="https://www.manning.com/books/mlops-engineering-at-scale">MLOps
Engineering at Scale</a> (Manning)</li>
<li><a
href="https://link.springer.com/book/10.1007/978-1-4842-9642-4">MLOps
Lifecycle Toolkit</a> (Apress)</li>
<li><a
href="https://www.packtpub.com/product/practical-deep-learning-at-scale-with-mlflow/9781803241333">Practical
Deep Learning at Scale with MLflow</a> (Packt)</li>
<li><a
href="https://www.oreilly.com/library/view/practical-mlops/9781098103002">Practical
MLOps</a> (O’Reilly)</li>
<li><a
href="https://www.packtpub.com/product/production-ready-applied-deep-learning/9781803243665">Production-Ready
Applied Deep Learning</a> (Packt)</li>
<li><a
href="https://www.oreilly.com/library/view/reliable-machine-learning/9781098106218">Reliable
Machine Learning</a> (O’Reilly)</li>
<li><a
href="https://www.packtpub.com/product/the-machine-learning-solutions-architect-handbook/9781801072168">The
Machine Learning Solutions Architect Handbook</a> (Packt)</li>
</ul>
<h2 id="events">Events</h2>
<ul>
<li><a href="https://www.applyconf.com/">apply() - The ML data
engineering conference</a></li>
<li><a
href="https://www.youtube.com/playlist?list=PLH8M0UOY0uy6d_n3vEQe6J_gRBUrISF9m">MLOps
Conference - Keynotes and Panels</a></li>
<li><a href="https://mlopsworld.com/">MLOps World: Machine Learning in
Production Conference</a></li>
<li><a href="https://normconf.com/">NormConf - The Normcore Tech
Conference</a></li>
<li><a href="https://mlsys.stanford.edu/">Stanford MLSys Seminar
Series</a></li>
</ul>
<h2 id="other-lists">Other Lists</h2>
<ul>
<li><a href="https://github.com/eugeneyan/applied-ml">Applied
ML</a></li>
<li><a
href="https://github.com/hibayesian/awesome-automl-papers">Awesome
AutoML Papers</a></li>
<li><a href="https://github.com/windmaple/awesome-AutoML">Awesome
AutoML</a></li>
<li><a href="https://github.com/academic/awesome-datascience">Awesome
Data Science</a></li>
<li><a href="https://github.com/kelvins/awesome-dataops">Awesome
DataOps</a></li>
<li><a
href="https://github.com/ChristosChristofidis/awesome-deep-learning">Awesome
Deep Learning</a></li>
<li><a href="https://github.com/leomaurodesenv/game-datasets">Awesome
Game Datasets</a> (includes AI content)</li>
<li><a
href="https://github.com/josephmisiti/awesome-machine-learning">Awesome
Machine Learning</a></li>
<li><a href="https://github.com/visenger/awesome-mlops">Awesome
MLOps</a></li>
<li><a
href="https://github.com/EthicalML/awesome-production-machine-learning">Awesome
Production Machine Learning</a></li>
<li><a href="https://github.com/vinta/awesome-python">Awesome
Python</a></li>
<li><a
href="https://github.com/ahkarami/Deep-Learning-in-Production">Deep
Learning in Production</a></li>
</ul>
<h2 id="podcasts">Podcasts</h2>
<ul>
<li><a href="https://how-ai-built-this.captivate.fm/">How AI Built
This</a></li>
<li><a href="https://kubernetespodcast.com/">Kubernetes Podcast from
Google</a></li>
<li><a
href="https://podcasts.google.com/?feed=aHR0cHM6Ly9zb2Z0d2FyZWVuZ2luZWVyaW5nZGFpbHkuY29tL2NhdGVnb3J5L21hY2hpbmUtbGVhcm5pbmcvZmVlZC8">Machine
Learning – Software Engineering Daily</a></li>
<li><a
href="https://podcasts.google.com/?feed=aHR0cHM6Ly9hbmNob3IuZm0vcy8xNzRjYjFiOC9wb2RjYXN0L3Jzcw">MLOps.community</a></li>
<li><a href="https://podcast.zenml.io/">Pipeline Conversation</a></li>
<li><a href="https://changelog.com/practicalai">Practical AI: Machine
Learning, Data Science</a></li>
<li><a href="https://twimlai.com/">This Week in Machine Learning &amp;
AI</a></li>
<li><a
href="https://www.youtube.com/playlist?list=PL4-eEhdXDO5F9Myvh41EeUh7oCgzqFRGk">True
ML Talks</a></li>
</ul>
<h2 id="slack">Slack</h2>
<ul>
<li><a href="https://kubeflow.slack.com/#/">Kubeflow Workspace</a></li>
<li><a href="https://mlops-community.slack.com">MLOps Community
Wokspace</a></li>
</ul>
<h2 id="websites">Websites</h2>
<ul>
<li><a href="http://featurestore.org/">Feature Stores for ML</a></li>
<li><a href="https://github.com/GokuMohandas/Made-With-ML">Made with
ML</a></li>
<li><a href="https://ml-ops.org/">ML-Ops</a></li>
<li><a href="https://mlops.community/">MLOps Community</a></li>
<li><a href="https://mlops-guide.github.io/">MLOps Guide</a></li>
<li><a href="https://mlopsnow.com">MLOps Now</a></li>
</ul>
<h1 id="contributing">Contributing</h1>
<p>All contributions are welcome! Please take a look at the <a
href="https://github.com/kelvins/awesome-mlops/blob/main/CONTRIBUTING.md">contribution
guidelines</a> first.</p>
Reference in New Issue View Git Blame Copy Permalink