383 lines
18 KiB
HTML
383 lines
18 KiB
HTML
<div data-align="center">
|
||
<!-- title -->
|
||
<!--lint ignore no-dead-urls-->
|
||
<h1 id="awesome-xai-awesome">Awesome XAI <a
|
||
href="https://awesome.re"><img src="https://awesome.re/badge.svg"
|
||
alt="Awesome" /></a></h1>
|
||
<!-- subtitle -->
|
||
<p>A curated list of XAI and Interpretable ML papers, methods,
|
||
critiques, and resources.</p>
|
||
<!-- image -->
|
||
<p><img src="https://raw.githubusercontent.com/altamiracorp/awesome-xai/main/images/icon.svg" width="256" style="max-width: 25% !important"/></p>
|
||
<!-- description -->
|
||
<p>Explainable AI (XAI) is a branch of machine learning research which
|
||
seeks to make various machine learning techniques more
|
||
understandable.</p>
|
||
</div>
|
||
<!-- TOC -->
|
||
<h2 id="contents">Contents</h2>
|
||
<ul>
|
||
<li><a href="#papers">Papers</a>
|
||
<ul>
|
||
<li><a href="#landmarks">Landmarks</a></li>
|
||
<li><a href="#surveys">Surveys</a></li>
|
||
<li><a href="#evaluations">Evaluations</a></li>
|
||
<li><a href="#xai-methods">XAI Methods</a></li>
|
||
<li><a href="#interpretable-models">Interpretable Models</a></li>
|
||
<li><a href="#critiques">Critiques</a></li>
|
||
</ul></li>
|
||
<li><a href="#repositories">Repositories</a></li>
|
||
<li><a href="#videos">Videos</a></li>
|
||
<li><a href="#follow">Follow</a></li>
|
||
</ul>
|
||
<!-- CONTENT -->
|
||
<h2 id="papers">Papers</h2>
|
||
<h3 id="landmarks">Landmarks</h3>
|
||
<p>These are some of our favorite papers. They are helpful to understand
|
||
the field and critical aspects of it. We believe this papers are worth
|
||
reading in their entirety.</p>
|
||
<ul>
|
||
<li><a href="https://arxiv.org/abs/1706.07269">Explanation in Artificial
|
||
Intelligence: Insights from the Social Sciences</a> - This paper
|
||
provides an introduction to the social science research into
|
||
explanations. The author provides 4 major findings: (1) explanations are
|
||
constrastive, (2) explanations are selected, (3) probabilities probably
|
||
don’t matter, (4) explanations are social. These fit into the general
|
||
theme that explanations are -contextual-.</li>
|
||
<li><a href="https://arxiv.org/abs/1810.03292">Sanity Checks for
|
||
Saliency Maps</a> - An important read for anyone using saliency maps.
|
||
This paper proposes two experiments to determine whether saliency maps
|
||
are useful: (1) model parameter randomization test compares maps from
|
||
trained and untrained models, (2) data randomization test compares maps
|
||
from models trained on the original dataset and models trained on the
|
||
same dataset with randomized labels. They find that “some widely
|
||
deployed saliency methods are independent of both the data the model was
|
||
trained on, and the model parameters”.</li>
|
||
</ul>
|
||
<h3 id="surveys">Surveys</h3>
|
||
<ul>
|
||
<li><a href="https://arxiv.org/abs/2004.14545">Explainable Deep
|
||
Learning: A Field Guide for the Uninitiated</a> - An in-depth
|
||
description of XAI focused on technqiues for deep learning.</li>
|
||
</ul>
|
||
<h3 id="evaluations">Evaluations</h3>
|
||
<ul>
|
||
<li><a href="https://arxiv.org/abs/2009.02899">Quantifying
|
||
Explainability of Saliency Methods in Deep Neural Networks</a> - An
|
||
analysis of how different heatmap-based saliency methods perform based
|
||
on experimentation with a generated dataset.</li>
|
||
</ul>
|
||
<h3 id="xai-methods">XAI Methods</h3>
|
||
<ul>
|
||
<li><a href="https://arxiv.org/abs/2102.07799">Ada-SISE</a> - Adaptive
|
||
semantice inpute sampling for explanation.</li>
|
||
<li><a
|
||
href="https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/rssb.12377">ALE</a>
|
||
- Accumulated local effects plot.</li>
|
||
<li><a
|
||
href="https://link.springer.com/chapter/10.1007/978-3-030-33607-3_49">ALIME</a>
|
||
- Autoencoder Based Approach for Local Interpretability.</li>
|
||
<li><a
|
||
href="https://ojs.aaai.org/index.php/AAAI/article/view/11491">Anchors</a>
|
||
- High-Precision Model-Agnostic Explanations.</li>
|
||
<li><a
|
||
href="https://link.springer.com/article/10.1007/s10115-017-1116-3">Auditing</a>
|
||
- Auditing black-box models.</li>
|
||
<li><a href="https://arxiv.org/abs/2012.03058">BayLIME</a> - Bayesian
|
||
local interpretable model-agnostic explanations.</li>
|
||
<li><a href="http://ema.drwhy.ai/breakDown.html#BDMethod">Break Down</a>
|
||
- Break down plots for additive attributions.</li>
|
||
<li><a
|
||
href="https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Zhou_Learning_Deep_Features_CVPR_2016_paper.pdf">CAM</a>
|
||
- Class activation mapping.</li>
|
||
<li><a
|
||
href="https://ieeexplore.ieee.org/abstract/document/4167900">CDT</a> -
|
||
Confident interpretation of Bayesian decision tree ensembles.</li>
|
||
<li><a
|
||
href="https://christophm.github.io/interpretable-ml-book/ice.html">CICE</a>
|
||
- Centered ICE plot.</li>
|
||
<li><a
|
||
href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.40.2710&rep=rep1&type=pdf">CMM</a>
|
||
- Combined multiple models metalearner.</li>
|
||
<li><a
|
||
href="https://www.sciencedirect.com/science/article/pii/B9781558603356500131">Conj
|
||
Rules</a> - Using sampling and queries to extract rules from trained
|
||
neural networks.</li>
|
||
<li><a
|
||
href="https://ieeexplore.ieee.org/abstract/document/6597214">CP</a> -
|
||
Contribution propogation.</li>
|
||
<li><a
|
||
href="https://dl.acm.org/doi/abs/10.1145/775047.775113">DecText</a> -
|
||
Extracting decision trees from trained neural networks.</li>
|
||
<li><a
|
||
href="https://ieeexplore-ieee-org.ezproxy.libraries.wright.edu/abstract/document/9352498">DeepLIFT</a>
|
||
- Deep label-specific feature learning for image annotation.</li>
|
||
<li><a
|
||
href="https://www.sciencedirect.com/science/article/pii/S0031320316303582">DTD</a>
|
||
- Deep Taylor decomposition.</li>
|
||
<li><a
|
||
href="https://www.aaai.org/Papers/IAAI/2006/IAAI06-018.pdf">ExplainD</a>
|
||
- Explanations of evidence in additive classifiers.</li>
|
||
<li><a
|
||
href="https://link.springer.com/chapter/10.1007/978-3-642-04174-7_45">FIRM</a>
|
||
- Feature importance ranking measure.</li>
|
||
<li><a
|
||
href="https://openaccess.thecvf.com/content_iccv_2017/html/Fong_Interpretable_Explanations_of_ICCV_2017_paper.html">Fong,
|
||
et. al.</a> - Meaninful perturbations model.</li>
|
||
<li><a
|
||
href="https://www.academia.edu/download/51462700/s0362-546x_2896_2900267-220170122-9600-1njrpyx.pdf">G-REX</a>
|
||
- Rule extraction using genetic algorithms.</li>
|
||
<li><a
|
||
href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3977175/">Gibbons,
|
||
et. al.</a> - Explain random forest using decision tree.</li>
|
||
<li><a
|
||
href="https://link-springer-com.ezproxy.libraries.wright.edu/article/10.1007/s10618-014-0368-8">GoldenEye</a>
|
||
- Exploring classifiers by randomization.</li>
|
||
<li><a href="https://arxiv.org/abs/0912.1128">GPD</a> - Gaussian process
|
||
decisions.</li>
|
||
<li><a
|
||
href="https://ieeexplore.ieee.org/abstract/document/4938655">GPDT</a> -
|
||
Genetic program to evolve decision trees.</li>
|
||
<li><a
|
||
href="https://openaccess.thecvf.com/content_iccv_2017/html/Selvaraju_Grad-CAM_Visual_Explanations_ICCV_2017_paper.html">GradCAM</a>
|
||
- Gradient-weighted Class Activation Mapping.</li>
|
||
<li><a
|
||
href="https://ieeexplore.ieee.org/abstract/document/8354201/">GradCAM++</a>
|
||
- Generalized gradient-based visual explanations.</li>
|
||
<li><a href="https://arxiv.org/abs/1606.05390">Hara, et. al.</a> -
|
||
Making tree ensembles interpretable.</li>
|
||
<li><a
|
||
href="https://www.tandfonline.com/doi/abs/10.1080/10618600.2014.907095">ICE</a>
|
||
- Individual conditional expectation plots.</li>
|
||
<li><a
|
||
href="http://proceedings.mlr.press/v70/sundararajan17a/sundararajan17a.pdf">IG</a>
|
||
- Integrated gradients.</li>
|
||
<li><a
|
||
href="https://link.springer.com/article/10.1007/s41060-018-0144-8">inTrees</a>
|
||
- Interpreting tree ensembles with inTrees.</li>
|
||
<li><a href="https://arxiv.org/abs/1611.04967">IOFP</a> - Iterative
|
||
orthoganol feature projection.</li>
|
||
<li><a href="https://arxiv.org/abs/1703.00810">IP</a> - Information
|
||
plane visualization.</li>
|
||
<li><a href="https://arxiv.org/abs/1810.02678">KL-LIME</a> -
|
||
Kullback-Leibler Projections based LIME.</li>
|
||
<li><a
|
||
href="https://www.sciencedirect.com/science/article/abs/pii/S0031320398001812">Krishnan,
|
||
et. al.</a> - Extracting decision trees from trained neural
|
||
networks.</li>
|
||
<li><a href="https://arxiv.org/abs/1606.04155">Lei, et. al.</a> -
|
||
Rationalizing neural predictions with generator and encoder.</li>
|
||
<li><a
|
||
href="https://dl.acm.org/doi/abs/10.1145/2939672.2939778">LIME</a> -
|
||
Local Interpretable Model-Agnostic Explanations.</li>
|
||
<li><a
|
||
href="https://amstat.tandfonline.com/doi/abs/10.1080/01621459.2017.1307116#.YEkdZ7CSmUk">LOCO</a>
|
||
- Leave-one covariate out.</li>
|
||
<li><a href="https://arxiv.org/abs/1805.10820">LORE</a> - Local
|
||
rule-based explanations.</li>
|
||
<li><a href="https://dl.acm.org/doi/abs/10.1145/2487575.2487579">Lou,
|
||
et. al.</a> - Accurate intelligibile models with pairwise
|
||
interactions.</li>
|
||
<li><a
|
||
href="https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0130140">LRP</a>
|
||
- Layer-wise relevance propogation.</li>
|
||
<li><a
|
||
href="https://www.jmlr.org/papers/volume20/18-760/18-760.pdf">MCR</a> -
|
||
Model class reliance.</li>
|
||
<li><a
|
||
href="https://ieeexplore.ieee.org/abstract/document/7738872">MES</a> -
|
||
Model explanation system.</li>
|
||
<li><a href="https://arxiv.org/abs/1611.07567">MFI</a> - Feature
|
||
importance measure for non-linear algorithms.</li>
|
||
<li><a
|
||
href="https://www.sciencedirect.com/science/article/abs/pii/S0304380002000649">NID</a>
|
||
- Neural interpretation diagram.</li>
|
||
<li><a href="https://arxiv.org/abs/2006.05714">OptiLIME</a> - Optimized
|
||
LIME.</li>
|
||
<li><a
|
||
href="https://dl.acm.org/doi/abs/10.1145/3077257.3077271">PALM</a> -
|
||
Partition aware local model.</li>
|
||
<li><a href="https://arxiv.org/abs/1702.04595">PDA</a> - Prediction
|
||
Difference Analysis: Visualize deep neural network decisions.</li>
|
||
<li><a
|
||
href="https://projecteuclid.org/download/pdf_1/euclid.aos/1013203451">PDP</a>
|
||
- Partial dependence plots.</li>
|
||
<li><a
|
||
href="https://academic.oup.com/bioinformatics/article/24/13/i6/233341">POIMs</a>
|
||
- Positional oligomer importance matrices for understanding SVM signal
|
||
detectors.</li>
|
||
<li><a href="https://arxiv.org/abs/1807.07506">ProfWeight</a> - Transfer
|
||
information from deep network to simpler model.</li>
|
||
<li><a
|
||
href="https://dl.acm.org/doi/abs/10.1145/2858036.2858529">Prospector</a>
|
||
- Interactive partial dependence diagnostics.</li>
|
||
<li><a
|
||
href="https://ieeexplore.ieee.org/abstract/document/7546525">QII</a> -
|
||
Quantitative input influence.</li>
|
||
<li><a
|
||
href="https://content.iospress.com/articles/ai-communications/aic272">REFNE</a>
|
||
- Extracting symbolic rules from trained neural network ensembles.</li>
|
||
<li><a href="https://arxiv.org/abs/1608.05745">RETAIN</a> - Reverse time
|
||
attention model.</li>
|
||
<li><a href="https://arxiv.org/abs/1806.07421">RISE</a> - Randomized
|
||
input sampling for explanation.</li>
|
||
<li><a
|
||
href="https://link.springer.com/article/10.1007%2Fs11063-011-9207-8">RxREN</a>
|
||
- Reverse engineering neural networks for rule extraction.</li>
|
||
<li><a href="https://arxiv.org/abs/1705.07874">SHAP</a> - A unified
|
||
approach to interpretting model predictions.</li>
|
||
<li><a href="https://arxiv.org/abs/2101.10710">SIDU</a> - Similarity,
|
||
difference, and uniqueness input perturbation.</li>
|
||
<li><a href="https://arxiv.org/abs/1312.6034">Simonynan, et. al</a> -
|
||
Visualizing CNN classes.</li>
|
||
<li><a href="https://arxiv.org/abs/1611.07579">Singh, et. al</a> -
|
||
Programs as black-box explanations.</li>
|
||
<li><a href="https://arxiv.org/abs/1610.09036">STA</a> - Interpreting
|
||
models via Single Tree Approximation.</li>
|
||
<li><a
|
||
href="https://www.jmlr.org/papers/volume11/strumbelj10a/strumbelj10a.pdf">Strumbelj,
|
||
et. al.</a> - Explanation of individual classifications using game
|
||
theory.</li>
|
||
<li><a
|
||
href="https://www.academia.edu/download/2471122/3uecwtv9xcwxg6r.pdf">SVM+P</a>
|
||
- Rule extraction from support vector machines.</li>
|
||
<li><a href="https://openreview.net/forum?id=S1viikbCW">TCAV</a> -
|
||
Testing with concept activation vectors.</li>
|
||
<li><a
|
||
href="https://dl.acm.org/doi/abs/10.1145/3097983.3098039">Tolomei, et.
|
||
al.</a> - Interpretable predictions of tree-ensembles via actionable
|
||
feature tweaking.</li>
|
||
<li><a
|
||
href="https://www.researchgate.net/profile/Edward-George-2/publication/2610587_Making_Sense_of_a_Forest_of_Trees/links/55b1085d08aec0e5f430eb40/Making-Sense-of-a-Forest-of-Trees.pdf">Tree
|
||
Metrics</a> - Making sense of a forest of trees.</li>
|
||
<li><a href="https://arxiv.org/abs/1706.06060">TreeSHAP</a> - Consistent
|
||
feature attribute for tree ensembles.</li>
|
||
<li><a href="https://arxiv.org/abs/1611.07429">TreeView</a> -
|
||
Feature-space partitioning.</li>
|
||
<li><a
|
||
href="http://www.inf.ufrgs.br/~engel/data/media/file/cmp121/TREPAN_craven.nips96.pdf">TREPAN</a>
|
||
- Extracting tree-structured representations of trained networks.</li>
|
||
<li><a href="https://dl.acm.org/doi/abs/10.1145/3412815.3416893">TSP</a>
|
||
- Tree space prototypes.</li>
|
||
<li><a
|
||
href="http://www.columbia.edu/~aec2163/NonFlash/Papers/VisualBackProp.pdf">VBP</a>
|
||
- Visual back-propagation.</li>
|
||
<li><a
|
||
href="https://ieeexplore.ieee.org/abstract/document/5949423">VEC</a> -
|
||
Variable effect characteristic curve.</li>
|
||
<li><a href="https://dl.acm.org/doi/abs/10.1145/1014052.1014122">VIN</a>
|
||
- Variable interaction network.</li>
|
||
<li><a href="https://arxiv.org/abs/1508.07551">X-TREPAN</a> - Adapted
|
||
etraction of comprehensible decision tree in ANNs.</li>
|
||
<li><a href="http://proceedings.mlr.press/v37/xuc15">Xu, et. al.</a> -
|
||
Show, attend, tell attention model.</li>
|
||
</ul>
|
||
<h3 id="interpretable-models">Interpretable Models</h3>
|
||
<ul>
|
||
<li><a
|
||
href="https://christophm.github.io/interpretable-ml-book/rules.html">Decision
|
||
List</a> - Like a decision tree with no branches.</li>
|
||
<li><a href="https://en.wikipedia.org/wiki/Decision_tree">Decision
|
||
Trees</a> - The tree provides an interpretation.</li>
|
||
<li><a href="https://www.youtube.com/watch?v=MREiHgHgl0k">Explainable
|
||
Boosting Machine</a> - Method that predicts based on learned vector
|
||
graphs of features.</li>
|
||
<li><a
|
||
href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">k-Nearest
|
||
Neighbors</a> - The prototypical clustering method.</li>
|
||
<li><a href="https://en.wikipedia.org/wiki/Linear_regression">Linear
|
||
Regression</a> - Easily plottable and understandable regression.</li>
|
||
<li><a href="https://en.wikipedia.org/wiki/Logistic_regression">Logistic
|
||
Regression</a> - Easily plottable and understandable
|
||
classification.</li>
|
||
<li><a href="https://en.wikipedia.org/wiki/Naive_Bayes_classifier">Naive
|
||
Bayes</a> - Good classification, poor estimation using conditional
|
||
probabilities.</li>
|
||
<li><a
|
||
href="https://christophm.github.io/interpretable-ml-book/rulefit.html">RuleFit</a>
|
||
- Sparse linear model as decision rules including feature
|
||
interactions.</li>
|
||
</ul>
|
||
<h3 id="critiques">Critiques</h3>
|
||
<ul>
|
||
<li><a href="https://arxiv.org/abs/1902.10186">Attention is not
|
||
Explanation</a> - Authors perform a series of NLP experiments which
|
||
argue attention does not provide meaningful explanations. They also
|
||
demosntrate that different attentions can generate similar model
|
||
outputs.</li>
|
||
<li><a href="https://arxiv.org/abs/1908.04626">Attention is not –not–
|
||
Explanation</a> - This is a rebutal to the above paper. Authors argue
|
||
that multiple explanations can be valid and that the and that attention
|
||
can produce <em>a</em> valid explanation, if not -the- valid
|
||
explanation.</li>
|
||
<li><a href="https://arxiv.org/abs/1903.11420">Do Not Trust Additive
|
||
Explanations</a> - Authors argue that addditive explanations (e.g. LIME,
|
||
SHAP, Break Down) fail to take feature ineractions into account and are
|
||
thus unreliable.</li>
|
||
<li><a href="https://arxiv.org/abs/1905.03151">Please Stop Permuting
|
||
Features An Explanation and Alternatives</a> - Authors demonstrate why
|
||
permuting features is misleading, especially where there is strong
|
||
feature dependence. They offer several previously described
|
||
alternatives.</li>
|
||
<li><a
|
||
href="https://www.nature.com/articles/s42256-019-0048-x?fbclid=IwAR3156gP-ntoAyw2sHTXo0Z8H9p-2wBKe5jqitsMCdft7xA0P766QvSthFs">Stop
|
||
Explaining Black Box Machine Learning Models for High States Decisions
|
||
and Use Interpretable Models Instead</a> - Authors present a number of
|
||
issues with explainable ML and challenges to interpretable ML: (1)
|
||
constructing optimal logical models, (2) constructing optimal sparse
|
||
scoring systems, (3) defining interpretability and creating methods for
|
||
specific methods. They also offer an argument for why interpretable
|
||
models might exist in many different domains.</li>
|
||
<li><a
|
||
href="https://link.springer.com/chapter/10.1007/978-3-030-28954-6_14">The
|
||
(Un)reliability of Saliency Methods</a> - Authors demonstrate how
|
||
saliency methods vary attribution when adding a constant shift to the
|
||
input data. They argue that methods should fulfill <em>input
|
||
invariance</em>, that a saliency method mirror the sensistivity of the
|
||
model with respect to transformations of the input.</li>
|
||
</ul>
|
||
<h2 id="repositories">Repositories</h2>
|
||
<ul>
|
||
<li><a href="https://github.com/EthicalML/xai">EthicalML/xai</a> - A
|
||
toolkit for XAI which is focused exclusively on tabular data. It
|
||
implements a variety of data and model evaluation techniques.</li>
|
||
<li><a href="https://github.com/MAIF/shapash">MAIF/shapash</a> - SHAP
|
||
and LIME-based front-end explainer.</li>
|
||
<li><a
|
||
href="https://github.com/PAIR-code/what-if-tool">PAIR-code/what-if-tool</a>
|
||
- A tool for Tensorboard or Notebooks which allows investigating model
|
||
performance and fairness.</li>
|
||
<li><a href="https://github.com/slundberg/shap">slundberg/shap</a> - A
|
||
Python module for using Shapley Additive Explanations.</li>
|
||
</ul>
|
||
<h2 id="videos">Videos</h2>
|
||
<ul>
|
||
<li><a href="https://www.youtube.com/watch?v=93Xv8vJ2acI">Debate:
|
||
Interpretability is necessary for ML</a> - A debate on whether
|
||
interpretability is necessary for ML with Rich Caruana and Patrice
|
||
Simard for and Kilian Weinberger and Yann LeCun against.</li>
|
||
</ul>
|
||
<h2 id="follow">Follow</h2>
|
||
<p>Their views aren’t necessarily our views. :wink:</p>
|
||
<ul>
|
||
<li><a href="https://ethical.institute/index.html">The Institute for
|
||
Ethical AI & Machine Learning</a> - A UK-based research center that
|
||
performs research into ethical AI/ML, which frequently involves
|
||
XAI.</li>
|
||
<li><a href="https://twitter.com/tmiller_unimelb">Tim Miller</a> - One
|
||
of the preeminent researchers in XAI.</li>
|
||
<li><a
|
||
href="https://www.microsoft.com/en-us/research/people/rcaruana/">Rich
|
||
Caruana</a> - The man behind Explainable Boosting Machines.</li>
|
||
</ul>
|
||
<p>Who else should we be following!?</p>
|
||
<h2 id="contributing">Contributing</h2>
|
||
<p><a href="contributing.md">Contributions of any kind welcome, just
|
||
follow the guidelines</a>!</p>
|
||
<h3 id="contributors">Contributors</h3>
|
||
<p><a
|
||
href="https://github.com/altamiracorp/awesome-xai/graphs/contributors">Thanks
|
||
goes to these contributors</a>!</p>
|
||
<p><a href="https://github.com/altamiracorp/awesome-xai">xai.md
|
||
Github</a></p>
|