awesome-awesomeness/html/auditalgorithms.html

<h1 id="awesome-audit-algorithms-awesome">Awesome Audit Algorithms <a
href="https://awesome.re"><img src="https://awesome.re/badge-flat.svg"
alt="Awesome" /></a></h1>
<p>A curated list of algorithms for auditing black-box algorithms.
Nowadays, many algorithms (recommendation, scoring, classification) are
operated at third party providers, without users or institutions having
any insights on how they operate on their data. Audit algorithms in this
list thus apply to this setup, coined the “black-box” setup, where one
auditor wants to get some insight on these remote algorithms.</p>
<p><img src="https://github.com/erwanlemerrer/awesome-audit-algorithms/blob/main/resources/audit.png" width="600" alt="banner" class="center"></p>
<blockquote>
<p>A user queries a remote algorithm (eg, through available APIs), to
infer information about that algorithm.</p>
</blockquote>
<h2 id="contents">Contents</h2>
<ul>
<li><a href="#papers">Papers</a></li>
<li><a href="#related-events">Related Events</a></li>
</ul>
<h2 id="papers">Papers</h2>
<h3 id="section">2024</h3>
<ul>
<li><a href="https://arxiv.org/pdf/2402.12572v1.pdf">FairProof:
Confidential and Certifiable Fairness for Neural Networks</a> -
<em>Proposes an alternative paradigm to traditional auditing using
crytographic tools like Zero-Knowledge Proofs; gives a system called
FairProof for verifying fairness of small neural networks.</em></li>
<li><a
href="https://grodino.github.io/projects/manipulated-audits/preprint.pdf">Under
manipulations, are some AI models harder to audit?</a> - (SATML)
<em>Relates the difficulty of black-box audits to the capacity of the
targeted models, using the Rademacher complexity.</em></li>
<li><a href="https://arxiv.org/pdf/2310.07219.pdf">Improved Membership
Inference Attacks Against Language Classification Models</a> - (ICLR)
<em>Presents a framework for running membership inference attacks
against classifier, in audit mode.</em></li>
<li><a href="https://arxiv.org/pdf/2305.17570.pdf">Auditing Fairness by
Betting</a> - (Neurips) <a
href="https://github.com/bchugg/auditing-fairness">[Code]</a>
<em>Sequential methods that allows for the continuous monitoring of
incoming data from a black-box classifier or regressor.</em> ###
2023</li>
<li><a href="https://arxiv.org/pdf/2206.04740.pdf">XAudit : A
Theoretical Look at Auditing with Explanations</a> - <em>Formalizes the
role of explanations in auditing and investigates if and how model
explanations can help audits.</em></li>
<li><a href="https://arxiv.org/pdf/2305.12620.pdf">Keeping Up with the
Language Models: Robustness-Bias Interplay in NLI Data and Models</a> -
<em>Proposes a way to extend the shelf-life of auditing datasets by
using language models themselves; also finds problems with the current
bias auditing metrics and proposes alternatives – these alternatives
highlight that model brittleness superficially increased the previous
bias scores.</em></li>
<li><a href="https://dl.acm.org/doi/pdf/10.1145/3580305.3599454">Online
Fairness Auditing through Iterative Refinement</a> - (KDD) <em>Provides
an adaptive process that automates the inference of probabilistic
guarantees associated with estimating fairness metrics.</em></li>
<li><a
href="https://people.cs.umass.edu/~amir/papers/CCS23-LM-stealing.pdf">Stealing
the Decoding Algorithms of Language Models</a> - (CCS) <em>Steal the
type and hyperparameters of the decoding algorithms of a LLM.</em></li>
<li><a
href="https://link.springer.com/epdf/10.1007/s13278-023-01105-9?sharing_token=h-O-asHI49VUWS9FxN1Gsve4RwlQNchNByi7wbcMAY6I98PKW1PqhFQJ_JqQyk3TrB05qDb3LUzMDmKOgrupccQliViDle-rwKEi2MZ8xBViaAQhyN41oZBKLLeXchoeIW2kklVHC094I5KD8pxja4-if6-iB0uAI1FnqnYoxjU%3D">Modeling
rabbit‑holes on YouTube</a> - (SNAM) <em>Models the trapping dynamics of
users in rabbit holes in YouTube, and provides a measure of this
enclosure.</em></li>
<li><a href="https://dl.acm.org/doi/full/10.1145/3568392">Auditing
YouTube’s Recommendation Algorithm for Misinformation Filter Bubbles</a>
- (Transactions on Recommender Systems) <em>What it takes to “burst the
bubble,” i.e., revert the bubble enclosure from
recommendations.</em></li>
<li><a href="https://arxiv.org/pdf/2308.02129.pdf">Auditing Yelp’s
Business Ranking and Review Recommendation Through the Lens of
Fairness</a> - (Arxiv) <em>Audits the fairness of Yelp’s business
ranking and review recommendation systems, with demographic parity,
exposure, and statistical tests such as quantile linear and logistic
regression.</em></li>
<li><a
href="https://openreview.net/pdf?id=iIfDQVyuFD">Confidential-PROFITT:
Confidential PROof of FaIr Training of Trees</a> - (ICLR) <em>Proposes
fair decision tree learning algorithms along with zero-knowledge proof
protocols to obtain a proof of fairness on the audited server.</em></li>
<li><a href="https://arxiv.org/pdf/2302.03251.pdf">SCALE-UP: An
Efficient Black-box Input-level Backdoor Detection via Analyzing Scaled
Prediction Consistency</a> - (ICLR) <em>Considers backdoor detection
under the black-box setting in machine learning as a service (MLaaS)
applications.</em> ### 2022</li>
<li><a
href="https://ojs.aaai.org/index.php/ICWSM/article/view/19300/19072">Two-Face:
Adversarial Audit of Commercial Face Recognition Systems</a> - (ICWSM)
<em>Performs an adversarial audit on multiple systems APIs and datasets,
making a number of concerning observations.</em></li>
<li><a
href="https://journals.sagepub.com/doi/10.1177/01655515221093029">Scaling
up search engine audits: Practical insights for algorithm auditing</a> -
(Journal of Information Science) <a
href="https://github.com/gesiscss/WebBot">(Code)</a> <em>Audits multiple
search engines using simulated browsing behavior with virtual
agents.</em></li>
<li><a href="https://openreview.net/pdf?id=OUz_9TiTv9j">A zest of lime:
towards architecture-independent model distances</a> - (ICLR)
<em>Measures the distance between two remote models using
LIME.</em></li>
<li><a
href="https://proceedings.mlr.press/v162/yan22c/yan22c.pdf">Active
Fairness Auditing</a> - (ICML) <em>Studies of query-based auditing
algorithms that can estimate the demographic parity of ML models in a
query-efficient manner.</em></li>
<li><a
href="https://proceedings.neurips.cc/paper/2021/file/da94cbeff56cfda50785df477941308b-Paper.pdf">Look
at the Variance! Efficient Black-box Explanations with Sobol-based
Sensitivity Analysis</a> - (NeurIPS) <em>Sobol indices provide an
efficient way to capture higher-order interactions between image regions
and their contributions to a (black box) neural network’s prediction
through the lens of variance.</em></li>
<li><a href="https://arxiv.org/pdf/2204.10920.pdf">Your Echos are Heard:
Tracking, Profiling, and Ad Targeting in the Amazon Smart Speaker
Ecosystem</a> - (arxiv) <em>Infers a link between the Amazon Echo system
and the ad targeting algorithm.</em> ### 2021</li>
<li><a href="https://arxiv.org/pdf/2102.00141.pdf">When the Umpire is
also a Player: Bias in Private Label Product Recommendations on
E-commerce Marketplaces</a> - (FAccT) <em>Do Amazon private label
products get an unfair share of recommendations and are therefore
advantaged compared to 3rd party products?</em></li>
<li><a href="https://arxiv.org/pdf/2105.02980.pdf">Everyday Algorithm
Auditing: Understanding the Power of Everyday Users in Surfacing Harmful
Algorithmic Behaviors</a> - (CHI) <em>Makes the case for “everyday
algorithmic auditing” by users.</em></li>
<li><a
href="https://www.cs.bu.edu/faculty/crovella/paper-archive/minimization-audit-Neurips21.pdf">Auditing
Black-Box Prediction Models for Data Minimization Compliance</a> -
(NeurIPS) <em>Measures the level of data minimization satisfied by the
prediction model using a limited number of queries.</em></li>
<li><a href="https://arxiv.org/abs/2012.05101">Setting the Record
Straighter on Shadow Banning</a> - (INFOCOM) <a
href="https://gitlab.enseeiht.fr/bmorgan/infocom-2021">(Code)</a>
<em>Considers the possibility of shadow banning in Twitter (ie, the
moderation black-box algorithm), and measures the probability of several
hypothesis.</em></li>
<li><a href="https://arxiv.org/pdf/2012.07805.pdf">Extracting Training
Data from Large Language Models</a> - (USENIX Security) <em>Extract
verbatim text sequences from the GPT-2 model’s training data.</em></li>
<li><a
href="https://www.sciencedirect.com/science/article/pii/S030645732100145X?casa_token=oyjFKij269MAAAAA:w_ohScpMPNMnkDdzBqAIod5QfBgQlq5Ht9mMRSOydZpOgNG-i1yuqEmBjWN__38gOGmjNL7dVT0">FairLens:
Auditing black-box clinical decision support systems</a> - (Information
Processing &amp; Management) <em>Presents a pipeline to detect and
explain potential fairness issues in Clinical DSS, by comparing
different multi-label classification disparity measures.</em></li>
<li><a
href="https://dl.acm.org/doi/abs/10.1145/3447535.3462491">Auditing
Algorithmic Bias on Twitter</a> - (WebSci).</li>
<li><a
href="https://proceedings.mlr.press/v139/neiswanger21a.html">Bayesian
Algorithm Execution: Estimating Computable Properties of Black-box
Functions Using Mutual Information</a> - (ICML) <em>A budget constrained
and Bayesian optimization procedure to extract properties out of a
black-box algorithm.</em> ### 2020</li>
<li><a
href="https://proceedings.neurips.cc/paper/2020/file/e8d66338fab3727e34a9179ed8804f64-Paper.pdf">Black-Box
Ripper: Copying black-box models using generative evolutionary
algorithms</a> - (NeurIPS) <em>Replicates the functionality of a
black-box neural model, yet with no limit on the amount of queries (via
a teacher/student scheme and an evolutionary search).</em></li>
<li><a
href="https://dl.acm.org/doi/pdf/10.1145/3351095.3372879">Auditing
radicalization pathways on</a> - (FAT<em>) </em>Studies the reachability
of radical channels from each others, using random walks on static
channel recommendations.*</li>
<li><a href="https://arxiv.org/abs/1912.07721">Adversarial Model
Extraction on Graph Neural Networks</a> - (AAAI Workshop on Deep
Learning on Graphs: Methodologies and Applications) <em>Introduces GNN
model extraction and presents a preliminary approach for this.</em></li>
<li><a href="https://rdcu.be/b6qB4">Remote Explainability faces the
bouncer problem</a> - (Nature Machine Intelligence volume 2,
pages529–539) <a
href="https://github.com/erwanlemerrer/bouncer_problem">(Code)</a>
<em>Shows the impossibility (with one request) or the difficulty to spot
lies on the explanations of a remote AI decision.</em></li>
<li><a
href="https://openaccess.thecvf.com/content_CVPR_2020/papers/Rahmati_GeoDA_A_Geometric_Framework_for_Black-Box_Adversarial_Attacks_CVPR_2020_paper.pdf">GeoDA:
a geometric framework for black-box adversarial attacks</a> - (CVPR) <a
href="https://github.com/thisisalirah/GeoDA">(Code)</a> <em>Crafts
adversarial examples to fool models, in a pure blackbox setup (no
gradients, inferred class only).</em></li>
<li><a
href="https://github.com/erwanlemerrer/erwanlemerrer.github.io/raw/master/files/imitation_blackbox_recommenders_netys-2020.pdf">The
Imitation Game: Algorithm Selectionby Exploiting Black-Box
Recommender</a> - (Netys) <a
href="https://github.com/gdamaskinos/RecRank">(Code)</a> <em>Parametrize
a local recommendation algorithm by imitating the decision of a remote
and better trained one.</em></li>
<li><a
href="https://ojs.aaai.org/index.php/ICWSM/article/view/7277">Auditing
News Curation Systems:A Case Study Examining Algorithmic and Editorial
Logic in Apple News</a> - (ICWSM) <em>Audit study of Apple News as a
sociotechnical news curation system (trending stories
section).</em></li>
<li><a
href="https://dl.acm.org/doi/pdf/10.1145/3375627.3375852">Auditing
Algorithms: On Lessons Learned and the Risks of DataMinimization</a> -
(AIES) <em>A practical audit for a well-being recommendation app
developed by Telefónica (mostly on bias).</em></li>
<li><a href="https://arxiv.org/pdf/2012.07805">Extracting Training Data
from Large Language Models</a> - (arxiv) <em>Performs a training data
extraction attack to recover individual training examples by querying
the language model.</em> ### 2019</li>
<li><a href="https://arxiv.org/abs/1711.01894">Adversarial Frontier
Stitching for Remote Neural Network Watermarking</a> - (Neural Computing
and Applications) <a
href="https://github.com/dunky11/adversarial-frontier-stitching">(Alternative
implementation)</a> <em>Check if a remote machine learning model is a
“leaked” one: through standard API requests to a remote model, extract
(or not) a zero-bit watermark, that was inserted to watermark valuable
models (eg, large deep neural networks).</em></li>
<li><a href="https://arxiv.org/abs/1812.02766.pdf">Knockoff Nets:
Stealing Functionality of Black-Box Models</a> - (CVPR) <em>Ask to what
extent can an adversary steal functionality of such “victim” models
based solely on blackbox interactions: image in, predictions
out.</em></li>
<li><a href="https://par.nsf.gov/servlets/purl/10101277">Opening Up the
Black Box:Auditing Google’s Top Stories Algorithm</a> - (Flairs-32)
<em>Audit of the Google’s Top stories panel that pro-vides insights into
its algorithmic choices for selectingand ranking news
publisher</em></li>
<li><a href="https://arxiv.org/pdf/1906.03397.pdf">Making targeted
black-box evasion attacks effective andefficient</a> - (arXiv)
<em>Investigates how an adversary can optimally use its query budget for
targeted evasion attacks against deep neural networks.</em></li>
<li><a
href="https://research.fb.com/wp-content/uploads/2019/05/Online-Learning-for-Measuring-Incentive-Compatibility-in-Ad-Auctions.pdf">Online
Learning for Measuring Incentive Compatibility in Ad Auctions</a> -
(WWW) <em>Measures the incentive compatible- (IC) mechanisms (regret) of
black-box auction platforms.</em></li>
<li><a href="https://arxiv.org/abs/1903.00317">TamperNN: Efficient
Tampering Detection of Deployed Neural Nets</a> - (ISSRE) <em>Algorithms
to craft inputs that can detect the tampering with a remotely executed
classifier model.</em></li>
<li><a href="https://arxiv.org/pdf/1903.03916.pdf">Neural Network Model
Extraction Attacks in Edge Devicesby Hearing Architectural Hints</a> -
(arxiv) <em>Through the acquisition of memory access events from bus
snooping, layer sequence identification bythe LSTM-CTC model, layer
topology connection according to the memory access pattern, and layer
dimension estimation under data volume constraints, it demonstrates one
can accurately recover the a similar network architecture as the attack
starting point</em></li>
<li><a
href="https://ieeexplore.ieee.org/abstract/document/8851798">Stealing
Knowledge from Protected Deep Neural Networks Using Composite Unlabeled
Data</a> - (ICNN) <em>Composite method which can be used to attack and
extract the knowledge ofa black box model even if it completely conceals
its softmaxoutput.</em></li>
<li><a href="https://dl.acm.org/citation.cfm?id=3354261">Neural Network
Inversion in Adversarial Setting via Background Knowledge Alignment</a>
- (CCS) <em>Model inversion approach in the adversary setting based on
training an inversion model that acts as aninverse of the original
model. With no fullknowledge about the original training data, an
accurate inversion is still possible by training the inversion model on
auxiliary samplesdrawn from a more generic data distribution.</em> ###
2018</li>
<li><a href="https://arxiv.org/abs/1711.00399">Counterfactual
Explanations without Opening the Black Box: Automated Decisions and the
GDPR</a> - (Harvard Journal of Law &amp; Technology) <em>To explain a
decision on x, find a conterfactual: the closest point to x that changes
the decision.</em></li>
<li><a href="https://arxiv.org/abs/1710.06169">Distill-and-Compare:
Auditing Black-Box Models Using Transparent Model Distillation</a> -
(AIES) <em>Treats black box models as teachers, training transparent
student models to mimic the risk scores assigned by black-box
models.</em></li>
<li><a href="https://arxiv.org/abs/1711.01768">Towards
Reverse-Engineering Black-Box Neural Networks</a> - (ICLR) <a
href="https://github.com/coallaoh/WhitenBlackBox">(Code)</a> <em>Infer
inner hyperparameters (eg number of layers, non-linear activation type)
of a remote neural network model by analysing its response patterns to
certain inputs.</em></li>
<li><a
href="https://www.sciencedirect.com/science/article/pii/S092523121830136X">Data
driven exploratory attacks on black box classifiers in adversarial
domains</a> - (Neurocomputing) <em>Reverse engineers remote classifier
models (e.g., for evading a CAPTCHA test).</em></li>
<li><a href="https://arxiv.org/pdf/1806.08867.pdf">xGEMs: Generating
Examplars to Explain Black-Box Models</a> - (arXiv) <em>Searches bias in
the black box model by training an unsupervised implicit generative
model. Thensummarizes the black-box model behavior quantitatively by
perturbing data samples along the data manifold.</em></li>
<li><a href="https://arxiv.org/pdf/1801.07386">Learning Networks from
Random Walk-Based Node Similarities</a> - (NIPS) <em>Reversing graphs by
observing some random walk commute times.</em></li>
<li><a
href="https://rd.springer.com/chapter/10.1007/978-3-030-00374-6_6">Identifying
the Machine Learning Family from Black-Box Models</a> - (CAEPIA)
<em>Determines which kind of machine learning model is behind the
returned predictions.</em></li>
<li><a href="https://arxiv.org/pdf/1812.11720.pdf">Stealing Neural
Networks via Timing Side Channels</a> - (arXiv)
<em>Stealing/approximating a model through timing attacks usin
queries.</em></li>
<li><a href="https://arxiv.org/abs/1806.05476">Copycat CNN: Stealing
Knowledge by Persuading Confession with Random Non-Labeled Data</a> -
(IJCNN) <a href="https://github.com/jeiks/Stealing_DL_Models">(Code)</a>
<em>Stealing black-box models (CNNs) knowledge by querying them with
random natural images (ImageNet and Microsoft-COCO).</em></li>
<li><a href="https://dl.acm.org/doi/10.1145/3178876.3186143">Auditing
the Personalization and Composition of Politically-Related Search Engine
Results Pages</a> - (WWW) <em>A Chrome extension to survey participants
and collect the Search Engine Results Pages (SERPs) and autocomplete
suggestions, for studying personalization and composition.</em> ###
2017</li>
<li><a href="https://dl.acm.org/authorize.cfm?key=N21772">Uncovering
Influence Cookbooks : Reverse Engineering the Topological Impact in Peer
Ranking Services</a> - (CSCW) <em>Aims at identifying which centrality
metrics are in use in a peer ranking service.</em></li>
<li><a href="https://arxiv.org/abs/1704.08991">The topological face of
recommendation: models and application to bias detection</a> - (Complex
Networks) <em>Proposes a bias detection framework for items recommended
to users.</em></li>
<li><a href="http://ieeexplore.ieee.org/document/7958568/">Membership
Inference Attacks Against Machine Learning Models</a> - (Symposium on
Security and Privacy) <em>Given a machine learning model and a record,
determine whether this record was used as part of the model’s training
dataset or not.</em></li>
<li><a href="https://dl.acm.org/citation.cfm?id=3053009">Practical
Black-Box Attacks against Machine Learning</a> - (Asia CCS)
<em>Understand how vulnerable is a remote service to adversarial
classification attacks.</em> ### 2016</li>
<li><a
href="https://www.andrew.cmu.edu/user/danupam/datta-sen-zick-oakland16.pdf">Algorithmic
Transparency via Quantitative Input Influence: Theory and Experiments
with Learning Systems</a> - (IEEE S&amp;P) <em>Evaluate the individual,
joint and marginal influence of features on a model using shapley
values.</em></li>
<li><a href="https://arxiv.org/abs/1602.07043">Auditing Black-Box Models
for Indirect Influence</a> - (ICDM) <em>Evaluate the influence of a
variable on a black-box model by “cleverly” removing it from the dataset
and looking at the accuracy gap</em></li>
<li><a href="https://arxiv.org/abs/1611.04967">Iterative Orthogonal
Feature Projection for Diagnosing Bias in Black-Box Models</a> - (FATML
Workshop) <em>Performs feature ranking to analyse black-box
models</em></li>
<li><a href="http://datworkshop.org/papers/dat16-final22.pdf">Bias in
Online Freelance Marketplaces: Evidence from TaskRabbit</a> - (dat
workshop) <em>Measures the TaskRabbit’s search algorithm rank.</em></li>
<li><a
href="https://www.usenix.org/conference/usenixsecurity16/technical-sessions/presentation/tramer">Stealing
Machine Learning Models via Prediction APIs</a> - (Usenix Security) <a
href="https://github.com/ftramer/Steal-ML">(Code)</a> <em>Aims at
extracting machine learning models in use by remote services.</em></li>
<li><a href="https://arxiv.org/pdf/1602.04938v3.pdf">“Why Should I Trust
You?”Explaining the Predictions of Any Classifier</a> - (arXiv) <a
href="https://github.com/marcotcr/lime-experiments">(Code)</a>
<em>Explains a blackbox classifier model by sampling around data
instances.</em></li>
<li><a href="http://ieeexplore.ieee.org/document/7546497/">Back in
Black: Towards Formal, Black Box Analysis of Sanitizers and Filters</a>
- (Security and Privacy) <em>Black-box analysis of sanitizers and
filters.</em></li>
<li><a href="http://ieeexplore.ieee.org/document/7546525/">Algorithmic
Transparency via Quantitative Input Influence: Theory and Experiments
with Learning Systems</a> - (Security and Privacy) <em>Introduces
measures that capture the degree of influence of inputs on outputs of
the observed system.</em></li>
<li><a href="https://mislove.org/publications/Amazon-WWW.pdf">An
Empirical Analysis of Algorithmic Pricing on Amazon Marketplace</a> -
(WWW) <a href="http://personalization.ccs.neu.edu">(Code)</a>
<em>Develops a methodology for detecting algorithmic pricing, and use it
empirically to analyze their prevalence and behavior on Amazon
Marketplace.</em> ### 2015</li>
<li><a href="https://arxiv.org/abs/1412.3756">Certifying and Removing
Disparate Impact</a> - (SIGKDD) <em>Proposes SVM-based methods to
certify absence of bias and methods to remove biases from a
dataset.</em></li>
<li><a href="https://dl.acm.org/citation.cfm?id=2815681">Peeking Beneath
the Hood of Uber</a> - (IMC) <em>Infer implementation details of Uber’s
surge price algorithm.</em> ### 2014</li>
<li><a href="">A peek into the black box: exploring classifiers by
randomization</a> - (Data Mining and Knowledge Discovery journal) (<a
href="https://github.com/tsabsch/goldeneye">code</a>) <em>Finds groups
of features that can be permuted without changing the output label of
predicted samples</em></li>
<li><a href="https://www.usenix.org/node/184394">XRay: Enhancing the
Web’s Transparency with Differential Correlation</a> - (USENIX Security)
<em>Audits which user profile data were used for targeting a particular
ad, recommendation, or price.</em> ### 2013</li>
<li><a href="https://dl.acm.org/citation.cfm?id=2488435">Measuring
Personalization of Web Search</a> - (WWW) <em>Develops a methodology for
measuring personalization in Web search result.</em></li>
<li><a
href="https://www.cs.bgu.ac.il/~sabatos/papers/SabatoSarwate13.pdf">Auditing:
Active Learning with Outcome-Dependent Query Costs</a> - (NIPS)
<em>Learns from a binary classifier paying only for negative
labels.</em></li>
</ul>
<h3 id="section-1">2012</h3>
<ul>
<li><a href="http://www.jmlr.org/papers/v13/nelson12a.html">Query
Strategies for Evading Convex-Inducing Classifiers</a> - (JMLR)
<em>Evasion methods for convex classifiers. Considers evasion
complexity.</em> ### 2008</li>
<li><a href="https://dl.acm.org/citation.cfm?id=1455806">Privacy Oracle:
a System for Finding Application Leakswith Black Box Differential
Testing</a> - (CCS) <em>Privacy Oracle: a system that uncovers
applications’ leaks of personal information in transmissions to
remoteservers.</em> ### 2005</li>
<li><a href="https://dl.acm.org/citation.cfm?id=1081950">Adversarial
Learning</a> - (KDD) <em>Reverse engineering of remote linear
classifiers, using membership queries.</em></li>
</ul>
<h2 id="related-events">Related Events</h2>
<ul>
<li><a href="https://algorithmic-audits.github.io">Workshop on
Algorithmic Audits of Algorithms (WAAA)</a></li>
<li><a href="https://regulatableml.github.io/">Regulatable ML Workshop
(RegML’23)</a></li>
</ul>