M3: Adversarial Machine Learning

Machine learning has seen a significant rate of adoption in recent years across a wide range of domains and applications. Deep learning with neural networks has become a highly popular machine learning method due to recent breakthroughs in computer vision, speech recognition, and other areas. However, machine learning algorithms can be fragile and easily fooled. For example, an attacker could add adversarial perturbations often invisible to human vision to an image to cause a deep neural network to misclassify the perturbed image. Such attacks go beyond image classification, and are effective across different neural network architectures and applications. An adversary with very weak access to a system, and little knowledge about the machine learning systems, can devise powerful attacks against such systems if he can interact with the system. This does not require direct access to the system and attacks can be launched against machine learning as service systems.

There is growing recognition that machine learning exposes new vulnerabilities and in response to these concerns, there is an emerging literature on adversarial machine learning. However, the community’s understanding of the nature and extent of the vulnerabilities in machine learning algorithms remains limited. This course will survey a broad array of these issues and techniques from both the cybersecurity and machine learning research areas. We will discuss the problems of adversarial classifier evasion, both the evasion and data poisoning attacks and the associated defensive techniques. We then consider specialized techniques for both attacking and defending deep learning techniques. Another group of attacks that will be discussed deals with stealing private data by interacting with the model. In this scenario the attacker can obtain data that was used during training, can infer if a particular data point was used in training or obtain the parameters of the model. Finally, we discuss some applications in cybersecurity domain.

Prerequisites: No specific prerequisite is required. A basic understanding of machine learning is enough.

Text: No textbook is required; online references will be provided.

Outline:

I.    Introduction to adversarial machine learning (15 min)

We will first review a taxonomy of threat models and potential attacks against machine learning algorithms. Generally, attacks against machine learning algorithms are categorized based on the effect they have on the classifier, the security violation they cause, how specific they are and the level of access the attacker has to the machine learning system.

II.    Evasion attacks and defenses (45 min)

Evasion attacks are the most prevalent type of attack that may be encountered in adversarial settings during system operation. For instance, spammers and hackers often attempt to evade detection by obfuscating the content of spam emails and malware code. In the evasion setting, malicious samples are modified at test time to evade detection; that is, to be misclassified as legitimate. These attacks require no influence over the training data. We will discuss how the evasion attacks work and some potential defense mechanisms.

III.    Data poisoning attacks (15 min)

Machine learning algorithms are often re-trained on data collected during operation to adapt to changes in the underlying data distribution. An attacker may poison the training data by injecting carefully designed samples to eventually compromise the whole learning process. Poisoning may thus be regarded as an adversarial contamination of the training data. We will discuss how the poisoning attacks work and some potential defense mechanisms.

IV.    Model inversion, membership inference and obtaining private training data (30 min)

Obtaining good machine learning models is not an easy process. Companies engaged in this process often wish to keep their developed models secret, such that the end user gets access only to the prediction interface and not to the model itself.. This is commonly known as machine learning as a service. Even access to the prediction interface, though, is often enough access for an attacker to obtain sensitive data, whether that be the model itself or the data on which the model was trained. Training data often contains highly sensitive data from other users of the system, e.g. medical data or private photos. We will discuss how attackers can extract a model through the prediction interface. We will also discuss how attackers can obtain sensitive training data through the prediction interface, and how an attacker may be able to infer whether a given piece of data was used during the training of the model.

V.    Attacks on deep neural networks (45 min)

Not only are deep neural networks also attackable with adversarial examples some suggest that current models are very easy to attack. We will discuss some hypothesis that have been brought forward on why adversarial examples exist. Additionally we will discuss how the creation of adversarial examples can be formulated as an optimization problem and different strategies for solving this problem. We will also discuss attempted defenses against adversarial examples and why is it hard to defend against adversarial examples. Furthermore, we will describe generative adversarial networks (GANs), how they can be used to create adversarial examples, how they can be attacked and further implications.

VI.    Application to Security (30 min)

In the past few years, many researchers have begun to apply machine learning techniques to various security problems. Security, however, is a difficult area because adversaries actively manipulate training data and vary attack techniques to defeat new systems. We will discuss adversarial machine learning problems across different security applications to see if there are common problems and effective solutions, and to determine if machine learning can indeed work well in adversarial environments.

About the Instructors:

Dr. Hassan Takabi is an Assistant Professor in the Department of Computer Science and Engineering at University of North Texas where he directs the INformation Security and Privacy: Interdisciplinary Research and Education (INSPIRE) Lab. He received his PhD from University of Pittsburgh in 2013. His research interests span a wide range of topics in cybersecurity and privacy including secure privacy preserving machine learning, advanced access control models, insider threats, and usable security and privacy. He has published three book chapters and more than 80 papers in renowned conferences and journals and is recipient of best paper award at ACM CODASY 2011. Dr. Takabi serves on organizing/ program committee of several top security conferences including ACM CCS, IEEE Security and Privacy, ACM CODASPY, and ACSAC. He is a member of IAPP, ACM, and IEEE.

Robert Podschwadt is a PhD student in computer science at the University of North Texas, where his research focuses on machine learning for cybersecurity, and particularly on attacking and defending machine learning systems from adversarial examples. His Masters thesis (2012, Hochschule der Medien, Stuttgart, Germany) presented innovations in GPU-aided machine learning with applications to energy grid management. Before starting the PhD program at UNT in fall 2017, Mr. Podschwadt worked for Sirrix, now Rohde & Schwarz Cybersecurity, where he designed and developed cybersecurity products for industry.