Full Program »
Learning from Authoritative Security Experiment Results (LASER) Workshop
Tuesday, 6 December 2022
08:30 - 12:00
Classroom 116
See the original Call for Participation
Workshop Overview
The LASER workshop series focuses on learning from and improving cybersecurity experiment results. The workshop strives to provide a highly interactive, collegial environment for discussing and learning from experimental methodologies, execution, and results. Ultimately, the workshop seeks to foster a dramatic change in the experimental paradigm for cybersecurity research, improving the overall quality and reporting of practiced science.
The LASER workshop invites broad participation by the community, including (1) authors of accepted papers from major cybersecurity conferences to present and discuss the experimental aspects of their work, and (2) others interested in contributing to and learning from such discussions and interaction.
Conference papers all too often must focus on research results and contain limited discussion of the experimental aspects of the work (maybe a small section with a few paragraphs at the end of the paper). LASER provides an opportunity to focus on and explore the experimental approaches and methodologies used to obtain the research results.
The LASER workshop not only provides authors of accepted papers the opportunity to present and discuss the experimental aspects of their work with other workshop participants, but also the option to write new published papers that expand on the experimental aspects of their work.
Workshop Format
The workshop will be structured as a true “workshop” in the sense that it will focus on discussion and interaction around the topic of experimental methodologies, execution, and results with the goal of encouraging improvements in experimental science in cybersecurity research. Authors will lead the group in a discussion of the experimental aspects of their work.
Areas of interest include, but are not limited to, the following:
- Research questions and/or hypothesis
- Experimental methodologies used and/or developed
- Experiment design
- Use of simulation, emulation, virtualization, and/or physical testbeds
- Use of specialized hardware including CPS and IoT devices
- Modeling of human-behavior characteristics
- Software tools used and/or developed to perform experimentation
- Approaches to experiment validation, monitoring, and data collection
- Datasets used and/or developed to perform experimentation
- Measurements and metrics
- Analytical techniques used and/or developed to evaluate experimental results
As a group, participants will discuss these areas and answer interesting questions such as:
- Did you use experimentation artifacts borrowed from the community?
- Did you attempt to replicate or reproduce results of earlier research as part of your work?
- What can be learned from your methodology and your experience using your methodology?
- What did you try that did not succeed before getting to the results you presented?
- Did you produce any intermediate results including possible unsuccessful tests or experiments?
Program
| TUESDAY, DECEMBER 6, 2022 (ALL TIMES CST) | |
| 8:15 am - 8:30 am | Gathering |
| 8:30 am - 8:45 am | Welcome, Introductions, Workshop Goals and Agenda SLIDES |
| 8:45 am - 10:15 am | Session 1: Paper Discussions |
| Threats in Crowdsourcing Threat Intelligence for Practical Threat Triaging Afsaf Anwar, Northeastern University SLIDES |
|
| Exploring Backdoors in Federated Graph Neural Networks Stefanos Koffas, TU Delft SLIDES |
|
| 10:15 am - 10:45 am | Break |
| 10:45 am - 12:15 pm | Session 2: Paper Discussions |
| Simulation of Differentially Private Federated Meta-learning Systems Ning Wang, Virginia Tech SLIDES |
|
| Torches on Pitchfork: Multi-feature Evaluation of a Security-oriented Programming Toolchain Nik Sultana, Illinois Institute of Technology SLIDES |
|
| 12:15 pm - 1:30 pm | Lunch |
| 1:30 pm - 3:00 pm | Keynote Talk and Discussion |
| Towards True Reproducibility of Findings in Cybersecurity Research Emma Tosch, Researcher, Northeastern University SLIDES |
|
| 3:00 pm - 3:30 pm | Break |
| 3:30 pm - 5:00 pm | Session 3: Paper Discussions |
| Design and Methodology of a Longitudinal Honeypot Study Shreyas Srinivasa, Aalborg University SLIDES |
|
| Performance Analysis: Robust Combiners vs. Secret Sharing Moe Sabry, McMaster University SLIDES |
|
| 5:00 pm - 5:15 pm | Wrap Up SLIDES |
| 5:15 pm | Adjourn |
Workshop Papers
Participants in the LASER Workshop are invited to write new papers on their experimental work. The papers will be published in post-workshop proceedings. The new papers will be driven and guided, in part, by the discussions and interactions, and possibly even new collaborations, forged at the workshop.
Draft papers will be due approximately two months after the workshop. The program committee will review papers and provide notifications and feedback one month after submission. Final camera-ready papers will be due approximately one month later.
Important Dates
LASER Workshop @ ACSAC: December 6, 2022
Draft Papers Submitted: February 6, 2023
Paper Reviews and Feedback: March 6, 2023
Final Papers Submitted: April 6, 2023
Papers Published: May 6, 2023
Organizers
David Balenson (SRI International)
Laura S. Tinnel (SRI International)
Further Information
Please see www.laser-workshop.org for more information about the LASER Workshop. Send questions to info@laser-workshop.org.
Keynote Talk
Towards True Reproducibility of Findings in Cybersecurity Research
Emma Tosch, Researcher, Northeastern University
Abstract: True reproducibility of empirical software research involves significant effort: in addition to being able to run new tools, researchers must also reproduce experimental results in order to validate findings. Unfortunately, descriptions of experiments in scientific papers can be underspecified. Thus, artifact evaluation often focuses on replicability via VMs or containers, which allow evaluators to repeat experiments under near-identical conditions to those reported in scientific papers. These replications fall short of reproducibility because they may still hold constant conditions that would otherwise vary in practice.
In this talk I argue that formal language support for experimentation can address these concerns. I will outline how a formal specification of hypotheses and experimental procedures can be used to abstract over empirical evaluations, lifting their lower-level specifications into the domain of causal reasoning. Such specifications would provide an underlying structure that can aide in experiment search, while also serving as a kind of regression testing for new experiments. Furthermore, researchers could share their experimental designs, lowering the barrier to entry of sharing software artifacts. Finally, I will end on a discussion of the limits of reproducibility in cybersecurity research, in light of both the statistical limitations of certain research questions as well as the role of non-scientific knowledge in cybersecurity.
Emma Tosch works in applied programming languages (PL) research, where she treats the process of language formalization — especially the design of domain-specific languages — as a methodological approach to problems not ordinarily considered the domain of PL. She is particularly interested in building languages and tools for data scientists and social scientists. Her work has won several awards and was most recently featured as a Research Highlight in the September 2021 issue of the Communications of the ACM.
Emma Tosch earned her B.A in English Literature from Wellesley College in 2008 before working at a healthcare IT start up. She earned her PhD from the University of Massachusetts Amherst in 2020. She was an Assistant Professor of Computer Science in the College of Engineering and Mathematical Sciences at the University of Vermont during the pandemic. She is currently a researcher with the POEM group at Northeastern University.
Detailed Paper Information
Threats in Crowdsourcing Threat Intelligence for Practical Threat Triaging
Afsaf Anwar, Northeastern University
Abstract: With the increasing evolution of threats, centralized monitoring is an arduous task. Recently, there has been an inclination in the community to share their analysis and findings with the larger security community. These decentralized efforts can be leveraged to understand threat mutations across the globe. However, these analysis results are often plagued with noise, such as temporal inconsistencies. In this talk, we will discuss the challenges in using Open-Source Cyber Threat Intelligence (OSCTI) for practical threat identification at the network. We will also discuss our experiences with using honeypot data and combining it with OSCTI to understand the attack vectors and campaign inferences.
ACSAC PAPER ABSTRACT, ACSAC PAPER, SLIDES
Afsah Anwar is an Assistant Professor at the Department of Computer and Information Sciences at Florida A&M University. He received his PhD from University of Central Florida and previously worked as a Post-Doc at Northeastern University. His research interests lie in malware analysis, vulnerability management, and threat intelligence. His research aims to ease efforts at the different stages of the incident response life cycle.
Exploring Backdoors in Federated Graph Neural Networks
Stefanos Koffas, TU Delft
Abstract: Due to privacy concerns, regulation restrictions, and commercial competitions, Federated Graph Neural Networks are proposed to train GNNs over isolated graph data. However, limited access to local clients' data due to privacy concerns or regulatory constraints may facilitate backdoor attacks on the global GNN model trained in federated learning (FL). In our ACSAC '22 paper, we are the first to explore such attacks against federated GNNs.
In this talk, we focus on the experimental part of our work. First, we discuss our methodology and the platform we used to run our experiments. Then, we present the implementation details of our two attacks (distributed and centralized backdoor attack) and show the experiments we run to fairly compare their performance. We also discuss how we designed our ablation studies. Along the way, we show the lessons we learned from this process.
ACSAC PAPER ABSTRACT, ACSAC PAPER, SLIDES
Stefanos Koffas is a Ph.D. candidate in the cybersecurity group at the Delft University of Technology under the supervision of Dr. Stjepan Picek and Prof. Mauro Conti. His research focuses on the security of AI and especially on backdoor attacks in neural networks. Before that, he obtained his MSc. in Computer Engineering from the Delft University of Technology and a bachelor's in electrical and computer engineering from the National Technical University of Athens. Stefanos has also worked for several years in the industry in Greece and the Netherlands.
Jing Xu is a Ph.D. candidate in the Cybersecurity group at the Delft University of Technology under the supervision of Dr. Stjepan Picek. Her main research focuses on the security of graph neural networks, backdoor attacks, and explainable AI.
Stjepan Picek is an associate professor at Radboud University, The Netherlands. His research interests are security/cryptography, machine learning, and evolutionary computation. Prior to the associate professor position, Stjepan was an assistant professor at TU Delft, and a postdoctoral researcher at MIT, USA and KU Leuven, Belgium. Stjepan finished his PhD in 2015 with a topic on cryptology and evolutionary computation techniques. Stjepan also has several years of experience working in industry and government. He is a program committee member and reviewer for a number of conferences and journals, and a member of several professional societies. His work has been featured in the mainstream media and on popular technology blogs.
Simulation of Differentially Private Federated Meta-learning Systems
Ning Wang, Virginia Tech
Abstract: Federated meta-learning has emerged as a promising AI framework for today's mobile computing scenes involving distributed clients. It enables collaborative model training using the data located at distributed mobile clients and accommodates clients that need fast model customization with limited new data. We proposed DP-Fedmeta which utilizes differential privacy for protecting clients’ training data privacy in a federated meta-learning system. We simulated DP-Fedmeta in PyTorch and evaluated it on three well-known datasets, including Omniglot, CIFAR-FS, and Mini-ImageNet. The results demonstrated that DP-FedMeta accomplishes better privacy protection while maintaining comparable model accuracy compared to the state-of-the-art solution.
ACSAC PAPER ABSTRACT, ACSAC PAPER, SLIDES
Ning Wang is a fifth-year Ph.D. student in Computer Engineering at Virginia Tech. Her research interests include differential privacy, federated learning, adversarial machine learning, and machine learning-based intrusion detection.
Torches on Pitchfork: Multi-feature Evaluation of a Security-oriented Programming Toolchain
Nik Sultana, Illinois Institute of Technology
Abstract: This talk will describe the experimental methodologies that were used in the Pitchfork project on privilege separation (http://pitchfork.cs.iit.edu/). The project developed a library and software tool to analyze, transform, and execute C programs as software compartments that consist of separate processes or binaries. The talk will outline the conceptual and practical challenges that were encountered when designing Pitchfork, building it for reusability, and evaluating the Pitchfork toolchain for various features: security, functionality, generality, performance, and programmer convenience.
ACSAC PAPER ABSTRACT, ACSAC PAPER, SLIDES
Nik Sultana is an assistant professor of Computer Science at Illinois Tech in Chicago. His research focuses on distributed system techniques that leverage programming theory, formal logic, and practical systems engineering. He completed his PhD at Cambridge University's Automated Reasoning Group, where he worked on a compiler-based approach to proof translation. Before joining Illinois Tech Nik postdoc'd at the UPenn Distributed Systems Lab and at the Cambridge Systems Research Group.
Design and Methodology of a Longitudinal Honeypot Study
Shreyas Srinivasa, Aalborg University
Abstract: The Internet of things (IoT) and critical infrastructure utilizing operational technology (OT) protocols are nowadays common attack targets and/or attack surfaces used to propagate malicious actions further. Deception techniques such as honeypots have been proposed for both IoT and OT, but they either lack an extensive evaluation or are subject to fingerprinting attacks. In this session, we present an overview of the design and methodology followed in the longitudinal honeypot study from our accepted work in ACSAC 2022. We describe the imposed parameters, design decisions, challenges, and entailing limitations of the experiment. Furthermore, we outline the analysis and the resulting datasets.
ACSAC PAPER ABSTRACT, ACSAC PAPER, SLIDES
Shreyas Srinivasa is nearing the completion of his Ph.D. at Aalborg University (AAU) in Copenhagen, Denmark. His research interests include evaluating Cyber Deception mechanisms for Defensive Security, Threat Research, Internet Security Measurements, and Digital Forensics. Before his Ph.D., Shreyas headed the IT Infrastructure and Network team at acarda GmbH, Frankfurt, Germany, for six years. He also worked on the Incident Response Team to analyze incidents that impacted the organization's daily operations. Shreyas obtained his Masters's degree from Technische Universität Darmstadt in 2015 and a Bachelors's degree from India in 2011. Shreyas is actively working on counter-cyber deception techniques and studying the impact of war on the critical infrastructure in Ukraine.
Performance Analysis: Robust Combiners vs. Secret Sharing
Moe Sabry, McMaster University
Abstract: ArchiveSafe LT provides secure long-term archiving through robust combiners utilizing standard encryption schemes, in contrast to the state-of-the-art solutions where secret sharing is utilized. In our experiment, we evaluate the performance of ArchiveSafe LT against the SOTA solutions and against the identified secure long-term archiving requirements, in real life imitated scenarios. Our experiments utilize standard and publicly available encryption libraries and are developed entirely in Python. In this workshop presentation, we discuss the challenges we faced in developing our experiments in three areas: (1) How to simulate real-life scenarios for long-term archiving? (2) How to reproduce and compare the performance results with the SOTA solutions where standard encryption schemes are not being used or the details of the implementations are not publicly available? (3) How to find comparable datasets?, and (3) The lessons learned for developing an efficient set of security experiments.
ACSAC PAPER ABSTRACT, ACSAC PAPER, SLIDES
Reza Samavi is currently a faculty member in the Department of Electrical, Computer, and Biomedical Engineering at Toronto Metropolitan University (formerly Ryerson University) and a faculty affiliate with the Vector Institute for Artificial Intelligence. Reza's research interests include data security and the security of machine learning algorithms. For his research on security and machine learning, Reza has received several research grants and awards from NSERC, SOSCIP, MITACS, HHS, IDEaS, IBM, and the Information and Privacy Commissioner of Ontario.
Moe Sabry is a Ph.D. candidate at McMaster University in the Computing and Software department. He has a masters degree in software engineering from McMaster university and a bachelor degree in computer science. Moe's research is focused on secure archiving systems.