°Õ¾±³Ù±ô±ð:ÌýPrivate Prompt Learning for Large Language Models
³§±è±ð²¹°ì±ð°ù:ÌýAdam Dziedzic
Date:ÌýFriday,ÌýApril 12th, 2024
Time:Ìý1:00 PMÌý- 2:00 PM
Location:ÌýDC 1302
Zoom Link:Ìý
Passcode:DC2597
Abstract:ÌýLarge language models (LLMs) are excellent in-context learners. However, the sensitivity of data contained in prompts raises privacy concerns. Our work first shows that these concerns are valid: we instantiate a simple but highly effective membership inference attack against the data used to prompt LLMs. To address this vulnerability, one could forego prompting and resort to fine-tuning LLMs with known algorithms for private gradient descent. However, this comes at the expense of the practicality and efficiency offered by prompting. Therefore, we propose to privately learn to prompt. We first show that soft prompts can be obtained privately through gradient descent on downstream data. However, this is not the case for discrete prompts. Thus, we orchestrate a noisy vote among an ensemble of LLMs presented with different prompts, i.e., a flock of stochastic parrots. The vote privately transfers the flock’s knowledge into a single public prompt. We show that LLMs prompted with our private algorithms closely match the non-private baselines.
Speakers Bio:ÌýAdamÌýis a Tenure Track Faculty Member at CISPA Helmholtz Center for Information Security, co-leading the SprintML group. His research is focused on secure and trustworthy Machine Learning as a Service (MLaaS).ÌýAdamÌýdesigns robust and reliable machine learning methods for training and inference of ML models while preserving data privacy and model confidentiality.ÌýAdamÌýwas a Postdoctoral Fellow at the Vector Institute and the University of Toronto, and a member of the CleverHans Lab, advised by Prof. Nicolas Papernot. He earned his PhD at the University of Chicago, where he was advised by Prof. Sanjay Krishnan and worked on input and model compression for adaptive and robust neural networks.ÌýAdamobtained his Bachelor's and Master's degrees from Warsaw University of Technology in Poland. He was also studying at DTU (Technical University of Denmark) and carried out research at EPFL, Switzerland.ÌýAdamÌýalso worked at CERN (Geneva, Switzerland), Barclays Investment Bank in London (UK), Microsoft Research (Redmond, USA), and Google (Madison, USA).