Reflecting on 100 Issues of Projects to Know

Today, I’m in awe as we celebrate the 100th issue of Projects to Know. I never planned to write a newsletter. A few years ago our Director of Marketing, Malia Powers, suggested that I write a blog post on data quality. After experiencing writer’s block 3/4 through a draft, I proposed that I write a short newsletter instead. Certainly, a short newsletter would be easier to an author than a comprehensive review of data testing, monitoring, and preparation tools and technologies! What’s more, I felt frustrated existing newsletters on data science and ML, many of which focused on attention-grabbing headlines about the impact of AI on climate change, cancer, and space exploration. I saw the need for a publication that would highlight research and projects with more practical relevance – content that data practitioners and developers would and could apply.

100 issues later, I’ve confirmed that writing a newsletter is, in fact, more challenging than writing a blog post. Nonetheless, I have zero regrets. More importantly, I’ve validated that data practitioners and developers want to learn more about the projects that their peers are building in academia and industry; as research papers, internal platforms, and OSS tools; in machine intelligence, distributed systems, and data management.

While I sometimes opine about summarizing lengthy academic papers, surfing engineering blogs, and sifting through GitHub repos every weekend; I’m so inspired those who create and consume projects. Projects to Know has proven to me that the impact of a paper is deeper than an academic conference; the impact of an OSS project cannot be measured with GitHub stars, and the impact of internal initiatives often extends far beyond a single company. I’m motivated to continue writing the readers from unicorn tech companies who share how they’re applying the models described in featured papers; or the creators of OSS database technologies who connect with academic collaborators through PTK.

When I started my career in data in 2009, it was a lonely profession. When I became a manager in 2012, I had so few peers to turn to for mentorship and guidance. But things have changed and now there are so many data practitioners and developers who want to communicate and collaborate; who galvanize each other to try more experimental approaches to managing teams or to reveal the skunkworks project they’ve been working on between calls and meetings. Now, a community exists and there are so many more Projects to Know.

Below, we’ve highlighted a few projects from this expansive compilation – the most popular Papers, Projects, and Content from 4 sets of issues. You’ll see that these projects span a range of topics – from privacy-preserving machine learning to literate programming to serverless prediction serving. They’re created authors from Tennessee to Singapore and from institutions ranging from F500 companies like Nike to seed stage startups like Ponder. It’s hard to skim through this list without feeling awe – there’s just so much to learn and so many people to learn from.

I’m so excited to celebrate an ever-growing community and corpus of knowledge today. Thanks for your contributions, readership, and support.

Not a subscriber? Subscribe here and get 3 academic papers and 3 open source projects that are playing a meaningful role in advancing machine intelligence and data science in your inbox on a weekly basis! You can view past issues here.

Papers

Self-Supervised GANs via Auxiliary Rotation LossSelf-Supervised GANs via Auxiliary Rotation LossTing Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, Neil HoulsbyConditional GANs are at the forefront of natural image synthesis. The main drawback of such models is the necessity for labeled data. In this work we exploit two popular unsupervised learning techniques, adversarial training and self-supervision, and take a step towards bridging the gap between conditional and unconditional GANs. In particular, we allow the networks to collaborate on the task of representation learning, while being adversarial with respect to the classic GAN game. The role of self-supervision is to encourage the discriminator to learn meaningful feature representations which are not forgotten during training. We test empirically both the quality of the learned image representations, and the quality of the synthesized images. Under the same conditions, the self-supervised GAN attains a similar performance to state-of-the-art conditional counterparts. Finally, we show that this approach to fully unsupervised learning can be scaled to attain an FID of 23.4 on unconditional ImageNet generation.arXiv:1811.11212v2View paper (Ting Chen, Xiaohua Zhai, Marvin Ritter, Mario Lucic, Neil Houlsby)
Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models InsteadStop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models InsteadCynthia RudinBlack box machine learning models are currently being used for high stakes decision-making throughout society, causing problems throughout healthcare, criminal justice, and in other domains. People have hoped that creating methods for explaining these black box models will alleviate some of these problems, but trying to \textit{explain} black box models, rather than creating models that are \textit{interpretable} in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm to society. There is a way forward -- it is to design models that are inherently interpretable. This manuscript clarifies the chasm between explaining black boxes and using inherently interpretable models, outlines several key reasons why explainable black boxes should be avoided in high-stakes decisions, identifies challenges to interpretable machine learning, and provides several example applications where interpretable models could potentially replace black box models in criminal justice, healthcare, and computer vision.arXiv:1811.10154v3View paper (Cynthia Rudin)
Recommending Podcasts for Cold-Start Users Based on Music Listening and TasteRecommending Podcasts for Cold-Start Users Based on Music Listening and TasteZahra Nazari, Christophe Charbuillet, Johan Pages, Martin Laurent, Denis Charrier, Briana Vecchione + 1 moreRecommender systems are increasingly used to predict and serve content that aligns with user taste, yet the task of matching new users with relevant content remains a challenge. We consider podcasting to be an emerging medium with rapid growth in adoption, and discuss challenges that arise when applying traditional recommendation approaches to address the cold-start problem. Using music consumption behavior, we examine two main techniques in inferring Spotify users preferences over more than 200k podcasts. Our results show significant improvements in consumption of up to 50\% for both offline and online experiments. We provide extensive analysis on model performance and examine the degree to which music data as an input source introduces bias in recommendations.arXiv:2007.13287v1View paper (Zahra Nazari, Christophe Charbuillet, Johan Pages, Martin Laurent, Denis Charrier, Briana Vecchione, Ben Carterette)
An Overview of Privacy in Machine LearningAn Overview of Privacy in Machine LearningEmiliano De CristofaroOver the past few years, providers such as Google, Microsoft, and Amazon have started to provide customers with access to software interfaces allowing them to easily embed machine learning tasks into their applications. Overall, organizations can now use Machine Learning as a Service (MLaaS) engines to outsource complex tasks, e.g., training classifiers, performing predictions, clustering, etc. They can also let others query models trained on their data. Naturally, this approach can also be used (and is often advocated) in other contexts, including government collaborations, citizen science projects, and business-to-business partnerships. However, if malicious users were able to recover data used to train these models, the resulting information leakage would create serious issues. Likewise, if the inner parameters of the model are considered proprietary information, then access to the model should not allow an adversary to learn such parameters. In this document, we set to review privacy challenges in this space, providing a systematic review of the relevant research literature, also exploring possible countermeasures. More specifically, we provide ample background information on relevant concepts around machine learning and privacy. Then, we discuss possible adversarial models and settings, cover a wide range of attacks that relate to private and/or sensitive information leakage, and review recent results attempting to defend against such attacks. Finally, we conclude with a list of open problems that require more work, including the need for better evaluations, more targeted defenses, and the study of the relation to policy and data protection efforts.arXiv:2005.08679v1View paper (Emiliano De Cristofaro)
Optimizing Prediction Serving on Low-Latency Serverless DataflowOptimizing Prediction Serving on Low-Latency Serverless DataflowVikram Sreekanti, Harikaran Subbaraj, Chenggang Wu, Joseph E. Gonzalez, Joseph M. HellersteinPrediction serving systems are designed to provide large volumes of low-latency inferences machine learning models. These systems mix data processing and computationally intensive model inference and benefit from multiple heterogeneous processors and distributed computing resources. In this paper, we argue that a familiar dataflow API is well-suited to this latency-sensitive task, and amenable to optimization even with unmodified black-box ML models. We present the design of Cloudflow, a system that provides this API and realizes it on an autoscaling serverless backend. Cloudflow transparently implements performance-critical optimizations including operator fusion and competitive execution. Our evaluation shows that Cloudflow's optimizations yield significant performance improvements on synthetic workloads and that Cloudflow outperforms state-of-the-art prediction serving systems by as much as 2x on real-world prediction pipelines, meeting latency goals of demanding applications like real-time video analysis.arXiv:2007.05832v1View paper (Vikram Sreekanti, Harikaran Subbaraj, Chenggang Wu, Joseph E. Gonzalez)
Interpretable Machine Learning: Fundamental Principles and 10 Grand ChallengesInterpretable Machine Learning: Fundamental Principles and 10 Grand ChallengesCynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, Chudi ZhongInterpretability in machine learning (ML) is crucial for high stakes decisions and troubleshooting. In this work, we provide fundamental principles for interpretable ML, and dispel common misunderstandings that dilute the importance of this crucial topic. We also identify 10 technical challenge areas in interpretable machine learning and provide history and background on each problem. Some of these problems are classically important, and some are recent problems that have arisen in the last few years. These problems are: (1) Optimizing sparse logical models such as decision trees; (2) Optimization of scoring systems; (3) Placing constraints into generalized additive models to encourage sparsity and better interpretability; (4) Modern case-based reasoning, including neural networks and matching for causal inference; (5) Complete supervised disentanglement of neural networks; (6) Complete or even partial unsupervised disentanglement of neural networks; (7) Dimensionality reduction for data visualization; (8) Machine learning models that can incorporate physics and other generative or causal constraints; (9) Characterization of the "Rashomon set" of good models; and (10) Interpretable reinforcement learning. This survey is suitable as a starting point for statisticians and computer scientists interested in working in interpretable machine learning.arXiv:2103.11251v1View paper (Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong)
Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the LoopFits and Starts: Enterprise Use of AutoML and the Role of Humans in the LoopAnamaria Crisan, Brittany Fiore-GartlandAutoML systems can speed up routine data science work and make machine learning available to those without expertise in statistics and computer science. These systems have gained traction in enterprise settings where pools of skilled data workers are limited. In this study, we conduct interviews with 29 individuals from organizations of different sizes to characterize how they currently use, or intend to use, AutoML systems in their data science work. Our investigation also captures how data visualization is used in conjunction with AutoML systems. Our findings identify three usage scenarios for AutoML that resulted in a framework summarizing the level of automation desired by data workers with different levels of expertise. We surfaced the tension between speed and human oversight and found that data visualization can do a poor job balancing the two. Our findings have implications for the design and implementation of human-in-the-loop visual analytics approaches.arXiv:2101.04296v1View paper (Anamaria Crisan, Brittany Fiore-Gartland)
What are the most important statistical ideas of the past 50 years?What are the most important statistical ideas of the past 50 years?Andrew Gelman, Aki VehtariWe review the most important statistical ideas of the past half century, which we categorize as: counterfactual causal inference, bootstrapping and simulation-based inference, overparameterized models and regularization, Bayesian multilevel models, generic computation algorithms, adaptive decision analysis, robust inference, and exploratory data analysis. We discuss key contributions in these subfields, how they relate to modern computing and big data, and how they might be developed and extended in future decades. The goal of this article is to provoke thought and discussion regarding the larger themes of research in statistics and data science.arXiv:2012.00174v5View paper (Andrew Gelman, Aki Vehtari)
Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible DevelopmentMachine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible DevelopmentAspen Hopkins, Serena BoothPractitioners from diverse occupations and backgrounds are increasingly using machine learning (ML) methods. Nonetheless, studies on ML Practitioners typically draw populations from Big Tech and academia, as researchers have easier access to these communities. Through this selection bias, past research often excludes the broader, lesser-resourced ML community -- for example, practitioners working at startups, at non-tech companies, and in the public sector. These practitioners share many of the same ML development difficulties and ethical conundrums as their Big Tech counterparts; however, their experiences are subject to additional under-studied challenges stemming from deploying ML with limited resources, increased existential risk, and absent access to in-house research teams. We contribute a qualitative analysis of 17 interviews with stakeholders from organizations which are less represented in prior studies. We uncover a number of tensions which are introduced or exacerbated by these organizations' resource constraints -- tensions between privacy and ubiquity, resource management and performance optimization, and access and monopolization. Increased academic focus on these practitioners can facilitate a more holistic understanding of ML limitations, and so is useful for prescribing a research agenda to facilitate responsible ML development for all.arXiv:2110.02932v1View paper (Aspen Hopkins, Serena Booth)
DAG Card is the new Model CardDAG Card is the new Model CardJacopo Tagliabue, Ville Tuulos, Ciro Greco, Valay DaveWith the progressive commoditization of modeling capabilities, data-centric AI recognizes that what happens before and after training becomes crucial for real-world deployments. Following the intuition behind Model Cards, we propose DAG Cards as a form of documentation encompassing the tenets of a data-centric point of view. We argue that Machine Learning pipelines (rather than models) are the most appropriate level of documentation for many practical use cases, and we share with the community an open implementation to generate cards from code.arXiv:2110.13601v2View paper (Jacopo Tagliabue, Ville Tuulos, Ciro Greco, Valay Dave)