Publications

To ensure accountability and mitigate harm, it is critical that diverse stakeholders can interrogate black-box automated systems and find information that is understandable, relevant, and useful to them. In this paper, we eschew prior expertise- and role-based categorizations of interpretability stakeholders in favor of a more granular framework that decouples stakeholders' knowledge from their interpretability needs. We characterize stakeholders by their formal, instrumental, and personal knowledge and how it manifests in the contexts of machine learning, the data domain, and the general milieu. We additionally distill a hierarchical typology of stakeholder needs that distinguishes higher-level domain goals from lower-level interpretability tasks. In assessing the descriptive, evaluative, and generative powers of our framework, we find our more nuanced treatment of stakeholders reveals gaps and opportunities in the interpretability literature, adds precision to the design and comparison of user studies, and facilitates a more reflexive approach to conducting this research.

We have developed an ontology based framework that evaluates compatibility between processing modules within an end user development framework, using MIT Lincoln Laboratory's Composable Analytics environment as a test case. In particular, we focus on inter-module semantic compatibility as well as compatibility between data and modules. Our framework includes a core ontology that provides an extendible vocabulary that can describe module attributes, module input and output requirements and preferences, and data characteristics that are pertinent to selecting appropriate modules in a given situation. Based on the ontological description of the modules and data, we first present a framework that takes a rule based approach in measuring semantic compatibility. Later, we extend the rule based approach to a flexible fuzzy logic based semantic compatibility evaluator. We have built an initial simulator to test module compatibility under varying situations. The simulator takes in the ontological description of the modules and data and calculates semantic compatibility. We believe the framework and simulation environment together will help both the developers test new modules they create as well as support end users in composing new capabilities. In this paper, we describe the details of the framework, the simulation environment, and our iterative process in developing the module ontology.

The Visualization for Cyber Security research community (VizSec) addresses longstanding challenges in cyber security by adapting and evaluating information visualization techniques with application to the cyber security domain. In this paper, we survey and categorize the evaluation metrics, components, and techniques that have been utilized in the past decade of VizSec research literature.