The diagnosis, treatment and study of psychiatric illnesses relies primarily on subjective impressions and clinical ratings of behavior. Unlike other fields of medicine, psychiatry does not have objective tests that can be used for diagnosis and treatment. In psychiatry, therefore, there is a clear need to operationalize and quantify behavior, in particular behavior related to social communication. Human communication entails a rich repertoire of expression, including spoken language, face expression and gaze. Language has evolved in humans within a social context, beginning with the parent-infant dyad, and mirroring of sounds and facial expressions. Its natural ecology is face-to-face interpersonal interactions, both in-person and increasingly remotely via platforms such as Zoom, including telepsychiatry. Human dyadic social communication entails a rich repertoire of expression, including not only language, but also face expression, acoustics (prosody), pauses and turn-taking, gaze (shared attention), and gestures, highly coordinated between speakers, with regular turn-taking and alignment of facial expression, gesture, semantic content, and speech rates. Overall, given the synchronization and speed needed to successfully communicate with others, it is remarkable how well this process normally works. However, in schizophrenia and its risk states, communication frequently breaks down, leading to frustration, isolation and significant difficulties in social and role function.  This presentation will describe the development of objective measures of dyadic social communication behavior using natural language processing and computer vision, with schizophrenia and its risk states as a use case.

Learning Objectives:

1. Describe language disturbances in schizophrenia and its risk states.

2. Describe face expression disturbances in schizophrenia and its risk states.

3. Describe ethical and technical issues related to computational phenotyping.