#consumer electronics #learning #storytelling #interfaces #social science #machine learning #affective computing #industry
#interactive #computer science #interfaces #learning #art #computer vision #design #creativity #technology
Dancing Plants, 2019-20
A sculptural platform/display for computer vision analysis of pepper plant growth. Four chambers with embedded growth lights, automated watering system, pepper plants in hydroponic planting pods, depth cameras, web cameras, and embedded microcontrollers. The container opens and closes and captures plants in total darkness for long-term plant growth monitoring. Animations produced from time-lapse capture are then processed with machine vision algorithms and sonified into symphonic outputs.
As lead creative technologist for a creative research project, Dancing Plants, I collaborated with colleagues across Plant Science, Computer Science and the Performing Arts to study plant movement- the “dance of plants”. I designed a system for capturing stop-motion media of the growth of pepper plants, afterwards, analyzing the data with computer vision- in order to generate data sonifications resulting in “plant science symphonies”.
Using a variety of sophisticated imaging techniques, I outfitted a “mobile plant capture sculptural platform”, utilizing 16 webcameras, 4 Realsense Depth Cameras, and a FLIR thermal camera.
Through this collaboration of specialists from multiple disciplines, we captured a complex dataset describing each plant’s movement in relationship to varying controlled conditions, with the goal of aligning creative outputs with real scientific research. Our platform used two different kinds of lighting conditions, which enabled us to capture differences in growth patterns- one, with a full spectrum LED light, the other, with mixed blue/red LED lights.
The platform I developed enabled us to capture multiple modalities of visualization of pepper plant growth, in response to various stress stimuli based on how we controlled the lighting.
I developed software that interfaced with the multi-camera capture system, enabling us to remotely monitor each plant’s response to their environment. The resulting dataset of stop-motion animations captured the slow and subtle movements of pepper plant growth, as they moved in relation to light and hydration.
Genetically Modified Plants
Through evolution, plants developed mechanisms to move their roots, leaves, and stems to capture nutrients or sunlight.
These micro-movements are a response to microbial pathogen and stress. Our goal was to capture these micro-movements, and analyse them from an artistic and structural perspective.
Computer Vision Sonifications
Collaborator Jia-Bin Huang, a computer vision specialist, extracted a wealth of data about plant movement by applying computer vision algorithms to the stop-motion images.
Afterwards, Huang produced sonifications of the data into sounds, later, arranged into scores.
Events + Experiences
The resulting multi-camera time-lapse content, in combination with the audio sonification, produced an artful animation and score, which has been displayed and exhibited at numerous public events.
This “symphony of pepper plant growth” was an experimental study into the multiple modalities of media, ever-present in the natural patterns of biological systems.
