SYNCHRONOUS VISUALIZATION OF VIDEOTAPED SOUNDS AND MOTIONS OF INSECTS BY

High-speed cinematography with synchronized oscillography has been used to analyze accurately the sound production by a rapidly moving structure during communicative behavior of insects (Walker & Dew, I972 Willey, I973, I974). However, this method involves expensive equipment and perhaps kilometers of film containing only a few hundred feet of analyzable sequences. Nevertheless, the accuracy and resolution of audio-visual synchrony by this method has been exceeded only by the use of minute magnetometers using the Hall-effect (Elsner, I97O; Elsner & Huber, I973) and by an ingenious method of flash-photography (Morris & Pipher, I972). On the other hand, studies needing an average synchrony accuracy of only 5 msec (essentially equivalent to motion picture framing rates of 60 pictures per sec) can utilize inexpensive or already available video-systems used commonly as educational tools. Video has several advantages over cinematographic methods" ) sound and picture are routinely recorded synchronously, 2) videotape can be erased and reused, 3) some video-cameras have great sensitivity and can be used even for nocturnal insects at very low light levels, and 4) the system can be used to monitor behavior for hours and then sequences can be selected, duplicated on another tape and then the original tape can be used again. There are disadvantages of video for high resolution work and these will be discussed in the Analysis (q.v.).

. Block schematic diagram of the equipment connected for line- by-line synchronization of a video field with a simultaneous full sweep of the cathode ray gun of the oscilloscope.The CRO is laid on its side so that the horizontal sweep passes from top to bottom of the superimposed field in VTR2 (see text, sect.8, Analysis).For ordinary horizontal sweep, the CRO is in normal position and the external trigger on the vertical sync is disconnected.
It seems strange that, with the exception of publications from our laboratory (Steinberg & Conant, 1974; Steinberg & Willey, I974), we have found only a few studies of insect behavior which have involved the analysis of video-taped sounds and motion (Loher & Chandrashekaran, I97O, 972; Hoeg-Guldberg, I972).Moreover, even these studies have neglected to capitalize on the audio-recording and electronic potential of the video-tape system.We have devised a simple method to superimpose the oscillographic trace of the sound already recorded on any pre-existing video-tape on a duplicate record- ing.Extensions of this method to synchronous oscillography trans- duced from any mode will make video monitoring of insect behavior amenable to more versatile and precise analysis.

METHOD
A sequence was video-recorded with the following sound-related     1974]   Wussow et al.Sounds and Motions of Insects 211 parameters noted: distance rom the microphone to the subect, and 2) the average temperature o air between subject and micro- phone.Then we played the recorded sequence on a compatible video-tape reproducer (VTR1) which was linked by a multichannel synchronizing circuit (SWITCHER) to another video-recorder (VTR.) which preerably used at least one-inch tape.Figure diagrams the set-up schematically.The audio circuit o the VTR1 was linked also to the input of an oscilloscope (CRO), the screen (CRT) o which has an image persistence on the order o micro- seconds (P phosphor coating).A video-camera (CAM) t:ocused on the CRT image which was positioned so that it would not mask the image o the subject in the superposition.I the trace was to be positioned along the bottom of the rame, the horizontal sweep ot: the CRO was set at o msec/cm so that a single sweep o I2.5 crn will encompass at least eight video fields (q.v., sect.7 & 8, Analysis   and Fig. or urther explanation and an alternative method).We also masked the negative potentials with opaque tape.The video- camera is connected with the VTR2 by the vertical interval switcher and the CRT screen is shielded by an improvised hood rom the room lights.The results of superposition are simultaneously video- monitored?
The resulting video-tape, minus all the inactive time, can be played on a VTR which has stop-action and single-field orwarding capabil- ity (or the tape can be moved orward by hand).The audio signal can still be heard as well as seen as an oscillo-trace when the VTR is running at normal speed.At low speed the audio circuit auto- matically turns o and, although an adustment can be made in the machine's circuitry to enable the sound to be heard, with the oscillo- grap'hic trace this is neither necessary nor desirable.With the VTR stopped on a single field, we marked with a wax crayon the positions of the moving parts on the projection screen of the monitor and made measurements of the angle changes with a protractor (a better method will be outlined later in section 9, Analysis).aEquipment used: "Quiet Room" (Suttle, Inc., Chicago), Audiometric Room, double-walled (Industrial Acoustics, New York), Sony EV-310 25 mm videocorder, Scotch Brand MT 20568 video-tape, General Electric model 4TE33D1 TV camera with a Soligor 10 cm lens, Sony AVC 3200 TV camera with Angenieux 7.5 cm lens with a +1 diopter auxiliary lens, AKG shotgun microphone model Dg00, vertical interval' switcher-sync generator with sync lock capability (Shintron 360), Tektronix 502A os- cilloscope with Pll phosphor, Panasonic video-monitors (3-inch), one '50W electric spot bulb.

ANALYSIS
Analysis ot audio-visual synchrony must consider the following actors: ) 2k video field is scanned rom top o the screen to the bottom once in 6.67 msec at a maximum resolution of 262.5 lines per eld; the horizontal sweep time or a line is 0.0635 msec.The vertical blanking interval (retrace time for the electron-gun to move rom bottom of the screen to the top) requires up to 2 lines, and sub- tracted rom the maximum resolution leaves a net resolution o 241.5 lines per ield with a net duration o 5.5 msec, and a vertical blank- ing interval of to .2 msec.
2) A video frame consists of two interdigitating video fields with a total net duration of 3 msec and a total net resolution of 483 lines.
3) Transduction of light and electronic impulses can be regarded as instantaneous, however the speed of sound in air is a variable.
Near sea level, at 37 C and no greater than o cm from the micro- phone, one can calculate that the maximum time interval between sound production and electronic transduction is no greater than o.4 msec and can be neglected as an important factor.4) A crucial discrepancy in visual-acoustic synchrony and visual analysis of movements is the time taken by the electron-gun of the video camera to trace any action within a field.This factor causes three time-related distortions in the image which must be considered in the analysis, namely (a) vertical loss of resolution, (b) excessive curvature of a rotating image, and (c) inability to record events which occur after a scanning trace has passed until the next field scan.5) Vertical loss of resolution: the image formed during a single field has a maximum vertical resolution of only 241.5 lines per field whereas a 6 mm optical-film frame has 5o-6o lines of resolution per mm (at the center) or about 7oo to 8oo lines of resolution overall (allowing for loss of about 2o lines of resolution at the front and rear edge of the frame).On the other hand, the horizontal resolution of a good video system probably approaches that of a comparable optical- film system.However evaluation of information content of a video field and an optical film frame encompassing 5 msec is obscured by the next considerations.6) Excessive curvature of rotating image: the time required for scanning a single field naturally produces a distorted image in any movement having lateral components.Plate a-e illustrates this  showing a pulse of sound in five photographs of consecutive fields (a-e).The male which is chirping is outlined in (a), (b) and (c) show the right femur blurred as it makes the downward stroke (left femur is missing).The two vertical black lines drawn on the trace in (d) bracket the portion of the oscillotrace which was scanned at the time the leg was moving in this field.To the right of the lines is the base of a trace which really was much higher, but its peaks will not be scanned until the next field (e).
The image persistence is due to retention by the video-camera pick-up tube and is about equal to one field (--16 reset).The image in (f) is the result of triggering and synchronizing the CRT sweep with the "front porch" of the vertical field (set up as in Fig. 1).The field is identical to (e), but only the most intense peaks were registered.(See text, sect.8) This is caused by the act that a video field is a metachronic series of 241.5 lines each representing a "shutter speed" o t/5,75o s.ec.The scanning time of the femur image in P1. ta is about 45 lines o 2.86 msec.In other words, the tip o.f the femur, which is moving in an arc to the let, is imaged nearly 3 msec earlier than that o the femur base.Further magnification on the screen would result in greater temporal displacements and greater artifact in the shape o the organ.Any attempt to measure angles o rotation or points o displacement must take into account the video scanning time of the specific image and its vector relative to the vertical trace.7) Inability to record events after passage of the scanning trace: it is usually desirable to superpose the oscillotrace along the bottom of the video field (P1.a-e) where the action is less likely to be obscured.However, the lower portion of the field will be scanned milliseconds after the action in the center.Nevertheless, visual- acoustic synchrony can be computed .asshown in P1. d.The selec- tion of the horizontal sweep setting of the CRO is determined by the duration of the longest acoustical signal and should be about 4o or 5o msec longer, so that the oscillotrace of the entire action can be visualized in one sweep, and traces in successive frames can be cross- referenced.The additional time allows some time before and after the signal on the screen so that the beginning or the end of the signal and its causative motion aren't lost in the CRT retrace time, the vertical blanking interval, nor at the sometimes distorted edges of the monitor screen.Sometimes traces are badly reproduced or oriented, but constant inspection of the VTR.monitor will allow such mistakes to be corrected immediately by duplicating the sequence again.
8) An alternative method to achieve visualization of motion-sound synchrony is shown in Fig. and P1.f.Here the CRO is laid on its side so the oscillotrace will sweep from top to bottom of the camera pickup tube.The external trigger of the CRT electron-gun is triggered on the electronic transition from the vertical sync to the start of the first horizontal scanning line of the video field.The hori- zonal sweep of the CRT is set to equal the time constant of the video field (5 msec per sweep or cm/msec), thus only the sound re- corded during the video scan will be oscillographed.The oscillotrace peaks appearing at the same horizontal plane as the video-recorded movement will be relatively synchronous with that movement, with  allowance or the time lag induced by the speed ot: the original sound in air (P1., arrows).9) Close examination o CRTs can be a health hazard and the thickness o the protective cover can induce parallax error.There- t:ore, i exact angle and displacement measurements are desired the fields should be photographed with careful ocussing on the video lines.Such photographs will orm a permanent working record o key fields o the rather ragile video-tape which can be damaged or destroyed by the constant riction o scanning in the stopped position.
DISCUSSION AND CONCLUSIONS This method allows relatively synchronous visualization ot motions and the sounds produced by the movements i the actions are relatively slow.For example, it is possible to determine whether the stri- dulation by a single up-down motion o a grasshopper's hind temur rubbing against the fore-wing is produced during the up-stroke or down-stroke, provided the entire motion has a duration of 6o msec.In the case of oedipodine grasshoppers, the chirps produced by males during courtship usually are produced by motions which last at least 60 msec and oten IOO msec (Willey,974).Since many such sig- nals are complicated by being segmented into pulses within a single up-down motion, normal-speed cinematographic and audiospectro- graphic investigations of Oedipodinae (Otte, I97O; Willey & Willey, I969) have concluded that the first pulse must be produced on the up-stroke.However, high-speed motion analysis (Willey,974) shows that Arlhia sullhurea produces the definitive chirp on the downstroke only.A very weak signal was produced on the up-stroke but was of such low amplitude that it had never been audiospectro- graphed.Video-tapes of Chimarocephala (Loher & Chandrashekaran, I97O) and Chortolhaga (Steinberg & Willey, 974) also show that stridulations are produced only on the downstroke of the femur in these species.However, the mechanism for production of two-pulsed chirps must be analyzed by the higher speed methods (Willey,974).
Further refinements of this system can involve direct synchrony of oscillotrace with the initial video-tape recording.Also the use o a light-emitting diode (LED) and a small high speed digital clock in the background, synchronized with each video /]eld, would simplify identification of particular fields.However, the method we have reported can be used for improving the analysis o already existing tapes.

SUMMARY
Oscillographic traces o sounds recorded on videotapes can be superimposed easily and synchronously on a duplicate o those tapes.
The audio recording remains intact in the duplicate and the oscillo- trace allows rather precise analysis o slower movements and the sounds they produce.This system was tested on pre-existing "tapes o courtship in the grasshopper Chortophaga viridifasciata (De Geer).The method promises to be useful in analysis o tapes used i:or monitoring behavior and also as a teaching device.
Fig.1.Block schematic diagram of the equipment connected for line- by-line synchronization of a video field with a simultaneous full sweep of the cathode ray gun of the oscilloscope.The CRO is laid on its side so that the horizontal sweep passes from top to bottom of the superimposed 8ounds and Motions of Insects 213 PI. 1.A portion of a stridulation (chirp) by Chortolhaga oiridifasciata, oscillographic trace and a barely detectable clock- wise curvature o the emur tip relative to the base in b and c.
ussow et al.Sounds and Motions of Insects