EMUSIC-L Digest Volume 42, Issue 03 This issue's topics: "muscle-memory" synth interfaces (part 3 of 3)(11 messages) Your EMUSIC-L Digest moderator is Joe McMahon . You may subscribe to EMUSIC-L by sending mail to listserv@american.edu with the line "SUB EMUSIC-L your name" as the text. The EMUSIC-L archive is a service of SunSite (sunsite.unc.edu) at the University of North Carolina. ------------------------------------------------------------------------ Date: Wed, 8 Jul 1992 17:46:00 EDT From: LNDR08@CLEMSON.BITNET Subject: synth interfaces-implementation issues Just wondering if anyone had thought of using a video camera and a computer to "watch" body movement instead of all the complex electronics that have been mentioned. Less electronics would be involved but most likely the software would be much more complex. I know there is work being done in visual perception with computers but I don't know how precise the output is (can it tell the difference between a arm movement and a hand or finger movement). Matt. ------------------------------ Date: Thu, 9 Jul 1992 07:40:00 MDT From: Jim Logue Subject: Re: synth interfaces-implementation issues That reminds me of the Amiga program that projected your video image onto the computer screen, where your image could activate screen zones linked to MIDI. Very wild, very wacky.... don't think it caught on, tho- Pity. Jim Logue Univ. of Utah ------------------------------ Date: Thu, 9 Jul 1992 10:35:00 EDT From: RAY BROHINSKY Subject: Re: synth interfaces-implementation issues >getting it to say okay this is a finger and it moved up... As Jonthan can certainly verify, the problem is even more difficult than this. Any action causes an equal and opposite reaction. Most of us are aware of that when we play `space wars', and forget about it entirely when we're swing our arms about on stage. There are reasons for this: first off, we're made wonderfully: each muscle has another or a group to counter-balance it. This counter-balancing gives us, in it's smallest form, muscle tone and tremor. We have both unconscious and conscious control over these muscles: when you ask an arm to raise by an amount (whether the amount is `that much' and is regulated by eye feedback, or `3 inches' or `that much regulated by feel') we may counteract the counter-motion of our bodies, and we may not. Getting a computer to guess how much motion has occured in the mind of the person controlling this arm, and translate that reliably into an amount of change in a sound is ghastly. It requires the kind of non-artificial intellegence that is traditionally limited to the likes of circus mind-readers (Try wiring one of them to your synth 8^) If you have a sensor that is anchored to your wrist above the hand, with feelers to guage the relative motion of the palm, you at least have something that has compensation built in, and you are measureing only the angle of the palm to the forearm. This is basically controllable, both from the computer's point of view, and from the anatomical point of view. Trying to guage even just that angle by video, with a person who exhibits more than 5 degrees of freedom, and a fixed camera with no degrees of freedom, is an immense task. Trying to do it with a camera capable of gimbaling and or on a cart...I'm not volunteering! Add to that the fact that the hand is attatched to an arm, and the arm to a torso, and a motion has the capacity to move or not move the whole or parts in a manner that is regulated by a)inertia, b)the unconscious efforts of the body to balance, c)the conscious efforts of the dancer/performer to make an expressive gesture, d) the current set of angles between the performer and the pickup (both of whom are free to move, turn, advance, retreat...) this problem approaches complexity beyond imagining. Then, you have to face the resolution problem, first of cameras, then of the analog electronics used to carry the signals (this may be the strongest link with modern video technology!) and of the video capabilities of the computational hardware. I know of few cameras with 1024x1024 resolution, and few of them operate at 30hz (or 60hz interlace, which is worse in lots of ways!) Those that do are _expensive!_ Faced with real-world physics, anatomy, and budgets, a really accurate video system is not likely in the near future. Even at that, I think (my OHO, mind you) that you'll still have to instrument the performer, and perhaps add visual clues (reflectors or lights) to him/her before you'll get very far. Before you shrug your shoulders and say `he's guessing', I'll state that I've worked in electronic holography for 7 years, (where the problems are not much different), and I've seen and manipulated some of the equations. For the most part, we were working with things that sat relatively still! raybro ------------------------------ Date: Thu, 9 Jul 1992 10:58:00 EDT From: RAY BROHINSKY Subject: Re: "muscle-memory" synth interfaces - implementation issues > > Blindfold the performer, move the synth around > > so that the selected knob isn't detectable by that performer, and you > > have, I beleive, the same situation a dance performer would have. How Actually, only if you moved the synth constantly, in a random, unlearnable pattern. Otherwise, one moment after the synthesist gets his/her hands on that synth, they'll know where everything is, and will probably do a creditable job of adjusting if they can't seat themselves before it in their `customary position'. >I heard recently about using lasers to project images in 3-D space. A system >like this would allow the dancer/musician to find those anchors. It would als >be a little more interesting for the audience. Imagine creating virtual >instruments out of light...that everyone can see! If you're referring to display holography, I am familiar with this. It has limitations, the worst of which is that the image is _not actually in space_, but appears to be from a fairly limited viewing angle. This angle works both ways..walk to the side of the hologram, and you can't see the image at all. Stand in front of the hologram, walk forward until your hand, stretched in front of you, appears to be in the image. Then put your head where your hand is, and look around. The image is not around you at all! Add to this the lack of tactile feedback, and you have a pretty nasty instrument to get into! >By the way...has anyone seen this done, or know how it is done. The place tha >I heard was doing it is Science North, in Sudbury, Canada. I haven't heard of what's going on at Sudbury, so they might have something new. However, here's some speculation: If you have a cloud of a flourescent vapor (vapour, I guess) and a set of laser beams which alone have less than the power necessary to trigger flourescence, you could, conceivably, come up with a system that would draw an image in a three dimensional space which could be viewed from within the image. Because of the way light doesn't interfere with matter (not to mention itself) this can be really hard to do. This borders on force fields. QED. If, in fact, this were possible/easy, we'd all have 3-d tanks in our homes! >Alan ------------------------------ Date: Thu, 9 Jul 1992 08:25:48 ADT From: Alan Edwards Subject: Re: "muscle-memory" synth interfaces - implementation issues > > Blindfold the performer, move the synth around > > so that the selected knob isn't detectable by that performer, and you > > have, I beleive, the same situation a dance performer would have. How > > would the dancer know that a particular motion will have the desired, > > repeatable, musical result? The dance motion may be correct, but > > lift the arm 5 cm and "a different knob is being turned." This may or > > may not be controllable by the dancer in the heat of the performance. (I > > don't know; I'm not a dancer.) > Yep, that's what I was talking about with perceptual anchors. It's > difficult to swing a baseball bat through a specific point in space > unless there's a baseball there to anchor the gesture. > > --- Joe M. > I heard recently about using lasers to project images in 3-D space. A system like this would allow the dancer/musician to find those anchors. It would also be a little more interesting for the audience. Imagine creating virtual instruments out of light...that everyone can see! By the way...has anyone seen this done, or know how it is done. The place that I heard was doing it is Science North, in Sudbury, Canada. Alan ------------------------------ Date: Thu, 9 Jul 1992 15:12:00 CDT From: John C Edwards Subject: Re: synth interfaces-implementation issues Easier, but less expressive through specific gestures, would be a system that could detect the extent (top/bottom/left/right) of the body and thus control parameters...using a plain background with two-toned garb would allow you to make rapid changes to a variable, (total visible "mass" could be a variable, altered by distance to camera, lighting (handheld, triggered), and position) while keeping analysis time to a bare minimum. specific locations could be parsed for number of contrasting interruptions, turning the dancer into a horizontal or vertical bar code...hmmm...Laurie Anderson's audio bow jumps to mind and I see dancers covered in UPC codes dancing over supermarket scanners and triggering sequences. but never mind that...the system described in the top paragraph could be done with mimimal programming, probably finished in a week. "I may be a bilateral X-ray technician, but I ain't no dan-cer." John ------------------------------ Date: Thu, 9 Jul 1992 10:25:26 -0400 From: Max Stevens-Guille Subject: Re: synth interfaces-implementation issues > > That reminds me of the Amiga program that projected your video image onto th > computer screen, where your image could activate screen zones linked to MIDI > Very wild, very wacky.... don't think it caught on, tho- Pity. > > Jim Logue > Univ. of Utah > The program you are talking about is commercially available. It's called Mandala and is marketted by "The Vivid Group". Contact: Sue Wyshynski The Vivid Group 317 Adelaide St. West Suite 302 Toronto, Ontario Canada, M5V 1P9 (416) 340-9290 fax: (416) 348-9809 WELL: catinhat I know for sure it works on Amigas. Last I heard a PC version was in the works. It's pretty economical. They are performance artists and there's lots of control over the system from a MIDI perspective. I have no ties with company -- I just hang out with them from time to time. Check 'em out. Max -- Max Stevens-Guille Research & Development __ Alias Research Inc. 110 Richmond St. East, / Toronto, Ontario, CANADA M5C 1P1 / Alias max@alias.com / \ Research vox: (416) 362-9181 fax: (416) 362-0630 / \_ Incorporated ------------------------------ Date: Thu, 9 Jul 1992 21:42:48 GMT From: srnorton@GALAXY.GOV.BC.CA Subject: Re: "muscle-memory" synth interfaces - implementation issues In article <9207082047.AA07701@twinpeaks.gsfc.nasa.gov>, xrjdm@CALVIN.GSFC.NASA.GOV (Joseph D. McMahon) writes: > wbf@alux2.att.com writes: >> >> I'm finding this discussion quite interesting. Do I understand that it >> is desirable to make body motions (i.e. a dance performance) generate a >> specific musical performance that is repeatable in a recognizable way >> like one can recognize the Beatles' "Yesterday" no matter who performs >> it? (That DOES leave room for style and interpretation.) > Originally, I was thinking only about why it seems to be easier to > control parameters on an analog synth than on a digital one, but we've > moved into such rarefied realms of speculation that the original > thought seems rather dull in comparison... >> >> Blindfold the performer, move the synth around >> so that the selected knob isn't detectable by that performer, and you >> have, I beleive, the same situation a dance performer would have. How >> would the dancer know that a particular motion will have the desired, >> repeatable, musical result? The dance motion may be correct, but >> lift the arm 5 cm and "a different knob is being turned." This may or >> may not be controllable by the dancer in the heat of the performance. (I >> don't know; I'm not a dancer.) > Yep, that's what I was talking about with perceptual anchors. It's > difficult to swing a baseball bat through a specific point in space > unless there's a baseball there to anchor the gesture. > > --- Joe M. I've heard of a similar experiment nicknamed "Frog's Eye" up here in Canada. They used 2 or 3 TV cameras to monitor the dancer's movements, and a computer played various synths, depending on what region of space the dancer moved into. Apparently, the dancer *did* learn to "play" the instrument, with practice. Unfortunately, I don't know how precise this "playing" was. Simon N. ------------------------------ Date: Fri, 10 Jul 1992 09:46:00 EDT From: ronin Subject: dancing synths two thoughts, one frivolous... the first: i think that much of the difficulty in the idea of controlling synthesis with large scale gesture is in domain matching. frankly, i would not expect to be able to move my arm in a similar manner to the way i did before and get exactly the same knob tweak as a result. the mapping between cause and effect does not seem to me to preserve the topology of either... there is a gross compression necessary. i might, however, expect that an arm sweep in general specify a timbral swell, that the rough measure of the extension of my arm correlate to the spectral region in which to perform the change, that the arc length correspond somewhat to the depth of the effect, and that the speed of the gesture affect the rapidity of the modulation. now, one need not provide high-resolution couplings for these kinds of controls. in fact, i propose that they are not desirable. one would, i think desire a certain amount of 'fuzziness' in the translation, so that similaritie of gesture whose features are within a boundary of significance are mapped to similarities of result. sort of like playing an acoustic instrument, whose infinite variety and imprecision do not make it 'wrong' every time you play the same note, just idiosyncratic. i do not think of this kind of thing as farfetched. a neural network system, while it may take quite some time to develop and train to the kinds of inputs that are expected, can run in real time once one stops updating its connection weights. such a system would be able to extract gestural features with the kind of floating tolerance that i suggest is necessary, given any of a number of input mechanisms. velocity, pressure, and flexion transducers on the body, as mentioned here, are one possibility. technically simpler, although harder to train for, would be two cameras at right angles to each other. edge detection issues can be minimized at the outset by simply taking binary images... no gray scale. basic body position, configuration, and motion would provide ample data for a wealth of control. and now the frivolous notion... the idea of swinging the synths around a stationary performer really amuses me. i conceive of an installation in which keyboards are hung and mounted on tracks, and the performer simply stands there, arms outstretched, as heavy keyboards are passed at high speeds around his body, and (oops) ... well, it still needs some work. -----------< Cognitive Dissonance is a 20th Century Art Form >----------- Eric Harnden (Ronin) or The American University Physics Dept. 4400 Mass. Ave. NW, Washington, DC, 20016-8058 (202) 885-2748 ---------------------< Join the Cognitive Dissidents >------------------- ------------------------------ Date: Fri, 10 Jul 1992 11:37:03 -0400 From: Max Stevens-Guille Subject: Re: "muscle-memory" synth interfaces - implementation issues > > In article <9207082047.AA07701@twinpeaks.gsfc.nasa.gov>, > xrjdm@CALVIN.GSFC.NASA.GOV (Joseph D. McMahon) writes: > > wbf@alux2.att.com writes: > >> > >> I'm finding this discussion quite interesting. Do I understand that it > >> is desirable to make body motions (i.e. a dance performance) generate a > >> specific musical performance that is repeatable in a recognizable way > >> like one can recognize the Beatles' "Yesterday" no matter who performs > >> it? (That DOES leave room for style and interpretation.) > > Originally, I was thinking only about why it seems to be easier to > > control parameters on an analog synth than on a digital one, but we've > > moved into such rarefied realms of speculation that the original > > thought seems rather dull in comparison... > >> > >> Blindfold the performer, move the synth around > >> so that the selected knob isn't detectable by that performer, and you > >> have, I beleive, the same situation a dance performer would have. How > >> would the dancer know that a particular motion will have the desired, > >> repeatable, musical result? The dance motion may be correct, but > >> lift the arm 5 cm and "a different knob is being turned." This may or > >> may not be controllable by the dancer in the heat of the performance. (I > >> don't know; I'm not a dancer.) > > Yep, that's what I was talking about with perceptual anchors. It's > > difficult to swing a baseball bat through a specific point in space > > unless there's a baseball there to anchor the gesture. > > > > --- Joe M. > > I've heard of a similar experiment nicknamed "Frog's Eye" up here in > Canada. They used 2 or 3 TV cameras to monitor the dancer's movements, and > a computer played various synths, depending on what region of space the > dancer moved into. Apparently, the dancer *did* learn to "play" the > instrument, with practice. Unfortunately, I don't know how precise this > "playing" was. > > Simon N. > The system that you're talking about here is David Rokeby's "Very Nervous System". David's located at: 349a Spadina Avenue, Toronto, Ont. M5T 2G3 (416) 596-1428. He's an incredibly talented individual. You can usually find him on tour at Computer Graphics/Human-Interaction conferences. I dunno if he's making SIGGRAPH this year. For the techy inclined, the system is basically a multi-camera frame grabber with some sort of convolution hardware. He showed me schematic once, but that was a long time ago... The truely neat stuff is the software. He built up a programming language that is used to interpret changes in the contents of his frame grabber. The system as a whole is not so much oriented towards absolute positional accuracy as noticing changes over time. Using his system one can define regions (which may move over time). Dancers work in a fixed space (usually 20' by 20' or so, but he's done very large outdoor events). Truely facinating stuff. Max -- Max Stevens-Guille Research & Development __ Alias Research Inc. 110 Richmond St. East, / Toronto, Ontario, CANADA M5C 1P1 / Alias max@alias.com / \ Research vox: (416) 362-9181 fax: (416) 362-0630 / \_ Incorporated ------------------------------ Date: Wed, 15 Jul 1992 19:41:02 GMT From: Bruce Becker Subject: Re: synth interfaces-implementation issues In article LNDR08@CLEMSON.BITNET writes: |Just wondering if anyone had thought of using a video camera |and a computer to "watch" body movement instead of all the complex |electronics that have been mentioned. Less electronics would be |involved but most likely the software would be much more complex. |I know there is work being done in visual perception with computers |but I don't know how precise the output is (can it tell the difference |between a arm movement and a hand or finger movement). It's called "Mandala" it runs on the Commodore Amiga (from Vivid Effects, 314 Adelaide W., Toronto, ON), and has very nice interface to associate graphic overlays etc. with trigger areas in the image - triggers can be set up to produce an amazing array of events... -- ,u, Bruce Becker Toronto, Ontario a /i/ Internet: bdb@becker.gts.org Uucp: ...!web!becker!bdb `\o\-e \*/*\*/*\*/*\*/*\*/*\*/*\*/*\*/*\*/*\*/*\*/*\*/*\*/*\*/ _< /_ "Awaken the Geek Within" - Anthony Robot ------------------------------ End of the EMUSIC-L Digest ******************************