EMUSIC-L Digest Volume 42, Issue 02 This issue's topics: "muscle-memory" synth interfaces (part 2 of 3)(11 messages) Your EMUSIC-L Digest moderator is Joe McMahon . You may subscribe to EMUSIC-L by sending mail to listserv@american.edu with the line "SUB EMUSIC-L your name" as the text. The EMUSIC-L archive is a service of SunSite (sunsite.unc.edu) at the University of North Carolina. ------------------------------------------------------------------------ Date: Tue, 7 Jul 1992 21:40:30 +0200 From: A.MULDER@ELSEVIER.NL Subject: "muscle-memory" synth interfaces - implementation issues Raybro wrote: > There is a company in PA whose name escapes me (isn't it always so?) > who make a piezoelectric sheet material. They call it Kynar, and > the company name is something like PenWalt (but I'm sure that's not > it!). Its Pennwalt. I've been fiddling with this piezo stuff (by glueing layers of sheets onto each other in specific manner it is possible to construct a flex sensor), but the thing is that it leaks. It leaks charge so to say. So you nee very high impedance opamps to interface them. It can be done but you need clever electronics, and I'm not really sure the result will be as good as with e.g. (piezo)resistive stuff that is sensitive to bending. I'm still working on these piezos, but if anyone knows more about e.g. where to get the flex sensors in the powerglove I'd be more than interested. > On fibre-optics...this stuff is a) expensive to play with, especially > for precision results, b) fragile, and c) requires some stuff which is > downright dangerous. OK, thanks, I won't pursue that path for now. Another possibility I'm investigating is the FSR sensors of Interlink (FSR stands for Force Sensing Resistors). They are supposed to change resistance to bending, but I haven't succeeded to achieve that result sofar. They certainly are sensitive to pressure ! > I encountered a few interesting stumblers on the way: most synths > are very snotty about changing parameters in mid-flight. They hafve > to stop and think about it before going on. That made some of the > thoughts I'd had about this interface somewhat pointless. [...] > Also, most synths are designed to be driven by keyboards. The whole idea > of MIDI poo-pooh's the paradigm of a wind instrument. It is very difficult > to make a sensor that can present to a MIDI synth the following two > natural-to-wind-instrument sounds: That's what I meant: no real time patch programming is possible on most MIDI synths. So why don't you go DSP. Do you have any more detailed info on this RTX project ? > The thing that has kept me from doing more on this is that I basically > don't see a way to do something that will be worth the rest of my life. > I do believe that it would take at least that long to master the equipment > necessary to get wind-like control from a MIDI synth. Well, again consider going DSP ! Forget about MIDI, it's outdated anyway. I'm sure something can be made that is worth the rest of my life (and even longer) There is some unexplored artistic universe here, bringing dance so close to music. I think I would call music coming from these experiments "transmodal art" or something. Axel ------------------------------ Date: Wed, 8 Jul 1992 08:50:00 MDT From: "JONATHAN SMITH" Subject: muscle-memory synth interfaces Hrm interesting discussion indeed, One note I AM a dancer :) and no its not at all feasable to get that mechanica in movement usually, especially if your doing more than just standing there moving small body parts. However, i dont think that discounts the possibilities of the effect still happening. After all one finds out how ones body works by using it, ie moving around in different ways. Whats to keep one from doing the same with the gear movment-music trasnlation. Obviously that i you have similiar setup similiar movements will get similiar sounds. And the performer does have feedback, more so than the usual dancer really does in som sense. He /she can HEAR what the results are.. All-in-all I thing something like this would be very interesting to do, I've tinkered around with synths an musical composition myself but dance is much more a part of my life than that stuff. One thought though, how big/heavy/cumbersome is the eq going to be? if its just a few sensors attached at places like joints that's probably ok, and hopefully we aren't dragging around cable with us that connects us to the musi stuff :) something along the lines of an short range infrared broadcaster or such? Jonathan ------------------------------ Date: Wed, 8 Jul 1992 09:10:00 EDT From: wbf@ALUX2.ATT.COM Subject: "muscle-memory" synth interfaces - implementation issues I'm finding this discussion quite interesting. Do I understand that it is desirable to make body motions (i.e. a dance performance) generate a specific musical performance that is repeatable in a recognizable way like one can recognize the Beatles' "Yesterday" no matter who performs it? (That DOES leave room for style and interpretation.) It seems to me that when one grabs a knob on a synth, its action is (somewhat) defined by which knob it is. Visual feedback and the knob's relative position on the front panel are involved. The actual motion, once a knob is selected, is the same motion used on some of the other knobs on that panel. Blindfold the performer, move the synth around so that the selected knob isn't detectable by that performer, and you have, I beleive, the same situation a dance performer would have. How would the dancer know that a particular motion will have the desired, repeatable, musical result? The dance motion may be correct, but lift the arm 5 cm and "a different knob is being turned." This may or may not be controllable by the dancer in the heat of the performance. (I don't know; I'm not a dancer.) Bill ------------------------------ Date: Wed, 8 Jul 1992 09:33:00 EDT From: RAY BROHINSKY Subject: Re: "muscle-memory" synth interfaces - implementation issues I said: > Also, most synths are designed to be driven by keyboards. The whole idea > of MIDI poo-pooh's the paradigm of a wind instrument. It is very difficult > to make a sensor that can present to a MIDI synth the following two > natural-to-wind-instrument sounds: Axel replied: |That's what I meant: no real time patch programming is possible on most |MIDI synths. So why don't you go DSP. Do you have any more detailed info on |this RTX project ? DSP? Might be a wonderful way to go. I wish I had a budget 8^) Of course, I also wish I had the calculus bacground and the signal processing background that would make understanding the algorithms possible (or at least a bit easier!) As for the RTX, it's a dead issue: Harris decided that it couldn't be both analog chip-master and microprocessor-developer/supplier, and they decided in favor of the established analog side of the world. Upon speaking to Phil Koopman, who designed the chip (and who now works for UTRC), I've discovered that the chip is still available, in a couple of different ways, but ya really have to want it. There are two companies, one in the US and one in GB who sell development systems, and Harris has a warehouse full of dies and a fab line that will probably go away, and can fill volume orders... Frankly, as good as the chip and the design are, This is probably not a good idea for current designs unless you are not only budgeted but bankrolled! > The thing that has kept me from doing more on this is that I basically > don't see a way to do something that will be worth the rest of my life. > I do believe that it would take at least that long to master the equipment > necessary to get wind-like control from a MIDI synth. |Well, again consider going DSP ! Forget about MIDI, it's outdated anyway. |I'm sure something can be made that is worth the rest of my life (and even longer). |There is some unexplored artistic universe here, bringing dance so close to |music. I think I would call music coming from these experiments |"transmodal art" or something. Well, in some ways, playing a wind instrument is pretty close to dance, also! However, most of the dancing happens inside someone's head...8^) If you have any advice for resources for getting into DSP, I'd be interested to hear. |Axel raybro ------------------------------ Date: Wed, 8 Jul 1992 16:31:07 PDT From: more dark than shark Subject: RE: synth interfaces-implementation issues Hey Johnithan, that video-based people watching midi input device already exists. I've seen implementations on amiga and mac. the question is that of resolution. casey ------- ------------------------------ Date: Wed, 8 Jul 1992 16:27:00 MDT From: "JONATHAN SMITH" Subject: RE: synth interfaces-implementation issues >Just wondering if anyone had thought of using a video camera >and a computer to "watch" body movement instead of all the complex >electronics that have been mentioned. Less electronics would be >involved but most likely the software would be much more complex. >I know there is work being done in visual perception with computers >but I don't know how precise the output is (can it tell the difference >between a arm movement and a hand or finger movement). > Matt. *OUCH* and you call all this hanging gizmo's on people complex?? Whew. OKay well for one thing getting a computer TO recognize a shape is nto very easily done, and requires a lot of compute power. Getting it to recogniz shapes like a finger and identifying that as a finger(to start up a drum or something) is an even greater task. Pile on top of that getting it to say oka this is a finger and it moved up, IS NOT a trivial task at all! Not to mentio all the processing power that you would need to do this period, let alone in real time. And depends on how you consider electronics.. the stuff thats being developed for precisly the purpose of recognizing shapes and patterns is usually very complex and more often than not there is a lot of said electronics.. Jonthan Smith (yea the dancer again , and im also a Comp Sci person, and EE) ------------------------------ Date: Wed, 8 Jul 1992 17:29:17 PDT From: more dark than shark Subject: RE: synth interfaces-implementation issues Hey, nice job with the numbers, Jonathan! What I've seen is of lower resolution, true. U r infront of a video camera. On a display U can see are a number of colored circles overlaying the video display of U. when you move you're in a virtual instrument; as your image intersects with the overlays MIDI is generated. the overlays can change position. ah, this is simplistic, no? nevermind. casey ------- ------------------------------ Date: Wed, 8 Jul 1992 17:56:00 MDT From: "JONATHAN SMITH" Subject: RE: synth interfaces-implementation issues >Hey Johnithan, > that video-based people watching midi input device already >exists. I've seen implementations on amiga and mac. > the question is that of resolution. > >casey >------- Well first off I didn't say it wasn't doable, just not hardly easy, secondly are you sure we are talking the same things? This device you saw, could actually determine that a person has turned his hand, raised a leg, stomped a foot , and so on with that much detail? IN real time no less? Well if so I find it extreemly hard to believe that its running on JUST an amiga or mac. Something much simplier could surely exist.. I was assuming you meant something that could handle the complexities and details of what most of the rest of this discussion was talking about. PS even with a simple low resolution device, and no grey scale (1 bit of data per pixel on the CCD) that would require almost 1 Mega byte/sec data handling ability, thats not even DOING any calculations at all!, and image recognition is not simple calculations.. and if we take that up to 256 grey scales thats bumps it up to 8 megabyte/sec at least (thats at 30 frames a second, at 60 double it). and we still have to process that entire array of data! to tell which part is the person, then which parts of the person changed from before.. Oops that means we have to store the frame before too! another 8meg so you haveto process 16 megabyte (at 256 grey scale levels) in 1/30th of a second.. (actually less than that..) And I haven't even mentioned any of the algorithims that are used to trace say edges so I can tell where the person ends and starts.. and where the backgroun begins.. I also have seen some impressive image recognition stuff. Including a machine that played checkers (which BTW took a bit of hardware, and thats NOT real tim nor something near as complex as a body, its just figuring out where some basically circular shaped objects are.) But again I seriously doubt such exists purely on a mac or amiga, both of whic have some nice machines out, but we are talking extensive calculations on 16+ megabytes every 1/30th of a second.. Most of the large amount of research being done on image recognition is being assisted by a myriad of custom device designed specifically for that express purpose. Again! something simplier by far that what I assumed you meant, and what I was talking about may exist on its own on a mac or amiga... Jonathan Smith ------------------------------ Date: Wed, 8 Jul 1992 16:27:57 EDT From: "Joseph D. McMahon" Subject: Re: "muscle-memory" synth interfaces A.MULDER@elsevier.nl writes: > > I sure think that there are conceptual anchors for gestures; aren't gestures > recognizable ? One can discuss the style of one's movements in sports or in > general, making comparisons between various persons. I can (try to) imitate > some person's behaviour: mimicry. I'm an ignorant on dance but I'm sure ther > must be various (sequences of) gestures/postures that have a name (how about > a double-axel in ice-skating ?). There have also been various efforts to > formalize dance (laban notation) and use computers to write choreographies. > Ah. A matter of scale. i was thinking of my studio where there is room for me, the synth, and a chair. Just. :-) Recognition is vastly different from abstraction. If I close my fist and chop it downward, what does it mean? A balance has to be struck between precision and sloppiness -- the control isn't as fine for the full body. There is no conceptual anchor to "pass your hand through the space 5 inches up and 12 inches to the left of your nose." You can move your hand through there but it's hard to hit a precise point -- unless you're trying to smack a handball. Now you've got a visual target to anchor the gesture. (Now there's another idea - put sensors in the ball and bounce it, or juggle it, or step on it...) > > them a little more "blurred" in conceptual space -- that is, the net could > > "watch" your gestures and "listen" to the output, perhaps? > > I agree, I think ... what do you mean with listening of the neural net ? Sensing some portion of the performance medium: the sound via samples and FFT; the instrument settings and comparing them with previous performances -- obviously you have the latitude to make this very important, or not important at all. The crux of it is ensuring that results are predictable: a step *here* does *this*. Obviously, this is going to be contextual. Raising your arm then rotating your wrist is obviously a much different gesture from doing both at the same time or rotating then raising, or hesitating during one or the other, or ... You get the picture. This is a very complex medium if one chooses to sense a lot of parameters. EMUSIC trivia: there was actually an instrument built called the Terpsitheremi . It was pretty much a standard instrument, except that the antennas were tuned so that it took a complete body movement, rather than a hand movement, to play the instrument. I vaguely remember reading a review of someone (*maybe* Clara Rockmore) who danced the "Ave Maria" on it... > > Is feedback, that is tactile feedback, really needed ? Expression exists in > many forms and gestures/postures are some of them. Dancers must have feedbac > on what they do (isn't it called proprioception ?); they know when they did > a good performance. Therefore it's probably a matter of preference; does > visual, auditory or another type of feedback work for you. Yes, that's true. However, the feedback isn't direct. You don't *feel* your hand hit the virtual drumhead, so to speak. The interface becomes very fuzzy and therefore much less controllable. The difficulty is achieving a delicate and precise control over something using parts of the body not designed for fine control. Ever try to turn a knob with your nose? > > In any case, knobs etc. are in the "finger"-domain, that is they hardly > involve full body motions. As a compromise I could imagine that you might > have plain motion sensors on all parts of your body except your fingers, whi h > would be equipped with some kind of tactile feedback devices (e.g. > piezobuzzerz zuch az in the Dataglove). Another way to get feedback is to > perform with someone else that has to counter your moves to some extent in > order to obtain his/her musical results. It could certainly be an interestin > choreography .... Wrestling with the instrument takes on whole new meanings, doesn't it? --- Joe M. ------------------------------ Date: Wed, 8 Jul 1992 16:39:39 EDT From: "Joseph D. McMahon" Subject: MIDI Dance Fever JONATHAN SMITH writes: > > Hrm interesting discussion indeed, > One note I AM a dancer :) Thank goodness, contact with reality :-). > However, I don't think that discounts the > possibilities of the effect still happening. After all one finds out how on s > body works by using it, ie moving around in different ways. For an improvisational piece it might work very well indeed. But I was thinkin of controlling predetermined musical events. or modifying sound parameters, originally. > Obviously that if > you have similiar setup similiar movements will get similiar sounds. And th > performer does have feedback, more so than the usual dancer really does in > some sense. He /she can HEAR what the results are.. As long as the motions *do* translate directly into a specific control (e.g. right hand low - slow, high - fast, unless fist is closed, in which case ignore)this will work well. It's when relative combinations do different thin s that it gets complex. Then again, complex can be very good. :-) > One thought though, how big/heavy/cumbersome is the eq going to be? if > its just a few sensors attached at places like joints that's probably ok, an > hopefully we aren't dragging around cable with us that connects us to the > music stuff :) something along the lines of an short range infrared > broadcaster or such? Last month's CMJ had an article on control of sequencers (I think) with dance The dancer-to-machine interface was a small FM transmitter, about the size of a couple of fat paperbacks. Obviously I didn't read it carefully, because that's about all I remember about it. Have to go back and look at it again. --- Joe M. ------------------------------ Date: Wed, 8 Jul 1992 16:47:35 EDT From: "Joseph D. McMahon" Subject: "muscle-memory" synth interfaces - implementation issues wbf@alux2.att.com writes: > > I'm finding this discussion quite interesting. Do I understand that it > is desirable to make body motions (i.e. a dance performance) generate a > specific musical performance that is repeatable in a recognizable way > like one can recognize the Beatles' "Yesterday" no matter who performs > it? (That DOES leave room for style and interpretation.) Originally, I was thinking only about why it seems to be easier to control parameters on an analog synth than on a digital one, but we've moved into such rarefied realms of speculation that the original thought seems rather dull in comparison... > > Blindfold the performer, move the synth around > so that the selected knob isn't detectable by that performer, and you > have, I beleive, the same situation a dance performer would have. How > would the dancer know that a particular motion will have the desired, > repeatable, musical result? The dance motion may be correct, but > lift the arm 5 cm and "a different knob is being turned." This may or > may not be controllable by the dancer in the heat of the performance. (I > don't know; I'm not a dancer.) Yep, that's what I was talking about with perceptual anchors. It's difficult to swing a baseball bat through a specific point in space unless there's a baseball there to anchor the gesture. --- Joe M. ------------------------------ End of the EMUSIC-L Digest ******************************