EMUSIC-L Digest Volume 42, Issue 01 This issue's topics: "muscle-memory" synth interfaces (part 1 of 3) (5 messages) Your EMUSIC-L Digest moderator is Joe McMahon . You may subscribe to EMUSIC-L by sending mail to listserv@american.edu with the line "SUB EMUSIC-L your name" as the text. The EMUSIC-L archive is a service of SunSite (sunsite.unc.edu) at the University of North Carolina. ------------------------------------------------------------------------ Date: Thu, 2 Jul 1992 17:08:48 +0200 From: A.MULDER@ELSEVIER.NL Subject: "muscle-memory" synth interfaces Joe McMahon wrote sometime ago in emusic-d (I think in volume 50 issue 2 or 3) [..] > I think I've finally gotten my finger on *why* timbral improvisationalists > find current synths useless - it's a right-brain/left-brain phenomenon. [..] and somewhat further down [..] > Conversely, CV machine used pots and patch cords because *they* were easy to > implement with the technology. Ask me where the modulation knob was on a > MicroMoog, and I can tell you it was on the top row (of two), and was on the > left of the oscillator frequency knob. But I don't remember what it was > labelled, or whether the wording was above or below the knob. It was operabl > by "muscle-memory". You didn't read all of the labels and say, "Well, here's > the modulators sub-panel, and then here is the filter cutoff control." You > just reached and got it. *That* is the strength of a lotsa-knobs interface, > the ability to change any parameter at any time without having to switch fro > right-brain to left-brain mode. It lends itself to associative memory, rathe > than "lookup" memory. [..] So, why don't we create such an interface to those stupid synths or DSP's ? Personally I get crazy of the idea of having vast musical expression potential y available and yet being unable to fully use it, especially in an improvisation l context, just because the user interface has been badly configured. How the he l can one concentrate on sound/music (right-brain) and at the same time (left-brain) have to remember to press button x for menu y to reach parameter z that will influence the sound such and such so that you need to remember to change parameter zz in menu yy also to achieve the acoustical result you desire. I believe very much in right-brain activity and detest the tremendous amount of left-brain activity that we need to perform these days, if you know what I mean. Perhaps in improvisation right-brain activity is even more necessary. Therefore, looking for real right-brain activity, I thought of dance (bodily motion, posture and gestures or joint motions) as an ideal generator of contro parameters for music and sound. 5 years ago my (right ?) brainwave was to simp y map synth parameters to joint motions, so I started working on a glove-like interface that would measure joint flexion and extension. Clearly technology has developed since upto the point that Power-/Cyber-/Data-/etc Gloves and Suits have become readily available products, but little effort has been done to actually formulate a way to relate these joint motions (or bodily gestures) to musical or sound parameters. On the one hand the problem of a well defined sound synthesis model useable for these kind of experiments remains (how the heck is bodymotion/bodily gesture/dance related cognitively to musical expression or sound perception ? Any refs are welcome). On the other hand it's not too difficult to hook up gloves to synths and experiment. Clearly this is what happens today and as far as I know most of the experiments are concerned with finger and hand motions mapped to all kinds of synths and DSPs (e.g. the Glove-List archive has a summary of people doing experiments with the Powerglove). My own experiments sofar do not reach further than mapping 4 joints of the han (one of the thumb, two of the index and one of the middle finger) with a homebrew glove via a homebrew MIDI card in a PC-AT running homebrew interrupt-driven Turbopascal code to parameters of a industrybrew Roland D10. The D10 is a mode t synth and unfortunately incapable of changing sys-ex parameters without having to send a new note-on msg. So either I had to keep on sending note-ons or I ha to use only non-sys-ex parameters, such as pitch, pitch-bend, modulation etc. However the musical effects were there. The piezoelectric sensors I used have tremendous precision so that control was really smooth. Mapping a sensor onto the parameter that selects the timbre had a stupid effect because you start jumpin around in soundspace like an idiot. So what's needed is some way to interpolat between all these timbres (something for a neural net ?), David Wessel (CNMAT, UC Berkeley) has given this some thought but I have no knowledge of other literature on ways to realise this. My bottomline conclusion sofar was that you really need realtime sound processing and an easily configurable mapping system, which means that I am in shortage of hardware/software (I have sold my PC-AT and now work with my Atari-ST equipped with an AT hardware emulator). But I want to go for the real thing and use the entire body articulation to generate signals as control values of synths/DSPs. The technology is there and what's more its affordable. Right now I'm trying to get hold of either the flexible resistance sensors used in the powerglove (anyone got any knowledge of a company called Amtec in Salt Lake City, Utah, who are supposed to have these sensors ?) or other such sensors to use them in a suit. If I'm unsuccesful there I still have my self-developed (somewhat unreliable) piezoelectric flex sensors. Then there seems to be a workable/cheap solution with fiberoptic cables (where does one buy these ?). Furthermore I was thinking to use neural nets not only for gesture recognition (more and more a maturing technology, and simple experiments have been conducted by various people also using the output for music/sound generation), but also for the remodeling or reorganization of the control parameters of synths/DSPs so that linking gestures/motion with DSPs/synths becomes easier. E.g. a neural net could provide interpolation between the various timbres (anyone have more info on this technique ?). I'm relatively unacquainted with soundprocessing boards, but would appreciate a good suggestion, that remains affordable (dutch thriftyness). So what's a good sound processing platform and what sound processing algorithm should I use, or should I go work in parallell, using various boards and various algorithms simultaneously ? I guess a Mac II with MAX or a NeXT would suffice but I really do not want to spend this as a private person. I like to do low-end/home-brew work first to see what I can achieve with that and then perhaps upgrade. I know very few people doing mapping experiments with an entire suit, but if you are out there, I (and presumably others also) would like to know their results. Axel (A.Mulder@Elsevier.NL) ------------------------------ Date: Thu, 2 Jul 1992 10:09:19 EDT From: "Joseph D. McMahon" Subject: Re: "muscle-memory" synth interfaces A.MULDER%ELSEVIER.NL@gibbs.gsfc.nasa.gov writes: > > I believe very much in right-brain activity and detest the tremendous amount > of left-brain activity that we need to perform these days, if you know what > mean. Perhaps in improvisation right-brain activity is even more necessary. > > Therefore, looking for real right-brain activity, I thought of dance (bodily > motion, posture and gestures or joint motions) as an ideal generator of control > parameters for music and sound. 5 years ago my (right ?) brainwave was to simply > map synth parameters to joint motions, so I started working on a glove-like > interface that would measure joint flexion and extension. Clearly technology > has developed since upto the point that Power-/Cyber-/Data-/etc Gloves and > Suits have become readily available products, but little effort has been don > to actually formulate a way to relate these joint motions (or bodily gesture ) > to musical or sound parameters. > The only problem is that dataglove/full-body interfaces lack the other really powerful thing that knobs have, which is *feedback*. You can feel the miniscule bit of friction that you're overcoming you turn the knob, and that helps you scale the motion. This is what I was obliquely referring to when I mentioned the fact that wide-throw and narrow-throw parameters are mapped to the exact same motion range on a typical digital synth with a single-slider interface. > On the one hand the problem of a well defined sound synthesis model useable > for these kind of experiments remains (how the heck is bodymotion/bodily > gesture/dance related cognitively to musical expression or sound perception > ? Any refs are welcome). On the other hand it's not too difficult to hook up > gloves to synths and experiment. Clearly this is what happens today and as f r > as I know most of the experiments are concerned with finger and hand motions > mapped to all kinds of synths and DSPs (e.g. the Glove-List archive has a > summary of people doing experiments with the Powerglove). > The problem being that a gesture is so much more personal than is a dial-twist I could probably teach someone to twiddle the dials *approximately* the same way as I did in a given musical passage; if the same thing took complex gestural movements to do, it would be much more difficult, because there is no conceptual anchor for them. If you are learning to do complex motion X, it helps to get *both* sides of the brain involved (that's why knobs should have labels!). It's still my contention that the knob interface relies heavily on muscle memory, but it's probably much deeper than that. I think it also depends on visual and kinesthetic feedback as well (whoever it was who was joking about the feel of the grit in the knob was actually more correct than he or she knew!). You *see* the knob you're reaching for, and where its little index mark is pointing, and you *feel* how fast or slow you're turning it, and you *hear* realtime what is happening, and based on all of that input, you decide when, how fast, and how much (and even whether) to turn a knob. This is a *very* complex interaction. We haven't gotten taste and smell in on this, but they're harder to interface with. :-) > > Furthermore I was thinking to use neural nets not only for gesture > recognition (more and more a maturing technology, and simple experiments > have been conducted by various people also using the output for music/sound > generation), but also for the remodeling or reorganization of the control > parameters of synths/DSPs so that linking gestures/motion with DSPs/synths > becomes easier. E.g. a neural net could provide interpolation between the > various timbres (anyone have more info on this technique ?). > Perhaps the neural nets could also help "conceptually anchor" gestures to make them a little more "blurred" in conceptual space -- that is, the net could "watch" your gestures and "listen" to the output, perhaps? This is a great topic. I imagine there are probably CS folks, cognitive psych folks, and EE types out there saying, "Look! It's a thesis!" :-) More! --- Joe M. ------------------------------ Date: Thu, 2 Jul 1992 10:29:00 EDT From: RAY BROHINSKY Subject: Re: "muscle-memory" synth interfaces Axel writes prolifically about motion-sensors for synth control. I have a few comments. First, on sensors: There is a company in PA whose name escapes me (isn't it always so?) who make a piezoelectric sheet material. They call it Kynar, and the company name is something like PenWalt (but I'm sure that's not it!). This stuff is both mechanical- and pyro- sensitive, and they've made some very interesting sensors out of them. They are not above experimentation, either, but you may find that their standard experimenter sheets and sensors meet your needs quite well. The bandwidth on this material can be quite high...one of their demos at Electro the last few years has involved a sheet of 5"x12" Kynar, bonded to the back of one of those silver helium balloons, being driven through an equalizer from a cassette machine. Makes the balloon into a speaker! On fibre-optics...this stuff is a) expensive to play with, especially for precision results, b) fragile, and c) requires some stuff which is downright dangerous. For instance, the best sensor fibers are twin-core glass. The material is expensive, it shatters into small, sharp, very dangerous splinters. It requires lasers to `read' properly. The lasers don't have to be high power, but can still cause you damage without your knowing because the light wavelength used is invisible. (The fibers that sense don't use visible laser light efficiently, or at all). Getting the light into the fibres requires special selfoc lenses, getting it out requires expensive sensors. This is one of the reasons that the NASA glove is so expensive. Second, on platforms: If you want to get multitasking, with some intrinsic sound and midi capabilities, consider an amiga500. They come with four channels of slightly-better than 8-bit sound, and a simple midi converter (basically connectors and a few opto's and resistors) are cheaply available. More expensive (of course) multi-channel midi interfaces are available, but still are less expensive than the midi cards you need for PC's. There are also patch editors and librarians already available for the amiga, which will assist in unifying the stetup, and the C compilers range from the Manx and SAS compilers (which are extremely capable and moderately costly to very expensive, depending on your taste in costs!) to DICE, which is written by Matt Dillon (really!) and available in a shareware version which is _not_ crippled and a fully supported version which is darn near a match for the commercial ones. (I believe that Matt isn't going to support some of the more esoteric optimizations, but I'm not up on that). There are also FORTHs available for the amiga, notable being JFORTH, which has available for it hmsl (Hierarchical Music Systems(?) Language) which has been used for auto-composers, intellegent controllers, etc. (I have no direct experience with hmsl, but I've used JFORTH thru version 2, and it is _very good!_) Although the amiga's intrinsic sound capabilities are not quite recording-quality, they have been used in performance (and the performers weren't stoned by the audience!) and they will provide a sufficient platform for proof-of-concept. Also, the A500 can be had for 3-4 hundred bucks for a 512K-1M machine. Add a monitor (or, god forbid, a TV), hook up your stereo, and go. Finally, another though or two: When Harris had the big RTX contests, (the RTX200x series are microprocessors that execute the FORTH language directly as their instruction set) I was going to make a `multi-interface for midi' as my entry. I obtained the entry kit, and then ran out of time. The idea of the machine was to be a `brain' with the RTX as translater/mapper/sensor-computer, and a bus into which various sensor-electronics modules would go. There were to be sensors for piezos, resistance, switches, capacitive, and just about anything else. The sensor board would provide the proper drive for the sensor, and translate the returned signal into something digital that the brain would read directly. The brain would have all the stuff necessary to translate the recieved digital info into sys-ex/non-sys-ex to change the parameters and drive the synth. I encountered a few interesting stumblers on the way: most synths are very snotty about changing parameters in mid-flight. They hafve to stop and think about it before going on. That made some of the thoughts I'd had about this interface somewhat pointless. Also, most synths are designed to be driven by keyboards. The whole idea of MIDI poo-pooh's the paradigm of a wind instrument. It is very difficult to make a sensor that can present to a MIDI synth the following two natural-to-wind-instrument sounds: one that starts very quietly with practically no attatck, and crescendos to triple forte, cutting off instantly... followed by... a string of quiet thirty-second notes with sharp attacks. The whole velocity/volume thing, with velocity being attatched to the note-on and determining an overall volume level, and volume being attatched to a `controller' and requiring the signal to be `non-zero' is directly in opposition to breath pressure, which determines force (velocity, sforzandi, attack, etc) and volume at once. To do so will require a different paradigm in the hardware. (Oh, yeah, if someone wants to prove me wrong, and actually gets their synth set up to do the above, I'll add one more: a string of sixteenths played at FF with a very smooth attatck. Remember, these have to be played one after the other without having your hands leave the controller. I suppose you could have a patch-changer machine in the line that changed from one patch to another, and you could actually have a hundred pedal switches to be able to randomly choose...but you still pay the price in patch change time, and such a machine begins to make having twenty-seven wind instruments on stage look simpler.) The thing that has kept me from doing more on this is that I basically don't see a way to do something that will be worth the rest of my life. I do believe that it would take at least that long to master the equipment necessary to get wind-like control from a MIDI synth. Hopefully, you'll find more to encourage you here than to discourage you, and you'll prove me wrong. I would be glad to see it! raybro raybro@holon.res.utc.edu PS, I don't have any `claim' filed on any of these ideas, and would rather see them implemented and used than worry about suing anyone anyway. So use what you can, and give away the rest. raybro ------------------------------ Date: Thu, 2 Jul 1992 14:06:28 EDT From: Brian Adamson Subject: Re: "muscle-memory" synth interfaces To those of you interested in this alternate controller thread, There is a picture & short blurb in this month's MacWorld showing a guy using a modified DataGlove with a Mac & some kind of neural net extension to the Max programming environment to control MIDI synths. The text blurb specifically mentioned that the neural net responded to general physical gestures is a consistent manner, eliminating the person's need to move in very specific (& rigid) motions to get the desired MIDI response. The blurb also said that this extension to Max is not yet available as part of the Max package (I don't know if this implies that it will be sometime). -- Brian Adamson NRL Code 5523 adamson@itd.nrl.navy.mil ------------------------------ Date: Tue, 7 Jul 1992 21:07:10 +0200 From: A.MULDER@ELSEVIER.NL Subject: Re: "muscle-memory" synth interfaces In order to somewhat tackle this topic in a way that it doesn't get to fuzzy I suggest to identify some subtopics. IMO there are a couple of things involved here: 1 - timbral abstraction (are there sounds/timbres that can be designated as 'most general'- or 'root'- sounds/timbres ? What does ethnomusicology say about this ?) 2 - timbral interpolation (allowing for a continuous as opposed to discretized timbral space) 3 - gestural/postural abstraction or conceptual anchors for gestures/postures (allowing for reproduction by others of musical passages resulting from gestures/postures) 4 - gestural/postural interpolation (allowing for continuous as opposed to discrete control of timbral space and/or allowing for smooth variation between musical passages) 5 - (multi-sensory) feedback (e.g. auditory, visual, kinesthetic) (allowing fo scaling of motions) 6 - implementation issues (motion-sensing, neural nets, sound-processing, MIDI etc.) Now Joe McMahon wrote On 3: > The problem being that a gesture is so much more personal than is a dial-twi t . > I could probably teach someone to twiddle the dials *approximately* the same > way as I did in a given musical passage; if the same thing took complex > gestural movements to do, it would be much more difficult, because there is > no conceptual anchor for them. If you are learning to do complex motion X, > it helps to get *both* sides of the brain involved (that's why knobs should > have labels!). I sure think that there are conceptual anchors for gestures; aren't gestures recognizable ? One can discuss the style of one's movements in sports or in general, making comparisons between various persons. I can (try to) imitate some person's behaviour: mimicry. I'm an ignorant on dance but I'm sure there must be various (sequences of) gestures/postures that have a name (how about a double-axel in ice-skating ?). There have also been various efforts to formalize dance (laban notation) and use computers to write choreographies. > Perhaps the neural nets could also help "conceptually anchor" gestures to ma e > them a little more "blurred" in conceptual space -- that is, the net could > "watch" your gestures and "listen" to the output, perhaps? I agree, I think ... what do you mean with listening of the neural net ? On 5: > The only problem is that dataglove/full-body interfaces lack the other > really powerful thing that knobs have, which is *feedback*. You can feel the Is feedback, that is tactile feedback, really needed ? Expression exists in many forms and gestures/postures are some of them. Dancers must have feedback on what they do (isn't it called proprioception ?); they know when they did a good performance. Therefore it's probably a matter of preference; does visual, auditory or another type of feedback work for you. In any case, knobs etc. are in the "finger"-domain, that is they hardly involve full body motions. As a compromise I could imagine that you might have plain motion sensors on all parts of your body except your fingers, which would be equipped with some kind of tactile feedback devices (e.g. piezobuzzerz zuch az in the Dataglove). Another way to get feedback is to perform with someone else that has to counter your moves to some extent in order to obtain his/her musical results. It could certainly be an interesting choreography .... > It's still my contention that the knob interface relies heavily on muscle > memory, but it's probably much deeper than that. I think it also depends It must be pretty deep; are there any cognitive whizzers or behavioural experts watching that can comment on this ? > This is a great topic. I imagine there are probably CS folks, cognitive > psych folks, and EE types out there saying, "Look! It's a thesis!" :-) > More! I have tried to turn this into a thesis, but I haven't been succesful in finding a place to do this as yet. Any suggestions are welcome !! I have a (Dutch) Masters in Physics with Computer Science and extensive wind instrument experience. Axel For anyone interested I have attached some pointers to literature (much of which I still need to read !) that relate to this discussion. (New electro-acoustical musical instruments) - Various articles in: Computer Music Journal 14 nr 1 and 2 (spring and summer 1990) - Machover T.; Chung, J., Hyperinstruments: Musically intelligent and interactive performance and creativity systems, ICMC Proceedings 1989, Computer music association, San Fransisco CA, USA, 1989 - Morita, H.; Hashimoto, S.; Ohteru, S., A computer music system that follows human conductor, IEEE Computer July 1991 pp 44-53 (EMG/EEG control of music) - Knapp, R. B.; Lusted, H., A bioelectric controller for computer music applications, Computer Music Journal 14 nr 1 (spring 1990) pp 42-47 - Rosenboom, D., Extended Musical Interface with the Human Nervous System, Leonardo Monograph Series (1990) (Abstract gestural control of music) - Chabot, X., Gesture interfaces and a software toolkit for performance with electronics, Computer Music Journal 14 nr 2 (summer 1990) pp 15-27 - Gibet, S.; Marteau, P-F., Gestural control of sound synthesis, ICMC Glasgow 1990 proceedings pp 387-391, Computer music association, San Fransisco CA, USA, 1990 - Krefeld, V., The Hand in the Web: An interview with Michel Waisvisz., Computer Music Journal 14 nr 2 (summer 1990) pp 28-33 - Pressing, J., Cybernetic issues in interactive performance systems, Computer Music Journal 14 nr 1 (spring 1990) pp 12-25 - Rubine, D.; McAvinney, P., Programmable finger tracking instrument controllers, Computer Music Journal 14 nr 1 (spring 1990) pp 26-41 - Wessel, D., Improvisation with highly interactive real-time performance systems, ICMC Montreal 1991 proceedings, Computer music association, San Fransisco CA, USA, 1991 (Gesture/posture recognition/analysis) - McAvinney, P., Vision-based gesture recognition systems, NCGA '90 Conference proceedings 1 pp 304-313, NCGA, Fairfax VA, USA, 1990 - Quam, D.L., Gesture recognition with a Dataglove, Proceedings of the IEEE 1990 national aerospace and electronics conference NAECON 1990 2 pp 755-760, IEEE, New York NY, USA, 1990 - Rubine, D., Specifying gestures by example, Computer graphics 25 4 pp 329-337 (1991) (Psychology, gesture/posture and music) - Deutsch, D., The psychology of music, Academic press, New York NY, USA, 1982 - Jackendoff, R., Consciousness and the computational mind, MIT Press, Cambridge MA, USA, 1989 - Todd, P. M.; Loy, G. D., Music and connectionism, MIT Press, Cambridge MA, USA, 1991 (Dance and computers) - Camurri, A.; Morasso, P.; Tagliasco, V. and Zaccaria, R., Dance and movement notation, In: Human movement understanding, Morasso, P.; - Tagliasco, V., Elsevier Science Publishers, Amsterdam, The Netherlands, 1986 - Politis, G., Computers and dance: A bibliography, (contains many references) Leonardo 23 1 pp 87-90, 1988 ------------------------------ End of the EMUSIC-L Digest ******************************