The "Fresh Look" essays
are an attempt to look at Life through the eyes of an engineer and as you read
through them you will now and then find reference to various concepts. This
section is an attempt to explain these concepts.
An amplifier
Think of a mountain water reservoir with an exit pipe
leading down into the valley. The end of the pipe is closed with a valve. You
slightly open the valve and water starts to jet out several feet. You open it a
bit more and even more water squirts out. Open the valve completely and you
have a raging torrent.
The valve you are turning is acting as an Amplifier - little
effort: big result. And notice that the result is proportional to the effort,
more effort: more result. Or as we say, the output is proportional to the
input. The ratio of the work you do opening the valve to the work the water
does when it squirts out is called the Gain of the amplifier.
Feedback
Imagine the valve is rather difficult to turn. With a little
ingenuity you can use some of the water coming out of the pipe to drive a
turbine which you use to help you to open the valve. This will make operation
of the valve much easier. You need to do less work to control the valve. In the
jargon, you have added some of the output of the amplifier to the input of the
amplifier in order to help or reinforce the input. Less input for the same
output means the gain of the amplifier has been increased. When some of the
output is fed-back to reinforce the input, we say we have Positive Feedback.
If you feel intuitively that the application can be a bit
tricky, you are right. As you increase the amount of positive feedback, the
amplifier gain increases until a point arrives when almost no input is sufficient
to drive the amplifier to maximum output. At this point the amplifier is
extremely unstable. Some little glitch on the input causes an output which is
fed back to increase the input, which gives more output which still more
increases the input, until the amplifier is hard on with no input. Having an
output for no input means the gain has become infinite. The Amplifier has
become a signal Generator. (The type of signal it generates depends a lot on
how the signal gets from output to input - on what the feedback path is like).
Going back to the valve and pipe analogy again, let us now
connect the turbine in such a way that it opposes you when you turn the valve.
In other words it makes the valve harder to turn and reduces the gain. A fairly
pointless exercise you would think. But in fact most amplifiers have more than
enough gain - their problem is that they have too much gain and that this gain
is unstable, varying with time, temperature etc. Many amplifiers are also
non-linear ie. the output is not proportional to the input. (If your hi-fi
amplifier were non-linear its output would sound distorted). Feeding back some
of the output in such a way as to oppose the input is called Negative Feedback
and is a wonderful way to reduce the gain of an amplifier to reasonable values
and at the same time improve its stability and linearity.
Feedback in real life
The effect of positive feedback
can be seen when a salesman watches a customer to see which features please him
and then feeds this back to him, amplifying its effect. A more dramatic example
of positive feedback is to be seen in a riot where everyone receives (and
transmits) excitement-increasing signals. An actor in front of an approving
audience is receiving positive feedback. He becomes "high", which can
make him give a better performance, which makes the audience approve more,
which ...
Negative feedback can be seen in the action of the Devil's
Advocate - a person who damps down the enthusiasm of a highly-creative and
somewhat unstable character, pointing out all the faults in his ideas. The
combination of Enthusiast and Devil's Advocate exists in us all and enables us
to progress without excess.
There are many interesting examples of feedback in Nature.
You see a strange object on the ground and lean forward to pick it up. How much
force should you apply? If you give a light tug and it is heavy you may raise
it a bit but then it will slip out of your grasp: if you give a heave and it is
very light you may fall on your back. So you give it first a little exploratory
tug and see what happens. What you are doing is weighing it by first measuring
the strain in your arm muscles - and then you switch on the main power and
lift. But if it is a well-known object you don't spend a lot of time measuring
its weight before you pick it up. In the middle of a tennis match you pick up
the tennis ball rapidly because you "feed-forward" the weight
information to your arm muscles (and also the "correct grip"
information to your hand). The amusing effect of wrong information being
"fed-forward" can be imagined if a tennis ball filled with lead were
left on the tennis court.
The importance of getting
the feedback right
In the
sixties, everyone in the aeronautic world was building vertical take-off
aircraft. The Germans were no exception and they built one in Munich, called
the VJ-101. For propulsion the VJ-101 had two large rotatable jet engines, one
at each wing-tip, which could be pointed downwards for hovering. There was also
a small fixed downward-pointing engine in the fuselage which gave extra thrust
during hovering and by giving it more or less fuel than the main jet engines,
it could be used to control the pitch axis (ie. nose up, nose down). Now
hovering is tricky in a vertical take-off aircraft. If the aircraft starts to
roll, say clockwise, you have to feed some more fuel to the right-hand engine
and less to the left-hand engine. This will stop the roll but if you want to
get back level, you have to roll back in the opposite direction and when in the
level position give a burst of fuel in the opposite direction to stop more
rolling. Like a space-craft. Of course you have to do this on the pitch axis as
well, and it's a good idea to keep your eye on the height above ground so you
need to be able to control the fuel supply to all three engines. Not surprisingly,
it was thought too much for the pilot to do this all at once. So
to help him, the fuel supply to the
wing-tip engines was modified by a gyroscope, in such a way that if the
aircraft started rolling, fuel would be given to the two engines so as to stop
it rolling - negative feedback. "All" the pilot had to do during
hovering was to keep the aircraft level in pitch, and control the height above
the ground (apart from all the other things a test-pilot does in a prototype
aircraft!)
Well, one day the VJ-101 blasted off down the runway on a Short Take-Off test.
(By having the wing-tip engines point down at 45 instead of 90 degrees, the
VJ-101 would take off in about 100m. Advantage - great saving in fuel as
opposed to true vertical take-off). Imagine the surprise
and horror of all concerned when the VJ-101, immediately after
take-off and still only 50m up, was seen to start to roll. It did a complete
roll and had started on another when the pilot understandably lost his nerve
and pulled the ejector. He made it, (the parachute just had time to open) but
the VJ-101 was a write-off.
The explanation was found by studying the telemetry records
(data radioed from the aircraft). At first, level flight so no correction
signal from the gyroscopes. Then a buffet of wind tipped down the port wing.
The gyroscope unaccountably fed less fuel to the port engine and more to the
starboard engine. Aircraft rolls a bit more. Increased angle of roll detected
by the gyroscope which feeds yet more fuel in the wrong direction. Aircraft
continues rolling at an accelerating rate. Now you cannot roll an aircraft at
just over its stalling speed: lift disappears and it will just fall out of the
sky, which is what happened.
What had occurred, and was confirmed later by examination of
the wreckage, was a very simple
wiring error - the leads to the gyroscope had been accidentally
connected the wrong way round. What was supposed to be Negative
Feedback had become Positive Feedback!
The only surviving
VJ-101 can today be seen in the Deutsches Museum in Munich.
Communication Theory
Bandwidth
Let us take a normal public telephone as an example of a communication link. The first
thing an engineer would want to do is to find its
"bandwidth", or the range of frequencies it could pass. A quick check would be to
whistle into the microphone starting at a very low frequency
and gradually rising. He would then ask the person at the other
end what had he heard and he would probably say "I didn't
hear anything until you got to 100Hz then it was OK until you got to 3500Hz and
then I couldn't hear anything more". Hz means cycles per second. The
engineer would say the overall Bandwidth of the telephone link was from 100 to
3500Hz or 3400Hz. Which is OK for a telephone as it corresponds to the loudest
part of the human voice when speaking normally.
Bandwidth is the rate at which a signal can be handled, the
rate at which data can be transferred. Indian smoke signals, flag semaphore,
and deaf and dumb language are low bandwidth signals. Morse code is wider
(higher signal rate) but still narrower than speech. A good hi-fi has a signal
bandwidth about 7 times wider than a telephone. And so on up to satellite links
which have signal bandwidth in the hundreds of megaherz.
Human beings can also be studied using the concept of bandwidth. Our ears for instance,
have a bandwidth of from 20 to 18kHz. We can move the lighter
parts of our body in 1/10 sec. which corresponds to a bandwidth
of 3Hz. There are limits to the rate at which we can track a
moving object in three dimensions, react to some stimulus, type,
follow a conversation, talk. Our brains are not infinitely fast, we can only
make a certain number of decisions per second. As is to be expected of a very
complex mechanism, humans have bandwidths which can be varied to optimise
performance under different conditions. Asked a question, we can give a rough
answer quickly, but require more time for a considered answer, when the slow
moving (narrow bandwidth) brain has had time to work on the problem.
Resonance
Put a child on a swing and give a push. The swing will go
back and forth at its "natural" frequency with ever decreasing
amplitude until after a certain number of swings it comes to rest. The swing is
an "oscillator". Another oscillator is a church bell. Neither of
these oscillators goes on for ever after the initial push or strike. This is
because they lose a little of the energy (by friction somewhere)
at each swing until there is none left. The child learns that it is possible to
top up the energy lost by swinging its legs at a special frequency. This
frequency is the "natural" frequency of the swing. The child also
finds that by pushing energetically it is possible to more than compensate for
the energy lost per swing. The amplitude of the swing then builds up until it
is limited by the suspension.
The child's kicking frequency is
said to be "in resonance" with the natural frequency of the swing and
the child has found the "resonant frequency".
You can get the same effect if you go into your bathroom and
whistle up and down the scale. At one frequency the amplitude will suddenly
increase. You have found the resonant frequency of the bathroom. People sing in
bathrooms because it magnifies their voice, makes it sound more
"resonant".
Notice that the bathroom resonant frequency is quite well
defined, you really have to hit the right note. In technical language we say it
has a high "Q" or Magnification Factor. Now hang some towels around
and try again. The bathroom still resonates but it is not so sensitive to one
frequency and also the effect is not so pronounced. It is more "flat
tuned". You have reduced the Q. The wall and ceiling of concert halls are
covered with sound absorbing material to avoid resonances, to reduce Q, to make
the hall flat tuned so that no note is falsely emphasised.
Filters
Another way of looking at the bathroom in the example above
is to think of it as something which is particularly sensitive to one frequency
over all others. Put in (sing) a lot of frequencies and it would filter out
just one - its resonant frequency. It is acting like a filter.
Filters are much used in communications.
In electronics, a filter is a box fitted with an input and
an output connector. Inside the box are one or a number of circuits which act
like oscillators. They usually have different resonant frequencies and
different Qs. Printed o the side of the box is what it does. If you put a
"440Hz pass filter" between your record player and the loudspeaker
and then play a record you would only hear when some instrument was playing the
note A above middle C. All the rest would be silence. Now that would be a
fairly "narrow" "pass"
(high Q) filter. It is possible to build a filter, which would pass all
the notes from the G below A to the B above A ie. from 392 to 494Hz. This
filter would have a rather "wider" "pass-band" (lower Q)
than the first filter.
It is also possible to build "reject" filters
which do the opposite of the above eg. pass everything except the frequencies
from 392 to 494Hz.
We can construct "reject" filters in our head
which push familiar sounds into the background and allow us to concentrate on
new, interesting information. In a quiet room you soon forget the ticking of
the grandfather clock - until it stops. Our brains must be full of constructs
which are similar to "reject" filters. When you consider the flood of
data arriving every second through our senses, we must filter out most of it.
All the filters so far are "fixed" tuned. There
are "tuneable" filters, and not only tuneable but there are
"variable band-width" (or variable Q) filters.
There are filters which can be shaped so they give maximum
output from some complex sound (like a clarinet playing middle C). Listen to an
orchestra through such a filter and hear the sound of the clarinet boom out
each time it hits middle C. This is called a "matched filter".
Filters could be built to follow a sequence of events eg. to
listen to an orchestra playing Beethoven's 9th and give an output only if
someone is playing "God Save the Queen" on a mouth organ in the
background. We build filters like this in our head very easily. Pass filters
have generally been constructed in our brain to emphasise something that is
important or gives pleasure. We can all hear if our name is mentioned in a
noisy crowded room. Playing an initially difficult piece of music over and over
again will gradually tune a filter to it until you begin to appreciate it. The
same for "dry" wines, which most people don't like the first time.
"Falling in love" must surely be the act of tuning yourself to
"her". The more you finely tune the filter, the greater pleasure you
will receive. But this has its dangers. Finely tuning yourself (to receive more
pleasure) to one of her moods may only be done at the expense of relative
displeasure during her other moods.
When we say "I got used to it" (and it is truly
amazing what humans can get used to) we are really saying that we have
constructed a complex group of reject and pass filters to remove from our
consciousness what we don't like and emphasise what we do. An important measure
of intelligence must surely be the ability to quickly change these filters to
match a new environment.
Noise
Let us return to the engineer testing the telephone. Having
measured the overall bandwidth he would now simply listen to the earpiece with
no one talking and measure the strength of the clicks, crackles, and hissing
noises that occur as background to a telephone conversation. This he would call
the "Noise Level".
Noise can come from many sources - bad connections, nearby
power lines, faulty components, or just noise getting into your ear from an
airplane flying overhead. In real life it can take many forms: a bad photocopy,
sun shining on the TV screen, the shimmering earth's atmosphere that distorts
astronomers' star images, coughing in a concert hall, diesel fumes in a scented
flower garden, a wine that has "gone off". Noise is any disturbing
signal that reduces the quality of the signal you want.
Signal to Noise Ratio
Our engineer, having measured the noise level, would now
ask the person at the other end to
send a constant signal so he could make an estimate of the relative loudness of
the noise and the signal. He would call this "Signal to Noise Ratio"
and it should obviously be as high as possible. As you can see from the noise examples given above,
noise is always with us. The only thing we can do is to make its influence as
small as possible, by having as high a signal to noise ratio as possible. This
can be done in one of two ways: 1.either increase the signal (shout louder at a
noisy cocktail party), read books with large print (if your eyes are failing)
or: 2. reduce noise somehow (only use pure components in your cooking, take
your telescope to a mountain peak so there is less trembling air between you
and the star, draw a curtain in front of the window to stop sun shining on the
TV screen).
Or both.
Use of filters to improve
signal to noise ratio
In a communication link, filters are often used to remove
noise while letting the signal pass through. They improve the signal to noise
ratio. Say we are trying to listen to a weak Morse code signal at 1kHz on a
radio with lots of noise in the background. Our ears have a bandwidth of from
20Hz to 18kHz and so a lot of unwanted and unnecessary noise in this band is
getting through and distracting us. The situation can be greatly improved if we
put a 1kHz pass filter on the output of the radio. This will let the signal
pass through undisturbed but block most of the noise. Any noise at 1kHz will
still get through, of course.
In Life the type of filter used to improve the signal to
noise ratio can take many forms. You can "direct your attention" to
something, as you do when you wait for someone at an airport. You construct a
pass filter in your head which corresponds to his face. All other faces are
filtered out.
More about noise
Noise is funny stuff. It represents the fundamental
randomness of Nature and was first studied in high-gain amplifiers where it
appeared as a hissing sound. (Hence the
word "noise" for all disturbing signals). It is actually produced by
random movements of electrons in the amplifier itself and limits the minimum
signal that can be amplified.
Now imagine we have a collection of narrow pass-band
filters. Number 1 allows through frequencies from 1-2Hz, number 2 from 2-3Hz,
etc. going up to the last one which passes from 10 000 to 10 001Hz. We wire
them all up to the output of a high-gain amplifier. So as not to complicate the
issue at the moment, we will assume the amplifier can amplify all frequencies -
it has an infinite bandwidth. Let's short the input out and turn up the gain.
Out comes the rushing sound of its self-generated internal noise. Now let us
listen to the output of each individual filter. To our surprise the amount of
noise from each filter would be the same.
Sure, it would sound different from each filter but the
actual amount (power) would be the same. From the 1000-1001Hz filter you would
hear a slowly wobbling 1000Hz whistle. From the 100-101Hz filter you would hear
a slowly wobbling hum of around100Hz. For a 100Hz filter to have an output, it
must have a 100Hz input. And the same for all the other filters. Conclusion -
noise contains all frequencies.
Noise is a wide-band signal.
This is a very important fact.
Uses of noise
There are many practical applications of this. We can put
wide-band noise into a communication link and measure what frequencies come
out. This will tell us the bandwidth of the link.
The complex mechanical structure of an airplane can be
looked on as a collection of resonators or filters. One way to test it is to
first fit vibration sensors to all the important parts (wings, control surfaces
etc.) and then fly around in turbulent air. The turbulence acts like a noise
signal input to all these filters. By analyzing the frequencies coming out of
the vibration sensors it is possible to measure the resonant frequency of the
various components and determine how stable the aircraft is.
Noise into a filter
Imagine we have built a complex filter that only passes the
frequencies present in the output of an oboe playing middle C. If we feed noise
into this filter and listen to the output, to your amazement you will hear the
sound of an oboe playing middle C! This is because noise contains all
frequencies and our filter has just selected out those corresponding to an oboe
playing middle C.
This is a very important effect and explains a lot of
mysterious phenomena.
We can build filters in our head. Look at the stars at
night, a badly plastered wall, tea leaves in a cup, the embers of a fire, and
you will see pictures. These pictures are the output of a matched filter you
have just built, filtering out the "cat's head" from all the other
possible images in the noiselike input.
More ways of increasing
signal to noise ratio.
To recapitulate. We can get a message through in a noisy
environment by increasing the signal to noise ratio by:
- increasing the amplitude of the signal,
- decreasing the noise with a noise filter,
- or both.
Redundancy
Alternatively we can use the time element and take longer
over sending the message.
Remember that even in the presence of noise some signal will get through. So another way is to repeat
the message more than once, or send it
in a different way, or parts of it more than once. This is called adding
"Redundancy". The idea is that if part of the message is missed you get it the next time
around or it can be reconstructed from what you did receive.
Perhaps because of the generally noisy level of Life, or the
inattentiveness of the average listener (noise in his head), our speech is on
purpose very redundant. We repeat ourselves, by making little puns and adding
picturesque allusions.
Redundancy is most easily recognized when we have to pay for
it. For example, if each word in a telegram costs $1, it is remarkable how we
are able to shrink an ordinarily long message by squeezing out all the
redundancy. Of course, the other side of the coin is that each word must be
received perfectly or noise free. No errors can be tolerated.
Noise reduction by feedback
This requires a transmitter and receiver at each end of the
communication link. The idea is that the receiver confirms reception of the
message in some way. The Navy use feedback as in:
Captain `Steer 090'
Seaman `Aye, aye Sir. Steer 090'
Captain `Full ahead both'
Seaman `Aye, aye Sir. Full ahead both'.
But we use feedback more subtly. We send out the first part
of our message and then wait for the feedback. The opening is usually routine:
John `Ah, Peter'
Peter `Yes?'
The link has been established and now data starts to flow.
If Peter sends back the "right" signals, gasp of incredulity, grunt
of confirmation etc., John knows the message has been correctly interpreted and
sends along the next item. But if he receives a blank stare, or something like
`I haven't the faintest idea what you're talking about' he obviously needs to
rephrase his first remark, to expand it, add redundancy. On the other hand, an
impatient `Yes, yes' would cause John to strip out a lot of the redundancy he
was planning to add to the next part of the message.
Feedback is very necessary as you will soon find if you try
to talk to someone wearing a funny mask at a noisy party. You will soon feel
you are talking to yourself and walk off in disgust.
Lack of feedback is also the reason given by some of my friends for not leaving a message
on my telephone answering machine. When the time comes for
them to speak their message they cannot talk to a "dead
microphone".
Channel capacity
Let us now can take another look at the famous formula which
Claude Shannon of the Bell Telephone Company discovered around 1948, and which
I quoted in the introductory essay "A Fresh Look". I show it again:
C = W log 2 (1+S/N)
where:
C = the channel
capacity in “bits” per second
W = channel bandwidth
in Herz (or cycles per second)
S/N = signal to noise ratio
The "bit"
You are driving down a road and come to a fork. Go left or
right? Your passenger, who has a street-plan on his knees says
"right". You have received one "bit" of data. A bit is the
smallest "piece" of data that exists. You had a choice of two roads
and no knowledge of which to take. One bit of data was all you needed to remove
the uncertainty of which direction to take.
Information vs. data
"The receipt of information in the form of a message
implies uncertainty in the mind of the recipient before the message arrives. A
measure of the information content of a message can thus be based on the amount
of uncertainty it has removed".
Note that "information" is "news", it is
data you didn't have. If your passenger is grumbling about the weather
conditions, you are receiving lots of data but very little information - you
can see the weather is bad yourself and there isn't much uncertainty about the
reason for your passenger's annoyance; you were both going to play tennis.
Information is therefore also measured in bits. If I have to
go through a town and there are ten left/right choices, and each one is a
"toss-up" for me, I am going to need 10 bits of information in order
to navigate the town successfully.
Channel capacity
As you are driving through an unknown town your
navigator/passenger is constantly feeding you "bits" of information,
left, left, right etc., perhaps at the rate of one a second. He is feeding you
information at the rate of 1 bit per second and obviously needs a "channel
capacity" (speech channel from his mouth to your ear) capable of passing
at least 1 bit per second. The Shannon formula can be used to tell you the
maximum noise that can be tolerated for you to hear the message clearly.
Use of the Shannon formula
When the Shannon formula appeared
it was immediately seen as very important for several reasons:
- It showed that there were two factors which determined how
quickly a message could be sent over a given channel – the signal bandwidth and
the signal to noise ratio. This was by no means new, everyone knows this
intuitively once the meaning of the words "bandwidth" and
"signal to noise ratio” have been explained to them (as I hope I have to
you), but it was deeply satisfying to have a theoretical basis for this
knowledge.
- It meant that as it was now possible to measure data, it
was now possible to measure information (information meaning data that removed
uncertainty). As an example - how much information do you need to find a given
name in the London telephone directory? To work this out you ask the person who
knows the name "Is it in the first half?". He will say
"yes" or "no". If he said "yes", it is in the
first half, now you ask him "Is it in the first half of the first
half?" And so on, dividing the last page into top half and bottom half
etc. until finally you ask him if it is the top name or the bottom name. If
there are 8 million names in the book you will need 22 bits of information from
him (22 yes/no answers) in order to find the given name.
As
information could now be seen to be (and measured) as a series of yes/no
questions, it started everyone thinking about information generally and
investigating (for instance) the information handling capacity of a human
being, how much information you received from a news bulletin, how much
information is required to bake a cake, what is the channel capacity of the
nerve connecting the brain to a finger, how many bits per second can the eye
receive etc. etc.
As far as
I know, no one has yet used Information Theory to study Art. Which is what
started me writing these essays, you will remember.