"Higher, directed forms of energy (e.g., mechanical, electric, chemical) are dissipated, that is, progressively converted into the lowest form of energy, i.e., undirected heat movement of molecules; chemical systems tend toward equilibria with maximum entropy; machines wear out owing to friction; in communication channels, information can only be lost by conversion of messages into noise but not vice versa, and so forth." (Ludwig von Bertalanffy, "Robots, Men and Minds", 1967)
"To adapt to a changing environment, the system needs a variety of stable states that is large enough to react to all perturbations but not so large as to make its evolution uncontrollably chaotic. The most adequate states are selected according to their fitness, either directly by the environment, or by subsystems that have adapted to the environment at an earlier stage. Formally, the basic mechanism underlying self-organization is the (often noise-driven) variation which explores different regions in the system’s state space until it enters an attractor. This precludes further variation outside the attractor, and thus restricts the freedom of the system’s components to behave independently. This is equivalent to the increase of coherence, or decrease of statistical entropy, that defines self-organization." (Francis Heylighen, "The Science Of Self-Organization And Adaptivity", 1970)
"The power and beauty of stochastic approximation theory is that it provides simple, easy to implement gain sequences which guarantee convergence without depending (explicitly) on knowledge of the function to be minimized or the noise properties. Unfortunately, convergence is usually extremely slow. This is to be expected, as 'good performance' cannot be expected if no (or very little) knowledge of the nature of the problem is built into the algorithm. In other words, the strength of stochastic approximation (simplicity, little a priori knowledge) is also its weakness."
"In a real experiment the noise present in a signal is usually considered to be the result of the interplay of a large number of degrees of freedom over which one has no control. This type of noise can be reduced by improving the experimental apparatus. But we have seen that another type of noise, which is not removable by any refinement of technique, can be present. This is what we have called the deterministic noise. Despite its intractability it provides us with a way to describe noisy signals by simple mathematical models, making possible a dynamical system approach to the problem of turbulence." (David Ruelle, "Chaotic Evolution and Strange Attractors: The statistical analysis of time series for deterministic nonlinear systems", 1989)
"Black-noise phenomena govern natural and unnatural catastrophes like floods, droughts, bear markets, and various outrageous outages, such as those of electrical power. Because of their black spectra, such disasters often come in clusters." (Manfred R Schroeder, "Fractals, Chaos, Power Laws", 1991)
"What we now call chaos is a time evolution with sensitive dependence on initial condition. The motion on a strange attractor is thus chaotic. One also speaks of deterministic noise when the irregular oscillations that are observed appear noisy, but the mechanism that produces them is deterministic." (David Ruelle, "Chance and Chaos", 1991)
"An essential element of dynamics systems is a positive feedback that self-enhances the initial deviation from the mean. The avalanche is proverbial. Cities grow since they attract more people, and in the universe, a local accumulation of dust may attract more dust, eventually leading to the birth of a star. Earlier or later, self-enhancing processes evoke an antagonistic reaction. A collapsing stock market stimulates the purchase of shares at a low price, thereby stabilizing the market. The increasing noise, dirt, crime and traffic jams may discourage people from moving into a big city." (Hans Meinhardt, "The Algorithmic Beauty of Sea Shells", 1995)
"Chaos can leave statistical footprints that look like noise. This can arise from simple systems that are deterministic and not random. [...] The surprising mathematical fact is that most systems are chaotic. Change the starting value ever so slightly and soon the system wanders off on a new chaotic path no matter how close the starting point of the new path was to the starting point of the old path. Mathematicians call this sensitivity to initial conditions but many scientists just call it the butterfly effect. And what holds in math seems to hold in the real world - more and more systems appear to be chaotic." (Bart Kosko, "Noise", 2006)
"'Chaos' refers to systems that are very sensitive to small changes in their inputs. A minuscule change in a chaotic communication system can flip a 0 to a 1 or vice versa. This is the so-called butterfly effect: Small changes in the input of a chaotic system can produce large changes in the output. Suppose a butterfly flaps its wings in a slightly different way. can change its flight path. The change in flight path can in time change how a swarm of butterflies migrates." (Bart Kosko, "Noise", 2006)
"I wage war on noise every day as part of my work as a scientist and engineer. We try to maximize signal-to-noise ratios. We try to filter noise out of measurements of sounds or images or anything else that conveys information from the world around us. We code the transmission of digital messages with extra 0s and 1s to defeat line noise and burst noise and any other form of interference. We design sophisticated algorithms to track noise and then cancel it in headphones or in a sonogram. Some of us even teach classes on how to defeat this nemesis of the digital age. Such action further conditions our anti-noise reflexes." (Bart Kosko, "Noise", 2006)
"Linear systems do not benefit from noise because the output of a linear system is just a simple scaled version of the input [...] Put noise in a linear system and you get out noise. Sometimes you get out a lot more noise than you put in. This can produce explosive effects in feedback systems that take their own outputs as inputs." (Bart Kosko, "Noise", 2006)
"This phenomenon, common to chaos theory, is also known as sensitive dependence on initial conditions. Just a small change in the initial conditions can drastically change the long-term behavior of a system. Such a small amount of difference in a measurement might be considered experimental noise, background noise, or an inaccuracy of the equipment." (Greg Rae, Chaos Theory: A Brief Introduction, 2006)
"Neural networks are a popular model for learning, in part because of their basic similarity to neural assemblies in the human brain. They capture many useful effects, such as learning from complex data, robustness to noise or damage, and variations in the data set. " (Peter C R Lane, Order Out of Chaos: Order in Neural Networks, 2007)
"Noise is bad for the network, if high and continuous noise levels disturb all network functions. So far, the take-home message is that we have to stop noise in order to survive. This assumption is wrong. Reducing the noise to zero would mean no interaction of the network with the environment. Isolation is clearly a bad strategy, since such an isolated network will die. However, zero noise is bad for another reason too. Noise can be helpful in many ways. The first documented observations of good noise were sailors’ reports on the peculiar phenomenon that disordered raindrops falling on the ocean can calm roughseas. Another example of the optimal level of noise is opinion formation. A low noise is not enough for modulation of opinion formation, while strong fluctuations prevent the formation of a definitive collective opinion." (Péter Csermely, "Weak Links: The Universal Key to the Stabilityof Networks and Complex Systems", 2009)
"Perturbations are often regarded as noise. What is the difference? Noise is usually understood from the point of the experimenter. If we measure it from the outside, noise is the fluctuation of the value we measure. However, from the point of view of the network, noise is a series ofperturbations changing its original status. Network perturbations can be called either signals or noise." (Péter Csermely, "Weak Links: The Universal Key to the Stabilityof Networks and Complex Systems", 2009)
"Self-organizing networks suffer various types of random damage. Therefore, if the network remained static, it would soon become dysfunctional. Some networks have developed highly specific screening systems which recognize and repair random damage. On the one hand, this process requires energy, which arrives in the form of perturbations or noise. On the other hand, noise-triggered network restructuring will repeat a few steps of the original self-organization and therefore constitutes a much cheaper way of providing a continuous repair function, with the additional advantage that it is always adaptive with respect to the actual environment of the network." (Péter Csermely, "Weak Links: The Universal Key to the Stabilityof Networks and Complex Systems", 2009)
"To understand, how noise is related to scale-freeness, we have to do some mathematics again. Noise is usually characterized by a mathematical trick. The seemingly random fluctuation of the signal is regarded as a sum of sinusoidal waves. The components of the million waves giving the final noise structure are characterized by their frequency. To describe noise, we plot the contribution (called spectral density) of the various waves we use to model the noise as a function of their frequency. This transformation is called a Fourier transformation [...]" (Péter Csermely, "Weak Links: The Universal Key to the Stabilityof Networks and Complex Systems", 2009)
"When some systems are stuck in a dangerous impasse, randomness and only randomness can unlock them and set them free. You can see here that absence of randomness equals guaranteed death. The idea of injecting random noise into a system to improve its functioning has been applied across fields. By a mechanism called stochastic resonance, adding random noise to the background makes you hear the sounds (say, music) with more accuracy." (Nassim N Taleb, "Antifragile: Things that gain from disorder", 2012)
"It is evident that chaotic behavior, in the new scientific sense of the term, is very different from random, erratic motion. With the help of strange attractors a distinction can be made between mere randomness, or 'noise', and chaos. Chaotic behavior is deterministic and patterned, and strange attractors allow us to transform the seemingly random data into distinct visible shapes." (Fritjof Capra, "The Systems View of Life: A Unifying Vision", 2014)
"Neural networks can model very complex patterns and decision boundaries in the data and, as such, are very powerful. In fact, they are so powerful that they can even model the noise in the training data, which is something that definitely should be avoided. One way to avoid this overfitting is by using a validation set in a similar way as with decision trees.[...] Another scheme to prevent a neural network from overfitting is weight regularization, whereby the idea is to keep the weights small in absolute sense because otherwise they may be fitting the noise in the data. This is then implemented by adding a weight size term (e.g., Euclidean norm) to the objective function of the neural network." (Bart Baesens, "Analytics in a Big Data World: The Essential Guide to Data Science and Its Applications", 2014)
No comments:
Post a Comment