t h e   p h y s i c s   o f   h a p p e n  i n g . . .

bump on a curve ¸¸¸.·´ ¯ `·.¸¸¸ Notepad for Dummies

(very rough draft) Principles of Natural Systems
Common sense, Key Methods, Physics Theory, Systems Thinking

.  .  .  .  .

the things having internal design and behavior of their own

 ... Closely observing local growth and decay processes reveals their internal 'germs' of organizational development, and the individual complex systems that develop around them, as the atoms of nature.   They're the connecting parts through which larger and smaller scales interact.  They emerge from their environments by development and interact though using their environments as mediums of exchange.  

 ... A related discovery by the larger scientific community is now being made.  The various established scientific disciplines all need a common reference as a requirement for collaboration on sustainability.   They're also finding it necessity for engaging with each other and other stakeholder groups to relate to a physical world composed of diverse  networks of individual systems with significant independent behavior.  That just does not work with representing nature as equations of controlled variables.   It seems no longer a fantasy, the unification of the sciences around a functional understanding of nature, but a watershed advance in sustainability science just emerging, and something of an exciting new world.   When would I guess the inflection point in that learning 'S' curve would come, indicating making it to the 'home stretch'?  Oh gosh, so hard to guess the path of things just begun.  Say, 20 to 50 years maybe? 

 ... That the networks within individual systems are hidden by their organization being internalized, and having different degrees of independent behavior, means a disciplined exploratory process is needed to identify and engage with them.  They turn out to have a variety of features that give them general long range predictability, within their individuality.    It's quite analogous to our own thoughts being hidden from the outside, but finding some human behavior quite reliable.   For hidden systems you wouldn't even know to inquire about them without reason to explore particular gaps in your information.    That happens in studying individual system beginnings and endings (¸¸¸.·´ ¯ `·.¸¸¸) and the particulars of their normal phases of growth and decay.    Beginning and ending are indications of a process of beginning and ending.   An exploration method using physics principles of continuity as a question generating tool helps explore and reveal their design and one's choices for engaging with them.   It also locates what seems to be the real reason mathematics is useful for modeling nature, the usual continuity of organizational development in independent physical processes.  

What's here....

 ... Discovering this particular method for how to study what happens inside emerging systems began with a curiosity about 'eventfulness'.  That led to closely examining the complex organized systems of nature that have continuity that begins and ends.   I watched indoor air current networks, spontaneous evolving climatic energy transfer systems, as they developed from scratch and  migrated from one form to another through each day.   In general an easy place to find individual emergent systems is wherever there are lag times between cause and effect, the time needed for local response systems to independently develop.   Beginning and ending turns out to require nature to invent whole new systems, and you can watch if you ask the right questions.   

 ... Even after study, how systems work may be hard to imagine without attributing magical properties, but careful exploration also uncovers layer upon layer of clearly physical progressions connecting their change over time.   That's the same observation that Darwin made, an accumulating succession of events.  Darwin's guess as to how that works needs updating I think, but the clear evidence of accumulative processes in both evolution and in growth system emergence seem to convincingly expose 'addition' as the central  natural behavior of our amazing world.   As I see it, beginning is the emergence of a learning process, that explores a limited network of opportunities, rather than a deterministic echo of distant events with no independent local behavior.  Any number of different kinds of learning aids (models and things) may be helpful, so long as learning about the physical thing is their common reference.  

 ... Today the key finding appears to be that many natural systems climax their period of 'run-away' growth at a level of maximum freedom and adaptability.  Stabilizing with freedom of movement, as many systems do, is a clear indication ending growth by some means other than meeting maximum environmental constraint.   It had long been assumed that all natural systems climaxed at a point of maximum constraint, controlled by their environments rather than altered inside.  That went along with the long held assumption that natural systems generally originated from remote forces rather than local self-organization.   The apparent  information source that independent systems use to stop growing short of engaging in resource 'wars' with each other at their resource limits is sensing environmental diminishing returns.   Diminishing returns is a true long range forecasting measure indicating the approach of terminal limits for resource using processes, and a warning that multiplying investment in them may lead to systemic failure.

 ... Perhaps it's laziness, liking the rough edges of things, or just being unsure how, but I haven't gotten around to fixing and weeding out and upgrading much of my old and less successful work here.   This site remains a kind of semi-organized accumulation of good & bad, updated & outdated, bits of experiment.   That may be valuable instead of misleading if you see part of the fun of the enormous challenges ahead as taking part in questioning the path we are on.   Going on a trip and having to get out and push when the car runs out of gas is nobody's idea of fun, and so it's often valuable if the job of questioning the road you're on is more widely shared.   Scientists doing fundamental research like nothing more than to question their direction of inquiry, a main attraction of the 'game'.   Curiously all people on earth, actually, are now having to become part of teams doing original fundamental research on the shifting natural systems of our lives.  It's both opportune and important, to share the path finding and questioning game of that discovery process.    Perhaps the scattered remains here of my own free thinking approach will seem hard to take when you're not a participant.  My apologies, then.  Still, I've tried to leave around some indication of the fun I've had as well, and though it may not be adequate, to also explain something of  how I think.   Judge for yourself.     

pfh 11/06 12/27/06 5/5/07 12/29/07 02/03/08 2/17/08

-  One Starting Point  -

8/6/06 11/6/06   I had a productive original question.  Why is nature missing abrupt transitions?   It turns out to be because of the developmental processes that cause change.  Nature isn't an image that can flit from one state to another without any work, and much of our thinking about nature fails to take into account that basic difference between images and reality.    That change takes a process of changing means that complex natural systems develop from relatively hidden local loops of opportunistic events, by a growth process (), and that most anything exhibiting growth (), is an emerging original complex natural system.  

.... The secret is in the data we discard, the information that approximated rules treat as useless, the transitions between the gaps between rules, and managing somehow to make some sense of it....

 It's often very hard to identify what's what in nature.  There are just too many things taking place with overlapping causes and results.   It's a problem.   Coherent individual whole systems do co-exist, overlap and intermingle, like the mingled systems of ecologies, or the untraceable cross fertilization of ideas in a new school of thought, but more than you'd think can be localized and individually identified by their characteristic dynamics of change.   It's displayed in the shapes of their curves of change over time.   They include storms, sparks, our own reflexes and thoughts, social movements, the growth to failure of great plagues and misguided civilizations, cosmic explosions and life.   It's a group of phenomena that includes everything in the macroscopic world that begins and ends.   It's surprising, but I think reasonably simple to demonstrate, that the growth of complex systems is the source of everything that is eventful around and within us.   Do they have any meaning independent of human questions and beliefs?    I'll stretch a lot of words but 'meaning' seems clearly to be an internal property of the minds that create it and the communities that share it, though no doubt it's meaningful to say there's quite a bit of reality we're missing.

Eventually most open natural systems may all be classed as 'living' things, with or without 'minds',  because their organization arises by a local unguided development process and the animation of their behavior is internal.  Life used to be a simple rigid category.   When you develop a continuous scale to replace a rigid category, a lot of relationships become visible that weren't there before.    What becomes obvious in my opinion is that the biggest difference between living and non-living systems is having an original interior design and behavior.   Lots and lots of things fit that description.   What deterministic science has been missing about them is that they're not just unpredictable, they're all also out of control.   We can't find what controls them because they're not controlled, but opportunistic.   Of course, I am bending perfectly good English words a little to accommodate new realities, and it's good to remember that.    Our observation methods have not been picking up the plentiful evidence importantly because they're designed to throw that data away.... 

The data problem also comes from the inherent stripping of connections involved in making any measurement.  It's impossible to measure a loop of relationships, and the emergence of new systems is a process of 'sticky' loops that accumulate new structures opportunistically.   It's more than a little confusing, and science is heavily invested in finding how everything is controlled, so we've ignored the things that are out of control.  Quite interestingly, it's a huge part of the world.     They're arguably the main part of what interests us.   They're not only the part that's whimsical, inventive and spontaneous, delightful and dangerous, but also the complex, independent as well as much of what's reliable, everything that begins and ends, i.e. most everything.   What we've been taught by science are the rules by which some things can be predicted, with the expectation that all aspects of nature would eventually be found to follow rules.    What we're discovering is that it was an illusion.  It's a useful one for some purposes, but the rules of science depend directly on locally opportunistic processes, not the reverse.   This work is about how to trace them, and some of things to expect.    There's much more to it, what would appear to be original new universes of relationships inside each one.   They're definitely there but hard to explore unless you're able to watch them quite closely.   That produces a kind of role reversal for science, since the people able to watch events most closely are the people involved with them.   Scientists can maybe make tools for exposing what's going on within the 'conspiracies' of nature making our lives eventful, but the participants have to do the actual original behavioral science.   We're made from and immersed in them in uncountable ways, and learning to read data for clues to 'what's happening' rather than taking someone else's 'rules for what should be', produces a highly satisfying, starkly different, image of the world.

evidence points to evolving         loops inside,      &     questions of transformation
one evolutionary   model fits them all
emergent systems organized around mediums of free exchange, places with stuff left lying around

My use of he term 'physics' here may seem misplaced to some.  It's not about the rules nature follows as much as the rules that might help us to see where the hidden loops of nature actually flow.   The main one is 'watch them develop'.    Maybe other familiar parts of physics begin to show in the rigor needed to justify treating separate data points as being connected by shapes, to define a new default hypothesis for shape, and  the various interesting structural and behavioral taxonomies.   You'll probably also note that as a method of recognizing 'the other half of the universe', the 'rule connecting' half as contrasted to the 'rule following' one, it rather  incomplete.   It's a discovery that follows, like most of the discoveries of physics, from finding clear evidence of something missing.   If the rules are all separated by gaps, what connects them?

Here's a short list of growth indicators which are likely to be false positives.
1) all measurement devices, light meters for example, have   artifacts at the limits of their sensitivity and while being turned on and off.   Those are not growth systems in the subject event, though it may look like it in a series of measurements of change over time.
2) the measures of many things are filtered through many layers of interceding processes.  If you look for the beginning and ending of light at sunrise and sunset, for example, you'll find growth curves to prove it, if they didn't more directly reflect the interceding influence of the shadow of the earth and the diffraction and filtering of light by the atmosphere.
3) there are various kinds of purely mechanical cascades which don't develop around the evolution of any sort of history dependent system.  The question is whether the events at any stage could have as easily occurred at any other time, without the accumulation of the events preceding them.  A cascade of dominos, for example, may seem to take on a life of its own by branching out and scouring the recesses of a region full of them, without any individual event depending on the accumulation of change or representing any change in kind.
4) when interpreting data, connecting dots with a continuous curve or any rule for one, turns any transition from one steady state to another (beginning and ending for example) into a chain linked by growth and decay curves.   The relation between dots and curves is a huge wonderful subject.
5) there are many varieties of secondary effects that require you to trace back upstream for kinds of complex growth system events they reflect.  The example that came to mind was measuring the rate at which theater goers arrive at the theater.   For every show there will be apparent growth and decay curves in the arrival rate, reflecting the distribution of choices people made in deciding when to leave home, but having no actual influence on each other.  The only remote connection is the sense of excitement about the show, a real community phenomenon with a real but very different growth and decay dynamic.  With a hot show folks will get there earlier because of the expectation of crowds, but that won't have to do with when other people actually arrive.

I think there are probably other good rules of thumb for false positive evidence of emerging complex systems.   Send me one worth posting here and I'll give you $10.

 ( Page History & 9/06 Current View )
. . . the Basics of Steering . . .
. . . the Principal Principle of Cybernetics . . .
Table of Contents...Related Subjects... Introduction,.. Publication
Ongoing Studies by P.F. Henshaw 
     To learn to think like nature, first learn to watch nature think.
    id @ synapse9.com

Always under construction,  some time I'll read the whole thing and fix everything that's out of date!
...NO FRAMES... for those really curious- My World (GET WHIP  )

4/22/00, 4/7/01,11/8/01,1/5/02, 3/9/04, 12/19/04, 7/22/05, 3/11/06 This site is perhaps a small baby step but has three purposes, discussing general theory, case studies, and scientific methods concerning the phenomenon of local growth systems and their uniquely evolving inner worlds of relationships.   The observational method concerns how to put a timeline under a mathematical microscope and identify developmental turning points in locally emergent structures and processes.    The theory concerns the consequent new understanding of nature, establishing useful global principles, and evolving the methods and models of physics.    The case studies bring the technique and theory together and inform both.

The subject area is the rapidly evolving organization of  event processes of all scales and kinds, sparks, ecological shifts, solar storms, earth quakes, heart beats, weather, epidemics, social change, life, etc., for any of these one can read any consistent measure over time, and draw an informative single story line within an epic tale.   You look for growth and decay and their phases of transition, for the loosely connected distribution of parts that act as a whole.    One of the first things you see is a mix of what seem to be continuous flows of discontinuous events, a tentative regularity of complete accidents.   Growth is the transition between discontinuity and continuity in nature, the development of the physical property of organizational continuity, that makes describing nature with mathematical functions useful.   Math behaves a little like nature in that it is meaningful to imagine filling in new points in-between your data points so that each short sequence telegraphs where the whole series is going, the property of flow.  That's the property of functions that was invented with calculus.   On close inspection all continuous physical flows are actually found to be granular, with emergent organization at a larger scale coming into focus as you imagine there might be points in-between.   Nature rarely builds machines like we do, with fixed structures, but assembles working parts ad hock from whatever is currently lying around.   It's quite amazing that it works at all, but seems to work extraordinarily well.

The beginning and end of smooth flows are the periods when emergent organization is being built up and dismantled, and when it is most directly exposed to view.     Time traces are only one dimensional and do not provide very good descriptions of the processes they reflect, but they do provide good markers, exposing when and where locally emergent organization and disorganization is occurring.   The set sequence is a four part series of cascades.    I call them Inflation (compound growth), Integration (climax/stabilization), Disintegration (destabilization) and Decay, the phases of rapid system evolution.   They typically represent  billions of molecules going through a collective process of reorganization without outside influence.   The designs of natural systems are always built locally and are never transferred from elsewhere.     The basic four phases of their development are also the minimum necessary set of simple continuous progressions forming a "bump on a curve" (¸¸¸¸.·´ ¯ `·.¸¸¸¸) .   They're to be found absolutely everywhere.

This study of locally emergent natural systems does not conflict with any empirical finding of traditional physics.   Physics is not about causation, and this is.    It's a matter of looking at the same world using different sets of questions.   Modern physics looks for fixed mathematical relationships between idealized measures, and has found many.    This approach to studying the structures of local happenings accepts that they are individually unique throughout and quite beyond full description.   Some useful information comes from learning how to watch them closely.

It's the perennial dance of nature.   What we find is that she does not actually 'follow' rules, but always makes them fresh, over and over again as she goes.   To observe it you first find a beginning or ending continuity and watch closely as the pieces come together or fall apart.    It's made easier with some technique.(continued)

collaboration: 3/11/06  One normal fact of collaborative research is that each of the disciplines and other communities involved develop and apply their own models, terms and methods for the common subject.  Slowly now I think everyone is recognizing that the 'Rosetta stones' are the cohesive physical systems of nature (when we can find them) that each approach looks at from a different perspective in a different way.  Joining different perspectives on the common complex subject, is highly useful.  Speaking about them in different languages is rather confusing, but we cope.

The basic formula for success is for everyone to be talking about the same physical thing, even if using different languages, i.e. system identification comes 1st.  One option is to accept the lead of the most articulate person in the group.  The hazard there is that he or she is likely to describe the physical system we're looking at as a model residing in his or her own mind...since that's how people think, leaving everyone else out of the loop in discussing it.

One of the ways to assure everyone is looking at different aspects of the same subject is by identifying the physical subject with data, and instead of first using the data for a statistical model, first use it to refer back to the larger whole system beyond the data in the physical world.  Then everyone can link up with the same thing using their own data, perhaps, and different perspectives.  

What data to use for that is perhaps a sorting and culling process, since the evidence of larger systems is in all sorts of things.   The one I've focused on, and developed some potentially good theory and technique for, is tracing system growth phases in time-series data. Complex systems are noticeable because they begin and end (have finite duration) and in those transitions exhibit a continuity of organizational development (growth curves) and then a continuity of systemic responses (homeostasis) in maturity.   The fact that strings of data points can often be treated as differentiable curves reflecting the physical and organizational continuity of a larger system is the trick. It's a physical property with mathematical landmarks that can identify the same physical subject from many perspectives.   If anyone sees ways to make these methods more accessible I'd naturally like to hear it.

Recent discussions, comment & a couple links

2006 Email Archive Blog - I post some letters & visitors can post or comment
Jun-Nov 06  FriAM Santa Fe complexity group list serve
Mar 06 NECSI re: capitalism, from R.Newman article in the Guardian
8/05&1/07 Stan Salthe email threads, seems familiar, easy to mark
95-2000 Sci.Physics news group conversations

Why we're all mostly out of the loop - more on home page
A Letter in the Science Times  on mathematical truth & beauty
Evolving Air Currents   The experimental origin
These Pages... Until recently were called 'Derivative Reconstruction'...
Notes & clippings... Some comments on related issues
Science Resources  My subject and resource bookmarks of 4/00

other explorers - links to interesting but unrelated scientific explorers
great time-series data source - Rob Hindman
 

Publication 

2007 Projecting Images of Complex Systems - research note for SASO-2007, see also PICS.htm for notes and supplemental materials that may accumulate

2007 Flowing processes in a punctuated species change, G. pleisotumida to G. tumida, displaying feedback driven evolution, Abstract & Intro  - draft for journal 12!  ...best so far -

1999 Features of derivative continuity in shape (344k), Abstract & Intro (15k), International Journal of Pattern Recognition and Artificial Intelligence (IJPRAI), for a special issue on invariants in pattern recognition, V13 No 8 1999 1181-1199

2000 (collection w/ above)  in Invariants for Pattern Recognition and Classification, M A Rodriguez Ed. World Scientific series in Machine Perception & Artificial Intelligence V 42., taken from IJPRAI V 13-8 1999 p1181-1189
· Math methods for finding underlying continuity and structural change in complex systems through the noise.

1985-1, Directed Opportunity, Directed Impetus: New tools for investigating autonomous causation, Proceedings, Society for General Systems Research, Louisville KY

1985-2, Unconditional Positive feedback in the economic system, Proceedings, Society for General Systems Research, Louisville KY

1979 An Unhidden Pattern of Events [collection of essays] Foreword (1.7M), title essay An Unhidden Pattern of Events (6.5M)
· The initial general model for understanding complex events.

On Natural Time: ..  (a little dated)

The general idea,... observing change 

What results are a some surprising discoveries, and a greater appreciation of the emergence and layering of  patterns of change in general, causation.  It is probable that what you'll find here will not live up to its promise, at least not until you are able to use it with the kinds of change you are most interested in and most familiar with.  Much of the focus here is also on mathematical methods and problems.  More on how to use the approach might be helpful, but it is given somewhat less attention.     Some of the techniques involve new areas of mathematics (empirical measures of derivative continuity, the paradox of indescribable simple pattern in the beginning and ending of events) and contribute to new methods in computer vision (defining  landmarks of organizational change), as well as providing a new empirical method of inquiry.   The method also involves a new point of view, a super-realism.   Why 'super'-realism?   It involves watching for, and draws you into, the deep inner workings of particular individual events.    There's lot's there.

For example, one might ask why intricate patterns develop in fluid flow, say watching cream in your coffee.  Even if we can see various scales of pattern, can name a cause such as heat or drag, and make some predictions, there's still nothing but the kinetic interaction of molecules to produce any complex pattern.  Somehow the molecules interact to build complex evolving pattern.   There are no other communicating forces, no imposed guides, no embedded memories, only accumulative original pattern development in molecular motion, with every part participating independently.

This is a common trait of nature's confusing but highly organized behaviors, they happen all at once, with unified orchestration but no central point of control.   So much is happening at the same time, and so smoothly going through changes, one has to watch it very very slowly....   A one dimensional trace, say temperatures at just one point, may provide particularly good information on when and at what rates organizational transitions develop.   That may point to exactly when, where and what kind of thing to look for to find the mechanism that develops to do it.   The same is true for tracing the inner workings of your business market, looking for particular kinds of turning points and then to look more closely for what it is and where it might go.

Measures of change over time directly reflect, but poor describe, the complex accumulations of events involved in any process of change.  They're not the answer, they're a guide you can use for looking beyond. The common indicator of  local organizational change in inner workings of a system is a growth curve, reflecting far more complex events, but accurately displaying their timing.   These are typically represented by the 'S' curves, the connecting shapes that are found in virtually every transition.   Their shape draws a smooth curve between the static measures as the underlying organizational processes develops a smooth changes between systems of behavior.   After identifying a growth curve a few simple checks usually identify what process it is reflecting and at least hint at some of its distributed system of interactions.

Beyond the effort to test scientific ideas, what drives the inquiry is the wonderful depth of unique detail that can be found in individual things and events of all kinds.    Nature is indescribably deep.   Individual events  of all sorts display a fantastic variety of local organization at many scales.    When you look at them in detail, it is obvious why any mathematical description needs to include uncertainty, there's simply too much there, too much going on to describe.   Patterns that can't be understood are called random, and in our data much of the intricacy of nature appears random.  Data is such a very poor story telling device.   The solution is not to abandon data, but to use it differently, i.e. not as the description of what's happening but as the map or guide for your own exploration of a territory of understanding the map can't directly provide.

ed Jan 99
The particular mathematical aspect of natural complexity focused on here is the continuity, or flow, of change.   Much of the work concerns a disciplined analytical tool, derivative reconstruction (DR).  It's not magic, but a tool which allows you to discover much more information about the history and progression of change than you would first expect to be available.   It's an extremely general technique, applicable to any subject of change over time whatever.  Change in anything usually progresses in a self consistent manner, developmentally, displaying the presence and accumulative modification of coherent systems of relationships.   By very sensitively representing smooth underlying flows of change DR helps make the details of these patterns more visible.

The property of infinitely smooth progression in mathematics is called 'derivative continuity'.   It is one of the most fundamental and useful properties of mathematical functions.  Physical measures of change often display a similar physical property, the organizational continuity of nature, call it flow.  DR uses the mathematical definition of the derivative in reverse to represent dynamic organizational change reflected in measures of physical systems .  It doesn't always work, but often does.  When it does it often exposes otherwise invisible natural structures and the links between them.

Recent investigation indicates that there is a distinct similarity between the primitive elements of DR and those being rapidly explored and developed in the fields of computer vision, artificial intelligence and parallel distributed processing. In the terms of those fields DR is a form of curve generalization, that treats randomly sampled data as compressed information about multi-scale continuities.

others.   Back to contents

............

Reconstructing the flow of individual events

Sampling a smooth, but complex flow of change may produce a pattern of points that appears randomly scattered. If you have no information about the underlying process then you can treat the data as form of compressed information about a flow. You might have some other reason to expect an underlying continuity or simply want to see if recognizable forms appear when it is looked at as if it had continuity. Using the implied progression of the derivatives to reconstruct some details of what went on in-between the data points is then a method of data decompression. When it works, it displays previously unseen fine dynamic structures directly, without requiring any preconceived theory of behavior. It also produces a new kind of mathematical entity called a 'proportional walk', having the mathematical property of derivative continuity, but no equation, composed instead of a finite sequence of points and a parsimonious rule for how to connect them with continuous derivatives.

Throughout all fields of scientific research there is a tendency for investigators to see whatever it is they are looking for. In a way that is both more and less of a hazard with derivative reconstruction. The curves produced are more strongly data driven than perhaps any other method, so the likelihood that they represent some real feature of the data is quite high. The problem lies in interpretation, generally requiring a thorough immersion in the detail of the subject, to know what the data represents and what assumptions about the different features of its shape can be made in the process.

There is also a philosophical problem. Reconstructing and studying the individual dynamic flows of change raises a question concerning whether to view nature with either a deterministic or probabilistic model of events. Neither seems appropriate. Individual processes are not probabilistic, since that paradigm applies only to classes of events. Flows of individual events are also not even remotely deterministic. You rarely find any fixed patterns. At the beginning and end of any apparently deterministic pattern (because they all begin and end somewhere) there is an indeterminate change in pattern. To make a possibly long story short, the inevitable conclusion is that individual events need to be seen as something else, call it 'opportunistic', displaying processes of discovery rather than of following rules. How individual events actually come to have the unique and intricate structures they display is clearly not by humanistic volition and intention etc., and it is not that sense of their being 'opportunistic' that is meant. It is only that new behavior begins to develop when, and only when, the appropriate circumstances arise, and then develops in a manner displaying continuity. A theoretical study shows that this implies that change begins with a very specific and recognizable mathematical shape.

The computation routines used (see Tools & StatTechnique) were developed in AutoLISP, the programming language of AutoCAD, and run on a PC. When proper care is taken the immediate usefulness is that the method tends to uncover evidence of surprising patterns of change. In long studied data these newly appearing patterns are often different from what others have found. For example DR offers a direct means of identifying dynamic coupling between complex systems, without having any behavioral theory for either one. (warming3.gif, econcync.gif). Sometimes this brings into question strongly held notions about very well studied processes.

(to Applications ) back to contents

Theory top

What one starts with is a series of points, with no necessary relationship between them. One needs to ask how to determine when a series of points can be treated as a sampling of points from a curve having derivative continuity. That can be done by simple inspection, knowing the specific process the data is from, using statistical measures, or by other means of determining that the data most likely represents some derivative continuities.

Another question is how you determine whether the dynamic structures you find in a data set are real. A few hundred years of experience with, and prospering from, the modeling of natural processes with mathematical functions suggests that derivative continuity (the central structural property of functions) is an useful enduring trait of comprehensible natural processes. The scientific method, though, applies to idealized classes of events, like all bouncing balls or all waves, etc., and we think of the mathematical models of these idealized classes of events as the 'real' structures of nature. It opens uncharted territory to investigate the unique mathematical structure of individual events, like a single bounce of a single ball or a single ripple rather than class of all waves. This is especially true when, as is often the case, there is no acceptable model of the individual or class of behaviors being studied at all.

back to contents

Tools

Commonly some noise suppression is used first, with the one hard and fast rule being to use the very least amount needed.    Validation of the process, like any other scientific method, is whether it's useful and everything knits together in the end.   For very noisy data there are good mathematical tests for the degree of smoothing that will reveal the continuities without erasing them.   The full methodology is not published as I've invested most of my time in applications, and trying to explain why anyone would want to closely study the real dynamics displayed by independent natural systems...  Contact for assistance: id @ synapes9.com

Scales of larger and smaller fluctuation are the most common finding.   Estimating the inflection points and growth rates for the larger scale fluctuating processes is usually the interest.   That can be greatly improved by representing larger scale fluctuating processes as curves drawn through the centers of the smaller scale fluctuation.   That can be done by connecting the inflection points (TLIN) of the smaller fluctuations, called 'integral interpolation', or finding the 'dynamic mean'.    The fluctuations in a process are usually symmetric and represent elastic variation in the underlying behavior.   Sometimes the variation is one sided and the maxima and minima of the fluctuations are used to represent the norms of the underlying process.  The effect is quite often to display surprising structural features of the real underlying processes that would not have been visible by any other kind of representation.

The routines available are written in AutoLISP (one of the programming languages of AutoCAD). This platform was chosen for the purpose because it allows curves with any number and spacing of points to be related in the same 'table', and is fully programmable. For conventional statistical analysis the data is transferred to a statistical package, such as JMP. Suggestions of alternate development platforms would be quite welcome. Current updates available on request.

Mar 97 reverse test - How it really works, a complete demonstration of the DR method.

Sep. 95 drtools.pdf -  description of the AutoLISP analytical command library

Apr 97 Curve.zip - a basic collection of AutoLISP DR command functions for AutoCad 13 or earlier

1/2/08 notes, back to contents

Applications
 - various examples of successes and failures -

Gamma Ray bursts

April 98 BATSE Trigger 551
May 99 The classic "backwards wave" shape of GRB's, perhaps indicating implosion
NASA Introduction to Gamma Ray Bursts
Information on NASA Compton Gamma Ray Observatory

back to contents

Plate Tectonic Flow...

Mar. 97 PBL1 , TIBB , Match

back to contents

Ice Core CO2

The most interesting feature of the ancient record is the amazingly smooth progression of the data, seen close up in fig. 2. The underlying process appears perfectly regular and does not appear to have any fluctuation more frequent on average than 3,000 years. That is extraordinary log term regularity! Perhaps it is driven by something like magma upwelling, or a glacial solar cycle, or there is some diffusion process within the ice which perfectly suppresses the shorter fluctuations without suppressing the larger ones over time. The 100,000 year decline might reflect the general rate at which carbon is withdrawn from the biosphere. The dramatic rise that began just 20,000 years ago might just possibly represent the extensive use of fire.

The study provides an interesting demonstration of the ability of DR to highlight, and repair, a bothersome calibration irregularity found in the original data. The second derivative of the data is still remarkably smooth, except for what appear to be occasional single point spikes due to calibration errors in data collection. The analysis of underlying behavior is that what appears as a general trend of decline interrupted by events of increase may actually be the reverse. This is shown by the second derivative of the trend of 12,500 year fluctuation showing a positive norm punctuated by transient negative periods.

Jan. 96, Sept 97, iceco2-1.gif , iceco2-2.gif , iceco2-3.gif

back to contents

Plankton Evolution

Bjorn. Malmgren gathered data on the size of a plankton species over 7 million years from an Indian ocean deep sea sediment core covering the transition from one plankton species to another (G. plesiotumidia to G. tumidia) during which the organism tripled in size. The data yields intriguing evidence of long term regular dynamic processes in evolution, an apparent growth process as evidenced by distinct periods of acceleration and deceleration in continuous transition between two steady states. There are other interpretations to consider, but the appearance is of mutation having proceeded by a feedback regulated process.

The validity of applying continuity analysis to evolution comes in part from finding a strong linear relationship between the mean sizes of each sample and their standard deviations.  That variation in size is correlated with size is not surprising perhaps, but it does contradict the commonly held 'null hypothesis', assuming that the data was produce by what is called a 'random-walk'.  The variance is neither invariant nor constantly increasing so the random walk assumption for individual lineages or the whole population does not fit.  A second, and more specific contradiction of random walk and demonstration of underlying continuity in the data is provided by the step variance test.    It shows that the variation between widely and narrowly spaced points does not differ like it would for random walks   In a random walk the variances would increase as a multiple of the point spacing.  For the variances of the data do not.  This indicates that the variation is noise, and that trends that are visible through the noise were the result of continuous processes.

Because the standard deviations are related to plankton size by a linear relation both can be used as indicators of the shape of the transitional events.   The values of the standard deviations were rescaled by that linear relation so that its shapes would be properly scaled for comparison.   Both the size and std. deviation curves were derivative smoothed and interpolated and both show an appearance of a similar progression.
 
 

The first derivatives of the two curves make the point much more clear.   They are remarkably similar despite seeming larger differences in the shapes of the curves themselves, and both display the presence of a dramatic transient underlying process.
 

• A slightly different approach to resolving the true shape underlying the noise in the data is provided by studying how widely the various shaped are distributed, using the curvature scale space approach.
• The detailed curves and steps are shown more clearly in the full figures.
• A curious appearance of clustering in the data has been given a very brief and incomplete study.

Global Warming -1

The average global surface temperature over the past 100 years has tended continually upward, but with dramatic fluctuation, and is currently rising at the highest average rate of the period. There is, however no visible evidence of dynamic linkage between it and the equally dynamic events of economic activity. Economic activity is related to energy use and fossil fuel consumption, and the contribution of CO2 and other pollutants to the atmosphere. The fact that the underlying dynamics of the two systems don't match, however, doesn't necessarily mean that atmospheric CO2 doesn't influence global surface temperature. It just means that in the past something else has had a larger effect. Having now found data (below) that does clearly demonstrate the dynamic link between warming and CO2, the question is what is the larger effect, and how long will it dominate.

(a 1996 mistake) I have speculated that it is some evolutionary process of energy transfer mechanisms, the kind that sometimes produce ice ages. Recent indications strongly implicate the solar magnetic cycle. Whatever the source the likelihood is that the dominant influence is cyclic, that we are at it's peak, and it will produce a period of cooling for the next 30-50 years. This would mask the warming effect of CO2, removing the urgency for reducing CO2 emissions, until the real crisis of global warming begins on the next cycle of the dominant influence.

(a 2006 comment) If you look at the old curves is does look like the long wave cycle of fluctuation should have produced a period of atmospheric cooling, or a pause in warming, to confuse us while CO2 continued to increase.   It doesn't seem to have.    ed. 2/4/06

Jan 96 warming1.gif , warming2.gif , warming3.gif

back to contents

Global Warming - 2

[I had thought that this little piece deserved credit for being the first definite human fingerprints on global warming in July 96...but it didn't get notice.   The real prize and credit go to Michael Mann, Ray Bradley (u.mass) and Malcolm Hughes (u.ariz).  Their 3/15/99 paper in Geophysical Research Letters shows a detailed 1000 year temperature record (based on a broad group of climate indicators) which quite abruptly changes direction in ~1900.][link to earlier 4 century curve] [link to Mann's 1999 study]

The study linked above displays an appearance of dynamic coupling between earth temperature and atmospheric CO2  in the alignment of first derivative turning points for recent temperature and CO2 measurements. The Hansen temperature index is a global aggregate of temperature measurements. The NOAA trends in CO2 used comes from measurements made at the top of Mauna Loa. The close correlation between changes in direction of increasing CO2 and changes in direction of earth temperature appears to be evidence of there being a connection (unspecified). The fact that during this period earth temperature tends to decline when the rate of CO2 increase slackens, indicates that there other factors were dominant at the time.   There appears to be a link, though, and atmospheric CO2 is increasing radically (see comparison with ice core CO2 data).  ed 2/4/06

back to contents

History of Economic Growth

The text file discusses principally statistical methodology involved, excerpted from "Reconstructing the Physical Continuity of Events".

Reconstructing the path of GNP (not all figures are available at this time)

Aug. 95 GNP08.gif , GNP10.gif , GNP12.gif , GNP13.gif

back to contents

Dynamic Synchrony between Economic Measures

Aug. 95 econcync.gif , Charts

Aug. 95 econcync.htm , An old methods reference

back to contents

Childhood Epidemics

2006 comment - I keep some of my more obviously problematic methods and interpretations around to keep me and others honest and and on their toes.   Nothing on the site is predigested for unthinking acceptance.   I tried constructing this cool statistical measure, I called 'DAR', to help identify when strings of dots were likely to represent a curve.   Great intent, but this version as described on my statistical methods page, is a mess.    My ~2000 step variance test that successfully disproves random walk in time series where variation is symmetric is my first really useful step in that direction, and I have some other ideas, but haven't gotten any further.   ed 2/4/06

Apr. 96 NYmeasl.gif , NYchick.gif , NYmumps.gif

back to contents

Crime

Sep. 99  Patterns in Crime a first look in a new area

In 2005 I finally got my hands on good data on what happened to the 60's inner city crime wave.   There's much to say.  A few of my notes and charts are at Crimewave's Collapse.

back to contents

Sparks

Nov. 99  Sparks; a first look in a new area.  I'd love it if someone would tell me where I can find detailed data on the growth phenomena of electrical discharges, sparks, lightening etc.

back to contents

top of file