"t h e   p h y s i c s   o f   h a p p e n  i n g" . . .
(a method of empirical cybernetics for understanding individual complex systems)

Curve studies - Method in brief  Applications

.  .  .  .  .

The development and decay of conserved processes of change,
marking where to look for the "seeds" and "explosions" of emerging change
and raising key questions
about the feedback networks of system organization developing,
and exposing them as exploring environmental pathways.
Leading to Foresight about what they'll run into and how they will themselves be changed by it.
 A view not available from hindsight, theories or models since
emergent systems are complex local processes for the environments in which they occur.  
What you gain
by watching for both the expected and unexpected turns in organizational development
key answers and an individual mental model of each individual process as a whole
filling in the chapters of the whole story of its turns in cybernetic structure. 06/08 05/09

( i.e. a very valuable adjunct for guiding the use of virtually any kind of behavioral model )
 

Note: This section of my web is mostly about the technical methods.  See Main Site for other things.   There seems to be some lag between my learning how to describe things and updating them old attempts, judge for yourself & ask questions

 ... Closely observing local growth and decay processes helps reveal their internal network 'germs' of organization as the 'atoms' of nature.    What appears to us to be responsible for growth and decay are internalized network cells, or 'circles', of relationships.    The trick is to shape your picture of them to the physical thing, rather than try to find the picture of them that 'is' the physical thing.   That doesn't work.   The physical thing is much too complicated to picture in more than one way.  The main way, of course, is that what you can picture a little is the behavior and what one can hardly see at all is the opportunities it is exploring in its environment.   The most reliable part you can picture seems to be the continuities that signal it's developing stability or change.

 ... Individual complex systems are the connecting parts through which larger and smaller scales of behavior interact.   It's how, when our molecular fields interact in touch, the message gets passed along from one complex of parts acting as a whole to another, from field to particle to molecule to membrane to cell to tissue to organ and bone to brain and back, that we then recognize as the warmth of a familiar hand and its grasp.   These accumulations of events along almost inconceivable scales of organization emerge from their environments by starting small.   They all develop with a two step, starting and completing, with continuity in-between.    When completed cells of organization interact it's by using their environments as mediums of exchange, where they deposit what they're done with and pick up what they work with.   Their form and pathways develop as they explore using variation at the fringe of their designs as their 'high speed' evolutionary design method.

 ... A related discovery by the larger scientific community is now being made.    We need to intervene to save the earth, and the disciplines all use different kinds of models and they all omit nature's individual path making systems, each using their own rules for what averaged measurements of them did in the past...!   The scientific disciplines all need to refer to the same physical systems as a requirement for collaborating for the sustainability of the earth.  That means learning to point to physical systems as physical things themselves rather only our explanations for them.   Yes, weathermen refer to individual storms, and ecologists to outbreaks of invasive species, but that does appear to help people see the consequences of using niche opportunities as unlimited resources until they have to throw up their hands after tragically colliding with other things.    

 ... An applied complex systems theory needs to become a real theory of discovery with our changing physical systems as the common subject.  It's also a necessity for engaging with other stakeholder groups to relate to the physical world of individual system networks with significantly independent local development behavior.   It seems no longer a fantasy, the unification of the sciences needs to develop around a functional understanding of the individual complex systems of nature, offering something of an exciting new world.   Given that all disciplinary learning processes, being emergent systems themselves, when would I guess the mid-point in that learning 'S' curve would come?   In my system that would indicate making it to the 'home stretch' and heading to a high plateau of refinement.    Oh gosh, so hard to guess the path of things just begun.  Say, 20 to 50 or 100 years maybe?    Maybe it depends on things like tiring of looking for good excuses and discovering the thrill of finding a whole new way of connecting with life by learning to look through our mistakes.

 ... The networks within individual systems are hidden by their very organization, being built around closed loops. They also have various degrees of independent behavior.   That means a disciplined exploratory process is needed to identify and engage with them.   They turn out to go through a variety of very predictable changes that give them general long range predictability that can be made useful as a guide to what to be watching for.  The cycle of independent processes is that they begin and end beginning with multiplying to then either stabilize or destabilize.   For hidden systems you wouldn't even know to look without some reason to explore.    You watch the directions of change, how they're changing, what process is doing that and how that process is making a path in its environment, and what that path will run into.   Reasons to explore are  beginnings and endings (¸¸¸.·´ ¯ `·.¸¸¸), and the expanding and shrinking network cells within them.    An exploration method using physics principles of continuity as a question generating tool helps explore where their designs change and so what to look for.   As you find how they work it gives you choices for how to engage with them.   It also locates what seems to be the real reason mathematics is useful for modeling nature, the continuity of organizational development that individual developmental processes seem to require. ed 11/07  5/23/08

0 1 2 3  4 56

What's here....

 ... Discovering this particular method for how to study what happens inside emerging systems began with a curiosity about 'eventfulness'.  That led to closely examining the complex organized systems of nature that have continuity that begins and ends.   I watched indoor air current networks, spontaneous evolving climatic energy transfer systems, as they developed from scratch and  migrated from one form to another through each day.   In general an easy place to find individual emergent systems is wherever there are lag times between cause and effect, the time needed for local response systems to independently develop.   Beginning and ending turns out to require nature to invent whole new systems, and you can watch if you ask the right questions.   

 ... Even after study, how systems work may be hard to imagine without attributing magical properties, but careful exploration also uncovers layer upon layer of clearly physical progressions connecting their change over time.   That's the same observation that Darwin made, an accumulating succession of events.  Darwin's guess as to how that works needs updating I think, but the clear evidence of accumulative processes in both evolution and in growth system emergence seem to convincingly expose 'addition' as the central  natural behavior of our amazing world.   As I see it, beginning is the emergence of a learning process, that explores a limited network of opportunities, rather than a deterministic echo of distant events with no independent local behavior.  Any number of different kinds of learning aids (models and things) may be helpful, so long as learning about the physical thing is their common reference.  

 ... Today the key finding appears to be that many natural systems climax their period of 'run-away' growth at a level of maximum freedom and adaptability.  Stabilizing with freedom of movement, as many systems do, is a clear indication ending growth by some means other than meeting maximum environmental constraint.   It had long been assumed that all natural systems climaxed at a point of maximum constraint, controlled by their environments rather than altered inside.  That went along with the long held assumption that natural systems generally originated from remote forces rather than local self-organization.   The apparent  information source that independent systems use to stop growing short of engaging in resource 'wars' with each other at their resource limits is sensing environmental diminishing returns.   Diminishing returns is a true long range forecasting measure indicating the approach of terminal limits for resource using processes, and a warning that multiplying investment in them may lead to systemic failure.

 ... Perhaps it's laziness, liking the rough edges of things, or just being unsure how, but I haven't gotten around to fixing and weeding out and upgrading much of my old and less successful work here.   This site remains a kind of semi-organized accumulation of good & bad, updated & outdated, bits of experiment.   That may be valuable instead of misleading if you see part of the fun of the enormous challenges ahead as taking part in questioning the path we are on.   Going on a trip and having to get out and push when the car runs out of gas is nobody's idea of fun, and so it's often valuable if the job of questioning the road you're on is more widely shared.   Scientists doing fundamental research like nothing more than to question their direction of inquiry, a main attraction of the 'game'.   Curiously all people on earth, actually, are now having to become part of teams doing original fundamental research on the shifting natural systems of our lives.  It's both opportune and important, to share the path finding and questioning game of that discovery process.    Perhaps the scattered remains here of my own free thinking approach will seem hard to take when you're not a participant.  My apologies, then.  Still, I've tried to leave around some indication of the fun I've had as well, and though it may not be adequate, to also explain something of  how I think.   Judge for yourself.     

pfh 11/06 12/27/06 5/5/07 12/29/07 02/03/08 2/17/08

-  One Starting Point  -

8/6/06 11/6/06   I had a productive original question.  Why is nature missing abrupt transitions?   It turns out to be because of the developmental processes that cause change.  Nature isn't an image that can flit from one state to another without any work, and much of our thinking about nature fails to take into account that basic difference between images and reality.    That change takes a process of changing means that complex natural systems develop from relatively hidden local loops of opportunistic events, by a growth process (), and that most anything exhibiting growth (), is an emerging original complex natural system.  

.... The secret is in the data we discard, the information that approximated rules treat as useless, the transitions between the gaps between rules, and managing somehow to make some sense of it....

 It's often very hard to identify what's what in nature.  There are just too many things taking place with overlapping causes and results.   It's a problem.   Coherent individual whole systems do co-exist, overlap and intermingle, like the mingled systems of ecologies, or the untraceable cross fertilization of ideas in a new school of thought, but more than you'd think can be localized and individually identified by their characteristic dynamics of change.   It's displayed in the shapes of their curves of change over time.   They include storms, sparks, our own reflexes and thoughts, social movements, the growth to failure of great plagues and misguided civilizations, cosmic explosions and life.   It's a group of phenomena that includes everything in the macroscopic world that begins and ends.   It's surprising, but I think reasonably simple to demonstrate, that the growth of complex systems is the source of everything that is eventful around and within us.   Do they have any meaning independent of human questions and beliefs?    I'll stretch a lot of words but 'meaning' seems clearly to be an internal property of the minds that create it and the communities that share it, though no doubt it's meaningful to say there's quite a bit of reality we're missing.

Eventually most open natural systems may all be classed as 'living' things, with or without 'minds',  because their organization arises by a local unguided development process and the animation of their behavior is internal.  Life used to be a simple rigid category.   When you develop a continuous scale to replace a rigid category, a lot of relationships become visible that weren't there before.    What becomes obvious in my opinion is that the biggest difference between living and non-living systems is having an original interior design and behavior.   Lots and lots of things fit that description.   What deterministic science has been missing about them is that they're not just unpredictable, they're all also out of control.   We can't find what controls them because they're not controlled, but opportunistic.   Of course, I am bending perfectly good English words a little to accommodate new realities, and it's good to remember that.    Our observation methods have not been picking up the plentiful evidence importantly because they're designed to throw that data away.... 

The data problem also comes from the inherent stripping of connections involved in making any measurement.  It's impossible to measure a loop of relationships, and the emergence of new systems is a process of 'sticky' loops that accumulate new structures opportunistically.   It's more than a little confusing, and science is heavily invested in finding how everything is controlled, so we've ignored the things that are out of control.  Quite interestingly, it's a huge part of the world.     They're arguably the main part of what interests us.   They're not only the part that's whimsical, inventive and spontaneous, delightful and dangerous, but also the complex, independent as well as much of what's reliable, everything that begins and ends, i.e. most everything.   What we've been taught by science are the rules by which some things can be predicted, with the expectation that all aspects of nature would eventually be found to follow rules.    What we're discovering is that it was an illusion.  It's a useful one for some purposes, but the rules of science depend directly on locally opportunistic processes, not the reverse.   This work is about how to trace them, and some of things to expect.    There's much more to it, what would appear to be original new universes of relationships inside each one.   They're definitely there but hard to explore unless you're able to watch them quite closely.   That produces a kind of role reversal for science, since the people able to watch events most closely are the people involved with them.   Scientists can maybe make tools for exposing what's going on within the 'conspiracies' of nature making our lives eventful, but the participants have to do the actual original behavioral science.   We're made from and immersed in them in uncountable ways, and learning to read data for clues to 'what's happening' rather than taking someone else's 'rules for what should be', produces a highly satisfying, starkly different, image of the world.

evidence points to evolving         loops inside,      &     questions of transformation
one evolutionary   model fits them all
emergent systems organized around mediums of free exchange, places with stuff left lying around

My use of he term 'physics' here may seem misplaced to some.  It's not about the rules nature follows as much as the rules that might help us to see where the hidden loops of nature actually flow.   The main one is 'watch them develop'.    Maybe other familiar parts of physics begin to show in the rigor needed to justify treating separate data points as being connected by shapes, to define a new default hypothesis for shape, and  the various interesting structural and behavioral taxonomies.   You'll probably also note that as a method of recognizing 'the other half of the universe', the 'rule connecting' half as contrasted to the 'rule following' one, it rather  incomplete.   It's a discovery that follows, like most of the discoveries of physics, from finding clear evidence of something missing.   If the rules are all separated by gaps, what connects them?

Here's a short list of growth indicators which are likely to be false positives.
1) all measurement devices, light meters for example, have   artifacts at the limits of their sensitivity and while being turned on and off.   Those are not growth systems in the subject event, though it may look like it in a series of measurements of change over time.
2) the measures of many things are filtered through many layers of interceding processes.  If you look for the beginning and ending of light at sunrise and sunset, for example, you'll find growth curves to prove it, if they didn't more directly reflect the interceding influence of the shadow of the earth and the diffraction and filtering of light by the atmosphere.
3) there are various kinds of purely mechanical cascades which don't develop around the evolution of any sort of history dependent system.  The question is whether the events at any stage could have as easily occurred at any other time, without the accumulation of the events preceding them.  A cascade of dominos, for example, may seem to take on a life of its own by branching out and scouring the recesses of a region full of them, without any individual event depending on the accumulation of change or representing any change in kind.
4) when interpreting data, connecting dots with a continuous curve or any rule for one, turns any transition from one steady state to another (beginning and ending for example) into a chain linked by growth and decay curves.   The relation between dots and curves is a huge wonderful subject.
5) there are many varieties of secondary effects that require you to trace back upstream for kinds of complex growth system events they reflect.  The example that came to mind was measuring the rate at which theater goers arrive at the theater.   For every show there will be apparent growth and decay curves in the arrival rate, reflecting the distribution of choices people made in deciding when to leave home, but having no actual influence on each other.  The only remote connection is the sense of excitement about the show, a real community phenomenon with a real but very different growth and decay dynamic.  With a hot show folks will get there earlier because of the expectation of crowds, but that won't have to do with when other people actually arrive.

I think there are probably other good rules of thumb for false positive evidence of emerging complex systems.   Send me one worth posting here and I'll give you $10.

Table of Contents...Related Subjects... Introduction,..
Ongoing Studies by P.F. Henshaw 
     To learn to think like nature, first learn to watch nature think.
Always under construction,  some time I'll read the whole thing and fix everything that's out of date!

4/22/00, 4/7/01,11/8/01,1/5/02, 3/9/04, 12/19/04, 7/22/05, 3/11/06 This site is perhaps a small baby step but has three purposes, discussing general theory, case studies, and scientific methods concerning the phenomenon of local growth systems and their uniquely evolving inner worlds of relationships.   The observational method concerns how to put a timeline under a mathematical microscope and identify developmental turning points in locally emergent structures and processes.    The theory concerns the consequent new understanding of nature, establishing useful global principles, and evolving the methods and models of physics.    The case studies bring the technique and theory together and inform both.

The subject area is the rapidly evolving organization of  event processes of all scales and kinds, sparks, ecological shifts, solar storms, earth quakes, heart beats, weather, epidemics, social change, life, etc., for any of these one can read any consistent measure over time, and draw an informative single story line within an epic tale.   You look for growth and decay and their phases of transition, for the loosely connected distribution of parts that act as a whole.    One of the first things you see is a mix of what seem to be continuous flows of discontinuous events, a tentative regularity of complete accidents.   Growth is the transition between discontinuity and continuity in nature, the development of the physical property of organizational continuity, that makes describing nature with mathematical functions useful.   Math behaves a little like nature in that it is meaningful to imagine filling in new points in-between your data points so that each short sequence telegraphs where the whole series is going, the property of flow.  That's the property of functions that was invented with calculus.   On close inspection all continuous physical flows are actually found to be granular, with emergent organization at a larger scale coming into focus as you imagine there might be points in-between.   Nature rarely builds machines like we do, with fixed structures, but assembles working parts ad hock from whatever is currently lying around.   It's quite amazing that it works at all, but seems to work extraordinarily well.

The beginning and end of smooth flows are the periods when emergent organization is being built up and dismantled, and when it is most directly exposed to view.     Time traces are only one dimensional and do not provide very good descriptions of the processes they reflect, but they do provide good markers, exposing when and where locally emergent organization and disorganization is occurring.   The set sequence is a four part series of cascades.    I call them Inflation (compound growth), Integration (climax/stabilization), Disintegration (destabilization) and Decay, the phases of rapid system evolution.   They typically represent  billions of molecules going through a collective process of reorganization without outside influence.   The designs of natural systems are always built locally and are never transferred from elsewhere.     The basic four phases of their development are also the minimum necessary set of simple continuous progressions forming a "bump on a curve" (¸¸¸¸.·´ ¯ `·.¸¸¸¸) .   They're to be found absolutely everywhere.

This study of locally emergent natural systems does not conflict with any empirical finding of traditional physics.   Physics is not about causation, and this is.    It's a matter of looking at the same world using different sets of questions.   Modern physics looks for fixed mathematical relationships between idealized measures, and has found many.    This approach to studying the structures of local happenings accepts that they are individually unique throughout and quite beyond full description.   Some useful information comes from learning how to watch them closely.

It's the perennial dance of nature.   What we find is that she does not actually 'follow' rules, but always makes them fresh, over and over again as she goes.   To observe it you first find a beginning or ending continuity and watch closely as the pieces come together or fall apart.    It's made easier with some technique. (continued)

collaboration: 3/11/06  One normal fact of collaborative research is that each of the disciplines and other communities involved develop and apply their own models, terms and methods for the common subject.  Slowly now I think everyone is recognizing that the 'Rosetta stones' are the cohesive physical systems of nature (when we can find them) that each approach looks at from a different perspective in a different way.  Joining different perspectives on the common complex subject, is highly useful.  Speaking about them in different languages is rather confusing, but we cope.

The basic formula for success is for everyone to be talking about the same physical thing, even if using different languages, i.e. system identification comes 1st.  One option is to accept the lead of the most articulate person in the group.  The hazard there is that he or she is likely to describe the physical system we're looking at as a model residing in his or her own mind...since that's how people think, leaving everyone else out of the loop in discussing it.

One of the ways to assure everyone is looking at different aspects of the same subject is by identifying the physical subject with data, and instead of first using the data for a statistical model, first use it to refer back to the larger whole system beyond the data in the physical world.  Then everyone can link up with the same thing using their own data, perhaps, and different perspectives.  

What data to use for that is perhaps a sorting and culling process, since the evidence of larger systems is in all sorts of things.   The one I've focused on, and developed some potentially good theory and technique for, is tracing system growth phases in time-series data. Complex systems are noticeable because they begin and end (have finite duration) and in those transitions exhibit a continuity of organizational development (growth curves) and then a continuity of systemic responses (homeostasis) in maturity.   The fact that strings of data points can often be treated as differentiable curves reflecting the physical and organizational continuity of a larger system is the trick. It's a physical property with mathematical landmarks that can identify the same physical subject from many perspectives.   If anyone sees ways to make these methods more accessible I'd naturally like to hear it.

The General Method In Brief:     ...an accumulation of different experiments and models .
[judge for yourself what old work is out of date...]

Publications Articles, Letters & Web Archives
Others - Who thinks like I do? Influences, Reading Lists & Fields of Interest
time-series data source - Rob Hindman

• Common sense, Key Methods, Physics Theory, Systems Thinking Principles 12/07
• the Basics of Steering -
• the Principal Principle of Cybernetics - pump it up... just enough
• What Approximation Leaves Out - Proposed For NECSI talk 10/06
• Why we're all mostly out of the loop 
• Page History & 9/06 Current View  View from 06 -
• Models, Emergence & Complexity View from 06
• Experimental Outline of a new Science Physics of natural time, (a bit old)
• A Letter in the Science Times  on mathematical truth & beauty
• Evolving Air Currents  The experimental origin - and all time favorite page
• These Pages... Until recently were called 'Derivative Reconstruction'...
• Notes & clippings... Some comments on related issues

The general idea,... observing change 

What results are a some surprising discoveries, and a greater appreciation of the emergence and layering of  patterns of change in general, causation.  It is probable that what you'll find here will not live up to its promise, at least not until you are able to use it with the kinds of change you are most interested in and most familiar with.  Much of the focus here is also on mathematical methods and problems.  More on how to use the approach might be helpful, but it is given somewhat less attention.     Some of the techniques involve new areas of mathematics (empirical measures of derivative continuity, the paradox of indescribable simple pattern in the beginning and ending of events) and contribute to new methods in computer vision (defining  landmarks of organizational change), as well as providing a new empirical method of inquiry.   The method also involves a new point of view, a super-realism.   Why 'super'-realism?   It involves watching for, and draws you into, the deep inner workings of particular individual events.    There's lot's there.

For example, one might ask why intricate patterns develop in fluid flow, say watching cream in your coffee.  Even if we can see various scales of pattern, can name a cause such as heat or drag, and make some predictions, there's still nothing but the kinetic interaction of molecules to produce any complex pattern.  Somehow the molecules interact to build complex evolving pattern.   There are no other communicating forces, no imposed guides, no embedded memories, only accumulative original pattern development in molecular motion, with every part participating independently.

This is a common trait of nature's confusing but highly organized behaviors, they happen all at once, with unified orchestration but no central point of control.   So much is happening at the same time, and so smoothly going through changes, one has to watch it very very slowly....   A one dimensional trace, say temperatures at just one point, may provide particularly good information on when and at what rates organizational transitions develop.   That may point to exactly when, where and what kind of thing to look for to find the mechanism that develops to do it.   The same is true for tracing the inner workings of your business market, looking for particular kinds of turning points and then to look more closely for what it is and where it might go.

Measures of change over time directly reflect, but poor describe, the complex accumulations of events involved in any process of change.  They're not the answer, they're a guide you can use for looking beyond. The common indicator of  local organizational change in inner workings of a system is a growth curve, reflecting far more complex events, but accurately displaying their timing.   These are typically represented by the 'S' curves, the connecting shapes that are found in virtually every transition.   Their shape draws a smooth curve between the static measures as the underlying organizational processes develops a smooth changes between systems of behavior.   After identifying a growth curve a few simple checks usually identify what process it is reflecting and at least hint at some of its distributed system of interactions.

Beyond the effort to test scientific ideas, what drives the inquiry is the wonderful depth of unique detail that can be found in individual things and events of all kinds.    Nature is indescribably deep.   Individual events  of all sorts display a fantastic variety of local organization at many scales.    When you look at them in detail, it is obvious why any mathematical description needs to include uncertainty, there's simply too much there, too much going on to describe.   Patterns that can't be understood are called random, and in our data much of the intricacy of nature appears random.  Data is such a very poor story telling device.   The solution is not to abandon data, but to use it differently, i.e. not as the description of what's happening but as the map or guide for your own exploration of a territory of understanding the map can't directly provide.

ed Jan 99
The particular mathematical aspect of natural complexity focused on here is the continuity, or flow, of change.   Much of the work concerns a disciplined analytical tool, derivative reconstruction (DR).  It's not magic, but a tool which allows you to discover much more information about the history and progression of change than you would first expect to be available.   It's an extremely general technique, applicable to any subject of change over time whatever.  Change in anything usually progresses in a self consistent manner, developmentally, displaying the presence and accumulative modification of coherent systems of relationships.   By very sensitively representing smooth underlying flows of change DR helps make the details of these patterns more visible.

The property of infinitely smooth progression in mathematics is called 'derivative continuity'.   It is one of the most fundamental and useful properties of mathematical functions.  Physical measures of change often display a similar physical property, the organizational continuity of nature, call it flow.  DR uses the mathematical definition of the derivative in reverse to represent dynamic organizational change reflected in measures of physical systems .  It doesn't always work, but often does.  When it does it often exposes otherwise invisible natural structures and the links between them.

Recent investigation indicates that there is a distinct similarity between the primitive elements of DR and those being rapidly explored and developed in the fields of computer vision, artificial intelligence and parallel distributed processing. In the terms of those fields DR is a form of curve generalization, that treats randomly sampled data as compressed information about multi-scale continuities.

Who else thinks like I do?... others.    Back to contents

............

Reconstructing the flow of individual events

Sampling a smooth, but complex flow of change may produce a pattern of points that appears randomly scattered. If you have no information about the underlying process then you can treat the data as form of compressed information about a flow. You might have some other reason to expect an underlying continuity or simply want to see if recognizable forms appear when it is looked at as if it had continuity. Using the implied progression of the derivatives to reconstruct some details of what went on in-between the data points is then a method of data decompression. When it works, it displays previously unseen fine dynamic structures directly, without requiring any preconceived theory of behavior. It also produces a new kind of mathematical entity called a 'proportional walk', having the mathematical property of derivative continuity, but no equation, composed instead of a finite sequence of points and a parsimonious rule for how to connect them with continuous derivatives.

Throughout all fields of scientific research there is a tendency for investigators to see whatever it is they are looking for. In a way that is both more and less of a hazard with derivative reconstruction. The curves produced are more strongly data driven than perhaps any other method, so the likelihood that they represent some real feature of the data is quite high. The problem lies in interpretation, generally requiring a thorough immersion in the detail of the subject, to know what the data represents and what assumptions about the different features of its shape can be made in the process.

There is also a philosophical problem. Reconstructing and studying the individual dynamic flows of change raises a question concerning whether to view nature with either a deterministic or probabilistic model of events. Neither seems appropriate. Individual processes are not probabilistic, since that paradigm applies only to classes of events. Flows of individual events are also not even remotely deterministic. You rarely find any fixed patterns. At the beginning and end of any apparently deterministic pattern (because they all begin and end somewhere) there is an indeterminate change in pattern. To make a possibly long story short, the inevitable conclusion is that individual events need to be seen as something else, call it 'opportunistic', displaying processes of discovery rather than of following rules. How individual events actually come to have the unique and intricate structures they display is clearly not by humanistic volition and intention etc., and it is not that sense of their being 'opportunistic' that is meant. It is only that new behavior begins to develop when, and only when, the appropriate circumstances arise, and then develops in a manner displaying continuity. A theoretical study shows that this implies that change begins with a very specific and recognizable mathematical shape.

The computation routines used (see Tools & StatTechnique) were developed in AutoLISP, the programming language of AutoCAD, and run on a PC. When proper care is taken the immediate usefulness is that the method tends to uncover evidence of surprising patterns of change. In long studied data these newly appearing patterns are often different from what others have found. For example DR offers a direct means of identifying dynamic coupling between complex systems, without having any behavioral theory for either one. (warming3.gif, econcync.gif). Sometimes this brings into question strongly held notions about very well studied processes.

(to Applications ) back to contents

Theory top 

Equations are usually made to fit many sets of data at once, to represent idealized behavioral structures. Here the proposal is to use the constructed curves fitting the detailed shape of individual sets of data to represent the unique structures of individual events. It is used where it can be reasonably assumed or concluded that the data reflects a continuous physical process which can be treated as having derivative continuity.  The more general principles of organizational continuity follow from general observation and a physics of conserved change.   The critical point of physics is a straight forward derivation from the conservation laws, assuming only that rates of energy transfer must be finite.   It demonstrates that all physical change must satisfy a principle of continuity & divergence.   It seems to imply that the only ultimate discontinuity in nature is in information.    That doesn't indicate when you have enough information to identify the continuities, or "little bangs" that may be present, of course.    It does point to where you'd need to look for them though.    Quantum mechanics, for example, concerns a range of behaviors beyond the known limit where a knowledge of continuity is possible.   There's no conflict in a lack of information, just various kinds of assumptions you can make about it.

What one starts with in observation is information without meaning, like a series of points with no necessary relationship between them.   It could be a history with a fascinating story to tell if you could connect the dots with the processes that produced them.    To interpret a curve as suggesting a process, one needs to ask how to determine when a series of points can be treated as information about a continuous process there's a series of general questions to ask.    Was each recorded the same way?  Are there any recognizable shapes?  Are there enough points for the kinds of mathematical tests you might try?   Is there any reason to think from the environment there was any accumulative change?.  Are there other things that might be affecting the points?    One exercise to begin learning the technique is to look at a time series graph and see if you can think of two or three possible explanations for each bump.   Having explored the context of the shapes and processes that might be involved, ask if there's any overall shape.    That can be by simple inspection, learning the specific steps taken to collect the data , using statistical measures, or by other means of determining that the data most likely represents some derivative continuities.

Just because there are breaks in the shape of a curve does not necessarily mean there is a break in the process producing the curve.    The activity of the process may have switched to somewhere else for example.   Studying complex systems involves developing an intuition for questions like that.    Others would have other lists.   Four of my lists of them are arranged as a list of principles.    some advanced methods are described for identifying cybernetic body parts and things in complex systems and learning from them.   This site is mostly about the work I've done with analyzing one dimensional curves for the questions they raise about the processes that produce them.    The analytical methods I developed are described below in 'Tools' and a little within the Autolisp program files in Curve.zip, and in a long 1995 paper. The discussion of analytical principles begins on p12 and is followed by an appendix describing the routines developed at that time starts of p35.

back to contents

Tools

Commonly some noise suppression is used first, with the one hard and fast rule being to use the very least amount needed.    Validation of the process, like any other scientific method, is whether it's useful and everything knits together in the end.   For very noisy data there are good mathematical tests for the degree of smoothing that will reveal the continuities without erasing them.   The full methodology is not published as I've invested most of my time in applications, and trying to explain why anyone would want to closely study the real dynamics displayed by independent natural systems...  Contact for assistance: id @ synapes9.com

Scales of larger and smaller fluctuation are the most common finding.   Estimating the inflection points and growth rates for the larger scale fluctuating processes is usually the interest.   That can be greatly improved by representing larger scale fluctuating processes as curves drawn through the centers of the smaller scale fluctuation.   That can be done by connecting the inflection points (TLIN) of the smaller fluctuations, called 'integral interpolation', or finding the 'dynamic mean'.    The fluctuations in a process are usually symmetric and represent elastic variation in the underlying behavior.   Sometimes the variation is one sided and the maxima and minima of the fluctuations are used to represent the norms of the underlying process.  The effect is quite often to display surprising structural features of the real underlying processes that would not have been visible by any other kind of representation.

The routines available are written in AutoLISP (one of the programming languages of AutoCAD). This platform was chosen for the purpose because it allows curves with any number and spacing of points to be related in the same 'table', and is fully programmable. For conventional statistical analysis the data is transferred to a statistical package, such as JMP. Suggestions of alternate development platforms would be quite welcome. Current updates available on request.

Mar 97 reverse test - How it really works, a complete demonstration of the DR method.

Sep. 95 drtools.pdf -  description of the AutoLISP analytical command library

Apr 97 Curve.zip - a basic collection of AutoLISP DR command functions for AutoCad 13 or earlier

1/2/08 notes, back to contents

Applications
 - various examples of successes and failures -

Gamma Ray Bursts

April 1998 BATSE Trigger 551
May 1999 The classic "backwards wave" shape of GRB's, perhaps indicating implosion
NASA Introduction to Gamma Ray Bursts
Information on NASA Compton Gamma Ray Observatory

back to contents

Plate Tectonic Flow...

Mar. 1997 PBL1 , TIBB , Match

back to contents

Ice Core CO2

The most interesting feature of the ancient record is the amazingly smooth progression of the data, seen close up in fig. 2. The underlying process appears perfectly regular and does not appear to have any fluctuation more frequent on average than 3,000 years. That is extraordinary log term regularity! Perhaps it is driven by something like magma upwelling, or a glacial solar cycle, or there is some diffusion process within the ice which perfectly suppresses the shorter fluctuations without suppressing the larger ones over time. The 100,000 year decline might reflect the general rate at which carbon is withdrawn from the biosphere. The dramatic rise that began just 20,000 years ago might just possibly represent the extensive use of fire.

The study provides an interesting demonstration of the ability of DR to highlight, and repair, a bothersome calibration irregularity found in the original data. The second derivative of the data is still remarkably smooth, except for what appear to be occasional single point spikes due to calibration errors in data collection. The analysis of underlying behavior is that what appears as a general trend of decline interrupted by events of increase may actually be the reverse. This is shown by the second derivative of the trend of 12,500 year fluctuation showing a positive norm punctuated by transient negative periods.

Jan. 1996, Sept 1997, iceco2-1.gif , iceco2-2.gif , iceco2-3.gif

back to contents

Punctuated Equilibrium in Plankton Evolution

Bjorn. Malmgren gathered data on the size of a plankton species over 7 million years from an Indian ocean deep sea sediment core covering the transition from one stable form of a plankton species to another (G. plesiotumidia to G. tumidia).   During the transition in shape the organism also tripled in size, giving a good corollary indicator of the progression. The data only hints at it at first, but yields clear evidence of continuous growth processes, repeated eruptions accelerating and decelerating progressive rates change, producing in continuous developmental transition between two steady states. There are other interpretations to consider, but the appearance is of mutation having proceeded by a complex feedback regulated process.

The validity of applying continuity analysis to evolution comes in part from finding a strong linear relationship between the mean sizes of each sample and their standard deviations.  That variation in size is correlated with size is not surprising perhaps, but it does contradict the commonly held 'null hypothesis', assuming that the data was produce by what is called a 'random-walk'.  The variance is neither invariant nor constantly increasing so the random walk assumption for individual lineages or the whole population does not fit.  A second, and more specific contradiction of random walk and demonstration of underlying continuity in the data is provided by the step variance test.    It shows that the variation between widely and narrowly spaced points does not differ like it would for random walks   In a random walk the variances would multiply with the number of points.   The variances of the data do not.  This indicates that the irregularity of the data is noise, and that recognizable shapes of continuous processes that are visible through the noise probably indicate the presence of a corresponding mechanism.    A rapid continuous developmental process that began and ended seems indicated and the kinds of mechanisms that could produce that are considered.

Because the standard deviations are related to plankton size by a linear relation both can be used as indicators of the shape of the transitional events.   The values of the standard deviations were rescaled by that linear relation so that its shapes would be properly scaled for comparison.   Both the size and std. deviation curves were derivative smoothed and interpolated and both show an appearance of a similar progression.
 
 

The first derivatives of the two curves make the point much more clear.   They are remarkably similar despite seeming larger differences in the shapes of the curves themselves, and both display the presence of a dramatic transient underlying process.
 

• A slightly different approach to resolving the true shape underlying the noise in the data is provided by studying how widely the various shaped are distributed, using the curvature scale space approach.
• The detailed curves and steps are shown more clearly in the full figures.
• A curious appearance of clustering in the data has been given a very brief and incomplete study.

Global Warming -1 01/1996, 02/2009

The average global surface temperature over the past 100 years has tended continually upward, but with both short and long wave fluctuation, and in the 90's was rising at the highest average rate of the period on the upslope of the long range curve.    Comparing the rate of change of warming and the rate of change of economic activity with US GDP, the derivative reconstruction curves(1) show the long term trends shows no apparent dynamic link between warming and the general increase in economic activity.   Economic activity is related to energy use and fossil fuel consumption, and the contribution of CO2 and other pollutants to the atmosphere.    This should be further studied.

1) [The derivative reconstruction method (dr) uses mathematical routines to identify continuities underlying natural process fluctuations, the waves underneath the ripples, in this case by taking the mid-points of the smaller scale fluctuations and using a special smoothing kernel to reduce the higher derivative fluctuations without changing the local integral of the curve.   It seems to be a very sensitive and effective means of indentifying natural system behaviors.]

One possible reason why the underlying dynamics of the two systems do not match is a variable lagging and leading response of the climate system responding to the forcing of CO2, visible in the data as a long wave fluctuation above and below a rising norm.   It doesn't mean that atmospheric CO2 doesn't influence global surface temperature.   It just means that something else that speeds and slows the process alternately is happening too.   Having now found data ( Next study below) that does appear to demonstrate a direct dynamic link between warming and CO2, the question is what is the larger effect, and how will it exaggerate or hide the warming effect..

The long wave curve looks like a climate system fluctuation that might have momentum and be repeated. That is quite common for system change, that there are multiple scales of fluctuation.   If so, that might begin to produce a period of actual atmospheric cooling, or a pause in warming, to hide continued  CO2 induced warming and give people a false impression that warming has stopped.   I've drawn what looks like the probable scale of the momentum of the large scale fluctuation, but this should be revisited with better data.    Click the images the temp. curve(A.1) and rate of change curve(A.2) to enlarge.

In 2007 a slowing of warming of just this type was noticed as associated with increasing cloud formation in the tropical latitudes, the Iris effect, Lindzen 2007 and seemingly substantiated by Spencer 2007.   Both authors consider the evidence of a pause in warming as a permanent effect, but I think, especially if its timing matches the 'long wave' seen here then it is probably a new kind of "El Nino" effect, but in the atmosphere.

Jan 1996 Air temp warming1.gif , 1st deriv (compared to GDP) warming2.gif , 2nd deriv (compared to GDP)warming3.gif

back to contents

Global Warming - 2

[I had thought that this little piece deserved credit for being the first definite human fingerprints on global warming in July 96...but it didn't get notice.   The real prize and credit go to Michael Mann, Ray Bradley (u.mass) and Malcolm Hughes (u.ariz).  Their 3/15/99 paper in Geophysical Research Letters shows a detailed 1000 year temperature record (based on a broad group of climate indicators) which quite abruptly changes direction in ~1900.][link to earlier 4 century curve] [link to Mann's 1999 study]

The study linked above displays an appearance of dynamic coupling between earth temperature and atmospheric CO2  in the alignment of first derivative turning points for recent temperature and CO2 measurements. The Hansen temperature index is a global aggregate of temperature measurements. The NOAA trends in CO2 used comes from measurements made at the top of Mauna Loa. The close correlation between changes in direction of increasing CO2 and changes in direction of earth temperature appears to be evidence of there being a connection (unspecified). The fact that during this period earth temperature tends to decline when the rate of CO2 increase slackens, indicates that there other factors were dominant at the time.   There appears to be a link, though, and atmospheric CO2 is increasing radically (see comparison with ice core CO2 data).  ed 2/4/06

back to contents

History of Economic Growth

Reconstructing the path of GNP discusses the statistical methodology involved,

Aug. 1995 GNP08.gif , GNP10.gif , GNP12.gif , GNP13.gif

back to contents

Dynamic Synchrony between Economic Measures

Aug. 1995 econcync.gif, Charts

Aug. 1995 econcync.htm, An old methods reference

back to contents

Childhood Epidemics

The reported cases of Measles, Chicken Pox and Mumps in New York City from 1928 to 1963 are data sets dominated by chaotic variation. (Olsen & Schaffer Science 3/8/90) DR technique was able to identify underlying regularities of some possible interest but with such poor confidence.

I keep some of my more obviously problematic methods and interpretations around to keep me and others honest and and on their toes.    Nothing on the site is predigested for unthinking acceptance.    I tried constructing this cool statistical measure, I called 'DAR', to help identify when strings of dots were likely to represent a curve.   Great intent, but this version doesn't do it.    My ~2000 step variance test that successfully disproves random walk in time series where variation is symmetric is my first really useful step in that direction, and I have some other ideas, but haven't gotten any further.   ed 2/4/06

Apr. 1996 NYmeasl.gif, NYchick.gif, NYmumps.gif

back to contents

Sparks

The development of a spark is a very complicated thing

Nov. 1999  Sparks; a first look in a new area.  I'd love it if someone would tell me where I can find detailed data on the growth phenomena of electrical discharges, sparks, lightening etc.

back to contents

The Great Crimewaves

Sep. 1999  Patterns in Crime the record shows clear events in youth culture that are separate from the society at large, thought this brief look only points to events that are local.   What that suggests is that their dynamics were local and internal, and raises the question of where they were occurring and what erupting to produce their dynamics.

Oct. 2005  Crimewave's Collapse  The great crack epidemic of the late 80's is studied in some detail, though the report is only a summary research note, to find some fascinating secrets.   I did 50 interviews on the street with people who lived in the neighborhoods affected.    The dynamics show a clear true collapse, starting about three years before the reputed 'causes'.   Might sound a bit 'unscientific' but it passed rather abruptly like an intense fever as the whole community of addicts and their families got sick of what they had become and it stopped being cool, ending the 30 year ghetto rage that began in the 60's with a dramatic universal silence.

back to contents

The Emergence of Sustainability

Oct 2006 Emergence of Sustainability  This brief research note shows how to trace the frequency of a key word or phrase to investigate the dynamics of changing ideas.    In this case the word 'sustainability' in NY Times articles shows a dynamic long term growth phenomenon punctuated by dramatic flurries of conversation linked to the individual articles discussing the subject.  

 The Collapse of General Systems Theory

Feb 2006 General System Theory's Collapse  This brief research note is another example of using word use frequency to trace the evolution of an idea.  

top of file