There are very few technologies that come along which actually make things easier for IT staff. This is particularly true with new technology introductions. Very often, the introduction of a new technology is problematic from a systems service up time perspective. With networking technologies in particular, new introductions often involve large amounts of intermittent down time and a huge amount of human resources to properly plan the outages and migration processes to assure minimal down time. More so than any other, network core technologies tend to be the most disruptive due to their very nature and function. Technologies like MPLS are a good example. It requires full redesign of the network infrastructure as well very detailed design within the network core itself to provide connectivity. While some argue that things like MPLS-TP helps to alleviate this, it is not without cost – and the distruption remains.

IEEE 802.1aq or Shortest Path Bridging (SPB for short) is one of those very few technologies that can introduced in a very seamless fashion with minimal disruption or down time. It can also be introduced with minimal redesign of the existing network if so desired. A good case point example is a recent project that we have been working on with a large health care provider up in the northeast US. This was a long time Avaya networking customer who had an installed base of existing ERS 8600 routing switches. There was particular portion of the topology that interconnected the customer’s two data centers which were located in separate geographic loctions. This was the portion of the network topology that they chose to upgrade and introduce shortest path bridging.

The original intention was to upgrade the existing backbone switches to code that could support shortest path bridging (v7.1). They would then build out a parallel routed core in the resulting new ISIS routing plane. The ISIS environment would be kept latent and secondary by the resetting of its global priority to something lower than OSPF. Typically, this value is set at 130 (the default for ISIS is 7). Once the parallel routed core is built out as a mirror to OSPF, the systems are checked for validity and then once assured of stability, the priority of ISIS is then reset back to its default value of 7. ISIS then becomes the primary routed plane and OSPF is relagated to a secondary role. After system checks and validation, the OSPF network can be kept as secondary for as long as required. Then, at a later point in time, it can be decommissioned to leave ISIS as the sole core routing protocol for the enterprise core. This is a very seamless migration that provides for zero downtime to the overall networking core.

After a survey of the equipment however, it became obvious that due to its age  and slot density requirements (circa- 2000-2001) would need to be completely upgraded – including the switch chassis. Rather than view it as an impediment we quickly realized that by implementing a parallel routed core infrastructure the upgrade and migration of the critical path could be accomplished with little or no down time to the network core. This was in comparison to a gradual swap out and upgrade of the existing core which would have meant multiple outage occurances for each chassis swap out.

The theory was based on the diagram below, which shows the existing OSPF routed core running in parallel to a new SPB based ISIS routed core. By using a series of migration techniques which we will cover shortly, both routed cores would work in tandem with networks gradually migrated over to the new ISIS routed core in a controlled and phased approach.

Figure 1. Parallel OSPF and ISIS routed cores

The first step in the project was to account for the various VLAN’s that were provisioned in the existing OSPF routed core. Part of this was to also identify if they were one of two types. The first being VLAN’s that did not traverse the routed core by the use of Q-tagged trunks. These we identified as ‘peripheral VLAN’s’ in that the only Q-Tagged tunks that they ran over were along the edge and over the SMLT ‘Inter-Switch Trunks’.  The second type was a VLAN that existed in multiple places in the routed core and hence traversed the routed core by the use of Q-tagged trunks. These we labeled as ‘traversal VLAN’s’. Figure 2 illustrates the difference between the two VLAN types. This was an important step in the investigation because as one will see it largely determined the migration method for a given VLAN.
As is noted in other white papers, SPB offers various provisioning options. These are listed below for the convenience of the reader.

                L2 Virtual Service Network

This is a provisioned path across SPB, known as an I-SID in IEEE terms that inter-connects VLAN’s at the SPB edge. Taken as such it can be termed as a VLAN extension method somehwat anologous to Q-tagged extensions.

                L3 Virtual Service Network

This is a provisioned path across SPB, known as an I-SID in IEEE terms that inter-connects VRFs at the SPB edge. Taken as such it can be termed as a IP VPN method somewhat anologous to VRF lite.

Inter-VSN routing

This is a method of interconnecting Virtual Service Networks by the use of external routers or other devices. A good usage example is in a data center topology where user or ‘dirty’ VSN’s interconnect to data center or ‘clean VSN’s by the use of security perimeter technolgies such as firewalls and intrusion protection type devices.

IP Shortcuts

This final method does not involve the use of VSN’s at all but instead works on the injection of IP routing directly into ISIS and utilizing ISIS as an actual internal gateway routing protocol or IGP.

For the purposes of the migration we chose to use a combination of IP shortcuts in order to implement ISIS as the replacement core routed topology and L2 VSN’s to facilitate the connectivity to support the ‘traversal VLAN’s’ which would require multiple points of presence across the routed core.

In essence the network core migration involved three major steps:

1). Build out parallel network segments that match in almost every sense topologically. The new segment will run ISIS/SPBm as its core protocol. A migration link will be set up between the two routed domains to provide for a communication path during the migration. This link will be a MLT configuration for both bandwidth capacity and resiliency.

2). Redistribute VLAN’s and IP routes into the SPBm ISIS core on a switch by switch and VLAN by VLAN basis. Both ISIS and OSPF routing domains will be utilized throughout the migration process.

3). After all network migrations are completed the OSPF network core is to be dismantled.

If properly orchestrated and implemented, we strongly felt that this could be accomplished with zero network downtime for the local core network. There would however be short outages for each switch as it is migrated over to the SPBm/ISIS core. There would also be short outages for the individual VLAN’s during the final migration steps over to the new ISIS core. These however would be minimal and could also be scheduled during opportune windows that the IT staff had on a regular basis. The rest of this document will provide a more detailed outline of the three project phases listed above.

The diagram below illustrates the various types of VLAN’s and how they relate to the overall parallel routed cores. Note that with the introduction of SPB there is an additional type of VLAN (subnet) that is introduced which is a traversal VLAN that is in the process of migrating to the new routed core but still uses OSPF as its IGP. This required a number of items to work successfully. First we need to interconnect the VLAN by the use of L2VSN’s (I-SIDs) across the SPB ISIS routed core. This provided for connectivity, but due to the L2 nature that provides extension back into the OSPF environment, NOT the use of ISIS in an L3 sense. Additionally, we added in OSPF to ISIS and ISIS to OSPF redistribution at the migration link interface between the new and existing cores. This provided for the ability for the migrating VLAN (subnet) to have routed connectivity into the new ISIS routed core via redistribution but still use OSPF as its IGP. As the resident switches and systems were migrated over, the VLAN (subnet) would eventually be redistributed direct into ISIS and effectively decommisioned from the OSPF routed core. Again, by the use of the OSPF to ISIS and ISIS to OSPF redistribution the completely migrated network would still have connectivity over to the older OSPF routed core and visa versa. With the exception of the actual movement of switches and decommissioning of the subnet from OSPF and redistribution into ISIS the network downtime would be zero. More importantly, there would never be a time when the network core was not functional in a holistic sense.

Figure 2. Various migration VLAN types

                Taking a closer look at the ISIS side in the illustration below will provide a better feel for the actual topology in action. As noted in the diagram, we show the three VLAN types in the new SPB ISIS environment. First, for the completely migrated dual homed VLAN; it is simply redistributed into ISIS and routed accordingly. Due to the fact that it is provisioned as a Q-Tagged VLAN over the edge SMLT IST there is no use of VSN’s, the peripheral VLAN is simply redistributed direct into ISIS by the use of IP shortcuts.

                In the case of the traversal VLAN’s the illustration shows VLAN A which is a completely migrated traversal VLAN that is set up with VRRP Master Backup at various points for router redundancy. The VLAN (subnet) is then redistributed direct into the ISIS routed core by the use of IP shortcuts. This provides for the multiple points of presence required in the routed core by the use of L2 VSN’s and for the IP connectivity into the ISIS routed core by the use of IP shortcuts. The third VLAN type (VLAN C) is a migrating VLAN (subnet). As pointed out above, this is a VLAN that is extended over from the old routed core by the use of Q-tagging (old side) and L2 VSN’s (new side). As the diagram also shows, the migrating VLAN C (subnet) will continue to use OSPF as its routing protocol until all systems are moved over to the new core. At that point in time, the subnet is decommisioned in OSPF and redistributed direct into ISIS and will mirror VLAN A with the exception that there will be four VRRP instances and not two.

Figure 3. A closer view of the new ISIS core and various VLAN types

                Normally is such a scenario, one would have to deal with prefix lists and route policies to suppress the advertisment of the networks on one side or the other as they are co-resident in both routed cores. We were able to avoid this by simply not assigning IP addresses to the VLAN’s in the new core during the migration. By not doing this, the VLAN’s would simply not be distributed direct into ISIS and all systems connected to the subnet would use OSPF for all IP routing until the final migration step.

                Prior to the actual migration project we thought it prudent to test out the migration scenario as well as use the environment to provide knowledge transfer to the customer. As a result we set up an OSPF environment in the lab prior to actual deployment that looked like the toplogy below in figure 4. Note that both VLAN types (peripheral and traversal) are represented in the diagram. The switch in the lower left hand portion of the illustration shows a switch that provided for the OSPF routed environment in the lab test. Note that OSPF also is supported on the SPB core in the form of an ASBR function. In this example, IP network 10.0.12.0/24 has connectivity to 10.0.11.0/24 via OSPF. (10.0.11.0/24 serves as the redistribution subnet). 10.0.12.0/24 also has connectivity to 10.16.110.0/24and 10.16.111.0/24 respectively by the use of OSPF to ISIS and ISIS to OSPF redistribution. In summary, all IP subnets have routed connectivity to one another.

Figure 4. Existing provisioned SPB ISIS core.

                The next step was to introduce a migration VLAN into the lab test. We did this by creating a new VLAN on the OSPF side and gave it the IP address of 10.0.13.0/24. As the figure below shows, we were able to extend that VLAN out across the migration link by the use of Q-tags on the OSPF side and L2 VSN’s in the new SPB ISIS core. We then emulated system moves over to the new core. Note that during this time 10.0.13.0/24 utilized the OSPF protocol as its IGP. As a final step to the migration, the VLAN was deleted from the OSPF environment including any Q-tag extensions and then assigned IP addresses, VRRP Master Backup on the ISIS side and redistributed direct into ISIS.

Figure 5. Migration VLAN case point example

                In summary, the migration steps can be summarized as follows:

1). VLAN is extended over migration MLT from OSPF side

2). VLAN is assigned at required points of presence. NO IP addresses configured yet!

3). Add port members as required and create I-SID to connect VLAN’s togther

4). Migration can now proceed (systems are moved over to new core)

5). Upon completion, decommission network from legacy OSPF side (short outage)

6). Assign IP addresses at required VLAN POP’S , set up & enable VRRP Master Backup

7). Remove VLAN from migration MLT (clean up)

The following shows the CLI sequence to perform these steps. Note that ISIS redistribute direct is already set up in the environment. For clarity and reference, the redistribution method for DC3-8800-1 is shown below:

#

# IP REDISTRIBUTION CONFIGURATION – GlobalRouter

#

ip ospf redistribute isis create

ip ospf redistribute isis enable

ip ospf redistribute direct create

ip ospf redistribute direct route-policy “suppress_IST”

ip ospf redistribute direct enable

ip isis redistribute ospf create

ip isis redistribute ospf metric 1

ip isis redistribute ospf enable

ip isis redistribute direct create

ip isis redistribute direct metric 1

ip isis redistribute direct route-policy “suppress_NETS”

ip isis redistribute direct enable

#

# OSPF ACCEPT CONFIGURATION – GlobalRouter

#

ip ospf accept adv-rtr 10.6.28.2 create

ip ospf accept adv-rtr 10.6.28.2 enable

ip ospf accept adv-rtr 10.6.28.2 route-policy “reject”

As a result to the above, as soon as the VLAN is assigned IP addressing and VRRP Master Backup it will have routed connectivity into ISIS, no other steps are required. Also note that the 10.0.13.0/24 network needs to be decommisioned in OSPF BEFORE being provisioned into the ISIS environment. This will involve a short outage (minutes) for the given subnet.

10.0.13.0/24 will then have connectivity back into the OSPF side by the use of the route redistribution occuring at the migration link point which again has already been configured as per the above.

1). Set up VID 3 on the MLT (new side) both DC3-8800-1 & DC3-8800-2 (assuming this is done on the 5510)

            config vlan 3 create byport 1 name “TEST_MIG1″

            config vlan 3 add-mlt 2

2). Set up port & I-SID configuration. Both DC3-8800-1&2

            config vlan 3 ports add <members> (i.e. 10/9-10/10)

            config vlan 3 i-sid 3

3). Set up port & I-SID configuration on each required DC1-8800-1 & DC1-8800-2.

config vlan 3 create byport 1 name “TEST_MIG1″

config vlan 3 ports add <members> (i.e. 10/9-10/10)             

config vlan 3 i-sid 3

4). Migrate systems as appropriate (note – during migration 10.0.13.0 still uses OSPF due to the fact that no IP addresses are yet assigned on ISIS side)

5). Once migration is complete 10.0.13.0/24 (VID3) is decommissioned from 5510 (legacy OSPF environment)

6). Assign IP addresses. Enable VRRP Master Backup and VRRP on DC3-8800-1&2, DC1-8800-1&2

            config vlan 3 ip create 10.0.13.*/255.255.255.0

            config vlan 3 ip dhcp-relay enable

            config vlan 3 ip vrrp 3 address 10.0.13.1

            config vlan 3 ip vrrp 3 backup-master enable

            config vlan 3 ip vrrp 3 enable

* Is 10.0.13.2,3,4 or  5 as required.

MIGRATION IS COMPLETE! 10.0.13.0/24 should now be visible to the 5510 across the OSPF-ISIS Redistribution on 10.0.12.0/24. Every subnet will have routed connectivity to the other.

Summary

            As can be seen by the example provided here, what can be a very complex migration project is now greatly simplified into a concise set of simple steps by the use of Shortest Path Bridging and VENA. OP/EX improvements when compared to other network virtualization technologies like MPLS are incomparible. Moreover, network downtime is predicatable, controllable and very short in comparison.

            Avaya’s VENA architecture facilitates a flexible yet powerful infrastructure that allows for this type of capability. It is also important to note that only a subset of the network services offered by VENA is used in this case point example. Very few technologies can claim such ease of introduction and actually ease the migration that they themselves require in order to be effectively used.

For more information please feel free to visit http://www.avaya.com/networking

Also please visit our VENA video on YouTube that provides further detail and insight. you can find this at: http://www.youtube.com/watch?v=ZSbycaOvy5I

 Happy Holidays to all!

With the very best wishes for the New Year!

 

The proper design of a network infrastructure should allow for a number of key traits that are very desirable in an overall network design. First, the infrastructure needs to provide redundancy and resiliency without a single point of failure. Second, the infrastructure must be scalable in both geographic reach as well as bandwidth and throughput capacity.

Ideally, as one facet of the network is improved, such as resiliency; it should also improve on bandwidth and throughput capacity as well. Certain technologies work on the premise of an active/standby method. In this manner, there is one primary active link – all other links are in a standby state that will only become active upon the primary links failure. Examples of this kind of approach are 802.1d spanning tree and its descendants rapid and multiple spanning trees in the layer 2 domain and non-equal cost distance vector routing technologies such as RIP.

While these technologies do provide resiliency and redundancy they do so at the assumption that half of the network infrastructure is unusable and that a state of failure needs to occur in order to leverage those resources. As a result, it becomes highly desirable to implement active/active resiliency wherever possible to allow for these resources to be used in the day to day operations of the network.

Active/Active Mesh Switch Clustering

The figure below illustrates a very simple active/active mesh fabric. As in all redundancy and resiliency methods, topological separation is a key trait. As shown in the diagram below the two bottom switches are interconnected by a type of trunk known as an ‘inter-switch trunk or IST, that allows for the virtualization of the forwarding database across the core switches. The best and most mature iteration of this technology is something known as Avaya’s Split Multi-Link Trunking or SMLT. First invented in 2001 and movning into its 3^rd generation, this effectively creates a virtualized switch that is viewed as single switch by the other edge switches in the diagram. Due to this fact, the edges switches can utilize defacto or industry standard multiple link technologies such as Multi-Line Trunks (MLT) or link aggregation (LAG) respectively. Because of the fact that the virtualized switch cluster appears as a single chassis these links can be dual homed to the two different switches at the top of the diagram to deliver active/active load balanced connectivity out to the edge switches.

Fig.1 A simple Active/Active Mesh Switch Topology

 Due to the fact that all links are utilized there is far better utilization of network resources. Additionally, because of this active/active mesh design, the resiliency and failover times offered are exponentially faster than comparative active/standby methods.

While the diagram above illustrates a very simple triangulated topology, active/active mesh designs can become much more sophisticated, such as box, full mesh and mesh ladder topologies. These additional topologies are shown in the diagram below. The benefit of these is that as the network topology is extended, both resiliency and capacity need not be sacrificed.

                                        box                   full mesh              ladder mesh

Fig. 2 Extended Active/Active Mesh Topologies

 As can be seen by the diagram above, these topologies can be very sophisticated and provide a very high degree of resiliency while enhancing the over all capacity of the network.

Topological Considerations for Active/Active Mesh Designs –

Most network topologies consist of various regions that provide certain functions. Depending on the region, there may be different features required that are specific to that region. As an example, within the network core high capacity load sharing trunks are a requirement where as at the network edge features like Power over Ethernet (PoE) are required in order to supply DC voltage to power VoIP handsets or other such devices.

Typically, these regions are divided into three sections of the topology; the network Core, Distribution and Edge. Below are short descriptions of each region and the role that they play. It should be noted that the distribution region is not required in all instances and should be viewed as an option.

The Network Core –

In a typical topology model, the individual network regions are interconnected using a core layer. The core serves as the backbone for the network, as shown in Figure 3. The core needs to be fast and extremely resilient because every network region depends on it for connectivity. Hence, active/active mesh topologies such as SMLT provide a very valuable role here. Even though the Core and Distribution Layer may be the same hardware, their role is different and should be looked as logically different layers. Also, as note above, the distribution layer is not always required. In the core of the network a “less is more” approach should be taken. A minimal configuration in the core reduces configuration complexity limiting the possibility for operational error. Ideally the core should be implemented and remain in a stable state with minimal adjustments or changes.

Fig 3. Simple Two Tier Switch Core

 The following are some of the other key design issues to keep in mind:

•Design the core layer as a high-speed, Layer 3 (L3) or Layer 2 (L2) switching environment utilizing only hardware-accelerated services. Active/active mesh core designs are superior to routed and other alternatives because they provide:

–Faster convergence around a link or node failure.

–Increased scalability because neighbor relationships and meshing are reduced.

–More efficient bandwidth utilization.

•Use active/active meshing as well as topological distribution to enhance the overall resiliency of the network design.

•Avoid L2 loops and the complexity of L2 redundancy, such as Spanning Tree Protocol (STP) and indirect failure detection for L3 building block peers.

If topology requires, utilize L3 switching in the active/active mesh core to provide for optimal sizing of the MAC forwarding table within the network core.

The Distribution Layer –

Due to the scale and capacity of active/active mesh core designs, the distribution layer is optional. It is far more efficient to dual home the network edge directly to the network core. This approach negates any aggregation or latency considerations that come in to play by the use of a distribution layer. The active/active mesh topology provides better utilization of trunk feeds and capacity can be scaled by multiple links in a dual homed fashion.

While the ideal topology is what is termed as a two tier design, it is some times necessary to introduce a distribution layer to address certain topology or capacity issues. Instances where a distribution layer might be entertained in a design are as follows:

• ·         Where the required reach is outside of available trunk distances.
• ·         Where the port count capacity in that portion of the network core can not support all of the edge connections without expansion and expansion in the core is not desired.
• ·         Where logical topology issues such as Virtual LAN’s or port aggregation require it

It should be noted though that all of the above instances could be addressed by the expansion of the network core. Examples if this are moving from a dual to a quad core design or going further, moving to a mesh ladder topology as shown in figure 2.
In any instance it is more desirable to maintain a two tier rather than a three tier design if possible. The overall design of the network is far more efficient and resiliency convergence times become optimized. The diagram below shows a three tier design that utilizes an intermediate distribution or aggregation layer.

Fig. 4. Simple Three Tier Network

Note that topologies can be hybrid. As an example, most of the network can be designed around a two tier architecture with one or two regions that are interconnected by distribution layers for one or more of the reasons noted above.

The Network Edge

The access layer is the first point of entry into the network for edge devices, end stations, and IP phones (see Figure 5). The switches in the access layer are connected to two separate distribution layer switches for redundancy. If the connection between the distribution layer switches is to an active/active mesh, then there are no loops and all uplinks actively forward traffic.

A robust edge layer provides the following key features:

High availability (HA) supported by many hardware and software attributes.

•Inline power for IP telephony and wireless access points, allowing customers to converge voice onto their data network and providing roaming WLAN access for users.

•Foundation services.

The hardware and software attributes of the access layer that support high availability include the following:

•Default gateway redundancy using dual active/active connections to redundant systems (core or distribution layer switches) that use industry standard or vendor specific Load Balancing or Virtual Gateway protocols such as VRRP or Avaya’s VRRP w/ Backup Master or R/SMLT. This provides fast failover of default gateway and IP paths. Note that with an active/active core or distribution mesh topology link and node resiliency and convergence are handled by the L2 topology which is much faster than any form of L3 IP routing convergence. As a result, any failover within the active/active mesh is well within the L3 routed timeout.

•Operating system high-availability features, such as Link Aggregation or Multi-Line Trunks, which provide higher effective bandwidth that leverages on the active/active mesh while reducing complexity.

•Prioritization of mission-critical network traffic using QoS. This provides traffic classification and queuing as close to the ingress of the network as possible.

In figure 5 the diagram illustrates a build out of a hybrid two/three tier network showing active/active load sharing interconnections with all network edge components.

Fig 5.  Full Resilient Active/Active Network Topology

Also note, that as shown in figure 5, active/active connections can also be established within the Data Center via top of rack switching to facilitate load sharing highly resilient links down to server nodes. Again, such resiliency is provided at L2 and is totally independent of the overlying IP topology or addressing.

Provisioned Virtual Network Topologies –

An evolution of active/active mesh topologies is provided by the ratification of IEEE 802.1aq “Shortest Path Bridging” or SPBm (the ‘m’ standing for MAC in MAC – IEEE 802.1ah) for short. This technology is an evolution of earlier carrier grade implementations of Ethernet bridging that were designed for metro and regional level reach and scale. The major drawbacks of these earlier methods were that they were based on modified spanning tree architectures that made the network complex to design and scale. IEEE 802.1aq resolves these issues with the implementation of link state adjacencies within the L2 switch domain in a manner that is the same as occurs by L3 link state adjacencies such as IS-IS and OSPF. All nodes within the SPB domain (which use ISIS to establish adjacencies) then run Dykstra to establish the shortest path to all other nodes in the active/active mesh cloud. Reverse Path Forwarding Checks provide for the ability to prevent loops in all data forwarding instances in a manner that is very similar to that provided in L3 routing. IEEE 802.1aq provides a cornerstone technology for Avaya’s Virtual Enterprise Network Architecture or VENA. The VENA framework utilizes SPBm as a foundational technology for many next generation cloud service models that either offerable today or currently under development at Avaya.

This next generation virtualization technology will revolutionize the design, deployment and operations of the Enterprise Campus core networks along with the Enterprise Data Center. The benefits of the technology will be clearly evident in its ability to provide massive scalability while at the same time reducing the complexity of the network. This will make network virtualization a much easier paradigm to deploy within the Enterprise environment.

Shortest Path Bridging eliminates the need for multiple protocols in the core of the network by separating the connectivity services from the protocol infrastructure. By reducing the core to a single protocol, the idea of build it once and don’t have to touch it again becomes a true reality. This simplicity also aides in greatly reducing time to service for new applications and network functionality.

The design of networks has evolved throughout the years with the advent of new technologies and new design concepts. IT requirements drive this evolution and the adoption of any new technology is primarily based on the benefit it provides versus the cost of implementation.

The cost in this sense is not only cost of physical hardware and software, but also in the complexity of implementation and on-going management. New technologies that are too “costly” may never gain traction in the market even though in the end they provide a benefit.

In order to change the way networks are designed, the new technologies and design criteria must be easy to understand and easy to implement. When Ethernet evolved from a simple shared media with huge broadcast domains to a switched media with segregated broadcast domains, there was a shift in design. The ease of creating a VLAN and assigning users to that VLAN made it commonplace and a function that went without much added work or worry. In the same sense, Shortest Path Bridging allows for the implementation of network virtualization in a true core distribution sense.

The key value propositions for IEEE 802.1aq SPBm include:

-          Standards-based

-          IEEE 802.1aq standard

-          Unmatched Resiliency

-          Single robust protocol with sub-second failover

-          Optimal network bandwidth utilization

-          Simplicity

-          One protocol for all network services

-          Plug & Play deployment reduces time to service

-          Scalability

-          Evolved from Carrier with Enterprise-friendly features

-          Separates infrastructure from connectivity services

-          Flexibility

-          No constraints on network topology

-          Easy to implement virtualization

There are some major features within SPBm that lend themselves well to a scalable and resilient enterprise design. Two major points are as follows:

1). Separation of the Core and the Edge

SPBm implements IEEE 802.1ah ‘MACinMAC’ which provides for a boundary separation between data forwarding methods in the network core versus the edge. It provides for a clear delineation between the normal Ethernet ‘learning bridge’ environment which is required for local area network operations and the SPBm Core network cut-through switching environment where performance and optimal path selection are the key most important criteria. As a result, the use of SPBm creates a core network that creates smaller edge forwarding environments where the MAC tables are effectively isolated. Within the actual SPBm core network itself the only MAC addresses within the forwarding tables are those of the SPBm switches themselves. As a result, the IEEE 802.1aq SPBm Core is very high performance and very scalable. It is also able to utilize multiple forwarding paths and provide for clear delineation between the network core and edge.

2). Virtual Provisioning Fabric

As noted earlier, IEEE 802.1aq evolved from earlier carrier grade implementations for Provider Based Bridging. There are two things that are key to a provider based offering. First, no customer should ever see another customer’s traffic. There needs to be complete and total services separation. Second, there must be a robust and detailed method for Operation and Maintenance (OAM) and Configuration and Fault Management (CFM) which is addressed by IEEE802.1ag and is used by SPBm for those purposes..

The first requirement is addressed by SPBm’s ability to create isolated data forwarding environments in a manner that are similar to VLAN’s in the traditional learning bridge fashion. In the SPBm core there is no learning function required. As such, these forwarding paths provide for total separation and allow for very determinate forwarding to associated resources across the SPBm core. These paths, termed as Instance Service Identifiers or I-SID’s allow for the ability to provision virtual network topologies that can be of a very wide variety of forms.

In addition, due to the established topology of the SPBm domain, the creation of these I-SID’s are provisioned at the edge of the SPBm cloud. There is no need to go into the core to any provisioning to establish the end to end connectivity. This contrasts with normal VLANs which require each and every node to be configured properly.

The figure below shows the dichotomy of these two features and how they relate to the network edge and in this case a distribution layer.

Fig. 6  MAC-in-MAC and I-SID’s within SPBm

As an example, I-SID’s can be used to connect Data Centers together with very high performance cut through dedicated paths for things such as Virtual Machine Migration, Stretch Server Clusters or Data Storage Replication. The figure below illustrates the use of L2 I-SID in this fashion

 Fig. 7. End to end IEEE 802.1aq L2 I-SID providing a path for V-Motion

Additionally, complete Data Center architectures can be built that provide for all of the benefits of traditional security perimeter design but with the benefits full virtualization of the network infrastructure. The figure below shows a typical Data Canter design implemented by inter-connected I-SID’s in a Shortest Path Bridging network. This effectively shows that not only is SPBm an ideal core network technology, it is also an optimal data center bridging fabric.

Fig. 8. Full Data Center Security Zone

Finally, complex L3 topologies can be built on top of SPBm that can utilize traditional routing technologies and protocols or can provide for the networks L3 forwarding requirements by the use of the native L2 link state routing within SPBm provided by IS-IS. The illustration below shows a network topology in which all methods are utilized to provide for a global enterprise design.

Fig. 9  Full end to end Virtualized Network Topology over an IEEE802.1aq cloud

Shortest Path Bridging Services Types

Avaya’s implementation of Shortest Path Bridging provides a tremendous level of flexibility to support multiple service types simultaneously, singly or in tandem.

One of the key advantages of the SPB protocol is the fact that network virtualization provisioning is achieved by just configuring the edge of the network, thus the intrusive core provisioning that other Layer 2 virtualization technologies require is not needed when new connectivity services are added to an SPB network.

Shortest Path Bridging Layer 2 Virtual Services Network (L2 VSN)

Layer 2 Virtual Services Networks are used to transparently extend VLANs through the backbone.  A SPB L2 VSN topology is simply made up of a number of Backbone Edge Bridges (BEB) used to terminate Layer 2 VSNs. The control plane uses IS-IS for forwarding at a Layer 2 level. Only the BEB bridges are aware of any VSN and associated edge MAC addresses while the backbone bridges simply forward traffic at the backbone MAC (B-MAC) level.

Figure 10. L2 Virtual Service Networks

A backbone service Instance Identifier (I-SID) used to identify the Virtual Services Network will be assigned on the BEB to each VLAN. All VLANs in the network sharing the same I-SID will be able to participate in the same VSN.

Shortest Path Bridging Inter-VSN Routing (Inter-ISID Routing)

Inter-VSN Routing allows routing between IP networks on Layer 2 VLANs with different I-SIDs. As illustrated in the diagram below, routing between VLAN 10, VLAN 100 and VLAN 200 occurs on one of the SPB core switches in the middle of the diagram. 

Figure 11. Inter-VSN routing

Although in the middle of the network, this switch provides “edge services” and has I-SIDs and VLANs provisioned on it, and therefore is designated as a BEB switch.  End users from the BEB switches as shown on the right and left of the diagram are able to forward traffic between their respective VLANs via the VRF instance configured on the switch shown.  For additional IP level redundancy, Inter-VSN Routing may also be configured on another switch and both can be configured with VRRP to eliminate single points of failure.

Shortest Path Bridging Layer 3 Virtual Services Network (L3 VSN)

A SPB L3 VSN topology is very similar to a SPB L2 VSN topology with the exception that a backbone service Instance Identifier (I-SID) will be assigned at a Virtual Router (VRF) level instead of at a VLAN level. All VRFs in the network sharing the same I-SID will be able to participate in the same VSN. Routing within a single VRF in the network occurs normally as one would expect.  Routing between VRF’s is possible by using redistribution policies and injecting routes from another protocol, i.e., BGP even if BGP is not used within the target VRF.

Figure 12. L3 Virtual Service Networks

Layer 3 Virtual Service Networks provide a high level of flexibility in network design by allowing IP routing functionality to be distributed among multiple switches without proliferation of multiple router-to-router transit subnets.

SPB Native IP shortcuts

The services described to this point require the establishment of Virtual Service Networks and their associated I-SID identifiers.  IP Shortcuts enables additional flexibility in the SPB network to support IP routing across the SPB backbone without configuration of L2 VSNs or L3 VSNs.

Figure 13. Native IP GRT Shortcuts

IP shortcuts allow routing between VLANs in the global routing table/network routing engine (GRT). No I-SID configuration is used.

Although operating at Layer 2, IS-IS is a dynamic routing protocol.  As such, it supports route redistribution between itself and any IP route types present in the BEB switch’s routing table.  This includes local (direct) IP routes and static routes as well as IP routes learned through any dynamic routing protocol including RIP, OSPF and BGP.

IP routing is enabled on the BEB switches, and route redistribution is enabled to redistribute these routes into IS-IS.  This provides normal IP forwarding between BEB sites over the IS-IS backbone.

 BGP-Based IP VPN and IP VPN Lite over Shortest Path Bridging

Avaya’s implementation of Shortest Path Bridging has the flexibility to support not only the L2 and L3 VSN capabilities and IP routing capabilities as described above, but also supports additional IP VPN types.  BGP-Based IP VPN over SPB and IP VPN Lite over SPB are features supported in the Avaya implementation of Shortest Path Bridging. 

Figure 14. BGP IP VPN over IS-IS

BGP IP VPNs are used in situations where it is necessary to leak routes into IS-IS from a number of different VRF sources.  Additionally, using BGP IP VPNs support over SPB, it is possible to provide hub and spoke configurations by manipulating the import and export Route Target (RT) values. This allows, for example, a server frame in a central site to have connectivity to all spokes, but, no connectivity between the spoke sites. BGP configuration is only required on the BEB sites where the backbone switches have no knowledge of any Layer 3 VPN IP addresses or routes.

Resilient Edge Connectivity with Switch Clustering Support

As earlier described, the boundary between the MAC-in-MAC SPB domain and 802.1Q domain is handled by the Backbone Edge Bridges (BEBs). At the BEBs, VLANs are mapped into I-SIDs based on the local service provisioning.

Figure 15. Resilient edge switch cluster

Redundant connectivity between the VLAN domain and the SPB infrastructure is achieved by operating two SPB switches in Switch Clustering (SMLT) mode. This allows dual homing of any traditional link aggregation capable device into a SPB network. 

Switch Clustering provides the ability to dual home any edge device that supports standards-based 802.1ad LACP link aggregation, Avaya’s MLT link aggregation, EtherChannel or any similar link aggregation method.  With Switch Clustering, the capability is provided to fully load balance all VLANs across the multiple links to the switch cluster pair.  If either link as depicted fails, all traffic will instantly fail over to the remaining link.  Although two links are depicted, Switch Clustering supports LAGs up to 8 ports for additional resiliency and bandwidth flexibility. 

Quality of Service Support and Traffic Policing and Shaping Support

Quality of Service (QoS) is maintained in a SPB network the same way any IEEE based 802.1Q network is operated. Traffic ingressing a SPB domain which is either already 802.1p bit marked (within the C-MAC header), or is being marked by an ingress policy (remarking), is getting its B-MAC header p-bit marked to the appropriate value.

Figure 16. QoS & Policing over SPB

The traffic in the SPB core is scheduled, prioritized and forwarded according to the 802.1p values in the backbone outside packet header. In the case where traffic is being routed at any of the SPB nodes, the IP Differentiated Services DSCP values are taken into account as well.

The number of I-SID’s available in an SPBm domain are virtually limitless (16 million). Additionally, this technology can be effectively extended over many forms of transport such as dark or dim optics, CWDM or DWDM, MPLS L2 pseudo-wires, ATM and others. This means that it can effectively cover vast geographies in its native form and provide all of the virtualization benefits where ever it reaches.

In instances where required however an SPBm domain can effectively interface to a traditional routed WAN by the use of standard interior and border gateway protocols.

Provider Type Services offerings and larger regional topologies

In instances where larger geographic coverage is desired to leverage IEEE 802.1aq and its inherent provisioned core approach the traditional mash topology has difficulty in scaling due to costs in optical infrastructure and point of presence. In these instances ring based topologies make the most sense. IEEE 802.1aq can not only support ring topologies but can also support various interesting iterations such as dual core rings or the more esoteric 3D torus topology which is intended to support very high core port densities.

The next section of this document will discuss the various ring topology options as well as the combination of their use. The diagram below illustrates the basic components for the dual core ring. There are two basic assumptions in the design. First, the core ring topology is populated with only Backbone Core Bridges (BCB’s). This optimizes one of the key traits of Shortest Path Bridging – separation of core and edge. The result is a design of immense scale from a services perspective. Second, all provisioned service paths are applied at the edge in the Backbone Edge Bridges (BEB’s) which provides the interface to the customer edge.

Figure 17. Basic Dual Core components

As we look below at a complete topology we can see that a very efficient design emerges which uses both minimal node and fiber counts as well as effectively leverage on shortest paths across the topology. Each BEB is dual homed back into the ring fabric by SPB trunks. As such there are multiple options for dual homing the BEB node back into the ring topology.

Figure 18.  A Basic Dual Core Ring

An additional level of differentiation can be provided by the use of a dual home active/active mesh service edge. In this type of edge shown below, there are two BEB’s which are trunked together with active/active Inter-Switch Trunks. These two switches then provide a clustered edge that interoperates with any industry standard dual homing trunk method such as MLT or LAG. The end result is a very high level of mesh resiliency directly down to the customer service edge.

Figure 19. Dual Homed Active/Active Mesh Edge

The diagram below shows a dual core ring design that implements various forms of dual homed resiliency. These can range from simple dual homing of the BEB to a very highly resilient inter-area active/active edge design that can provide sub-second failover into the provider cloud. Again, this supports industry standard methods for active/active dual homing of the Ethernet service edge.

 Figure 20. Dual Core Ring with various methods of dual homed resiliency

More complex topologies can be designed when higher densities of backbone core ports are required. The topology below illustrates a 3D torus design that links together triad nodal areas to build a very highly resilient and dense core port capacity ring.

Figure 21. 3D Torus Ring

As the diagram below shows, the basic construct of the 3D torus is fairly simple and is comprised of only six core nodes. The dotted lines show optional SPB trunks to provide enhanced shortest path meshing. With these optional trunks every node is directly connected for shortest path forwarding.

Figure 22. 3D Torus Section

These sections can be linked together to build a complete torus as shown above, or used in a hybrid fashion as shown below to build up or down core port densities as required by subscriber population. The illustration below shows a hybrid ring topology that scales up or down according to population and subscriber density requirements.

Figure 23. Hybrid Ring Topology

As this section illustrates, IEEE 802.1aq is an excellent technology for regional and metropolitan area networks. It allows for scalability and reach as well as a great degree of flexibility in supported topologies. Moreover, these different degrees of scale can be accomplished in the same network without any degree of sacrifice to the overall resiliency of the whole.

Provisioned Virtual Service Networks

As mentioned earlier, IEEE 802.1aq offers several methods of service connectivity across the SPB cloud. In the context of a service offering however, the use of I-SID’s will have a different focus. Rather than a departmental or organizational focus as was used in the above example, here we are concerned with shared service offerings or services separation. As an example, in the area of voice service offerings, a service may be shared in that it is much like the PSTN only over IP. In contrast, a service might be offered for a virtual PBX service for a private company that would expect that service to be dedicated. The figure below shows how IEEE 802.1aq can easily provide the dedicated service paths for both modes of service offering. The PSTN service I-SID offering is shown in green while the private virtual PBX service I-SID is shown in red.

Figure 24.  Shared vs. Dedicated Services

In typical deployment an offering of services might be as follows –

Private Sector – Voice/Shared – Video/Shared – Data/Shared

Business – Voice/Private – Video/Shared – Data/Private

These are of course general and can be customized to any degree. The diagram below shows how the use of IEEE802.1aq I-SID’s allows for the support of both service models with no conflict. Note that the private sector shares a common I-SID for video services with the business sector. Also note that the business sector profile allows for the use of a dedicated virtual PBX service that is private to that business.

Figure 25.  Voice and Video I-SID’s across SPB

Figure 26.  Multiple ‘Service Separated’ data service paths across SPB

The illustration above highlights the data networking services. Note that the private sector is using a shared I-SID (shown in green) much as is done today with DOCSIS type solutions. Note also that the business is using L3 I-SID’s with VRF’s to build out a separate private and dedicated IP topology over the IEEE 802.1aq managed offering. This creates separate and discrete data forwarding environment that are true ‘ships in the night’. They have no ability to support end to end communications unless the routing topology explicitly allows it. As such all of the traditional IT security frameworks such as firewalls and intrusion detection and prevention come into play and are used in a rather traditional fashion to protect key corporate resources. On the private residential space, end point anti-virus & protection as is typical with ISP’s today.

IP Version 6 Support

Introducing new technology is always a move into the unknown. IPv6 is no different. While the technology has been under development so some time (over ten years), there has been no great impetus that has been the motivation for large scale adoption. This is changing now that IANA/ARIN has announced that the last contiguous block of IPv4 addresses has been sold. Now it is down to non-contiguous blocks and recycling of address blocks. These efforts will not provide any significant extension to the availability of IPv4 addresses. With these events, many organizations are now actively investigating how IPv6 can be deployed into their networks.

This section is intended to provide an overview of a tested topology over shortest path bridging (IEEE 802.1aq) environments for the distribution of globally routable IPv6 addressing using L2 VSN’s and inter-VSN routing.

The high level results of the work demonstrate that an enterprise can effectively use SPB to provide for the overlay of a routed IPv6 infrastructure that is incongruent to the existing IPv4 topology. Furthermore, with IPv4 default gateways resident on the L2 VSN’s, dual stack end  stations can have full end to end hybrid connectivity without the use of L3 transition methods such as 6to4, ISATAP, or Teredo. This results in a clean and simple implementation that allows for the use of allocated globally routable IPv6 addresses in a native fashion.

IPv6 in General –

IPv6 is the next generation form of IP addressing. Replacing IPv4 it is intended to provide greatly enhanced address space as well as end to end transparency which was becoming more and more difficult by the increasing use of Network Address Translation (NAT) in IPv4. NAT was created in order to provide for the use of ‘private’ IPv4 addressing within an organization and then allow for a gateway device to interface out to the public Internet. Even this technology however could not forestall the unavoidable event that occurred earlier this year contiguous blocks of IPv4 addresses have run out.

Currently, there are address recycling efforts that will provide some reprieve but in the immanent future even this effort will be exhausted.

These events have caused a recent surge in the interest in IPv6. Many enterprises that had it on the back burner are now taking a new look at this technology and the requirements that need to met for their organizations to deploy it. For the first time investigator this can be a daunting task. Beyond the knowledge of IPv6 itself, one needs to learn all of the methods required in order to co-exist in an IPv4 network environment. This is a strict requirement because no one will completely forklift their complete communications environment and even if they could there are issues with contact to the outside world that need to be addressed. The reason for this is that the IPv6 suite is NOT directly backwards compatible to IPv4. This complication has caused quite a bit of effort within the IETF to resolve. There are a number of RFC’s, drafts as well as deprecated drafts that cover a wide variety of translation or transition methods. Each has its own set of complications and security or resiliency issues that need to be dealt with. At the end of the day, most IT personnel walk away with a headache and wish for the good old days of just IPv4.

During the time since IPv6 was first introduced different schools of thought evolved as to how this co-existence between IPv4 and IPv6 could be addressed. Network and Port Translation (NAT-PT) came into vogue but has since faded off into deprecation as the approach has largely proved to be intractable. Other methods have stayed and even become ‘default’. As an example, all Microsoft OS’s running IPv6 run 6to4, ISATAP and Teredo tunneling methods.

So it has become clear. One school has won out and that school of thought is… dual stack in the end stations and tunneling across the IPv4 network to tie IPv6 islands together. These methods work, but as I pointed out earlier, they all have complications and issues that need to be dealt with.

If one looks at the evolution long enough though something else becomes apparent. If you could provide the paths between IPv6 islands by Layer 2 methods, things like 6to4, ISATAP and Teredo are no longer required. Furthermore, without these methods an enterprise is free to use formally allocated globally routable address space. The only requirement for the dual stack host is that they have clear default routes for both IPv6 and IPv4. With typical VLAN based networks however, this design while feasible does not scale and quickly becomes intractable due to the complications of tagged trunk design within the network core. With the evolution of Shortest Path Bridging (IEEE 802.1aq) this scalable layer two method is now available. The rest of this solution guide will describe the test bed environment and then discuss ramifications that this work has on larger network infrastructures.

The IPv6 over SPB Example Topology –

The figure below shows the minimal requirements for a successful hybrid IPv6 deployment over shortest path bridging. As can be seen the requirements are fairly concise and simple. You require an SPB Virtual Service Network configured which is then associated with edge VLAN’s. These VLAN’s will host dual stack end stations.

Addtionally, this VSN will need to attach to default IPv6 and IPv4 default gateways. Again, this would occur by the use of edge VLAN’s that interface to the relevant devices.

Figure 27. Required elements for a native hybrid IPv6 deployment over SPB

So as one can see the requirements are straightforward and easy to understand. We implemented the following topology in a lab to demonstrate the proposed configuration.

The diagram below illustrates this topology in a simplified form for clarity. 

 Figure 28. Native IPv6 Dual Stack over L2 VSN Test bed

In the test bed we implemented a common VSN that would support the IPv6 deployment. This was for simplicity only. More complicated IPv6 routed topologies can easily be achieved by using inter-VSN routing. Examples later in the brief will be shown where this is illustrated. In the lab we created VLAN ID 500 at the three different key points at the edge of the SPB domain. A Virtual Service Network was created within the SPB domain (also using 500 as its identifier) that ties the different VLAN’s together. At one edge VLAN a Win7 end station running dual stack had the IPv4 address of 10.40.99.2 and the IPv6 address of 3000::2. For IPv4 the end stations default gateway was 10.40.99.1 and for IPv6 the Default Gateway was 3000::1. The IPv6 default gateway is also attached to VLAN 500 and is able to provide directly routable paths in and out of the VSN. Additionally, the IPv4 default gateway is also attached and reachable as well. The dual stack end station enjoys end to end hybrid connectivity to both IPv6 and IPv4 environments without the use of any L3 transition method. In the topology shown in figure 3, we show that from the dual stack end stations perspective, there is complete hybrid connectivity and available routed paths to both IPv4 and IPv6 environments. Due to the fact that formally allocated global addressing is used there is connectivity out into INET2 to native IPv6 resources.

Figure 29. Dual Stack end stations perspective on default routed paths

The ramifications on larger IPv6 deployments

One of the major drawbacks of L3 transition methods for IPv6 is that they bind the IPv6 topology to IPv4. Many find this as undesirable. After all, why implement a new globally routed protocol and then lock it down to an existing legacy topology? As a result, it was realized very early on that if you could run IPv6 as ships in the night with IPv4 it would be a very good solution. The problem with this was that the only method to accomplish this was by the use of VLAN’s and tagged trunks or with routed overlays. As a result, while the previous test bed shown in figure 2 was feasible and provable, the approach quickly suffers from complexity in larger topologies and does lend itself well to scale.

With Shortest Path Bridging these issues are vastly simplified making this approach tractable on an enterprise scale. The reason for this is that the IPv6 deployment becomes an overlay L3 environment that rides on top of SPB. As such, there is no need to make detailed configuration changes to the network core to deploy it. This original ‘ships in the night’ vision can now be realized in real world designs.

The diagram below shows a large network topology that interconnects two data centers. The topology in blue shows the IPv6 native dual stack deployment. The topology in green shows the IPv4 legacy routed environment. Note that while there are common touch points between the two environments for legacy dual stack IPv4 use, the two IP topologies are quite independent of one another.

Figure 30. Totally Independent IP topologies

In Summary –

This document has provided a review of active/active mesh network topologies and the significant benefits that they bring to an overall network design. With networking speeds now at plus 10 Gb/s it is no longer sufficient to have very high speed expensive switch ports sitting in a totally passive state waiting for a network failure. It is also no longer sufficient to tolerate failover times in the range of seconds or even tenths or hundreths of seconds. The amount of data loss and the performance impacts are just too serious. Active/active mesh networking addresses this by providing for multiple load sharing paths across the network topology. Additionally, due to the active nature of the trunking method, SMLT can very easily provide for failovers in the subsecond range. As a note, recent testing of Avaya’s 3^rd generation of SMLT reliably shows failovers in the range of 6 ms. This is practically instantaneous from the persapective of the overall network. This failover speed is unrivaled in the industry and is a testament to Avaya’s dedication to this technology space.

Additionally, newer active/active mesh technologies are being introduced such as IEEE 802.1aq Shortest Path Bridging – a key foundational component of Avaya’s VENA framework that promise to take active/active mesh network topologies into a new era of scale and flexibility never before realized with switched Ethernet topologies. The provisioned virtual network capability of VENA allows for one touch provisioing of the network serivce paths with zero touch requirements to the transport core. This new innovation not only vastly simplifies administration and reduces configuration errors. It can provide for dramatic improvements in IT OP/EX costs in that changes that would normally take hours are brought down to minutes with an exponential reduction in the probablity for error.

In addition, this paper has shown that this new addition to active mesh networking is totally complatible and complimentary with older active/active mesh switched Ethernet topologies such as SMLT. The results of the combination are a flexible core meshing technology that allows for almost unlimited permutations of topologies and a very highly resilient dual homed edge with sub-second failover.

Another more mundane but equally important aspect of Avaya’s SPBm offering is that it can be easily migrated to within their existing Ethernet Routing Switch 8600. The result of this upgrade is to make it the equivalent of an Ethernet Routing Switch 8800 which can participate in an SPBm domain as either a Backbone Edge Bridge (BEB) or a Backbone Core Bridge (BCB), including all service modes detailed earlier in this article. This mean that an existing ERS 8600 customer can implement the technology without the needs for a forklift upgrade.

Even when considering networks with alternative vendors, Avaya’s SPBm VENA framework – due to it’s strict compliance to IEEE 802.1aq and other IEEE standards – allows for the seamless introduction of SPBm into the network as a core distribution technology with minimal disruption to the network edge. Additionally, network edges that are Spanning Tree based today because of core networking limitations can then move to implement the active/active dual homing model spoken to earlier by the use of LAG or MLT at the edge, both of which are widely supported throughout the industry.

The end result is a technology that brings immense value.  It is easy to implement in both new and existing networks, and migration can be virtually seamless.

Could it be that the days of spanning tree have finally passed?

I would like to extend both credit and thanks to my esteemed Avaya colleagues, Steve Emert and John Vant Erve for both input and use of facilities for solution validation.

Communications technologies are evolving rapidly. This pace of evolution, while slowed somewhat by economic circumstances, still moves forward at a dramatic pace. This is indicative to the fact that while the ‘bubble’ of the 1990’s is past, society and business as a whole has arrived to the point where communications technologies and their evolution are a requirement for proper and timely interaction with the human environment.

This has profound impact on a number of foundations upon which the premise of these technologies rest. One of the key issues is that of the Internet Protocol, commonly referred to simply as ‘IP’. The current widely accepted version of IP is version 4. The protocol, referred to as IPv4 has served as the foundation to the current Internet since its practical inception in the public arena. As the success of the Internet attests, IPv4 has performed its job well and has provided the evolutionary scope to adapt over the twenty years that has transpired. Like all technologies though IPv4 is reaching the point where further evolution will become difficult and cumbersome if not impossible. As a result, IPv6 was created as a next generation evolution to the IP protocol to address these issues.

Many critics cite the length of time that IPv6 has been in development. It is after all, a project that has over a ten year history in the standards process. However, when one considers the breadth and complexity of the standards involved a certain maturity is conveyed that the industry can now leverage upon. The protocol has evolved significantly since the first proposals for its predecessor, IPng. Many or most of the initial shortcomings and pitfalls have been addressed to the point where actual deployment is a very tractable proposition. Along this evolution several benefits have been added to the suite that directly benefits the network staff and end user populous. Some these benefits are listed below. Note that this is not an inclusive list.

• Increased Addressing Space

• Superior mobility

• Enhanced end to end security

• Better transparency for next generation multimedia applications & services

Recently, there has been quite a bit of renewed activity and excitement around IP version 6. The recent announcements by the United States Federal Government for IPv6 deployment by 2008 and the White House Civilian Agency mandate by 2012 has helped greatly to fuel this. Also many, if not most of the latest projects being implemented by providers in the Asia Pacific regions are calling for mandatory IPv6 support. Clearly the protocols’ time is coming. We are seeing the two vectors of maturity and demand meeting to result in market and industry readiness.

There is a cloud on this next generation horizon however. It is known as IPv4. From a practical context all existing networks are either based on or in some way leverage IPv4 communications. Clearly, if IPv6 is to succeed, it must do so in a phased approach that allows hybrid co-existence with it. Fortunately, many in the standards community have put forth transition techniques and methodologies that allow for this co-existence.  A key issue to consider in all of this is that the benefits of IPv6 are somewhat (sometimes severely) compromised by their usage. However, like all technologies, if usage requirements and deployment considerations are considered prior to implementation the proposition is realistic and valid.

Setting the Foundation

IPv6 has several issues and dependencies that are common with IPv4. However, the differences in address format and methods of acquisition require modifications that need to be considered to them. Much of the hype in the industry is on the aspects of support within the networking equipment. While this is of obvious importance, it is critical to realize that there are other aspects that need to be addressed to assure a successful deployment.

The first Block – DNS & DHCP Services

While IPv6 supports auto-configuration of addresses, it also allows for managed address services. DNS does not require, or from a technical standpoint require DHCP, but the two are often offered in same product suite.

When considering the new address format (128 byte colon delimited hexadecimal), it is clear that it is not human friendly. A Domain Name System (DNS) infrastructure is needed for successful coexistence because of the prevalent use of names (rather than addresses) to refer to network resources.  Upgrading the DNS infrastructure consists of populating the DNS servers with records to support IPv6 name-to-address and address-to-name resolutions. After the addresses are obtained using a DNS name query, the sending node must select which addresses are used for communication. This is important to consider both from the perspective of the service (which address is offered as primary) and the application (which address is used). It is obviously important to consider how a dual addressing architecture will work with naming services. Again, the appropriate due diligence needs to be done by investigating product plans but also in limited and isolated test bed environments to assure predictable and stable behavior with the operating systems as well as the applications that are being looked at.

As mentioned earlier, DHCP services are often offered in tandem with DNS services in many products. In instances where IPv6 DHCP services are not supported, but DNS services are, it is important to verify that it will work with standard auto-configuration options.

The second Block – Operating Systems

Any of the operating systems that are being considered to use in the IPv6 deployment should be investigated for compliance and tested so that the operation staff are familiar with any new processes or procedures that IPv6 will require. Tests should also occur between the operating systems and the DNS/DHCP services using simple network utilities such as ping and FTP to assure that all of the operating elements, including the operating systems interoperate at the lowest common denominator of the common IP applications.

It is important to test behaviors of dual stack hosts (hosts that support both IPv4 and IPv6). Much of the industry supports a dual stack approach as being the most stable and tractable approach to IPv6 deployments. Later points in this article will illustrate why this is the case.

The third Block – Applications

Applications should be considered first off to establish the scope of operating systems and the extent to which IPv6 connectivity needs to be offered. Detailed analysis and testing however should occur last after the validation of network services and operating systems. The reason for this is that the applications are the most specific testing instances and strongly depend on the stable and consistent operation of the other two foundation blocks. It is also important to replicate the exact intended mode of usage for the application so that the networking support staff are aware of any particular issues around configuration and or particular feature support. On that note, it is important to consider if there are any features that do not work in IPv6 and what impact that they will have on the intended mode of usage for the application. Finally, considerations need to be made for dual stack configurations and how precedence is set for which IP address to use.

The forth Block – Networking Equipment

Up to this point all of the validation activity referred to can be performed on a ‘link local’ basis. As a result a typical layer two Ethernet switch would suffice. A real world deployment requires quite a bit more however. It is at this point where the networking hardware needs to be considered. It is important to note that many pieces of equipment, particularly layer two type devices will forward IPv6 data. If expressed management via IPv6 is not a requirement then these devices could be used in the transition plans provided they are used appropriately in the network design.

Other devices such as routers, layer three switches, firewalls and layer 4 through 7 devices will require significant upgrades and modification to meet requirements and perform effectively. Due diligence should be done with the network equipment provider to assure that requirements are met and timelines align with project deployment timelines.

As noted previously in the other foundation blocks, dual stack support is highly recommended and will greatly ease transition difficulties as will be shown later. With networking equipment things are a little more complex in that in addition to meeting host system requirements for IPv6 communications of the managed element, the requirements of data forwarding, route computation and rules bases need to be considered. Again, it is important to consider any features that will not be supported in IPv6 and the impact that this will have on the deployment. The figure below illustrates an IPv6 functional stack for networking equipment.

Figure 1. IPv6 network element functional blocks

As shown above, there are many modifications that need to occur at various layers within a given device. The number of layers as well as the specific functions implemented within each layer is largely determined by the type of networking element in question. Simpler layer two devices are only required to provide dual host stack support primarily for management purposes, products like routers and firewalls will be much more complex. When looking at IPv6 support in equipment it makes sense to establish the role that the device performs in the network. This role based approach will best enable an accurate assessment of the real requirements and features that need to be supported rather than industry or vendor hype.

The burden of legacy – Dual stack or translation?

The successful deployment of IPv6 will strongly depend on a solid plan for co-existence and interoperability with existing IPv4 environments. As covered earlier, the use of dual stack configurations whenever possible will greatly ease transition. Today this is an issue for any device supporting IPv6 to speak to IPv4 devices. As time moves on however, the burden will shift to the IPv4 devices to speak to IPv6 devices. As we shall see there are only a certain set of applications that require dual stack down to the end point. Most client server applications will work fine in a server only dual stack environment supporting both IPv4 and IPv6 only clients as shown in the figure below.

Figure 2. A dual stack client server implementation

As shown above both IPv4 and IPv6 client communities have access to the same application server each served by their own native protocol. In the next figure however we see that there are some additional complexities that occur with certain applications and protocols such as multimedia and SIP. In the illustration below we see that there are not only client/server dialogs but client to client dialogs as well. In this instance, at least one of the clients needs to support a dual stack configuration in order to establish the actual media exchange.

Figure 3. A peer to peer dual stack implementation

As shown above, with one end point supporting a dual stack configuration and the appropriate logic to determine protocol selection, end to end multimedia communications can occur. Note that this scenario will typically be lieu of IPv6 only devices as these will become more prevalent over time.

There are many benefits to the dual stack approach. By analyzing applications and mandating dual stack usage, a very workable transition deployment can be attained.

There are arguments that address space, one of the primary benefits of IPv6 is drastically compromised by this approach. After all, by using dual stack you do not remove any IPv4 addresses. In fact you are forced to add IPv4 addresses to accommodate an IPv6 deployment. The truth to this is directly related to the logic of the approach in deployment. By understanding the nature of the applications and giving preference to the innovative (Ipv6 only) population these arguments can be mitigated. The reason for this is that you are only adding IPv6 addresses to existing IPv4 hosts that require communication with IPv6. If this happens to be the whole IPv4 population, so be it. There are plenty of IPv6 addresses to go around! As new hosts and devices are deployed they should be IPv6 only preferentially, or dual stack if required but NOT IPv4 only.

An alternative to the dual stack approach is the use of intermediate gateway technologies to translate between IPv6 and IPv4 environments. This approach is known as NAT-PT. The diagram below illustrates a particular architecture for NAT-PT usage that will provide for the multimedia scenario used previously.

Figure 4. Translation Application Layer Gateway

In this approach the server is supporting a dual stack configuration and is using native protocols to support the client/server dialogs to each end point. Each end point is single stack, one is IPv4 the other is IPv6. In order to establish end to end multimedia communications, there is an intermediate NAT-PT gateway function that provides for the translation between IPv4 and IPv6. There are many issues and caveats with this approach. These can be researched in IETF records.  As a result to this, there is work towards deprecating NAT-PT to an experimental status.  It should be noted that a recent draft revision has been submitted so it is worth keeping on the radar map.

Tunnel Vision

There has been quite a bit of activity around another set of transition methods known as tunneling. In a typical configuration, there are two IPv6 sites that require connectivity across an IPv4 network. The use of tunneling would involve the encapsulation of the IPv6 data frames into IPv4 transport. All IPv6 traffic between the two sites would traverse this IPv4 tunnel. It is a simple and elegant, but correspondingly limited approach that provides co-existence not necessarily interoperability between IPv4 and IPv6. In order to achieve this we need to invoke one of the approaches (dual stack vs. NAT-PT) discussed earlier.  Tunneling by itself only provides the ability to link IPv6 sites and networks over IPv4.

This is a very important point. A point that, if taken to its logical conclusion, indicates that if the network deployment is appropriately engineered, the use of transition tunneling methods can be greatly reduced and controlled, if not eliminated. Before we take this course in logic however it is important to consider the technical aspects of tunneling and why it is something that needs to be thought out prior to using.

The high level use of tunneling is reviewed in RFC 2893 for those interested in further details. Basically there are two types of tunnels; the first is called configured tunnels. Configured tunnels are IPv6 into IPv4 tunnels that are set up manually on a point to point basis. Because of this, configured tunnels are typically used in router to router scenarios. The second type of tunnels is automatic. Automatic tunnels use various methods to derive IPv4/IPv6 address mappings on a dynamic basis in order to support an automatic tunnel setup and operation. As a result, automatic tunnels can be used not only for router to router scenarios but for host to router or even host to host tunneling as well. As a result we are able to build a high level summary table of the major accepted tunneling methods.

Method                Usage                               Risk

Configured          Router to router                 Low

Tunnels

Automatic           Router to router/             Medium

6 to 4                  Host to router

Automatic           Host to host                      High

ISATAP

With out going into deep technical detail on each automatic tunneling methods behavior, we can assume that there is some sort of promiscuous behavior that will activate the tunneling process on recognition of a particular pattern (IP packet type 41 (IPv6 in IPv4)). This promiscuous behavior is what warrants the increased security risk associated with the automatic methods. RFC 3975 goes into detail on the security related issues around automatic tunneling methods. At a high level there is the ability for Denial of Service attacks on the tunnel routers as well as the ability to spoof addresses into the tunnel for integrity breach. The document goes into recommendations on risk reduction practices but they are difficult to implement and maintain properly.

An effective work around to these issues is to use IPSEC VPN branch routing over IPv4 to establish secure encrypted site to site connectivity and then running the automatic tunneling method inside the IPv4 IPSEC tunnel.

The figure below shows a scenario where two 6 to 4 routers have a tunnel set up to establish site to site connectivity inside an IPv4 IPSEC VPN tunnel. With this approach any IP traffic will have site to site connectivity via the VPN branch office tunnels. The IPv6 hosts would have access to one another via the 6 to 4 tunnels. Any promiscuous activity required by 6 to 4 can now be used with relative assurances of integrity and security. The drawback to this approach is that additional features or devices are required to complete the solution.

Figure 5. Using Automatic Tunneling inside IPv4 IPSec VPN

The primary reason for using transition tunnel methods is to transport IPv6 data over IPv4 networks. In essence, the approach ties together islands of IPv6 across IPv4 and allows for connectivity to the IPv6 network.  If we follow this logic, then the use of transition tunneling can be reduced or even eliminated by getting direct connectivity to the IPv6 Internet by at least one IPv6 enabled router in a given organizations network. The figures below illustrate the difference between the two approaches. In the top example, the organization does not have direct access to the IPv6 Internet. As a result transition tunneling must be used to attain connectivity. In the lower example, the organization has a router that is directly attached to the IPv6 Internet. As a result there is no need to invoke transition tunneling. By using layer two technologies such as virtual LAN’s IPv6 hosts can acquire connectivity to the IPv6 dual stack native router.

Figure 6. Using transition tunneling to extend IPv6 connectivity

Figure 7. Using L2 VLAN’s to extend IPv6 connectivity

Within the organization – Use what you already have

As we established by providing direct connectivity to the IPv6 Internet the use of transition tunneling can be eliminated on the public side. Within the organization prior to implementing transition tunneling it makes sense to review the existing methods that may already exist in the network to attain connectivity.

All of the issues in dealing with IPv6 transition revolve around the use of layer 3 approaches. By using layer 2 networking technologies, transparent transport can be provided. There are multiple technologies that can be used for this approach. Some of these are listed below:

• Optical Ethernet
• Ethernet Virtual LAN’s
• ATM
• Frame Relay

As listed above there are many layer two technologies that can be used to extend IPv6 connectivity within an organizations network.

Virtual LAN’s can be used to extend link local connectivity to IPv6 enabled routers in a campus environment. The data will traverse the IPv4 network with out the complexities of layer 3 transition methods. For the regional and wide area, optical technologies can extend the L2 virtual LAN’s across significant distances and geographies again with the goal of reaching an IPv6 enabled router. Similarly, traditional L2 WAN technologies such as ATM and frame relay can extend IPv6 local links across circuit switched topologies. As the diagram above illustrates, by placing the IPv6 dual stack routers strategically within the network and interconnecting them with L2 networking topologies, an IPv6 deployment can be implemented that co-exists with IPv4 without any transition tunnel or NAT-PT methods.

The catch is of course that these layer two paths can not traverse any IPv4 only routers or layer 3 switches. As long as this topology rule is adhered to this simplified approach is totally feasible. By incorporating dual stack routers, both IPv4 and IPv6 Virtual LAN boundaries can effectively be terminated and in turn propagated further with virtual LAN’s or other layer two technologies on the other side of the routed element. A further evolution on this is to use policy based virtual LAN’s that determine membership according to IP version type of the data received on a given edge port. As the figure below illustrates, dual stack hosts will have access to all required resources in both protocol environments.

Figure 8. Using Policy Based VLAN’s to support dual stack hosts

In essence, where dual stack capability is provided end to end, layer three transition methods can be avoided all together. While it is unlikely that this can be made to occur in most networks, such logic can greatly reduce any layer three transition tunnel usage. By taking additional considerations regarding application network behaviors and characteristics as noted in the beginning of this article the use of intermediate protocol and address translation methods like NAT-PT can also be mitigated.

In conclusion

This article was written to clarify deployment issues for IPv6 with a particular focus on interoperability and co-existence with IPv4. A step by step summary of the deployment considerations can be now summarized as follows:

1). Build the foundation

There are four basic foundation blocks that need to be established prior to deployment consideration. Details on each particular foundation block are provided. In summary they are:

1). DNS/DHCP services

2). Network Operating Systems

3). Applications

4). Network Equipment

As pointed out several times, plan for dual stack support wherever possible in all of the foundation blocks. Such an approach will greatly ease the transition issues around deployment. Ongoing work in multiple routing and forwarding planes such as OSPF-MT (http://www.ietf.org/internet-drafts/draft-ietf-ospf-mt-04.txt)  and Multi-protocol BGP (MBGP) may have beneficial and simplifying merits to interconnect dual stack routing elements and exclusively identify them and build forwarding overlays or route policies based on the traffic type (IPv4 vs. IPv6). While the OSPF-MT work is in preliminary draft phases it has very strong merits in that it can in combination with MBGP effectively displace MPLS type approaches to accomplish the same goal. Again, no transition methods would be required within the OSPF-MT boundary as long as overlay routes exist between the dual stack routing elements.

2). Establish connectivity

Once the foundations have been provided for the next step is to establish how connectivity will be made between different sites. Assuming that dual stack routers are available, it makes sense to closely analyze campus topologies and establish methods that connectivity can be established in concert with layer two networking technologies. Once all available methods have been exhausted and it is clear that one is dealing with an IPv6 ‘island’. It is at this point where one should look at using one of the IPv6 transition tunneling methods with configured tunneling being the most secure and conservative approach and is appropriate for this type of site to site usage.. Host to router tunneling may have valid usage in remote access VPN applications, particularly where local Internet providers do not offer IPv6 networking services. Host to host tunneling applications should be used only in initial test bed or pilot environments and because of manageability and scaling issues is not recommended for general practice usage.

To connect sites across a wide area network, layer two circuit switched technologies such as frame relay and ATM can extend connectivity between the dual stack enabled sites. In some next generation wide area deployments, layer two virtual LAN’s can be extended across RPR optical cores to accomplish the end to end connectivity requirements. Again, only after all other options have been exhausted should the use of IPv6 transition tunneling methods be entertained.

At this point, a dual stack native mode deployment has been achieved with only the minimal use of tunneling methods. It is only at this point that the use of any NAT-PT functions should be entertained to accommodate any applications that do not comply to the deployment. It is strongly urged that such an approach be used in a very limited form and be relatively temporary in the overall deployment. Timelines should be established to move away from the temporary usage by incorporating a dual stack native approach as soon as feasible.

3). Test, test, test

As noted at several points throughout this article testing is critical to deployment success. The reason for this is that requirements are layered and they are interdependent. Consequently, it is important to validate all embodiments of an implementation. Considerations need to be made according to node type, operating system, application as well as any variations that need to be considered for legacy components. It is like the great law of Murphy, it is the implementation that you do not test that will be the one to have the problems.

There is a very good reason why cloud storage has so much hype. It simply makes sense. It has an array of attractive use case models. It has a wide range of potential scope and purpose making it as flexible as the meaning of the bits stored. But most importantly, it has a good business model that has attracted some major names into the market sector.

If you read the blog posts and articles, most will say that Cloud Storage will never be accepted due to the lack of security & accountability. The end result is that many CISO’s & CIO’s have decided that it is just too difficult to prove due diligence for compliance. As a result, they have not widely embraced the cloud model. Now while this is correct, it is not totally true. As a matter of fact most folks are actually using Cloud Storage within their environment. They just don’t equate it as such. This article is intended to provide some insight into the use models of SaaS as well as some of the technical and business considerations that need to be made in moving to a SaaS environment.

Types of SaaS Clouds

It is commonly accepted that there are two types of clouds; public and private. It is the position of this architect that there are in reality three major types of clouds and a wide range of manifestations of them. There are reasons for this logic and the following definitions will clarify why.

Public SaaS Clouds

Public clouds are clouds that are provided by open internet service providers. They are truly public in that they are equally available to anyone who is willing to put down a credit card number and post data to the repository. Examples of this are Google, Amazon & Storage Planet. While this is a popular model, as attested by its use, many are saying the honeymoon is fading along with issues of accountability, reports of lost data and lack of assurances for security and integrity of content.

Semi- Private SaaS Clouds

These are clouds that are more closed in that they usually require some sort of membership or prior business subscribership. As a result the service is typically less open to the general public. Also, the definition of semi-private can have a wide range of embodiments. Examples are, network service providers like cable and telco companies, then slightly more closed might be an educational clouds for higher education to store, post and share vast quantities of content; finally the most closed would be government usage where say in the example of a county that provides a SaaS cloud service to the various agencies within its area of coverage.

Private SaaS Clouds

These are the truly private SaaS services that are totally owned and supported by a single organization. The environment is totally closed to the outside world and access is typically controlled with the same level of diligence as corporate resource access. The usual requirements are that the user has secure credentials and his department is accounted for usage by some of type of cost center.

As indicated earlier these can occur in a variety of embodiments and in reality there is no hard categorization between them. Rather a continuum of characteristics that range from truly private to truly public.

While placing data up into a truly public cloud would cause most CISO’s and CIO’s to cringe, many are finding that semi-private and private clouds are totally acceptable in dealing with issues of integrity, security and compliance. Concern about security and integrity of content is one thing. Another more teasing issue is knowing exactly where your data is in the cloud. Is it in New York? California? Canada? Additionally, if the SaaS provider is doing due diligence in protecting your data then they are replicating it to a secondary site. Where is that? India? As you can see in a totally public cloud service there are a big set of issues that prevent large scale serious use. Additionally, often performance is a real issue. This is particularly the case for critical data or for system restores, when the disappointed systems administrator finds that it will be a day and a half before the system is back on line and operational. These are serious issues that are not easily addressable in a true public cloud environment. Semi-private and Private Clouds on the other hand can often answer these requirements and can provide fairly solid reporting about the security and location of posted content.

The important thing to realize is that it is not all or nothing. A single organization may use multiple clouds for various purposes, each with a different range of scope and usage. As an example, the figure below shows a single organization that has two private clouds one of which are used exclusively by a single department and one of which spans the whole organization. Additionally, that same organization may have semi-private clouds that are used for B2B exchange of data for use in partnerships, channel relationships, etc. Then finally, the organization may have an e-Commerce site that provides a fairly open public cloud service for its customer and prospect communities.

Figure 1. Multiple tiered Clouds

If you really boil it down, you come to a series of tiered security environments that control what type of data gets posted, by whom and for what purpose. Other issues include data type and size as well as performance expectations. Again, in a Semi-private to private usage model these issues can effectively be addressed in a fashion that satisfies both user and provider. The less public the service, the more stringent the controls for access and data movement and the tighter the security boundaries with the outside world.

It is for this reason that I think truly public SaaS clouds have too much stacked against them to be taken as a serious tool for large off site data repositories. Rather, I think that organizations and enterprises will more quickly embrace semi-private and private Cloud storage because of the more tractable environment to address the issues mentioned earlier.

There are also different levels of SaaS offerings. These can vary in complexity and offered value. As an example, a network drive service might be handy for storing extra data copies but might not be too handy as a tool for disaster recovery. As a result, most SaaS offerings can be broken into three major categories.

• Low level – Simple Storage Target

–        Easy to implement

–        Low integration requirements

–        Simple network drive

• Mid level – Enhanced Storage Target

–        VTL or D2D

–        iSCSI

–        Good secondary ‘off-site’ use model

• High level – Hosted Disaster Recovery

–        VM failover

–        P2V Consistency Groups

–        Attractive to SMB sector

As one moves from one level to the next the need for more control and security becomes more important. As a result, the higher the level of SaaS offering the more private it needs to be in order to satisfy security and regulatory requirements.

The value of the first Point of Presence in SaaS

As traffic leaves a particular organization or enterprise it enters either a private WAN and at some point there is boundary to the public Internet. Often these networks are depicted as clouds. We of course realize that there is in reality a topology of networking elements that handle the various issues of data movement. These devices are often switches or routers that operate at L2 or L3 and each imposes a certain amount of latency to the traffic as it moves from one point to another. As are result, the latency profiles to access data in a truly public SaaS becomes longer and less predictable due to increasing variables. The figure below illustrates this effect. As data traverses across the Internet it intermixes with other data flows at the various points of presence where these network elements route and forward data.

Figure 2. Various ‘points of presence’ for SaaS

In a semi-private or a private cloud offering, the situation is much more controlled. In the case of a network provider, they are the very first point of presence or ‘hop’ that their customer’s traffic crosses. It only makes sense that hosting a SaaS service at that POP will offer significantly better and more controlled latency and as a result far better throughput than will a public cloud service somewhere on the network. Also consider that the bandwidth of the connection to that first POP will be much higher than the average aggregate bandwidth that would be realized to the public storage provider on the Internet. If we move to a private cloud environment such as that hosted by a University as a billed tuition service for its student population, very high bandwidth can be realized with no WAN technologies involved. Obviously, the end to end latency in this type of scenario will be minimal when compared to pushing the data. This in addition to the security and control issues mentioned above will in the opinion of the author result in the dramatic growth in semi-private and private SaaS.

Usage models for SaaS

Now that we have clarified the issues of how SaaS can be embodied, what would someone use it for? The blatant response of ‘to store data stupid’ is not sufficient. Most certainly that is an answer, but it turns out that the use case models are much more varied and interesting. At this point, I think that it is fruitful to discern between two major user populations – Residential & Business, with business including education and government institutions. The reason for the division is the degree of formality in usage. In most residential use models, there are no legal compliance issues like SOX or HIPPA to deal with. There may be confidentiality and security issues but as indicated earlier these issues are easier to address in a semi-private or private SaaS.

Business and Institution use models

Virtual Tape Library SaaS

The figure below illustrates a simple VTL SaaS topology. The basic premise is to emulate a physical tape drive across the network with connectivity provided as an iSCSI target to the initiator, which is the customer’s backup software. With the right open system VTL, the service can be as easy as a new iSCSI target that is discovered and entered into the backup server. With no modifications to existing practices or installed software, the service matches well with organizations that are tape oriented in practice and are looking for an effective means of secondary off site copies. Tapes can be re-imported back across the network to physical tape if required in the future.

Figure 3. A simple VTL SaaS

D2D SaaS

Disk to disk SaaS offerings basically provide an iSCSI target of a virtual disk volume across the network. In this type of scenario the customers existing backup software simple points to the iSCSI target for D2D backup or replication. Again, the benefit is that because the volume is virtualized and hosted, it effectively addresses off site secondary data store requirements. In some instances that may require CPE, it can even be used in tandem with next generation technologies like continuous data protection and data reduction methods, which moves towards the Hosted Disaster Recovery end of the spectrum. The figure below shows a D2D SaaS service offering with two customers illustrated. One is simply using the service as a virtual disk target. The other has an installed CPE that is running CDP and data reduction resulting in a drastic improvement on the overall required bandwidth.

Figure 4. A D2D SaaS

Collaborative Share SaaS

Another use model that has been around for a long time is collaborative sharing. I say this because I can remember better than ten years ago placing a file up on an FTP server and then pasting the URL into an email that went out to a dozen or so recipients. Rather than plug up the email servers with multiple copies of large attachments. Engineers have a number of things in common regardless of discipline. First is collaboration. A close second though is the amount of data that they typically require in order to collaborate. This type of model is very similar to the FTP example except that it is enhanced with a collaborative portal that might even host real time web conferencing services. The storage aspect, though of primary importance to the collaboration is now a secondary supporting service that is provided in a unified fashion out to the customer via a web portal. The figure below shows an example of this type of service. Note that in reality there is no direct link between the SaaS and the Web Conferencing application. Instead they are unified and merged by a front end web portal that the customer sees when using the service. On the back end a simple shared virtual network drive is provided that receives all content that is posted by the collaborative team. Each may have there own view and sets of folders for instance and each can share them with one individual, or with a group, or with everyone. This type of service makes a lot of sense for this type of community of users. In fact, any user community that regularly exchanges large amounts of data would find value in the type of use model.

Figure 5. A Collaborative Share Service

Disaster Recovery as a Service (DRaaS)

There are times when the user is looking for more than simple storage space. There is a problem that is endemic in small and medium business environments today. There is minimal if any resident IT staff and even less funding to support back end secondary projects like disaster recovery. As a result many companies have BC/DR plans that are woefully inadequate and often would leave them with major or even total data loss in the event of a key critical system failure. For these types of companies using an existing network provider for warm standby virtual data center usage makes a lot of sense. The solution would most probably require CPE to be installed, but after that point the solution could offer a turnkey DR plan that could be tested at regular scheduled intervals for a per event fee.

The big advantage of this approach is that the customer can avoid expanding IT staff and addresses a key issue of primary importance, which is the preservation of data and system up time.

Obviously, this type of service offering requires a provider who is taking SaaS seriously. There is a Data Center required where virtual resources are leased out and hosted to the customer as well as the IT staff required to run the overall operations. As shown by the prevalence of vendors providing this type of service, even with the overhead, it does have an attractive business model that only improves with expanded customer base.

Figure 6. DRaaS implementation

Residential Use Models

PC Backup & Extra Storage

This type of SaaS offering is similar to the virtual disk service (D2D) mentioned above. The important difference is that it is not iSCSI based. Rather it a NAS virtual drive that is offered to the customer through some type of web service portal. Alternatively, it could be offered as a mountable network drive via Windows Explorer™. The user would then simply drag the folders that they want to store into the cloud onto that network drive. If they use backup software they can with a few simple modifications copy data into the cloud by pointing the backup application to the virtual NAS drive. Additionally, this type of service could support small and medium businesses that are NAS oriented from a data storage architecture perspective. In the figure below, a NAS SaaS is illustrated with a residential user who is using the service to store video and music content. Another user is a small business that is using the service for NAS based D2D backup. Both customers see the service as a mapped network drive (i.e. F or H:). For the residential customer it is a drive that content can be saved to, for the business customer it is a NAS target for its backup application.

Figure 7. NAS SaaS

Collaborative Share

More and more, friends and family are not only sharing content, but creating it as well. Additionally, most of it is in pictures, music and video. All files of huge size. This results in a huge amount of data that needs to be stored but also needs to be reference able in order to be shared with others. The widely popular YouTube™ is a good example of such a collaborative service. Another example is FaceBook™, where users can post pictures and video to their walls and share them with others as they see fit. As shown in the figure below, SaaS is an embedded feature of the service. The first user posts content into the service there by using the SaaS feature. Then the second user receives the content in a streaming CDN fashion. The first user would post the content via the web service portal (i.e. their wall).The second user would initiate the real time session via the web service portal by clicking on the posted link and view the content via their local installed media player. Aside from the larger industry players, there is a demand for more localized community based collaborative shares that can exist with art and book communities, student populations, or even local business communities.

Figure 8. Collaborative Share for Residential

Technologies for SaaS

The above use models assumed the use of underlying technologies to move the data, reduce it and store it. These are then merged with supporting technologies such as web services, collaboration and perhaps content delivery to create a unified solution to the customer. Again, this could be as simple as a storage target where data storage is the primary function or it could be as complex as a full collaboration portal where data storage is more ancillary. In each instance, the same basic technologies come into play. It is obvious that from the point of view of the customer, only the best will do. While from the point of view of the provider, it is providing what will meet the level of service required. This results in a dichotomy – as often results in a business model. The end result is an equitable compromise which uses the technologies below to arrive at an equitable solution that satisfies the interest of the user as well as that of the provider. The end result is a tenable set of values and benefits to all parties which is the sign of a good business model.

Disk Arrays

Spinning disks have been around almost as long a modern computing itself. We all have the familiar spinning and clicking (now oh so faint!) on our laptops as the machines chunks through data on its relentless task of providing the right bits at the right time. Disk technology has come a long ways as well. The MTBF rating for even lower end drives are exponentially higher than the original ‘platter’ technologies. Still though, this is the Achilles Heel. This is the place where the mechanics occur. Where mechanics occur, particularly high speed mechanics – failure is just one of the realities that need to be dealt with.

I was surprised to learn just how common it is that just a bunch of disks are set up and used for cloud storage services. The reason is simple, cost. It is far more cost effective to place whole disk arrays out for leasing than it is to take that same array and sequester a portion of it for parity or mirroring. As a result, many cloud services offer best effort service and with smaller services that pretty much works – particulary if the IT staff is diligent with backups. As the data volume grows however, this approach will not work as the MTBF rate of potential failure will out weigh the ability to pump the data back into the primary. That exact number is related to the network speed available and since most organizations do not infinite bandwidth available, that limit is a finite number.

Now one could go through the math to figure the probability of data loss and gamble, or one could invest into RAID and be serious about the offering they are providing. As we shall see later on, there are technologies that assist in the economic feasibility. In my opinion, it would be the first question I asked someone who wanted to provide me a SaaS offering. That is first beyond backup and replication or anything else. Will my data be resident on a RAID array? If so what type? Another question to ask is the data replicated? If so, the next question is how many times and where?

Storage Virtualization

While a SaaS offering could be created with just a bunch of disk space. Allocation of resources would have very rough granularity and the end result would be an environment that would be drastically over provisioned. The reason for this is that as space is leased out the resource is ‘used’ whether it has data or not. Additionally, as new customers are brought on line to the service additional disk space must be acquired and allocated in a discrete fashion. Storage virtualization overcomes this limitation by creating a virtual pool of storage resources that can consist of any number and variety of disks. There are several advantages that are brought about by the introduction of this type of technology. The most notable is that of thin provisioning. Which, from a service provider standpoint is some thing that is as old as service offerings itself. As an example, network service providers do not build their networks to be provisioned to 100% of the potential customer capacity 100% of the time. Instead they analyze and look at traffic patterns and engineer the network to handle the particular occurrences of peak traffic. The same might be said of a thinly provisioned environment. Instead of allocating the whole chunk of disk space at the time of the allocation, a smaller thinly provisioned chunk is setup but the larger chunk is represented back to the application. The system then monitors and audits the usage of the allocation and according to high water thresholds, allocate more space to the user based on some sort of established policy. This has obvious benefits in a SaaS environment as only very seldom will a customer purchase and use 100% of the space at the outset. The gamble is that the provider keeps enough storage resources within the virtual pool to accommodate any increases. Being that most providers are very familiar with type of practice in bandwidth provisioning, it is only a small jump to apply that logic in storage.

Not all approaches to virtualization are the same however. Some implementations are done at the disk array level. While this approach does offer pooling and thin provisioning, it only does so at the array level or within the array cluster. Additionally, the approach is closed in that it only works with that disk vendors’ implementation. Alternatively, virtualization can be performed above the disk array environment. This approach more appropriately matches a SaaS environment in that the open system approach allows any array to be encompassed into the resource pool which better leverages on the SaaS providers’ purchasing power. Rather than getting locked into a particular vendors approach, the provider has the ability to commoditize the disk resources and hence allow better pricing points.

There are also situations called ‘margin calls’. These are scenarios that can occur in thinly provisioned environments where the data growth is beyond the capacity if the resource pool. In those instances, additional storage must physically be added to the system. With array based approaches, this can run into issues such as spanning beyond the capacity of the array or the cluster. In those instances, in order to accommodate for the growth, the provider needs to migrate the data to a new storage system. With the open system approach, the addition of storage is totally seamless and it can occur with any vendors’ hardware. Additionally, implementing storage virtualization at a level above the arrays allows for very easy data migration, which is useful in handling existing data sets.

Data Reduction Methods

This is a key technology for the providers return on investment. Here remember that storage is the commodity. In typical Cloud Storage SaaS offerings the commodity is sold by the Gigabyte. Obviously, if you can retain 100% of the customers data and only store ten or twenty percent of the bits, the delta is revenue back to you for return on investment. If you are then able to take that same technology and not only leverage it across all subscribers but across all content types as well then it becomes something that is of great value to the overall business model of Storage as a Service. The key to the technology is that the data reduction is performed at the disk level. Additionally, the size of the bit sequence is relatively small (512 bytes) rather than the typical block levels. As a result, the comparative is large (the whole SaaS data store) while the sample is small (512 bytes) The end result, is that as more data is added to the system the context of reference is widened correspondingly meaning that the probability that a particular bit sequence will match another in the repository is hence  increased.

But beware, data reduction is not a panacea. Like all technologies it has its limitations and there is the simple fact that some data just does not de-duplicate well. There is also the fact that the data that is stored by the customer is in fact manipulated by an algorithm and abstracted in the repository. This means that some issues of regulatory legal compliance may come into play with some types of content. For the most part however, these issues can be dealt with and data reduction can play a very important role in SaaS architectures, particularly in the back end data store.

Replication of the data

If you are doing due diligence and implementing RAID rather than selling space on ‘just a bunch of disks’, then your most probably the type that will go further to create secondary copies of the primary data footprint. If you do this, you also probably want to do this on the back end so as not to impact the service offering. You also probably want to use as little network resource as possible to keep that replicated copy up to date. Here technologies like Continuous Data Protection and thin replication can assist in getting the data into the back end and performing the replication with minimal impact to network resources.

Encryption

There are more and more concerns about placing content in the cloud. Typically these concerns are from business users who see it as a major compromise of security policy. Individual end users are also broaching concerns around confidentiality of content. Encryption can not solve the issue by itself but it can go a long way towards it. It should be noted though that with SaaS encryption needs to be considered in two aspects. First is the encryption of data in movement. That is protecting the data as it is posted into and pulled out of the cloud service. Second is the encryption of data at rest, which is protecting the content once it is resident in a repository. The first is addressed by methods such as SSL/TLS or IPSec. The second is addressed by encryption at the disk level or prior to disk placement.

Access Controls

Depending on the type and intention of the service, access controls can be relatively simple (i.e. user name & password) to complex (RSA type). In private cloud environments, normal user credentials for enterprise or organization access would be the minimum requirement. Likely, there will be additional passwords or perhaps even tokenization to access the service. For semi-private clouds the requirements are likely to not be as intense but again, can be if needed. Also, there may be a wide range in the level of access requirements. As an example, for a backup service there only needs to be an iSCSI initiator/target binding and a monthly report on usage that might be accessible over the web. In other services such as collaboration, a higher level portal environment will need to be provided – hence the need for a higher level access control or log on. Needless to say, some consideration will need to be made for access to the service, even if it is for the minimal task of data separation and accounting.

The technologies listed above are not ‘required’, as pointed out above just a bunch of disks on the network could be considered cloud storage. Nor is the list exhaustive.  But if the provider is serious about the service offering and also serious about its prospect community, it will make investments into at least some if not all of them.

Planning for the Service

There are two perspectives to cover here. The first is that of the customer. When IT organizations start thinking about using cloud services they are either attempting to reduce cost or bypass internal project barriers. Most of these will plan on using the service to answer requirements for off site storage. Secondary sites are not cheap, particularly if the site is properly equipped as a data center. If this does not already exist, it can be a prime motivator for moving secondary or even tertiary data copies into a cloud service.

There are a number of questions and concerns that should be asked prior to using such a service though. The IT staff should create a task group to assemble a list of questions, requirements & qualifications as to what they expect out of the service. Individuals from various areas of practice should be engaged in this process. Examples are, Security, Systems Administrators, DB Administrators, IT Audit, Networking, etc… the list can be quite extensive. But it important to be sure to consider all facets of the IT practice in regards to the service in question. In the end a form should be created that can be filled out in dialogs with the various providers that are being entertained. Tests and pilots are also a good thing to arrange if it can be done. It is important to get an idea of how fast data can be pumped into the cloud. It is also very important to know how fast it can be pulled out as well. At the very least the service should be closely monitored by both storage and networking staff to be certain that the service works according to SLA (if there is one) and is not decaying in performance over time or increase in data. In either instance communication with the SaaS provider is then in order and may involve technical support and troubleshooting or service expansion. In any event, it should be realized that a SaaS service package, just like the primary data footprint, is not a static thing; and they usually do not shrink!

Some sample questions that might be asked of a SaaS vendor are the following:

Is the data protected by RAID storage?

Is the data replicated? If so, how many times and where will copies be located?

Is the data encrypted in movement? At rest?

What is the estimated ingestion capacity rate? (i.e. how much data can be moved in an hour into the cloud)

What is the estimated restore time? (i.e. how much data can be moved off of the cloud in an hour)

(The two questions above may require an actual test.)

What security measures are taken at the storage sites (both cyber and physical)?

These are only a few generic level questions that can help in getting the process started. You will quickly find that once you start bringing in other individuals into the process from various disciplines that list can get large and may need to be optimized and pared down. Once this process is complete, it is good to set up a review committee that will meet with the various vendors and move through the investigation process.

From the perspective of the SaaS provider the issues are similar as it is in the best interest to meet the needs of the customer. There is a spin of using the service to providing it however. There are two ways that this can occur. The first instance is where a prospective SaaS provider already has an existing customer base that it is looking to provide a service to. In this case the data points are readily available. A survey needs to be created that will assemble the pertinent data points and that then needs to be filled out by the various customers of the service. Questions that might be asked are, what is your backup environment like, what is the size of the full data repository, what is the size of the daily incremental backup, can you provide an estimated growth rate, what is your network bandwidth capacity? Once the data is assembled, it can be tallied up and sizing can occur in a rather accurate fashion.

The second method is in the case of a prospective provider who does not yet have a known set of data for existing customers. Here some assumptions must be made on a prospective business model. It needs to be determined what the potential target market is for the service launch. Once those numbers are reached a range or average needs to be figured on many of the data points above to create a typical customer profile. It is important that this is well defined and well known. The reason for this is that as you add new customers onto the service you can in the course of the service profile survey identify a relative size for the customer. (i.e. 1 standard profile or 3.5 times the standard profile) With that information predicting service impact and scaling estimations can be much easier. From there the system can then be sized according to those metrics with an eye to the future for growth. Capacity is added as the service deployment grows.

As a storage solution provider, my company will assist prospective SaaS providers in doing this initial sizing exercise. As an example, in the first case point we assisted a prospect in the creation of the service requirements survey as well as helped in actually administering it. Afterwords, we worked interactively with the provider to size out the appropriate system to meet the requirements of the initial offering. Additionally, we offered scaling information as well as regular consultative services so that the offering is scaled properly.

Like all service offerings, SaaS is only as good as its design. Someone can go out and spend the highest dollar on the ‘best’ equipment and then be some what slipshod in the way the system is sized and implemented and end up with a mediocre service offering. On the other hand one can get good cost effective equipment, size and implement them with care and wind up with a superior offering. The message here is that the key to success in SaaS is in the planning, both for the customer as well as the provider.

Not your mother’s Infiniband

The InfiniBand (IB) specification defines the methods & architecture of the interconnect that establishes the interconnection of the I/O subsystems of next generation of servers, otherwise known as compute clustering. The architecture is based on a serial, switched fabric that currently defines link bandwidths between 2.5 and 120 Gbits/sec. It effectively resolves the scalability, expandability, and fault tolerance limitations of the shared bus architecture through the use of switches and routers in the construction of its fabric. In essence, it was created as a bus extension technology to supplant the aging PCI specification.

The protocol is defined as a very thin set of zero copy functions when compared to thicker protocol implementations such as TCP/IP. The figure below illustrates a comparison of the two stacks.

Figure 1. A comparison of TCP/IP and Infiniband Protocols

Note that IB is focused on providing a very specific type of interconnect over a very high reliability line of fairly short distance. In contrast, TCP/IP is intended to support almost any use case over any variety of line quality for undefined distances. In other words, TCP/IP provides robustness for the protocol to work under widely varying conditions. But with this robustness comes overhead. Infiniband instead optimizes the stack to allow for something known as RDMA or Remote Direct Memory Access. RDMA is basically the extension of the direct memory access (DMA) from the memory of one computer into that of another (via READ/WRITE) without involving the server’s operating system. This permits a very high throughput, low latency interconnect which is of particular use to massively parallel compute cluster arrangements. We will return to RDMA and its use a little later.

The figure below shows a typical IB cluster. Note that both the servers and storage are assumed to be relative peers on the network. There are differentiations in the network connections however. HCA’s (Host Channel Adapters) refer to the adapters and drivers to support host server platforms. TCA’s (Target Channel Adapters) refer to the I/O subsystem components such as RAID or MAID disk subsystems.

Figure 2. An example Infiniband Network

At its most basic form the IB specification defines the interconnect as (Point-to-Point) 2.5 GHz differential pairs (signaling rate)- one transmit and one receive (full duplex) – using LVDS and 8B/10B encoding. This single channel interconnect delivers 2.5 Gb/s. This is referred to as a 2X interconnect. The specification also allows for the bonding of these single channels into aggregate interconnects to yield higher bandwidths. 4X defines a interface with 8 differential pairs (4 per direction). The same for Fiber, 4 Transmit, 4 Receive, whereas 12X defines an interface with 24 differential pairs (12 per direction). The same for Fiber, 12 Transmit, 12 Receive. The table below illustrates various characteristics of the various channel classes including usable data rates.

Table 1.

Also note that the technology is not standing still. The graph below illustrates the evolution of the IB interface over time.

Figure 3. Graph illustrating the bandwidth evolution of IB

As the topology above in figure 2 shows however, the effective distance of the technology is limited to single data centers. The table below provides some reference to the distance limitations of the various protocols used in the data center environment including IB.

Table 2.

Note that while none of the other technologies extend much further from a simplex link perspective, they do have well established methods of transport that can extend them beyond the data center and even the campus.

This lack of extensibility is changing for Infiniband however. There are products that can extend its supportable link distance to tens, if not hundreds of Kilometers, distances which rival well established WAN interconnects. New products also allow for the inter-connection of IB to the other well established data center protocols, Fibre Channel and Ethernet. These new developments are expanding its potential topology thereby providing the evolutionary framework for IB to become an effective networking tool for next generation Business Continuity and Site Resiliency solutions. In figure 4 below, if we compare the relative bandwidth capacities of IB with Ethernet and Fibre Channel we find a drastic difference in effective bandwidth both presently and in the future.

Figure 4. A relative bandwidth comparison of various Data Center protocols

Virtual I/O

With a very high bandwidth low latency connection it becomes very desirable to use the interconnect for more than one purpose. Because of the ultra-thin profile of the Infiniband stack, it can easily accommodate various protocols within virtual interfaces (VI) that serve different roles. As the figure below illustrates, a host could connect virtually to its data storage resources over iSCSI (via iSER) or native SCSI (via SRP). In addition it could run its host IP stack as a virtual interface as well. This capacity to provide a low overhead high bandwidth link that can support various virtual interfaces (VI) lends it well to interface consolidation within the data center environment. As we shall see however, in combination with the recent developments in extensibility, IB is becoming increasingly useful for a cloud site resiliency model.

Figure 5. Virtual Interfaces supporting different protocols

Infiniband for Storage Networking

One of the primary uses for Data Center interconnects is to attach server resources to data storage subsystems. Original direct storage systems were connected to server resources via internal busses (i.e. PCI) or over very short SCSI (Small Computer Serial Interface) connections, known as Direct Access Storage (DAS). This interface is at the heart of most storage networking standards and defines the internal behaviors of these protocols for hosts (initiators) to I/O device (targets). An example for our purposes is a host writing data to or reading data from a storage subsystem.

Infiniband has multiple models for supporting SCSI (including iSCSI). The figure below illustrates two of the block storage protocols used, SRP and iSER.

Figure 6. Two IB block storage protocols

SRP (SCSI RDMA Protocol) is a protocol that allows remote command access to a SCSI device. The use of RDMA avoids the overhead and latency of TCP/IP and because it allows for direct RDMA write/read is a zero copy function. SRP never made it into a formal standard. Defined by ANSI T10, the latest draft is rev. 16a (6/3/02).

iSER (iSCSI Extensions for RDMA) is a protocol model defined by the IETF that maps the iSCSI protocol directly over RDMA and is part of the ‘Data Mover’ architecture. As such, iSCSI management infrastructures can be leveraged. While most say that SRP is easier to implement than iSER, iSER provides enhanced end to end management via iSCSI management. Both protocol models, to effectively support RDMA, possess a peculiar function that results in all RDMA being directed towards the initiator. As such, a SCSI read request would translate into an RDMA write command from the target to the initiator; whereas a SCSI write request would translate into an RDMA read from the target to the initiator. As a result some of the functional requirements for the I/O process shift to the target and provides offload to the initiator or host. While this might seem strange, if one thinks about what RDMA is it only makes sense to leverage the direct memory access of the host. This is results in a very efficient leverage of Infiniband for use in data storage.

Another iteration of a storage networking protocol over IB is Fibre Channel (FCoIB). In this instance, the SCSI protocol is embedded into the Fibre Channel interface, which is in turn run as a virtual interface inside of IB. Hence, unlike iSER and SRP, FCoIB does not leverage RDMA but runs the Fibre Channel protocol as an additional functional overhead. FCoIB does however provide the ability to incorporate existing Fibre Channel SAN’s into an Infiniband network. The figure below illustrates a network that is supporting both iSER and FCoIB, with a Fibre Channel SAN attached by a gateway that provides interface between IB and FC environments.

Figure 7. An IB host supporting both FC & native IB interconnects

As can be seen, a legacy FC SAN can be effectively used in the overall systems network. Add to this high availability and you have a solid solution for a hybrid migration path.

If we stop and think about it, data storage is number two only to compute clustering for an ideal usage model for Infiniband. Even with this, the use of IB as a SAN is a much more real world usage model for the standard IT organization. Not many IT groups are doing advanced compute clustering and those that do already know the benefits of IB.

Infiniband & Site Resiliency

Given the standard offered distances of IB, it is little wonder that it has not been often entertained for use in site resiliency. This however, is another area that is changing for Infiniband. There are now technologies available that can extend the distance limitation out to hundreds of kilometers and still provide the native IB protocol end to end. In order to understand the technology we must first understand the inner mechanics of IB.

The figure below shows a comparison between IB and TCP/IP reliable connection. The TCP/IP connection shows a typical saw tooth profile which is the normal result of the working mechanics of the TCP sliding window. The window starts at a nominal size for the connection and gradually increases in size (i.e. Bytes transmitted) until a congestion event is encountered. Depending on the severity of the event the window could slide all the way back to the nominal starting size. The reason for this behavior is that TCP reliable connections were developed in a time when most long distance links were far more unreliable and of less quality.

Figure 8. A comparison of the throughput profiles of Infiniband & TCP/IP

If we take a look at the Infiniband throughput profile we find that the saw tooth pattern is replaced by a square profile that is the result of the transmission instantly going to 100% of the offered capacity and maintains as such until a similar event occurs which results in a halt to the transfer. Then after a period of time, it resumes as 100% of the offered capacity. This similar event is something termed as buffer starvation. Where the sending Channel Adapter has exhausted its available buffer credits which are calculated by the available resources and the bandwidth of the interconnect (i.e. 2X, 4X, etc.). Note that the calculation does not include any significant concept of latency. As we covered earlier, Infiniband was originally intended for very short highly dependable interconnects so the variable of transmission latency is so slight that it can effectively be ignored within the data center. As a result the relationship of buffer credits to available resources and offered channel capacity resulted in a very high throughput interconnect that seldom ran short of transmit buffer credits. Provided things were close.

As distance is extended things become more complex. This is best realized in the familiar bucket analogy. If I sit on one end of a three foot ribbon and you sit on the other end and I have a bucket full of bananas (which are analogous to the data in the transmit queue) where as you have a bucket that is empty (analogous to your receive queue) we can run the analogy. As I pass you the bananas , there is only a short distance which can allow for a direct hand off of the bananas. Remembering that this is RDMA, I pass you the bananas at a very fast predetermined speed (the speed of the offered channel) and you take them just as fast. At the end of passing you the bananas, you pass me a quarter to acknowledge the fact that the bananas have been received (this is analogous to the completion queue element shown in figure 1). Now imagine that there is someone standing next to me who is providing me bananas at a predetermined rate (this is the available processing speed of the system). Also, he will only start to fill my bucket if the following two conditions exist. 1). my bucket is empty and, 2). I give him the quarter for the last bucket. Obviously the time required end to end will impact that rate. If that resulting rate is equal to the offered channel, we will never run out of bananas and you and I will be very tired. If that rate is less than the offered channel speed then at some point I will run out of bananas. At that point I will need to wait until my bucket is full before I begin passing them to you again. This is buffer starvation. Now in a local scenario, we see that the main tuning parameters are a). the size of our buckets (available memory resources for RDMA) and, b). the rate of the individual placing bananas into my bucket (the system speed). If these parameters are tuned correctly, the connection will be of very high performance. (You and I will move a heck of a lot of bananas). The further we are from that optimal set of parameters, the lower the performance profile will be and an improperly tuned system will perform dismally.

Now let’s take that ribbon and extend it to twelve feet. As we watch the following scenario unfold it becomes obvious as to why buffer starvation limits distance. Normally, I would toss you a banana and wait for you to catch it. Then I would toss you another one. If you missed one and had to go pick it up off of the ground (the bruised banana is a transmission or reception error), I would wait until you were ready to catch another one. This in reality is closer to TCP/IP. With RDMA, I toss you the bananas just as if you were sitting next to me. What results is a flurry of bananas in the air all of which you catch successfully because hey – your good. (In reality, it is because we are assuming a high quality interconnect) After I fling the bananas however, I need to wait to receive my quarter and until my bucket is in turn refilled. At twelve feet if nothing else changes – we will be forced to pause far more often as my bucket refills. If we move to twenty feet the situation gets even more skewed. We can tune certain things like the depth of our buckets or the speed of the replenishment but these get to be unrealistic as we stretch the distance farther and farther. This is what in essence has kept Infiniband inside the data center.*

*Note that the analogy is not totally accurate with the technical issues but it is close enough to give you a feel of the issues at hand.

Now what would happen if I were to put some folks in between us who had reserve buckets for bananas I send to you and you were to do the same for bananas you in turn send to me? Also, unlike the individual who fills my bucket who deals with other intensive tasks such as banana origination (the upper system and application), this person is dedicated one hundred percent to the purpose of relaying bananas. Add to this the fact that this individual has enough quarters to give me for twice the size of his bucket, and yours in turn as well. If we give them nice deep buckets we can see a scenario that would unfold as follows.

I would wait until my bucket was full then I would begin to hand off my bananas to the person in front of me. If this individual were three feet from me I could hand them off directly as I did with you originally. Better than that, I could simply place the bananas in their bucket and they would give me quarter each time I emptied mine. The process repeats until their bucket is full. They then can begin throwing the bananas to you. While we are at it, why should they toss directly to you? Let’s put another individual in front of you that is also completely focused. But instead of being focused on tossing bananas, they would be focused on catching them. Now if these person’s buckets are roughly 4 times the size of yours and mine, and the relayed bananas occurred over six feet out to your receiver at the same rate as being handed by me, we in theory should never run out of bananas. There would be an initial period of the channel filling and the use of credit but after that initial period the channel could operate at optimal speed with the initial offset in reserve buffer credits being related to the distance or latency of the interconnect. The reason for the channel fill is that the person has to wait until their bucket is full before they can begin tossing, but importantly, after that initial fill point they will continue to toss bananas as long as there are some in the bucket. In essence, I always have an open channel for placing bananas and I always get paid and can in turn pay the guy who fills my bucket only on the conditions mentioned earlier.

This buffering characteristic has led to a new class of devices that can provide significant extension to the distance offered by Infiniband. Some of the latest systems can provide buffer credits equivalent to one full second, which is A LOT of time at modern networking speeds. If we add these new devices and another switch to the topology shown earlier we can begin to realize some very big distances that become very attractive for real time active-active site resiliency.

Figure 9. An extended Infiniband Network

As a case point, the figure above shows an Infiniband network that is extended out to support data centers that are 20Km in distance. The systems at each end, using RDMA are effectively regarding each other as local and for all intensive purposes in the same data center. This means that there are versions of fault tolerance and active to active high availability that otherwise would be out of the question, that are now quite feasible to design and work in practice. A common virtualized pool of storage resources using iSER allow for seamless treatment of data and bring a reduced degree of fault dependency between the server and storage systems. Either side could experience failure at either the server or storage system level and still be resilient. Adding further systems redundancy for both servers and storage locally on each side provides further resiliency as well as provide for off line background manipulation of the data footprint for replication, testing, etc.

Figure 10. A Hybrid Infiniband network

In order for any interface consolidation effort to work in the data center the virtual interface solution must provide for a method of connectivity to other forms of networking technology. After all, what good is an IP stack that can only communicate within the IB cluster? A new generation of gateway products provide for this option. As shown in the figure above, gateway products exist that can tie IB to both Ethernet and Fibre Channel topologies. This allows for the ability to consolidate data center interfaces and still provide for general internet IP access as well as connectivity to traditional SAN topologies and resources such as Fibre Channel based storage arrays.

While it is clear that Infiniband is unlikely to become a mainstream networking technology, it is also clear that there are many merits to the technology that have kept it alive and provided enough motivation (i.e. market) for its evolution into a more mature architectural component. With the advent of higher speed Ethernet and FCoE as well as the current development of lower latency profiles for DC Ethernet, the longer range future of Infiniband may be similar to that of Token Ring or FDDI. On the other hand, even with these developments, the technology may be more likened to ATM. Which, while being far from mainstream, is still being used extensively in certain areas.  If one has the convenience of waiting for these trends to sort themselves out then moving to Infiniband in the Data Center may be premature. However, if you are one of the many IT architects that are faced with intense low latency performance requirements that need to be addressed today and not some time in the future, IB may be the right technology choice for you. It has been implemented by enough organizations that best practices are fairly well defined. It has matured enough to provide for extended connectivity outside of the glass house and gateway technologies are now in place that can provide connectivity out into other more traditional forms of networking technology. Infiniband may never set the world on fire, but it has the potential to put out fires that are currently burning in certain high performance application and data center environments.

If we were to try to identify the most important component out of this substrate, most would agree that it is something known as virtualization. In the cloud, virtualization occurs at several levels. It can range from ‘what does what’ (server & application virtualization) to ‘what goes where’ (data storage virtualization) to ‘who is where’ (mobility and virtual networking). When viewed as such, one could even come to the conclusion that virtualization is the key enabling technology upon which all other components either rely on or embody in some subset of functionality.

As an example, at the application level Web Services and Service Oriented Architecture serve to abstract & virtualize the application resources required to provide a certain set of user exposed functions. Going further whole logical application component processes can be strung together in a work flow to create an automated complex business process that can be kicked off by the simple submittal of an on line form on a web server.

If we look further, underneath this we can identify another set of technologies where the actual physical machine is host to multiple resident ‘virtual machines’(VM) which house different applications within the data center. Additionally, these VM’s can migrate from one physical machine to another or invoke clones of themselves that can in turn be load balanced for improved performance during peak demand hours. At first this was a more or less local capability that was limited to the physical machines within the Data Center, but recently advances have been made by the use of something known as ‘stretch clustering’ to enable migrations to remote Data Centers or secondary sites in response to primary site failures and outages. This capability has been a great enabling tool in prompt Disaster Recovery plans for key critical applications that absolutely need to stay running and accessible.

In order for the above remote VM migration to work however there needs to be consistent representation and access to data. In other words, the image of the working data that VM #1 has access to at the primary site needs to be available to VM #2 at the secondary site. Making this occur with traditional data storage management methods is possible but extremely complex, inefficient and costly.

Virtualization is also used within storage environments to create virtual pools of storage resources that can be used transparently by the dependant servers and applications. Storage Virtualization not only simplifies data management for virtualized services but also serves to provide the actual foundation for all of the other forms of virtualization within the cloud in that the data needs to be always available to the dependant layers within the cloud. Indeed, without the data – the cloud is nothing but useless vapor.

This is painfully evident in some of the recent press around cloud failures, most notably the T-Mobile Sidekick failure that was the result of Microsoft’s Danger subsidiaries failure to back up key data prior to a storage upgrade that was being performed by Hitachi. Many T-Mobile users woke up one morning to find that their calendars and contact lists were non-existent. After some time, T-Mobile was forced to tell many of their subscribers that the data was permanently lost and not recoverable. This particular instance has had a multi-level reverberation that impacted T-Mobile (the Mobile Service Provider), Microsoft Danger (the Data Management Provider), Hitachi (the company performing the storage upgrade) and finally the thousands of poor mobile subscribers who arguably bore the brunt of failure. To be fair, Microsoft was able to restore most of the lost data, but this was only after days had passed. Needless to say, the legal community is now a buzz over potential law suits and some are already in the process of being filed.

The reasons for the failure are not really the primary purpose of the example. The example is intended to illustrate two things; first, while many think that Cloud Computing somehow takes us beyond the traditional IT practices – it does not. In reality, Cloud Computing builds upon them and is in turn dependent upon them for proper intended functionality. The responsibility for needs to perform them can be vague however and needs to be clearly understood by all parties. Second, Cloud Computing without data is severely crippled, if not totally worthless.  After all, the poor T-Mobile subscriber did not know who to meet or call, or even how to call to cancel or reschedule (unless they took the time to copy all of that information locally to the PDA – and some did).  What good is next generation mobile technology if you have no idea of where to be or who to contact!

If we view it as such then it could be argued that proper data storage management is the key foundation and enabler for Could Computing. If this is the case then it needs to be treated as such when the services are being designed. You often hear that security should not be an afterthought. It needs to be considered in every step of a design process. This is most definitely true. The point of this article is that the same thing needs to be said for data storage and management.

The figure below illustrates this relationship. The top layer, which represents the user leverages on mobility and virtual networking to provide access to resources anywhere, anytime. Key enabling technologies such as 3G or 4G wireless and Virtual Private Networking provide for secure almost ubiquitous connectivity into the cloud where key resources reside.

Figure 1. Cloud Virtualization Layers

In the next layer the enabling services are provided for by underlying applications. Some may be atomic like simple email in that they provide a single function from a single application. More and more however, services are becoming composite in that they may depend on multiple applications acting in concert to complete whole business processes. These types of services are typically SOA enabled in that they follow process flows that are defined by an overarching policy and rule set that is maintained and driven by the SOA framework. In these types of services there is a high degree of inter-dependency which, while enabling enhanced feature service offerings, also creates areas of vulnerability that can become critical outages if one of the component applications in the process flow were to suddenly become unavailable.  To accommodate for this, many SOA environments provide for recovery work flows which can provide for graceful rollback of a particular transaction. Optimally, any failure of a component application should be totally transparent to the composite service. If a server that is providing the application were to fail, another server should be ready to take over that function and allow the services process flow to proceed uninterrupted.

The layer below the service application layer is the layer that would provide for this transparent resiliency and redundancy.  Here physical servers provide hosting for multiple virtual machines which can provide for redundant and even load balanced application service environments.

In the figure below, we see that these added features provide the resource abstraction that allows one VM to step in for another’s failure so that a higher level business process flow can proceed without a glitch. Additionally, applications can be load balanced to allow for scale and higher capacity.

Figure 2. VM’s set up in a Fault Tolerant configuration

As we pointed out earlier however, this apparent Nirvana of application resiliency can only be met if there is consistent data that is available to both systems at the time of the failover at the VM level. In the case of a transaction database the secondary VM should ideally be able to capture the latest exchange so as to allow the application to proceed without interruption. In other words, the data has to have ‘full transactional integrity’. At the very least the user may have to fill out the present form page that they are currently working on once again. Without the availability to data any and all resiliency provided by the higher layers are null and void. The figure below builds upon figure two to illustrate this.

Figure 3. Redundant Data Stores key to service resiliency

As the user interacts with the service they ideally should be totally oblivious to any failures within the cloud. As we see in the figure above however, this can only be the case if there are consistent up to the current transaction data repositories that the failover VM can mount and carry on with the user service with as little interruption as possible. Doing this with traditional Direct Attached Storage (DAS) is a monumental task that is prone to vulnerabilities. The concept of transactional integrity in this approach is difficult. The use of Storage Virtualization helps solve this complexity by creating one large virtual storage space that can be leveraged at the logical level by multiple server resources within the environment. Shown below, this virtualized storage space can be divided up and allocated by a process known as provisioning. Once these logical storage spaces (LUN’s) are created, they can not only be allocated to physical servers but to individual VM’s as well as any higher level fault tolerance. The value to this is that failure at the VM level is totally independent of failure at the data storage level.

Figure 4. Failure mode independence

As shown in the figure above most VM level failures can be addressed at the local site. As a result, the failover VM can effectively mount the primary data store. Data consistency is not an issue in this case because it is the exact same data set. In instances of total site failure the secondary site must take over completely. In this instance the secondary storage must be used. It was pointed out earlier that this secondary store must have complete transactional integrity with the primary store and the dependent application.  In a remote secondary site scenario that is designed for disaster recovery, the costs for up to the minute traditional data backups is cost prohibitive and logistically impossible. Consquently, reliable backup data is in many instances 12 hours old or greater.

Newer storage technologies come into play here that allow for drastic reduction in the amount of data that has to be copied as well as optimization in the methods for doing so.

Thin Provisioning

One of the major reasons for the difficulties noted in the previous section is the prevalence of overprovisioning in the data storage environment. This seems counterintuitive. If there is more and more data, how can data storage environments be  overprovisioned? This occurs because of the friction between two sets of demands. When installing a server environment one of the key steps is in the allocation of the data volume. This is done at install and is not an easy allocation to adjust once the environment has been provisioned. As a result, most administrators will wiegh the risk and downtime to increase volume size against the cost of storage. In the end they will typically choose to over provision the allocation so that they do not have to be concerned about any issues with storage space later on.

This logic is fine in a static example. However, if we consider this practice in light of Business Continuity and Distaster Recovery it becomes problematic and costly. The reason for this is that using traditional volume management and backup methods require the backup of the whole data volume. This is the case even if the application is only actually using 20% of the allocated disk space. Now, size translates to WAN bandwidth. Suddenly disk space is not so cheap.

Storage virtualization enables the ability to do something known as thin provisioning. Because the virtualized storage environment abstracts the actual data storage from the application environment, it can be used to actually allocate a much smaller space than the application believes it has. The concept of pooling allows for the virtualized environment to allocate additonal space as the data store requirements grow for the application environment. This is all transparent to the application however. The end result is a much more efficient data storage environment and the need to re-configure the application environment is eliminated. The figure below illustrates an application that has been provisioned for 1 TeraByte of data storage. The storage virtualization environment however has only allocated 200 GigaBytes of actual storage. This translates into an 80% increase in the efficiency of storage usage.

Figure 5. Thin Provisioning & Replication

The real impact comes when considering this practice in Business Continuity and Disaster Recovery. At the primary site, only the allocated portion of the virtualized data store needs to be replicated for business continuity at the local site. This is something that is termed as thin replication. For disaster recovery purposes the benefits translate directly into an 80% reduction in the required WAN usage to provide for full resiliency. Now it becomes possible not only to seriously entertain network based DR (as opposed to the ‘tape and truck’ method), but to perform the replications at multiple times during the day rather than once at the end of the day during off hours. What enables this are two things, first the drastic reduction in the data being moved and second the fact that the server is removed from these tasks by the storage virtualization. This means that the application server environment can be up 24/7 and provide for a more consistent Business Continuity and Disaster Recovery practice.

Continuous Data Protection (CDP)

The next of these technologies is Continuous Data Protection. CDP is based on the concept of splitting writes to disk to a separate data journal volume. This process is illustrated below. While the write primary storage occurs as normal, a secondary write occurs which is replicated into the CDP data journal.  This split can occur in the host, within the Storage Area Network, in an appliance or in the storage array itself. If the added process is handled by the host (via a write splitter agent), the host must support the additional overhead.

Figure 6. Continuous Data Protection split on writes to disk

If the split is done in the disk array the journal must be local within that array or within an array that is local, hence its use in DR is somewhat limited. If the split occurs within the SAN Fabric or in an appliance the CDP data journal can be located in a different location than the primary store.  This can be supported in multiple configurations but the main point is that on primary storage failover there is a consistent data set that has full transactional integrity available and the secondary VM can take over in as transparent a fashion as possible regardless of which site it’s located at.

Figure 7. CDP and its use in creating ongoing DR Backup

As shown above with less than the original volume size, data consistency can be provided in any minute density that the administrator requires for historical purposes and up to the minute for real time recovery with data journaling. Also consider that disk space is cheap in comparison to bandwidth and even cheaper in comparison to lost business. With only the used disk deltas being copied, far less bandwidth is used. Additionally, with a complete consistent data set always available, off line backups can occur to archive Virtual Tape Libraries (VTL) or directly to tape at any time – even during production hours – to provide for complete DR compliance in the event of total catastrophe at the primary site.

Data De-Duplication

Full traditional backups will usually store a majority of redundant data. This means that every initial image will mostly be of redundant data that was already contained in the last full image. The replication of this data seems pointless and it is.* Data De-duplication works by the assumption that most of the data that moves into backup is repetitive and redundant. While CDP works well towards reducing this for database & file based environments by its very nature of operation, most tape based backups will simply save the whole file if any change has been recorded (typically done by size or last modification date).

*There may be instances where certain types of data cannot be de-duplicated due to regulatory requirements. Be sure that the vendor can support such exceptions.

Data De-Duplication works at the sub block level to identify only the sections of the file that have changed and thereby only backup the delta sub blocks to maintain complete consistency of not only the most recent, but also of all archived versions of the file. (This is accomplished by an in depth indexing that occurs at the time of the de-duplication that preserves all versions of the data file for complete historical consistency.) As an example, when a file is first saved obviously the de-duplication ratio is 1:1 as this is the first time that data is saved. However, over time as subsequent backups occur, file based repositories can realize de-duplication ratios as high as 30:1. The chart below illustrates some of the potential reduction ratios for different types of data files.

Document type    De-dupe ratio    % of data backed up

New working documents                                             2:1                                                          50% less data

5:1                                                          80% less data

Active working documents                                        10:1                                                        90% less data

20:1                                                        95% less data

Archived inactive documents                                  30:1                                                        97% less data

As can be seen, these technologies can drastically reduce the amount of data that you need to move over the wire to provide data consistency as well as greatly reduce the storage requirements for maintaining that consistency. The result is an ROI that is unprecedented and simply cannot be found in traditional storage and networking investments.

In reality, in data de-duplication the reduction ratios occur in ranges. More active data will show less reduction ratios than data that are largely historical. As a data set matures and goes into archive status the ratio for data reduction becomes quite high because there is no change to the data pattern within the file. This leads to the point that data de-duplication is best done at various locations, not only across its end to end path but from a life cycle perspective as well.  For instance, de-duplication provides great value in WAN usage reductions for remote site backups if the function is performed at the remote site. It would also find value within the replication and archive process, particularly to VTL or tape store, knowing that what goes onto this medium typically can be viewed as static and is for archive purposes.

Some of the newer research in the industry is around the management of the flow of data through its life cycle. As new data is created its usage factor is high as well as the amount of change that it undergoes. Imagine a new document that is created at the beginning of a standards project. As the team moves through the flow of the project the document is modified. There may even be multiple versions of the same document at the same time which would be considered valid to the overall project.

Figure 8. Project Data Life Cycle

As the project matures and the standard solidifies however, more and more of these documents will become ‘historical’ and will no longer change. Even the final valid document that the project delivers as its end product will not change without due process and notification. Then at such a time the whole parade begins anew. The main point is that as these pieces of data age they should be moved to more cost effective storage. The end result is that as the de-duplication hit gets higher, that piece of data should be moved to more cost effective storage. Eventually, that piece of data would end up in a VTL where it would act as a template for de-duplication against all further input to those final archives. The end result is the reduction of data amount as well as the lowering of the overall retention cost.

While it may be true that data storage is the key foundation and consequently Achilles heel for Cloud Computing services, there are technologies available to enable data storage infrastructures to step up to the added requirements for a true Cloud service environment. This is why the term Cloud Storage makes me uneasy when I hear it used without any qualification. Consider after all, any exposed disk in a server that is attached to a cloud could be called ‘cloud storage’. Just because it is ‘storage in the cloud’ does not mean that it is resilient, robust, or cost effective. Consequently, I would prefer to differentiate ‘Cloud Storage’, (i.e. storage as a cloud service) and ‘Storage architectures for Cloud Services’ which are the technologies and practices of data storage management to support all cloud services (of which Cloud Storage is one). The technologies reviewed in this article enable storage infrastructures to provide the resiliency and scale that are required for true secure and robust data storage solutions for cloud service infrastructures.  Additionally, they help optimize the IT cost profile both in capital as well as operational expense perspectives.  These technologies also work towards vastly improving the RPO (Recover Point Objectives) and RTO (Recovery Time Objectives) of any Business Continuity and Disaster Recovery plan. As cloud computing moves into the future its fate will depend upon the integrity of the data on which it operates. Cloud Service environments and perhaps the companies that provide or use them will succeed or fail based on whether or not they are built upon truly solid data management practices and solutions. The technology exists for these practices to be implemented. As always it is up to those who deploy the service to make sure that they consider secure and dependable storage in the overall plan for Business Continuity and Disaster Recovery as well as business regulatory compliance.

We typically think of trust as something that spans between two or more humans and provides a basis for their interactions. While this is a true characterization of trust it is not an exclusive one. Recent advances in technology within the past 20 years have served to greatly change both the scope and meaning of this paradigm. One of the primary reasons for this is that the scope and capability of interaction has increased in a like manner. Interactions occur not only between humans, but human to machine as well as machine to machine. Furthermore, these interactions can be chained by way of a conditional policy basis to allow for complex communication profiles that in some instances may not involve the direct participation of a human at all.

This paper is intended to analyze the subject of trust and its close association to other subjects such as risk, assurance and identity and the impact that it has on technology and the dynamics of human interaction. We will begin by looking at trust in the basic definition and historical (as well as pre-historical) context. This will serve to set the stage for later focus into the impact on areas of technology and advanced communication capabilities that have become prevalent in our lives. It is the hope of the author that this diatribe will allow for a better understanding of the subject from both a philosophical and practical standpoint.

How do I trust you?

This is the classic question, and one that is hard to quantify. Indeed, the answer may be different for different people. The reason for this is that some people are simply more ‘trusting’ (the cynical reader might think ‘gullible’) than others. There is also a degree of context, which is very closely related to assumed risk on behalf of the trusting party that comes into play with every decision of trust. If we think about it, the manifestations can quickly become boggling. After all, there is a big difference between trusting your neighbors’ kid to cut your grass versus trusting that same kid to baby-sit your own. There are certain pieces of additional information that you will typically need to extend your trust into the deeper context. This additional information will typically (if you decide to let him or her baby-sit) provide you with the additional level of assurance to extend the trust into the new scenario.

So while the possible manifestations are quite numerous and complex, we can already see that there are some common elements that are present in every instance. The first point being that trust is always extended based on some level of assurance. The second point being that this relationship between trust and assurance is dependant upon the context of the subject matter on which trust is established. Going further, this context will always have an element of risk that is assumed by the extension of trust. This results in a threefold vector relationship that is shown in figure 1. What the diagram attempts to illustrate is that the threefold vector is universal and that the subjects of trust (the context of its extension, if you will) fall in relative positions on the trust axis.

Figure 1. The vector relationship between context, assurance & trust

 As the figure above illustrates there is a somewhat linear relationship between the three vectors. It is the subject of trust that provides for the degree of non-linearity. Some subjects are rather relative. As an example, I might not me too picky about my lawn but others might be as sensitive as to rate the level of trust to be in close equivalence to baby-sitting their kids. Some parents may be so sensitive to the issue of baby-sitting that they will require a full back ground check prior to the extension of trust. In other instances, things are rather absolute. A good example of this is the trusting of an individual with the ‘Foot Ball’, which is a top secret attaché that covers the instructions, authorization & access codes for the nuclear warheads involved in the defense of the United States of America. For this subject, we are assuming that the individual is a member of the Department of Defense with relatively high rank and has passed the integrity and background checks as well as psychological stability testing to provide the level of assurance to extend what could be perceived as the ultimate level of trust. Also consider that there is no ‘one’ person that has this type of authority, it is a network of individuals that needs to act in a synchronous fashion according to well defined procedures. This reduces the possibility of last minute ‘rogue’ behavior.

There is another thing to consider as well and this is something known as an assurance cycle. For some extensions of trust, a one time validation of assurance is all that is required. As an example, even the pickiest of yard owners will typically validate someone’s skill just once. After that, there is the assumption that that skill level is appropriate and is unlikely to change. This is often the case as well for baby-sitting. Seldom will even the most selective parents do a full background check every time the same kid is brought in to do the job. It will usually take some exception, such as a poor mowing job or a bad event during baby-sitting that causes this degree of trust to be compromised and hence require re-validation. There are some positions however that are extremely sensitive, have huge potential impact and are non-reversible. A good example is the extension of trust to handle the ‘Foot Ball’. In this instance, there are several regular security and psychological tests that occur as well as random spot testing and background checks to assure that the individual’s integrity as well as those that support him or her are not in any way compromised.

So from the above we can assume that there are four major elements to trust. First is the aspect of context, what is it that the trust is about. In this there is always an element of risk that the party who extends the trust assumes; Second is the level of assurance, what will it take to enable and establish the extension of trust; The third is the element of validation, how often will I need assurance to keep the extension of trust and then finally there is the element of trust itself.

There are also modes of trust that occur, some of which are deemed to be more solid that others. These modes are found in three basic types. First there is what is termed as ‘initial trust’. This is the trust that you need to get up out of bed in the morning to face the world. This is basically the concept that the world is not outright hostile and that while still a jungle you have trust in your own ability to make progress in it. A good example is that in most neighborhoods you can pass someone on the sidewalk and ‘trust’ in the fact the individual will not try to attack you. Note that this requires a two way equation, the other individual has to have the same perception. This is a key ingredient and provides the bootstrap for the other two – more sophisticated forms of trust. Another commonly used term for this is trusting at ‘face value’. Second is something termed as ‘associative trust’. This is the extension of trust to someone or something based on the reference and recommendation of another individual in which you have already established a trusting relationship. Both initial trust and associative trust could be classified as temporary states of trust that require the third and last mode which is ‘assured trust’. This is where the initial trust is then validated by actual experience or some other system of assurance. This and associative trust provide a degree of historical context to the paradigm and begins to develop the concept of reputation. In essence, (though perhaps not always true) if you were trustworthy in the past it is likely that you will be trustworthy in the future. As an example, if my neighbor told me that a certain kid was great lawnmower, I am more likely to extend the initial trust based on this recommendation. Once the kid performs the job well and up to expectations the mode of trust then becomes extended to ‘assured’. I have seen the job that the kid does with my own eyes (note I have extended some degree of risk here – he could have scalped my lawn) and I am now happy with the job. The relationship with the kid is now direct, between my self and him or her. The neighbor has faded off as the relationship has matured. Although the neighbor’s opinion may still carry some value; for instance if I were told of something being damaged or stolen I might experience a compromise in the degree of assured trust that has been established between myself and the rumored individual. This can begin to uncover the potential corrosive effects of gossip and hearsay in inter-personal relationships but it also shows the capability of social systems to create feedback loops in which trust can be built up or eroded based on an individual’s behavior.

One last aspect to consider is the fact of identity. This may seem out of place in this face to face example. Obviously, I do not need identification to be assured of the fact the neighborhood kid is who he says he is. I can see this with my own eyes and establish it with easy conversation. However, there is something known as abstraction that becomes prevalent in more complex examples of trust. Also, as the assumed risk gets higher along with the increase in abstraction, the need to be certain of an individual’s identity becomes a requirement. As we shall see though, this is not required, or rather it is more implicit in the simpler examples of trust. However, as human interaction becomes more indirect and the relationship of worth to risk becomes higher, getting assurance of an individual’s identity becomes explicitly paramount.

I have this goat that I would like to trade for your cow

Since it is established that trust is a major requirement for a human societies, it makes sense to look at the phenomenon in the context of human societal evolution. For this, we need to look at the historical use of trust, particularly prior to the recent era of technological innovation. This will serve two purposes; first it will provide once again a simplified view of the paradigm. In a sense, it provides a form a reductionism because all of the newer trappings and manifestations of trust that technology requires are removed because they simply did not exist yet. Second, it will serve to provide a view of the phenomenon of trust in context of both social and commercial scopes. As an additional note, the following historical analysis is decidedly ‘western’ in its recourse and perspectives. This is not to indicate that the concept of trust or any of its resulting paradigms are solely western. The focus on western culture is done for one simple reason. Covering all cultural manifestations of trust and their evolutions would be exhaustive and well beyond the intended scope of this paper. Additionally, most if not all of the foundational concepts such as credit and currency are, aside from cultural trappings, largely the same.

If we go back to the time of hunter-gatherers, trust was something that was somewhat limited and narrow in scope. The limitation to the scope was simply because of the fact that humans had contact lists that were numbered in ten or perhaps twenty individuals. These were the individual’s tribe. This is where literally one hundred percent of social interaction took place. Additionally, these individuals were most often direct relations to the individual so there was still a grey area between genetic familial interactions versus interactions of a true non-familial social context. The scope was limited simply because humans did not ‘do’ a lot. We pretty much spent most of our time gathering roots and tubers as well as hunting.

While things were admittedly more limited back then, it could be argued that the table stakes were much higher. A single individual who makes a mistake in a large animal hunt could injure or kill themselves and perhaps several other prime members of the tribe. A single individual who did not know the difference between benign and poisonous plant species could endanger the whole tribe. So while the scope was both limited and narrow, the context was everything. In the high stakes game of Neolithic hunter-gatherer societies a single error would often spell disaster for everyone. For this reason the time of education was often well past adolescence and into young adulthood. Accompanying this were (and still are) complex initiation processes and ceremonies which are basically symbols of the tribes extension of trust to the individual as a fully functioning member of the society.

Here we still have the basic three vector relationship of figure one. Indeed, in order to be universal to the paradigm it needs to be so chronologically as well. There is still, 1). The context of the trust – I will trust you next to me with a spear; 2). The level of assurance – I worked with your father to teach you; and 3). The resultant extension of trust – let’s go hunt something that is ten times our size together. While the whole paradigm is much simpler, the stakes are very high. In some ways they are ultimate. Almost equivalent to the level of trust extended in the example of the ‘Foot Ball’.

With the invention of agriculture the phenomenon of trust had to change and evolve. At first, this was a simple extension to the Neolithic hunter-gatherer model. If you lived in a village on the Asian steppes at the end of the Stone Age it is likely that you were very isolated. It was probably unlikely that you ever saw an individual from a neighboring village as these other villages were often hundreds of miles in the distance. Consequently, the scope of trust was still limited to the tribe. The scope while still limited was becoming less narrow however. The reason for this is the element of possession. With the advent of agriculture and animal husbandry came the concept of possession. After all, if one and ones family spent their time and energy to raise crops and herds. There would undoubtedly evolve a sense of worth and ownership of that worth. With this came the concept of trading and bartering. The introduction of this simplest form of commerce occurred simply because it allowed individuals to specialize and thereby maximize the resources available to the tribe. At first this may have been communal, but as time passed and certain trades became differentiated, a sense of value for those trades became evident. We can see this from the archeological evidence of the early Bronze Age.

Trading in this context almost always happened within the tribe. External trading between tribes did not really occur in mainstream until the advent of the chiefdom. There are several reasons for this as we will later see. At the earlier stage, because of the limited scope, trust was often established on a handshake basis. If an individual wanted to trade an animal for some grain or another animal. The individual in that tribe who specialized in that trade was approached. There was often direct personal relationships that went back several if not dozens of generations between families. Trust you might say was embedded.

Something interesting also happened around the same time. Gradually, it came to light that there was not only a sense of worth for what an individual owned, like a goat or cow; but it began to extend to services that one could render. Such skills as medicine, metal smith, and yes even religion and tribal leadership (often which were synonymous at this stage) could be classified as such. With this splintering of occupations came the abstract concept of a contract. Even though the agreement was more often than not implicit and verbal, it was typically done with witnesses, was based on familial honor and the tribal penalties for breaking good faith were often severe.

As societies embraced all forms of agriculture there resulted in every instance a surge in population within the societies. This created a positive feedback loop that actually better enabled the tribe to survive and in turn grow further. That is… at first. It is commonly assumed that resource shortages are something that is new to humanity. This most definitely is not the case.  Many early societies quickly outstripped their surroundings of one resource or another. Often this resource was water. It is not a coincidence that the first advent of organized chiefdoms occurred in semi-arid regions that were tipping towards further arid conditions. Whether this happened because of communal agreement by all members or by force through a stratification of society (it was usually a combination of both), it is undeniable that this was a trend that occurred globally at various times in pre-history. As this happened throughout the Bronze Age there was an implicit extension of trust to the leader of the tribe that came along with it. It was not always given willingly, but in most instances it was absolute. With this came the evolution of the ‘divine’ rights of chiefs and their families and the quasi-religious merging of tribal leadership and religion that is often a signature of this stage in societal development. Even with this however, most chiefs did not long survive breaches of trust with the populous, at least at first. As ruling classes became more powerful, rule by force became possible and indeed many times attempted. Many things changed at this point as we shall see. Humankind had reached a sort of critical mass.

I want my silk and I am willing to do what it takes to get it

As human society progressed these isolated communities began to reach out and establish contact with one another. The reasons for this were varied but there is no doubt that pre-historic trade was widespread and would even traverse continental boundaries in some instances. There is one thing that is true with primates and humans are no exception. Once different societies or cultures establish contact, ignoring one another is not a long term option. Sooner or later they will interact. Whether this interaction is peaceful or warlike is to a great degree determined by trust. Societies that trust one another tend to establish trade and share cultural traits and ideals. Societies that do not trust one another tend to avoid contact and when they do have contact it tends to be of a violent nature. Again, we can get into boggling possible iterations that might occur for a virtually unlimited set of reasons. In some instances there may be vast ideological differences that cause the animosity. In other instances, (and it should be noted that this is by far the predominant cause) it was based on something known as circumscription. This is when one society sees another in a predatory sense. Most often the reason for predation was for territory or resources, both natural and human. What is important is that this trend again was self reinforcing. As the prevalence of aggressive societies increased there in turn increased the need for strong leadership and military capability within societies as a whole to either carry out the acts of circumscription or defend oneself from it.

At this stage of societal development we see each of the great civilizations enter to the empire phase. This phase which some would argue not to be a phase but an integral characteristic of human culture has dominated our history. As we shall see however, any empire that withstood the test of time realized that in order to do so one must have willing, or at the very least submissive subjects. These subjects must see the empire as the greater good or at least the lesser of two evils. Here we see the beginnings of the concept of a social contract known as citizenship. Where there are certain benefits, privileges and rights to being one. This is something that reached an ancient epitome with the Roman Empire. The wiser emperors were very astute to this concept. Some were masters at public display and acts of imperial benevolence done in a public fashion to assure wide reaching knowledge of the act. Such acts were cheap in relation to the revenue and value that it served to continue securing for the empire. In addition, there was the constant presence of hostile neighbors, which the emperor did not have to necessarily manufacture to create the additional rationale of keeping distant kingdoms within the fold. After all, if the emperor placed enough legions in the locale to defend it, it was often of the dual purpose of keeping it subdued as well.

None the less however, the Romans were keen on extending citizenship. It was once boasted of the Romanized Britain’s that they were ‘more Roman than the Romans themselves’. They were certainly no exception. It was very common across the empire to see a sense of membership in it. Some kingdoms were more willing subjects that others but by and large an entity as large as the Roman Empire simply could not be ruled by force alone. Again, the wiser emperors understood this and leveraged it to the hilt. There was a sense of pride and trust in being a Roman citizen. Particularly if you were a free merchant who looked at trade abroad (across the Mediterranean) as desirable.

Parallel to this is the development and maturity of two other concepts. One is the independent representation of worth. This is the development of a system of currency. This was certainly not new with the Romans but they did bring it to a level of maturity and perfection that can rival the process of today’s mints. Another thing that they did was remove any local intermediary to imperial allegiance. Roman citizens were to declare direct allegiance to the emperor, not to the local king who then claimed allegiance in turn. Each citizen was to take the oath directly. In this sense, a king was no different from his subjects. This way allegiance was not to local kings who could come and go (and be deposed at will by the emperor) but to Rome itself which stood ‘forever’ and was the greater good or greater obligation depending on your perspective. In either case, it superseded any allegiance at a local level.

With such systems in place trade was seen to prosper within the empire. Along with this surge in trade came the relative prosperity of the provinces that participated in it. Aside from the benefits however, there was the required abstraction of worth that came along with it. Within this more sophisticated commercial environment there were many intermediaries. With additional parties and complexities came the inevitable individuals who attempted to circumvent the system of governance. In the simple Neolithic village trade, it was very difficult if not impossible to subvert the trade. The trade was face to face, based on the trust of family to family and the transaction was solid not abstract. It was a real time exchange. There simply was no opportunity for infringement on the transaction. With the introduction of sophisticated monetary based commerce, this was no longer the case. There was now plenty of opportunity for enterprising but less than honest individuals who could now make a ‘little extra’ on the side within the normal flurry of business transactions. As this occurred, more formal systems of governance were created to provide the additional assurance that goods and services were rendered fairly and appropriately. Again, this is not new with the Romans but it could be said that they brought the concept of governance and law to a level of true maturity that there to fore had not been attained by any civilization (perhaps with the exception of China). Indeed, today many countries still base their legal systems on the precepts of Roman law.

If we look at all of this we can begin to see a resonant balance of concepts. Some, like the legal system are positive and reinforcing; others like thievery and embezzlement are negative and corrosive. Others can be either such a reputation. It is the delicate balance of these negative and positive influences that create an ecosystem of trust with the ultimate trust ecosystem being the very existence of civilization itself.

In the late 5^th century the emperor Justinian had an issue with getting access to certain eastern products. Justinian tried to find new routes for the eastern trade, which was suffering badly from the wars with the Persians. One important luxury product was silk and the famed purple dye used to color imperial robes, which was imported and then processed in the empire. In order to protect the manufacture of these products, Justinian granted a monopoly to the imperial factories in 541 AD. In order to bypass the Persian land route, Justinian established friendly relations with the Abyssinians, whom he wanted to act as trade mediators by transporting Indian silk to the empire; the Abyssinians, however, were unable to compete with the Persian merchants in India. Then, in the early 550’s, two monks succeeded in smuggling eggs of silk worms from Central Asia back to Constantinople, and silk then became an indigenous Byzantine product.

What we see here is a natural progression of steps that served to provide stronger assurance to Rome that it would get the products that it valued. The first set of steps attempted to remove unpredictable and hostile trade paths with those which were more friendly and stable. The final steps moved to remove intermediaries all together and thereby attain the highest level of assurance by direct control of the product.

All of this was for naught however. Despite all these measures to protect trade, the empire suffered several major setbacks in the course of the 6th century. The first one was the plague, which lasted from 541 to 543 and, by decimating the empire’s population, probably created a scarcity of labor and a rising of wages. The lack of manpower also led to a significant increase in the number of “barbarians” in the Byzantine armies after the early 540s. The protracted war in Italy and the wars with the Persians themselves laid a heavy burden on the empire’s resources, and Justinian was criticized for curtailing the government-run post service, which he limited to only one eastern route of military importance, the silk highway. Also under Justinian I, the army which had once numbered 645,000 men in Roman times, shrank to 150,000 men.

What this in essence shows is that even whole civilizations can collapse under the weight of history, bad circumstance and limited decisions by the ruling party. As the trust in the systems of governance waned, individuals tended to seek security at more local levels. As this happened the implosion of the culture was a certain result. The imperial contract was broken. Feudal society became the method de jure for the next one thousand years.

Adam Smith’s hidden (but shaky) Hand – the rise of the Market

It could be said that as the Roman Empire fell there was a pulling back of trust to the more local and limited scope that was prevalent prior to its existence. It would take several hundred years before economies and systems of trust and governance extended beyond the castle walls once again. With the advent of the renaissance and the rise of the merchant class much of the momentum that had been lost with the fall of Rome began to be regained. Gradually and with an accelerating pace Merchant and Guild classes began to develop.  Modern nationalistic attitudes began to appear and the concept of a ‘marketplace’ began to evolve where trading could occur with the assurance that transactions would happen in a lawful and orderly fashion. Once again we find the threefold vector relationship of context, assurance and trust that served to set the foundations of an independent but entirely abstract entity known as the Market. At first, these early markets were largely under the control of the trading companies. Individuals or businesses could gain a stake into the lucrative potential gains (and associated risks) of ‘global’ trade by investing in shares of the trading company. With this revenue, the trading company would be able to pay for the building of the required ships and crews for the expanding trade routes. The investors made their investment based on the trust in the worth of the shares that they bought. At some point in the future, if the trading expedition went well, the shares would be worth some value above what was invested.

Back at home, less adventurous individuals would focus on crafts trades by gaining access to one of the many Guilds that were springing up across Europe. Again there was an element of trust here. In this instance there is trust in the organization. There was trust in the fact that if one joined a Guild and went through the appropriate training and apprenticeship, one was more or less assured of getting a job upon completion.

As these social constructs began to gain momentum they found an eventual convergence in the industrial revolution and the rise of the modern trading market place. During this time a new branch of science began to be developed known as economics. One of the practitioners of this discipline known as Adam Smith noticed that there was a resonant feedback mechanism between profit and competition that seemed to keep the market balanced so that products and services were levied at fair rates of exchange. This he coined ‘the invisible hand’ of the marketplace. As the concept evolved, several practitioners began to assume that the market was predictable and could be ‘trusted’. This was based on the assumption that market behavior was essentially Gaussian and that it, in combination with this ‘invisible hand’ would serve to provide an overall stability to the marketplace.

As we all know now, this assumption was largely incorrect. The stock market crash and the following deep depression were largely fueled by an overextension in the market that was based on this false assumption of predictability. As a matter of fact, one week prior to crash of October 1929, Irving Fisher of Yale University who was perhaps the most revered US economist of the time claimed that the American economy had reached a “permanently high plateau”.  As little as three years later the national income had fallen by over fifty percent. In essence, no one, not single economist saw it coming. This was a prime example of misplaced trust and overconfidence that had been built up over the centuries from the initial days of the cognizant risk that was assumed by those investing into the early trade expeditions. What served to allow this? Again, it was the abstraction of worth and also of the risk assumed on that worth.

When early investors bought into a voyage, there was a direct one to one relationship to the success or failure of that voyage. If the ship went down so did your profits along with the initial investment. There was very little present to abstract or protect from the risk. In the modern marketplace however, wealth could be moved and transferred from one interest to another. This capability gave the impression of lessened risk. In reality the overall risk was spread among various interests, so it did reduce the risk, but in a single investment, and this is a key point. If the whole market crashed as it did on that fateful Monday morning and all of your assets were in the market at that time it did not matter how well spread out your investments were, the market crashed and so did your assets! There was no difference. In essence, the market was your ship.

What this serves to illustrate is that while abstraction allows for greater scale, volume, and agility; it reduces the overall visibility of assumed risk but does not eliminate the risk itself. This is an important principle that we will re-visit once again as we begin to look at the recent trends of trust in e-commerce.

 The new commerce paradigm

When you purchase something on the web today, you very seldom if ever get a chance to interact with another human being. When you think about it, there is a great degree of abstraction in the e-commerce model that the on line purchaser simply needs to accept. This is nothing new. It has been happening gradually over the years. It was even occurring back in Justinian’s day. After all, it is highly unlikely that Justinian ever met the actual proprietors of the dyes or silks in person. He had emissaries that handled his relationships with them. Note also that in the end he chose to remove all intermediaries to the product including the proprietor.

If we think about it, currency is the first level of abstraction that allows for all the others to occur. The concept of independent representation of worth allows for trading at a distance without moving huge hordes of product as barter or direct trade would require. One party could pay for product with currency, typically gold or silver. As time progressed, the concept of currency evolved into a ‘certificate’ paper form that represented an amount of gold or silver, which is then held in a reserve by some organization. One of the first organizations to do this were the Knights Templar in Europe to provide for safe transfer of wealth to the Holy Land for would be pilgrims. This added an additional level of abstraction, but with this new approach a business deal could happen in a totally separate occurrence from the actual movement of product or gold and this is more often than not the case. This is one of the primary tenets of commodity trading. For many centuries, currency through banking and a postal capacity addressed the requirements of distant trade and commerce. (Remember that Justinian kept the postal service to the east.) In more recent times, we can reference the use of the Pony Express and soon after the locomotive that allowed for the significant growth the countries of North America experienced, but the basic paradigm did not change. It was still a combination of currency and postal service. The only thing that was happening was that the information regarding commerce and the product being traded was moving faster.

All of this changed with the invention of the telegraph and soon afterwards the telephone and the further abstraction of worth, the ‘wiring of currency’. At this point the delta of time between information and product truly diverged. It could be argued that it is easier and faster to move a letter versus goods. However, in most instances, particularly with the locomotive, both moved on the same train. Telecommunications made its big impact by the ability to communicate far faster than the movement of goods. As a matter of fact, it allowed for the total separation of commerce information and product flow. This is the primary feature that has allowed for our modern world.

Everything is Virtual (in its own way)

The inception of the Internet could be viewed as a continuation of the telecommunications commerce paradigm. There is however a critical difference. There is a critical set of additional abstractions that it allows for true e-commerce to occur. The first is that commerce is no longer limited to physical commerce, whether it be products or services. Think about it, with a telephone even of the highest quality channel, the only thing I can do is talk to you. Now granted, there are some things of value here. Perhaps even valuable enough to pay for if I happened to be a lawyer, accountant, or some other form of consultant. The list is pretty narrow though because it has to be limited to talking. The fax machine changed this slightly so that now I can send a facsimile (hence the term ‘fax’) of a document and then talk to you about it over the telephone. There is more value for the service here. In the case of legal consultation, it might be a contract or agreement. In the case accounting it might be a balance sheet or cash flow statement. In either instance the value of the service is increased because you did not have to wait for two or three days for the letter or document to reach you by mail before I can call you about it. For quite some time, this was the state of the art for business communications.

With the Internet however whole processes and services can be productized in a virtual fashion and sold electronically. In essence, currency moves (virtually as well – we shall discuss this next) and nothing happens physically. No product is shipped; no person picks up a hammer or a shovel as a result. Something happens in cyberspace instead. More importantly, something happens in cyberspace that creates an eventual real world result.

There are many companies that serve as examples for this. Paychex™ provides electronic outsourcing of company payrolls. EBay™ provides an on line auctioning service where folks and companies can sell their belongings and products in a virtual garage auction type of setting. In all of this though, on line stock trading is the one with perhaps the biggest impact on the movement of wealth in today’s world. This ability has greatly improved the trader’s response time to market trends. This is accomplished not only by the use of the Internet and computing but by the removal of the intermediaries. (Sounds like Justinian doesn’t it?) While this has certainly been a boon for the typical individual many economists have indicated that the implications can be a knee-jerk economy, where herding behavior among trading communities can be greatly accelerated, sometimes to the detriment of the market.

Along with the virtualization of products and services there has been an equal and parallel trend in the virtualization of wealth. Much of our wealth today is paid out to us and then relayed to those we are indebted to without ever being realized physically. In other words whole cycles of revenue transfer happen in a totally virtual context. As an example, my mobile phone bill is automatically paid by my corporate card, and my corporate card is in turn paid electronically out of my checking account which is funded by electronic deposit by the company’s payroll service. None of the monies ever becomes physically realized. It is the transfer of the balance (in essence nothing more than a number) that moves the wealth. Indeed, at the very base reality it is the manipulation of numbers in different account records that represents the transfer of that wealth. I never touch the gold, but I realize the values of the benefits.

When we put these concepts together we arrive at the contemporary paradigm of e-commerce. Let’s take the example of an individual that buys a product on line and uses a credit card. The e-vendor charges to the account number and the individual incurs a charge on their account. They may have the card set up on an automatic payment from a checking account which in turn is funded by electronic payroll deposit from the company they work for. Everything in the end to end commerce flow is virtual. The only tangibles in the whole end to end commerce model are the hours worked by the individual and the product that (hopefully) eventually arrives at his home in good condition. This is something that most folks simply take for granted. They trust the paradigm. There are others who are more cautious, those who only trust a part or portion of the paradigm. An example would be an individual who is completely comfortable with electronic deposit from their company but prefers to write a check (which is in turn a paper abstraction of wealth that could be viewed as a precursor to the current paradigm) to pay their credit card bill. This same individual however, might be totally amicable with purchasing a product on line from an on line vendor using that card.

Then of course, there are those who would trust no such abstractions. Indeed, there are those who insist on being paid in cash and would not relinquish that cash to any entity for holding. All of their charges and bills they incur and pay on a personal basis. One has to wonder, in today’s society how limiting and restrictive this approach is. Any extension out of the normal day to day life would require significant effort and expense. As well as risk, this individual is carrying his whole wealth on his person. He is at extreme risk on the physical side. He could be mugged and most probably harmed, perhaps killed for the wealth he carries. So any extension of the constricted life style would be more costly, even if it went as projected. So there can be a cost for not trusting as well.

From this we can see a spectrum of trust, one that runs from total trust where everything is virtual to total mistrust where everything is physical. We could also argue to extend this to say that both are extremes and that as such they would represent the population according to Gaussian distribution with the majority of the population lying somewhere in the middle. At both ends of the spectrum there are extremes of risk as well. On the virtual side, all of the risks are in turn virtual (There is however the real loss of wealth in cyber-crime and identity theft. Most credit companies will protect their customers from any charges incurred – this begins to touch on the concepts of insurance and the spreading of the risk factor which we will discuss shortly), on the physical side all of the risks are physical including one critical difference – the risk of physical harm. Indeed, it is most probable that this was one of the primary motivations for abstraction (virtualization) of wealth to begin with. Recall the Templars, who founded the first embodiment of modern banking. They became powerful and wealthy on the holdings and transferal of wealth for pilgrims to the Holy Land so that risk was reduced on the individual who made the trip. In essence, the wealth was ‘virtualized’ during the trip. There was a degree of separation of the individual and their associated wealth. Over course of the sojourn the individual was fed and defended (for a substantial fee) and when they arrived at the Holy Land they could cash in their deposit checks and they were flush once again. The revenues were transferred by more secure military means or more ideally, the revenues existed in Jerusalem prior. Either way, the pilgrim received their gold at the end of the trip, less the substantial fee of course.

 Go ahead – everything will be alright…

If the aspect of risk is somehow primary to trust then there is a related value in the level of assurance provided to the individual entity that enters into the relationship as well. Again these are related in a vector relationship that is exactly that as shown in figure one. As the level of risk gets higher in the trust relationship the level of assurance must in turn be sufficient to ‘cover’ it.  There are more dimensions to consider however. We need to consider the aspect of reward.

Reward could be considered to be a positive dimension of risk. The two exist in opposition. As the ratio of reward to assumed risk becomes higher, it is more likely that an individual will move forward and assume the risk. It is almost as if an individual reduces the risk factor in their own mind when taken in context of reward. This is what causes individuals to do things that they would otherwise not ordinarily do, such as clicking on an icon on a questionable web page. In instances where the degree of risk is higher than the potential reward an individual is likely to pass the opportunity by. This relationship is shown by the diagram below. Note that there are two vectors in this diagram one is the lower risk or liberal risk vector because the expected level of assurance is lower per given equivalency in context. The higher risk vector represents the more conservative risk vector, as stronger expectation of assurance is expected for relatively lower extension of trust. The sinusoidal line in the middle represents the decision vector of the individual or entity. It is represented as such because it could be described as a waveform that is unique to the entity. Some individuals or organizations may be fairly liberal, other may be more conservative, but each one will be sinusoidal in that the decision hinges between perceived potential risk and reward. It is also important to note that at the nexus of the graph the sinusoidal pattern is smaller and increases in relation to the absolute boundary vectors which illustrate the potential range of decision.

Figure 2. The relationship of reward and risk in trust

Note that as the risk and reward grow more significant the sinusoid grows in relation; which represents the state of ‘indecision’ that we typically encounter in high stakes affairs where the risk and reward potentials are exceptionally high.

This is common sense to some degree. Few of us would argue this. However, there are a few important points to consider that are pertinent in today’s ecommerce environment. First, when we say assumed risk or potential reward, we mean ‘perceived’ assumed risk or potential reward. What an individual perceives and what is really occurring are two totally different things. Herein lies the root to all scamming and racketeering activities and the addition of a cyber environment only provides another level of cover for further abstractions between perceptions and truth.

The second important consideration is that assurance (or insurance) can change this relationship. Both can serve to decrease the degree of risk assumed and hence push the individual in the direction of a positive decision.

As an example, neither you nor I would purchase a book from an unknown vendor on line with no validation and no privacy. The level of risk (placing your credit card number on line unprotected) versus the reward (a book – that you must want otherwise we wouldn’t be having this thought exercise) is simply too high. However, if it is a well known vendor and your credit card information is held in a profile that does not go on line, the level of risk is minimal and the purchase becomes a very trivial decision that is almost equivalent to standing in an actual book store. This is even more the case if you happen to have coverage on your credit card for fraudulent activity. This is illustrated by a modification of the figure below. As systems of assurance are put in place they provide a positive ‘pressure’ on a given situation. This pressure serves to reduce the perceived (and hopefully actual) degree of risk.

Figure 3. The positive influence of increased assurance or insurance

From this we can deduce that providing increased assurance to individuals who participate in ecommerce is a good thing and will produce positive results. This is indeed the case. It also means however that individuals can be misled. They can be misled either by the degree of the perceived reward (think fake lotteries and sweepstakes) or by the degree of perceived assurance (anonymous SSL is the main avenue here). Many scams will try to do both. A good example is a sweepstakes email from a seemingly reputable company name that has the happy news that you are the winner and you only need to fill in some required information on a ‘secure’ web site. You even get the SSL connection with the lock icon in the bottom on the browser screen! So assurance is a two edged sword. If the potential reward is big enough and the ‘illusion’ of assurance can be provided, then the basic ingredients for a scam are present.

This can be carried further by the ingenious but nefarious use of software code that can provide the ability to place key loggers, bots and Trojans on a users’ PC as a result to the mere visiting of a web page. Once the code is resident, all sorts of information can be garnered off of that compromised system. With this approach there is no need to dupe the user into entering anything on-line. The malignant party need only wait for the scheduled updates from its cyber-minion. That is all that is needed in this scenario is a moment of indiscretion on behalf of a user who is ‘dazzled’ momentarily by the perception of some great potential reward. The code does the rest.

So what is a user to do? It seems that we are going back in a cyber sense to the days immediately following the fall of the Roman Empire or in the days of the Old West where your very survival often depended on the whims of the environment. Interestingly, there are many analogies about the Internet and the Old West. We are now at a point in evolution where the analogy to the time following the Roman Empire (known as the Dark Ages) may be more appropriate. Many of the malicious parties are no longer just college kids or folks looking for a quick buck. As systems automation has become more prevalent many malicious activities are being sourced against infrastructure. Some of these activities can even be traced back to national, religious or political interests. So things are getting into the big leagues and like a good ball player, we need to change our mentalities to play in the league.

In this model, you might view the typical enterprise as a feudal kingdom that lies behind solid defenses of rock and earth. From these ramparts an enterprise does its business via various ways of securely providing for access across its defenses. As we carry the analogy further, the single Internet user is like a peasant in a mud hut outside the walls. Their defense is only as good as the probability of contact with malicious forces. They may run anti-virus software and have security check updates, but the real bottom line is that there is always a lead with malice ware, just as there is always a lead in weapons versus defense. If the user is frequenting unclean sites then it is only a matter of time before they contract something that neither the security checks nor the anti-virus software recognizes… that is until it is too late. So the analogy is very good. In the Dark Ages, if you were living in a mud hut you where at very similar odds. If no one came along, you were fine (the analogy here is that your software is up to date and recognizes the threat)… if not, then not; because most often your defenses were paltry in comparison to those who threatened you.

 So what does all this mean?

 What we will do now is take a look at the information regarding the subject of trust that we have gathered by our walk through history and see how it relates to these modern day issues. Some of the results that we will find will be obvious, other results may be startling. Some may even provide discredit to some major industry trends. In all of this it is important to keep an open mind and to remember that history often does repeat itself – it just happens in a different context.

First, let’s be clear. The Internet was never like the Roman Empire, except perhaps in the earliest days of DARPA. From the outset, the analogy of the old west or the dark ages was the most appropriate way to describe the environment. What I would like to do however is bring the analogy a level higher in scope and say that the typical enterprise is the typical empire or kingdom and that each enterprise is responsible for its own domains and the interests that its enterprise represents. This is certainly a valid analogy in that even Rome co-existed with other empires though not always peacefully. Persia and Carthage are two examples. So in a similar fashion different enterprises may be seen to interact, sometimes friendly such as a supplier relationship, other times not; such as a competitive relationship. This however is not the point. The point is that each enterprise is responsible for securing its own domains, just as each empire was responsible for theirs. Here the analogy is true. As an enterprise, my organization can not be made responsible for the security of my suppliers or even my customers. It is up to them to make sure that their own house is in order. The bottom line is that some may be more diligent than others.

So what is the first thing that we can draw from this? Well, first off empires existed by virtue of the ability to leverage wealth. They did this by maintaining well protected trade routes to the various other empires or nations that provided or desired products for trade. We might view Virtual Private Networking and data encryption as the modern day equivalent of this. Business to business connections happen securely when they are properly administered as their widespread use can testify. (Note however that recent attacks on IPSec VPN gateways have been documented, just as attacks on well protected trade routes occurred.) Secure remote connections can happen for end users within enterprises (I am using one now) as well. All of this can occur because the enterprise, like the empire has the ability to set the policies for its security practices.

Like well protected trade routes to the empire, VPN’s are only a part of the answer for the enterprises defense. Each Enterprise also has a well protected border that is maintained by threat protection and security devices just as empires maintained well protected borders by the use of armies or legions.

In the industry today there is a major push to an end to end security model. In this model, everything is authenticated and encrypted directly from the user’s device to the server that they are accessing. This approach has it’s benefits but it also has a drawback in that intermediate security devices such as threat detection and firewalls are blind to the traffic that is coming across the border. As such, encryption could provide a cover of darkness for a would-be attacker instead of providing the protections that it was intended for. Parallel to this is a major thrust for the decomposition of the security framework within the enterprise. In this paradigm, intermediate security devices are labeled as antiquated and not up to the challenge of protecting the enterprise in today’s e-commerce environment. Instead, the function of security becomes increasingly resident in the server and the client in the end to end scenario. If we carry this analogy to the empire, this is equivalent to leaving the borders less protected in lieu of depending purely on trade routes. This brings to mind Justinian’s reduction of the armies and the resultant reduction in control of territory that the empire experienced.

Perhaps a clearer analogy is the foot soldier. This is a paradigm I like to term as the ‘Naked Samurai’. In this analogy, the trend of security decomposition can be made equivalent to a Samurai who disrobes of all armor prior to entering into battle. (While this was never a practice of the Samurai, it was known to happen with Scots, which scared the devil out of the Romans – but didn’t do well for the attrition levels of the Scots. It should be noted that they eventually abandoned the practice and started to use armor like everyone else.) In order to survive the endeavor, the soldier must be flawless in his reactions. Each response must be perfect because ‘any’ error would be grievous. Even a minor injury would prove fatal as it would likely lead to further errors via pain and blood loss that eventually would prove to be his demise. As a result, no sane soldier would enter into the thick of battle without armor and yes medieval Japan had its share of armor. In many ways, this is equivalent to the current decomposition trend. In the end to end encryption paradigm, the first point of defense is the last point of defense. As a result, any threats that the server experiences it must be perfect in its response to. As we covered earlier, it is not always possible for security code updates to catch the latest mal-ware. In this model it is also not possible to always monitor or protect the client end system because the client may very well visit sites that are compromised. As the client system gains access to the server, it can then in turn infect the more important system. Without intermediate security, there is nothing that can be done to rectify the situation.

To carry the analogy further, this is parallel to the fall of the empire and the rise of the feudal kingdom in its place where the feudal kingdom becomes analogous to the server. Arguably the feudal kingdom like the server is less able to defend itself than the empire like the enterprise. Most certainly, any defense it does have is much more local and as a result much more easily compromised. More so, once it is compromised there is no cavalry to rescue it because the intermediate security devices are blind to the encrypted traffic. Also consider that the compromised system is now an enemy outpost within the enterprise data center where it can further entrench and infect other systems. This is analogous to the Dark Ages castle opening its drawbridge and filling in its moats. All folks coming into the castle are escorted by a secure squad of guards to their place of business. All of this sounds well and good, but no one did it. Why? Because, such a practice would have been construed as insane.

It is clear that a good security practice involves a combination of components. It is also clear that security has strong impacts on degrees of assurance, whether it is for medieval merchants or for e-commerce enterprises. Secure borders, rock walls, earthen ramparts, armed guards and armed trade caravans, all of these were required in order to fully secure a domain of interest which was the empire. The very same thing holds true for the enterprise. To succumb to the notion that defending the border is just too difficult is to succumb to the notion that destruction or at the very least fragmentation of the larger entity in question is eminent. No enterprise would accept such a notion, just as no empire would. Yet, empires have fallen for these very precise reasons. Ominously enterprise networks, particularly those that depend on e-commerce within their business models, could be viewed in very close analogy here.

Fortunately, there are differences in the fact that unlike empires, enterprises do not have to control all of the territory connecting their sites in a physical sense. They do however have to deal with the secure inter-connections across vast geographic domains. As a result enterprises require multiple layers in the security model to properly protect its resources and interests. Firewalls, VPN gateways, Threat Detection & Remediation to name a few, as well as end to end security are required to totally secure an enterprise. All of them provide value, the question then becomes – ‘How do two paradigms like end to end encryption and intermediate security devices co-exist and provide value to the enterprise?’ Well, the answer is rather straight forward. It is the same as that which provided the answer for the empire. It is a term known as ‘Federation’.

I’ll trust you if you’ll trust me

Merriam-Webster dictionary defines a ‘federation’ as an encompassing political or societal entity formed by uniting smaller or more localized entities: as a:) a federal government b:) a union of organizations 2: the act of creating or becoming a federation; especially : the forming of a federal union. Extending this into the area of security technology it is interpreted as a system for common governance and implementation of consistent policy for the domains of interest. I say ‘domains’ in plural because this is one of the major uses of federation, the tying of enterprises for B2B usage. Such an approach allows for the ability to extend trust across domain boundaries for very specific reasons as well as the ability to limit any such trust only to those services that are made open. This is analogous to the opening of the draw bridge or the border to a trading party that has established friendly intentions. The figure below shows such a relationship. In the diagram we show an enterprise (enterprise A) that has a relationship with three other companies (B, C &D). One is a supplier to enterprise A and is connected to enterprise A over a provider network. In this scenario, the two companies use an actual VPN with dedicated gateways. Both enterprises extend basic trust and each one administers their own relevant firewalls and access control policies but they will trust the credentials of the other enterprise by the use of federated digital identity.

Figure 4. An example of a Federated Business Ecosystem

In the other relationship Enterprise A is the e-commerce vendor and has business relationships with enterprises C & D as a supplier of products. For these relationships enterprise A provides a secure web services portal over a provider network. In this scenario, there are no VPN gateways. Instead, enterprise A provides directory services for its customers based on a federated B2B relationship. As a result to the federation, the enterprise trusts the credentials that enterprise C & D users offer when they access enterprise A’s secure web portal. As they gain access to the portal they are in turn offered a certificate based secure encrypted transport via SSL or some similar method. Once that occurs they have access to the secure portal and can do their business within the allowance of the access control policies. Note that while Enterprise A has relationships with all companies, there is no provision for direct connectivity between Enterprises B, C & D in the context of ‘this’ business ecosystem. Other contexts may allow it.

Further federation of the internal security frameworks would allow for the autonomic modification of security policies (i.e. Firewalls) and access according to the higher level governance of the policy environment of the larger Federation. Federation allows all of these companies to interact and execute a business ecosystem in a relatively secure fashion that does not demand undue opening of each company’s security border.

Sidebar – The Neurobiology of Trust

Recent studies have shown that the phenomenon of trust is strongly related to the quantity of the hormone oxytocin in the bodily system. A monitored test with a variety of a game of trust indicated that during periods of relatively trusting interactions the hormone was seen to markedly increase in particular portions of the brain that revolve around facial recognition and social interactions. Conversely, the hormone was seen to decrease in instances where the other players actions illicit a feeling of mistrust. Along with this decrease in oxytocin, there are also telltale ‘fight or flight’ indicators such as colder hands – which reflect the surge of blood to the body core. Furrowed brows are another key indicator along with escalated heart rate and corresponding increase in blood pressure.

Additionally, other studies have shown that facial expressions or genuflections that are meant to indicate friendly intentions such as waving or smiling will also cause a marked increase in the presence of the hormone.

The question remains on how whole heartedly trust can be generated and maintained with the at best indirect human interactions that are often the case in ecommerce situations. These studies do indicate that there are biological reactions that can actually be measured within the human brain. This fact leads to the possibility of designing ecommerce sites where test users are monitored for the presence of oxytocin in the system as they navigate through the prototype site. Such design approaches will allow for the redesign of ecommerce sites that are better suited to the human aspects of trust. In the future, real-time biometric sensors may be able to report some of these indicators back to the ecommerce site to provide feedback of the customer’s level of comfort as they use the ecommerce site.

 What about the guy in the mud hut?

 All of this is well and good for enterprises, but what about individual users who are not affiliated with an enterprise? Unlike the enterprise, these individuals do not have the convenience of large budgets for security. The analogy here is very close to the farmer who lived in the mud hut and traded his wares with larger kingdoms in return for the needs of life. When you think about it, the e-commerce paradigm is quite frightening for these users. They are using a network that they do not administrate or control to gain access to services that they also do not control to purchase products. Very often they are required to put fairly sensitive data into the web interface that they are using. All with very little level of assurance that no foul play will occur. When put this way, it is a wonder that anyone does anything on line that has to do with credit or financing. Yet, many do. The convenience outweighs the perception of risk. Even with this motivation however, the level of internet sales during the Christmas holiday season has experienced a sharp decline with many folks opting to investigate on line but actually get in the car and physically go to the store to buy the product in person. Internet sales were shown to be down forty percent during the 2006 holiday shopping season. While the numbers are not yet in for 2007, many fear that it will reflect further depressed numbers. When asked why through the use of surveys and such, many users cited fears about identity theft and the commandeering of credit cards for illicit use, and this concern is to some degree validated. A study by the Federal Trade Commission (This study can be found at: http://www.ftc.gov/os/2007/11/SynovateFinalReportIDTheft2006.pdf ) has shown that Identity Theft reports hover at around 4% of the surveyed population with losses totaling 15 billion dollars in the 2006 time frame with an average cost per user to be around five hundred dollars. These statistics are intimidating. Moreover the experience of identity theft is even more so. Most users become very leery of e-commerce of any kind once they become victims. Indeed, many psychologists are saying that the same post stress symptoms that individuals experience after a mugging or robbery are being experienced by folks that are unfortunate enough to experience cyber fraud or identity theft. Obviously, there is no threat of physical harm, but the feeling of violation and loss of control are just as acute. As more users undergo this type of experience, they take it into their social context. They tell friends and family of the ordeal and by word of mouth provide a dampening effect on e-commerce activity by the reduction in the perception of assurance. This is very similar to my neighbor telling me about the shoddy job that the neighborhood kid did on his or her lawn. As a result, I will tend to be more diligent and inspect the job more thoroughly when it is completed and even perhaps pick out some thing I might otherwise disregard. My degree of trust has been compromised because of the reduction in assurance by my neighbor’s comments. This would be even more so if it were a baby-sitter because if the increase in the level of assumed risk. We can find a direct analogy in e-commerce that speaks to some of the reasons behind the downturn in activity.

Clearly, there needs to be some sort of governance for security within domains of public Internet access. Internet Service Providers are increasingly moving to meet this new set of requirements. Many will provide SPAM protection, anti-virus updates, free firewalls and other security related code and services as a bundled part of the access service. The movement to security governance in the provider space is the only way to further secure the guy in the mud hut. Many Internet purists bristle at any such proposal. I would argue that Internet purists are not ‘real world’ just as Adam Smith wasn’t on the economy. There is a real threat to the common Internet user, and security domains of interest (i.e. by the companies who provide Internet access) are the only way to combat the problem. Software updates to users PC’s are only part of the answer however. Providers need to incorporate stronger security policies based on histograms of problematic sources. As users become known perpetrators in cyber-crime or even spamming activities, providers need to crack down and revoke access as well as if appropriate, forward the incident to legal authorities. Believe it or not this actually does occur. During the 2005 timeframe the FBI executed a significant string of arrests on child pornography trafficking; all with the cooperation of Internet Service Providers. Other arrests have occurred in the areas of identity theft and cyber-fraud that show that it is possible to do enforcement, which after all, is a key ingredient in any system of governance.

Given all of this there is still something more. We have thought about protection of identity and privacy. We have talked about active components that can police and provide this boundary of security domain. We have also talked about the role that the user’s machine can play in the security paradigm. There was also the discussion of the federation of these systems and methods so as to provide a coordinated system of governance for infrastructure and policy. What is missing is a key element that goes back to the days of the Templar Knights; reducing the element of reward or temptation.

What’s in your wallet?

 If I have a credit card account that is in good standing is there really any reason to put the number of the account on line when I buy something on the web? Really, think about it… do I really need to do that? Would it not be better to hash out a string that is unique for the transaction and then share it with my card provider (via a dedicated secure connection between us) which I in turn then present in equivalence to the e-commerce vendor? Which would of course, occur over a different secure connection. The e-vendor (if I may use the term) would then in turn present the hashed token to the card provider. The card provider would then research its record of transactions for the user and then (hopefully) find that it is ‘open’ from a transaction standpoint. The card provider would then honor the credit and then update the account. Now yes, there is the argument that someone could steal that hash, but it is limited to the value of the transaction only. It will likewise be a one time only occurrence within space & time that can only be valid for the transaction at hand. Given the speed of most levels of Internet access, there will potentially be only micro-seconds worth in time where a potential thief could ‘steal’ the transaction. Consequently, a strong level of assurance would be provided to the user that ensures their trust.

The key concept of the above is that while abstraction has been the enemy of the commerce paradigm from the standpoint of this paper so far; it is also an avenue for further entrenchment of security services into the e-commerce paradigm. While abstraction from the original concrete transaction (remember the village trading example) has caused a series of potential security holes where criminal activity can occur, in a very real sense further abstraction of certain aspects can help alleviate them.

This concept of the digital wallet yields a system which simply generates credential hashes that are used in tandem with Identity assertion tokens to ‘point’ to entities that can in turn validate the transaction. These ‘pointers’ are only valid for the context of the transaction, with the vendor that it is intended for and for a limited duration. All of this closes the window of risk exposure considerably. The direct credit card information is never out on the wire. There is never any instance of where it needs to be presented. This thereby attains complete abstraction from the actual credit card number information. This is a critical move that greatly reduces the assumed risk for the purchaser. It also significantly lessens the level of temptation for any would be cyber-thief. The level of assurance increases, or more so, the level of required assurance decreases in like (recall figure 3). Provided that there can be a solid way to identify valid e-vendors, the level of assurance with existing technologies could be enough to provide the boost in activity that e-commerce needs at this point in its growth as a market sector.

If such a system could be built that not only incorporated the abstraction concepts described above but also included a consortium of e-vendors and credit card providers, a cyber shopper could then look for the ‘brand’ label that provides the added level of assurance that this is a safe site that participates in the business ecosystem consortium. They will know that they can enter the site and buy something by the hash generation technique mentioned above and that they will not at any time during the course of the shopping cart experience ever be asked to put in a credit card number. But this only works if there is the assurance of participation in the system of governance and the ability to identify oneself as such.

 Just who are you?

 In all of this what is consistent? From the initial Stone Age village trade to the next generation e-commerce transaction what shifts and what doesn’t? Well, as we have seen almost everything shifts. The concepts of representation of worth and the methods for doing so have definitely changed. The methods of advertisement and business have most definitely changed as well. In both instances, the changes have led to more abstract models of function. In turn, the aspect of identity has been necessarily been abstracted to fit into this new environment. But interestingly, it is the one thing that, at the end of the day, has not changed. After all, the human that traded his cow for grain in Neolithic times could be viewed to be no different from us modern humans outside of all of the additional trappings of civilization. In all of this garbled abstraction that has gotten added to the commerce model there is still the humble human who is looking to buy or sell something and of course during the course of business make a few bucks! Even in light of complicated autonomic business processes where the human who is buying something is not acting with another human ‘at the other end of the line’, but is instead working with fairly inhuman process oriented flows; there is still a group of humans who set up the automated process environment. It is also assumed that these humans did so with the intention of making a few bucks. So the fact of identity does not go away with automation. As a matter of fact, it has now become one of the most critical pieces to indicating the success or failure of the e-commerce model. To be clear, while the need for identity has been consistent, what it means has had to change drastically.

If we recall the Neolithic village trading example, we were in a village in central Asia before the advent of the Bronze Age or perhaps right at it inception. The whole population of the village was most probably around one hundred and fifty individuals. Comparing this with most isolated villages in central Asia today would give credence to such an estimate. Given these numbers, it is highly likely if not almost certain that the two individuals knew each other well. It is also highly probable that each individual’s families knew each other as well. In other words, identity was part and parcel to the Neolithic trade. If someone came in from across the tundra with a cow that they wanted to trade for grain the result would probably not be a good one for the ‘would be’ trader.  In real life, he would probably be killed quickly and the cow simply taken by the family that did the killing. At the very best, it would probably work out that the village would simply take the cow and leave the stranger, perhaps bantered about a bit. In any case, the least probable outcome would be for everyone to sit down at the fire and draw up an equitable trade agreement for the animal. Why? The answer is simply that the stranger is not ‘one of them’ and because of his singularity has no leverage. He is not part of the social fabric of the village, so unless he had something really outstanding and had the ability to defend it – and there were points in pre-history where things like this did occur – he would usually be turned away or worse, killed

This is really no different today. We do not kill folks that are not part of our social circle any longer, but someone who is not part of the normal social eco-system will usually find it harder to do business in person to person exchanges. The problem is, with e-commerce it is very hard to hold this kind of line at all. As soon as you go on line, you are dealing with folks you don’t know and probably never will meet. Granted there could be a small percentage of folks who you know who own e-commerce companies, but I think that you will find the list to be quite short. The real fact of the matter is that in most instances you do not know the folks that you are doing business with. This has been cited as one of the major issues that folks have with e-commerce. The fact that there is very little that can be provided to assure the user that they are talking to who they think that they are talking to and that there is no one in the middle.

Identity may be a consistent historical feature in assurance, but in the new e-commerce model the concept needs to change. Clearly, if any real capability for identity is to be brought into the e-commerce paradigm we need to consider the human in the cyber environment. First, all instances of human presence on the Internet are composite instances. The reason for this is that no human can access the Internet directly. All humans require some type of device as well as some type of network access with that device to get on line. The composite goes further as well, there are the aspects of the capabilities of the device; the bandwidth available, the type of video or audio supported, perhaps even the location of the individual as they access the network as well as the application they are using! All of these characteristics build up the composite entity that is a human being on line. The figure below illustrates this concept, note that there is a layered instance of the human over some type of interface into an application which is in turn supported by an operating system for the device and lastly the device hardware itself. All of this together adds up to the complete instance of a human presence on the network. Does this mean that a human with one device is different for that same human using another type of device? The answer is yes. Particularly if there is a significant difference is device capability, particularly in the area of security.

Figure 5. The Composite on-line Entity

 Going further, we could extrapolate this out to non-human instances of presence as well. It would apply to application servers or to thinner types of devices as well such as sensors of the physical environment like video surveillance cameras. In these cases, there is no human sitting at the ‘other end of the line’. Instead there is just a machine. But the machine is also a complex of composite elements. It also has an application, an operating system, hardware elements and many other items that make it a server or sensor device. As the figure below shows, the same could even be held true for the simple video surveillance camera. As the figure below shows, both the server and the camera have interfaces so that a user can log into that entity. It is by this logging in that an association then occurs between the entity and the human, which we must remember is in turn a composite instance on the network. So things can get fairly complex and convoluted in terms of who is who and who is running what. In order to clarify how these relationships can be embodied we will go through a couple of mundane examples of network resource usages and how the aspects of identity are inherited.

Perhaps the best and most clear example of the transference of identity by system log on is in the case of video surveillance. The reason for this is that by logging into the system, the direct visual perception of the individual at the console is literally extended on a virtually unlimited basis. In essence, a person could be sitting in Europe watching real time video (less the latency for delivery of the data) of camera feeds in the United States or elsewhere. This relationship is shown by the diagram below. This is a rather obvious fact. However, one of the things that needs to be considered is that the systems intentions and integrity are directly associated with the whims and motives of the human being that is logged into it. In other words, there is a big difference between law enforcement personnel, illegal voyeurism and potential terrorists.

Figure 6. An example of how identity transits composite entities

The issues get more complicated with automated process flows. In reality all process flows have initial human sources. Even process flows that are completely automated and self configured were designed by humans for a particular purpose. A good example is the recent flurry of Service Oriented Architectures (SOA) that are now the IT industry vogue. Based on web services concepts, a given process or application is packaged into a ‘service’ definition which is in turn represented into the SOA framework as a ‘service’. A service would typically represent some sort of application that drove a business process or a function for an overall business process. An example could be an application that performs order processing or billing within an end to end business transaction. A simple SOA process flow is shown in the figure below.

Figure 7. A Simple High Level SOA Process Flow

It illustrates a simple e-commerce order process flow. Each part of the end to end process is represented as a service within the overall process flow. Each is a web service application or a legacy application that has been adapted to a web services architecture. Each was created by a human being or multiple human beings for a specific purpose. Indeed a good degree of equivalence could be drawn between the old time order clerk, who manually fulfilled the order by paper and the application that now processes the order electronically. Just as there was the old time possibility of the clerk fudging the order and embezzling the remainder, so too there exists today the possibility of an embezzling web service that is purposely designed to accomplish that end. Perhaps more feasibly, a rogue web service could be designed by less than honest staff that could be inserted into the process that might behave perfectly well on the front end. This is shown in the figure below.

Figure 8. An example of a ‘Dark’ SOA Service extension

On the back end that same service that checks and validates credit accounts might export customer credit card numbers to a dark server somewhere on the network before being taken off site or otherwise forwarded. This ‘dark’ portion of the service is not represented in its services description to the SOA environment. It is for all intensive purposes an invisible portion of the service due to the abstraction that SOA infrastructures provide. In essence the only way for this service is by monitoring its conversations and data exchanges directly.

The whole point of this is that systems and process automation do not by themselves, address the issue of trust. In some respects, the issue is made more difficult by process automation. This is particularly true if systems of governance for Web Services within the organization are lax. There is an additional point to this however. Each of the web services is a composite entity. Each entity possesses the capability for damaging activity. How damaging depends on what the service does in the end to end business process. This means that identity is just as critical here as it is in the human interaction model. Additionally, histograms of activity for a service need to be monitored so that any unknown or undefined communications coming from it or going to it are quickly analyzed and dealt with. It must be considered that in this environment, a lot of damage could happen in a very short period of time. Hence, the systems of identity and governance must in turn be automated and extremely dynamic.

The Dark Delta – The difference between perception and reality

Trust is clearly something that is related to our perception of risk. The main problem here is that our perception may not always be totally accurate. Furthermore, it could be argued that our perception can ‘never’ be totally accurate. This gets into some very important aspects of the physical universe and our consciousness and awareness of it and the events within it. In essence, we never see reality as it is. We only see our representations of it within our own minds. At one time cognitive theorists proposed that our minds were in essence reflections of the events that transpire. This implies total accuracy of recall in those events. Subsequent findings have shown that our pictures of reality are in essence pictures that we generate in our head against an inventory of symbols and images that we learned and hold in our heads. What this means is that we do not totally recall events as they transpire but rather will swap and integrate the perceptions of events with the memories and symbols that are pre-resident in our minds. It is by this ‘strange loop’ (similar as in the mathematical concepts of Godel’s theorem) that allows for us to ruminate and in turn induct new symbols and perhaps even create new symbols in light of information received. If this were not the case then insight and invention would be impossible for us. We would simply be mirrors to what we see and react accordingly. This is obviously not the case. But while this innovative twist in cognition plays a critical role in what makes us human, it also introduces some thing that I term as the ‘Dark Delta’. Between the actual physical universe and our perceptions of it there is always a potential delta of information and as you can readily see there is no way to eliminate it. We can only narrow it.

Now we could go as far as to dismiss this to philosophical conjecture. After all, for every day occurrences this delta is very small. Generally, what we see and what we think we see align fairly well. However, let us consider how perception can be thwarted. First off, one commonly unknown fact that because of the latency in perception there is an inherent sub-second delta between what we see and what is there. For normal speeds, the delta is negligible, going at a speed of 60 miles an hour down the highway however our minds perceive us to be 11 feet ‘behind’ where we actually are. This translates to a critical subtraction of the time delta for decision making. The faster you go the more the delta expands, so that if you are piloting a jet going at six hundred miles per hour, your perception is 110 feet off from reality. This is just in speed and the latency of perception. Let’s now add in interpretation. Going back to our Neolithic ancestors, or even to a modern human in the jungles, a Tiger’s stripes can readily be perceived to be part of a tall stand of grasses. Lack of proper match between what you think is there and what is really there could get you killed. This carries forth into our modern world. Recently in New Orleans there was a woman who was approached by a well dressed and manicured man as she exited a quick mart and got into her car. He held out a five dollar bill and said that she dropped it in the parking lot on her way to the car. Fortunately, she had not yet put the change back into her purse and was able to quickly see that he was mistaken. She indicated so and went to close the door. The man quickly attempted to prevent her and insisted that she in turn was mistaken. After she managed to get the door closed the man began banging on the window. She quickly pulled out and away from the location. Shaken, she decided to call 911 and report the incident. As a result, she was contacted by the police and called down to the station. Puzzled by being called down for a seemingly odd but non-criminal event, she soon found out that a serial killer had been operating in the area and was somehow gaining access to women in broad daylight and in a populated area. The police were puzzled at how the killer was gaining access. This woman very narrowly missed what could have been a fatal incident. What saved her? It was information. Because of the fact that she did not put the money back into her purse, she was able to use this informational context to narrow the dark delta. By this narrowing of the delta she was able to arrive at the conclusion that ‘something’ was not quite right.

So we can see that the Dark Delta is not just philosophical mumbo-jumbo. It is something that we deal with every waking moment of our lives. (One could argue that during our sleep the delta is significantly widened – perhaps even infinite). When we move into a cyber-environment, this delta widens considerably. Importantly, it widens not only in the context of perception and interpretation because of the implied levels of abstraction we have spoken to previously, it also widens because of speed. Not the speed at what the user is traveling but in how fast transactions can occur in relation to the awareness of the user. In short, in a cyber- environment things happen fast and we are not always totally aware of exactly ‘what’ occurs. As a result there is a whole underground culture and industry that capitalizes on this expanded delta just as a whole culture and industry grew up around the various levels of abstraction that have evolved prior to cyber-commerce.

We can also show that the context of the delta shifts as well. In the case of the woman in New Orleans, the delta was in the perceived ‘intentions’ of the man. As noted in the previous section, in cyber-space this can extend to the very ‘identity’ of the man. The man can not only pretend to be nice, he can also pretend to be some one that she knows and trusts. This particular expansion of the delta pushes things into a third new critical dimension. Three dimensions being speed (latency), perception of intention, and perception of identity. The combination has fueled a surge of child predator’s that use the cyber-environment to gain the trust and to some degree control over youth that they would otherwise never gain from direct personal contact.

Information and Context – The light that narrows the delta

As we pointed out above, the fortunate woman in New Orleans was saved from what could been a fatal incident by information and context. As a result, information and context needs to be considered in the overall trust model. At first light, we could simply classify it as another issue of assurance and indeed it could be. As we look a little closer however we can see that the information and context more appropriately serves as a degree of ‘measurement’ in the ‘accuracy’ of the assurance. This is a key difference. In the case of the woman, the man’s appearance provided a sense of assurance that there was little risk to be assumed. This misunderstanding however led to the demise of many unfortunate victims. What the added information and context did for the woman was to highlight that fact that somewhere there were inaccuracies in this perception of low risk. Note that she did not know why – but it could be argued that that was not required. The inconsistency was enough to put her on guard and in an alarmed state. You could almost say that the context and information was like a torch or a flashlight that cut through the darkness and highlighted inconsistencies. This highlighted awareness perhaps saved her life.

We can draw the same analogy in the cyber-space environment. Many representations are made in cyber-space. Some are implicit in functions such as IP addressing and naming resolution, others are more explicit such as user identification and passwords. All of them can be manipulated, spoofed and stolen. There are also potential ambiguities as to what is actually on the wire versus what is perceived to be on the wire. Examples of this are Trojan payloads and masked XML data insertions. It is in drawing out these inconsistencies that provide us insight into potentially nefarious activities such as spoofing, insertion attacks and bot-nets. It should be noted that often attacks are caught by the symptoms of abnormality, not by the event itself. Searching for the attack instance itself or trying to find the exact event on the wire is like trying to find a needle in a haystack. This is part of the argument to do away with perimeter security. There is somehow the false impression that once you get authorized access and the appropriate health checks you are good to go of the rest of the time. There is also the false impression that you and your intentions and your machine and its intentions (for lack of a better term) are the same thing. They most certainly are not. You could be honestly accessing your systems and doing your job quite innocently while your machine is mounting attacks and/or running executables to pirate data. It is in highlighting the inconsistencies and abnormalities that where we find the best reference to the clues of such nefarious activity.

It could also be argued that if you wait for the attack and recognize it at the system that is being attacked, you are too late. This provides further argument against the total decomposition of the security perimeter to the server itself. The ubiquitous presence of the Dark Delta further exacerbates this model. The server is by analogy equivalent to the woman and the perimeter security systems to the information and context. The reason for this is that the Dark Delta applies for all entities, not just humans. By removing the perimeter security, the server is left to its own limited perceptions of what is actually going on or coming its way. Also consider that any element of time has also been removed, an attack is real-time and imminent. It also needs to be established that there is also a dark delta in known signatures for attack and virus recognition, so the server itself may not be able to discern a piece of malicious code or data because it has no context to reference and hence provide a match. Recall our symbol matching ability – as an example if you’ve never seen a poisonous snake you are much more likely to identify it incorrectly and perhaps even be willy-nilly in the way you choose to approach it. Such a mistake obviously could be fatal to you.  The security perimeter provides an additional perspective and informational context (equivalent to our internal symbol inventory) that can highlight and narrow the Dark Delta considerably.  It also provides the obvious role of intermediate remediation of any events which we typically attribute to such systems. By creating systems and architectures that can provide context and information that can be ‘cross referenced’ and validated, light can be shown into this Dark Delta and narrow it considerably by removing ambiguities and increasing the accuracy in the perception of ‘reality’ on the wire. Increasing this revealing light for users could potentially highlight inconsistencies in representation and intentions by highlighting unexpected address combinations, network ingress patterns, spoofed system names and addresses as well as whole web sites.

In a very real sense, the same elements that serve to save the Neolithic hunter or unsuspecting victim in a parking lot are the same elements that serve to protect and ‘save’ our information systems and infrastructures. A Neolithic hunter is saved by noticing an inconsistency in the textures and colors or in the shadows within the grass and moves well away prior. If he waits to find out it is a tiger he is probably too late. Next generation security architectures also ideally aim save the systems they protect by noticing and highlighting inconsistencies prior to finding the tiger in the grass first hand.

 In Summary

 As this paper closes on the subject of trust we find a number of parallels and traits that are characteristic and universal to the paradigm. As the human race moves into the next generation virtual world of cyber-commerce, these parallels will be extended and retrofitted to work into this new environment that same way that they have been retrofitted to monetary commerce and market based economies.

As shown early in the paper, the paradigm of trust has been challenged time and time again by increasing abstraction in the way we humans interact. What was initially a very concrete attribute to a relationship has become increasingly abstract and disjointed both spatially and temporally as we move into the 21^st century. As this evolution of commerce moved into a more virtual construct, we in turn developed methods of governance to provide assurance that transactions of commerce happened in a predictable fashion and with rules that insured participants complied. In addition to this element of governance there was an equal need that developed for enforcement so that the rules of governance were followed and those that violated the ‘contract’ were dealt with appropriately. This delicate balance has for the most part been maintained to allow for the sophisticated commerce culture that we have today. If one thinks about it, the culture relies on many things that are taken for granted. Once that balance is upset, many of those things fall asunder and a society can fall into severe and potentially fatal upset. We pointed out historical instances where this has occurred and provided insight into how the seed of demise came about. It became apparent that the lack in the ability to enforce the mandates of governance led to an overall reduction in the level of trust in the systems of the time. With this reduction of trust, the foundations of commerce began to implode and as a result the society as a whole reached a point of collapse.

I think that it should be apparent by now that trust is something that is inherent to the human condition. At the risk of extending the paper, it could be argued that trust is an integral ingredient to any social animal. Once an animal chooses to become social it ‘gives up’ certain things so that it can ‘gain’ others. Usually gain outweighs loss. A good example of this are the social evolution of wolves who give up independence in lieu of certain other benefits such as the superior hunting capability that they are so well known for. Each wolf ‘trusts’ in the system, and it works. This works the same way with us but it is made far more complex by the ‘strange loop’ phenomenon that was mentioned earlier. With humans, as we have seen – it is not so simple. Humans (and certain other primates) have this ability to intentionally deceive others within its social circle. This strategy has been successful over the millennia. This must be so, otherwise all people would be honest. This is obviously not the case.

This subversion of trust required systems of governance to assure proper bounds of behavior within the society and commerce system. Enforcement is therefore a key element to trust that may be somewhat indirect with it but directly related to the concept of assurance. This in turn shows that while we as social animals may have a magnetic tendency towards groups, we require rules and methods of enforcement to stay together in large groups for any length of time. We can view the modern requirement for network and systems security to be evolutionary results to this ‘arms race’ between subversion and governance that is as old as society itself.

There are some historical lessons to learn however. The first is that while decomposition and collapsing of the security boundary may seem more cost effective and scalable, it is not a feasible approach as it removes intermediate systems of defense that may prove to be critical during attack. Additionally, these systems add layers to the overall defense network as well as a different perspective that the server itself could never have. Rather than decompose and collapse, it makes sense to decompose and distribute security functions without removing critical layers of defense from the infrastructure. By doing so, there is necessarily the requirement that the server and application policy environment act in an orchestrated and federated fashion with the network if such coordinated services are available but then revert to a simple decomposed model when they are not. In the instances where it is not available, more constrained access policies may be put into place to assure that access is limited for the application called and nothing more. This approach can in turn offer the best of both worlds to the mobile user with the varying degrees of trust that are established.

We also discussed the delta that exists between perception and reality as well as how it relates to the concept of trust and assurance. We went on to illustrate that the level of perceived risk to the ratio of potential reward was the primary determinate in the trust decision process. It was shown that systems could be put into place to provide further assurance or insurance to the user. This in turn can push the level of perceived risk down and further encourage the user to continue with their on-line purchase. The proviso being that the user is secure in the fact that they are dealing with who they think they are.

This in turn led to the concepts of identity and the important foundational role that it plays in trust. We discussed how the aspect of identity gets fuzzy and rather complicated in the cyber environment as well as how identity can become smeared across the network by the user logging into different systems. We also discussed the fact that with systems automation we need to consider machines and the services they render in much the same way as we consider humans. Machines and their resident services need to be challenged, authenticated and authorized just as humans are required to do so. Systems of governance also need to be put into place to provide the right monitoring capabilities to assure proper behavior within the scope of authorization that has been allowed. Enforcement capabilities also need to be available so that entities that violate the scope of authorization are dealt with appropriately.

The delta between perception and reality was also discussed in both its inherency and its impact. We termed this the ‘Dark Delta’, which in essence represents the inherent aspect of the unknowable within a moment of space and time between what an entity (human or machine) sees or otherwise experiences and what is really there. We discussed the fact that there is always a nominal delta but that in most instances this minute difference is not enough to be of any significance. In instances where the delta widens, there is usually a strong cause for concern because decisions can be made by the entity in question that it might not otherwise make. In many cases, being in a scenario where decisions are made against incorrect or incomplete information can be dangerous. As with the tiger in the grass, it could be fatal.

Clearly, work to reduce the Dark Delta is required in order to establish and maintain a trusting environment that does not have undue risk for the individual extending it. In legacy commerce environments these systems have been in place since the birth of monetary based commerce. Many of these systems have simply been transposed into the ecommerce environment with little or no modification. This failure to evolve paradigms has resulted in a significant widening of the dark delta in ecommerce. This is reflected by the recent downturn in holiday season on line shopping – with fear or concern of identity theft being the number one cited reason.

One of the final premises of this paper is that in some case further well designed abstraction can in turn complicate things for the would be thief. Additionally working towards shortening the length of time and lessening the potential reward of pirating a transaction or its associated data will further reduce the window of opportunity to a level where it is longer worth the effort to subvert. By this further abstraction and by creating systems to reduce the dark delta within interactions (this includes all modes of interactions – person to person, person to machine and machine to machine) an environment can be reached where consumers will feel the degree of comfort that they require to move towards an ecommerce paradigm. Many would argue that the fate of the free market commerce system hangs on its success. Whether this is true or not will remain to be seen. It is however certain that the aspect of trust is foundational to human societal dynamics and its most recent embodiment in the Internet and ecommerce.

Epilogue

In light of the economic down turn of late 2008 it seems prudent to provide an epilogue to the summary and the conclusions that this paper reached. While many of the examples and analogies used in this paper seemed to be rather prophetic, it should not however be considered as special in any way. The reason for this is the fact that the basic elements of commerce and society have not changed. They are the same today as they were two thousand years ago. Technology has not served to change any of them. More so, it has served to enhance or inhibit them, but the basic elements have remained the same. Trust in the system requires trust in its governance, which extends to its rule of law and enforcement of it. Once these systems are eroded serious consequences are often the result. With the recent events of impropriety and even thievery at unprecedented levels, along with the long list of bail outs for firms that have come to the point they are at by mismanagement and overextension of risk. It is little wonder that trust is in short supply from the perspective of the common man.

It is not an exaggeration to say that very edifices and foundations of trust in our free market system have been severely shaken. Again history has shown that at such times, the collapsing system of commerce, if not corrected can result in follow on collapses in the trust of the systems of society. At these times, governments are often forced to implement martial law and strong centralize government to maintain order by rule of force. President Obama was quite correct when he alluded to the fact that stronger regulation and transparency were key elements in restoring faith in our systems of free commerce as well as our way of life.

As this paper has illustrated, while the basic elements of society and commerce have not changed, the dynamics are strongly affected by technology. On a closing note, history has shown that technology tends to ‘grease the skids’ for commerce and society. It can serve accelerate the rebound of such systems after down turn events. The reason for this is that human societies will tend to pull inward as a result to down turns. After the fall of the Roman Empire, both systems of commerce and society were in ruin. The pulling in of society was severe – perhaps the most severe in the history of mankind. Society and commerce often did not go beyond the walls of the castle or fortress. The pulling in at this time was also of a very long duration – lasting hundreds of years.

Subsequent down turns have not been so severe and in each instance technology served to allow for quick and more consistent rebounds to the economy. The reason for this is simple…communication. Each new innovation in the movement of information has served to re-establish the critical links of human communication that are so critical for the re-establishment of trust. It is the opinion of the author that this down turn is no different. As pointed out earlier, as a result to a down turn societies turn inwards in the way they operate. Commerce reverts to more local community levels. With the internet and modern communications ‘local’ no longer has to be geographically local but local in the form of context. The World Wide Web has allowed for the growth of communities of interest in which ‘local’ groups can interact on issue and motives of common interest. As an example, a vendor in North America can do business with a partner based out of Southeast Asia based on the fact that they were room mates in college. Now they are on opposite sides of the globe, but can leverage the personal relationship that they have just as if it were at a local level. Recent services such as LinkedIn™, Face Book™ and technology trends such as Cloud Computing and Service Oriented Architectures are good examples of this. On the web, local cyber communities can serve to re-establish on line commerce without requiring full blown trust in the monolithic world of high finance. By allowing technology to enhance traditional human patterns of interaction, the pulling inwards that accompany economic down turns can be accommodated without the severing of long distance and cross cultural ties that have typically been the result in the past. For the first time in history the term local is not limited to merely geography. This has had and will continue to have profound impact on human society, systems of commerce and the trust that these systems require in order to exist.

Recent work has been done in the study of game theory dynamics and the influence that it has had over the millennia on the process of both biological and cultural evolution. The theory cites that the underlying engine which drives evolutionary frameworks of any type is a mathematical abstraction known as non-zero sum dynamics. The previous statement sets the foundation for this paper. It will endeavor to investigate the premise that technology is an expression of culture and hence its forms and usages (its packaged solutions and architectures) are prone to the same dynamics as other forms of cultural evolution or even biological evolution.

What this analysis will indicate is that the future trends of technology can to some degree be predicted by use of non-zero sum dynamics much in the same fashion that future trends in culture can predicted. The basic premise being that non-zero sum dynamics is the mathematical ‘attractor’ to which all evolutionary process are driven, always to higher orders of ‘non-zero sumness’ if you will. It is the position of this executive white paper that by allowing for this unavoidable dynamic, some degree of pre-emptive capability can be garnered against the common technology market demands.

This is not ‘Black Magic’ however, nor is it a ‘Silver Bullet’. Consensus on a particular direction of technology can only be gained by knowledge of the technology, its latest facets and the industry and market dynamics that surround it. Once this knowledge is gained however, insight as to how it will evolve can be extrapolated with relative accuracy by predicting its evolution against a non-zero sum dynamic. Additionally, since non-zero sum dynamics lead to and leverage on cooperative behavior, an attractor can be provided which motivates various product and development teams within the organization to work for a set of common architectural and solution goals.

What is Game Theory?

As its name suggests, Game Theory is the mathematical description of game flow and outcome. There are three basic precedents to consider: 1). Zero sum dynamics, 2). Fixed Sum dynamics and lastly, 3). Non-zero sum dynamics. Simple examples of the first two instances can be provided by the sport of boxing. In this type of ‘game’ there are two opponents who face each other to ‘win’. In order for one player to win, the other player has to loose. At its face value, this is zero sum dynamics. Both players start with nothing, but one player ends up with the ‘win’. The potential for the win existed prior to the contest however and it is this potential which provides the zero value.

Most boxers however do not box for free. In most instances there is a prize for the winner and a ‘payout’ for the looser. At its face value, this is fixed sum dynamics of which zero sum dynamics is a part, for ‘zero’ is indeed a fixed sum. In more sophisticated examples, the payout and winning prize might be in ratio to the performance of each boxer on a round by round basis. This creates a scenario where even if a looser still looses, they can improve their lot and receive a larger portion of the prize by performing better during the contest. The sum is fixed however, there is only so much monies allocated for the event so that as the looser performs better the winner realizes less of the winning prize. This is the epitome of fixed sum dynamics. In essence, winning and loosing become a mathematical relationship that almost, but not quite, approaches a dynamic which can encourage cooperative behavior. It is does not because the winner still wants to hold onto as large a portion of the prize as possible while the looser wants to pull a larger payout. In essence, both sides ‘benefit’ from the contest, albeit one at the expense of the other.

The third precedent is quite a bit more complex, but can be readily be displayed at least in a rudimentary and limited form by the game of Monopoly. In this type of game there are a number of players that are participating in a sort of limited economic system. In instances where there are two players, the game is simply a very sophisticated fixed sum dynamic. As the number of players is increased however, the dynamic gets to be more complicated with the ability for alliances and coalitions to form where players ‘team up’ to reap benefits or to dislodge or take advantage of other players. While the maximum potential win is still fixed (this is why it is a rudimentary and limited example) there begins to appear ‘emergent’ benefits that would not have occurred if the alliances and coalitions were not formed. This is the essence of non-zero sum dynamics which is the ability to create emergent benefits which can either create positive sum gains or avoid negative sum losses.

There are a number of important points to consider before we move on. First, the dividing lines between these dynamics are somewhat soft. While zero sum dynamics is a form of fixed sum dynamics, non-zero sum dynamics is a ratio of available resources and the number of players fending for those available resources. The result is that as the ratio is spread out (expressed as available resources to the number of players); non-zero sum dynamics begin to occur. It should be noted at this point, that non-zero sum dynamics do not guarantee positive sum gains. Nor does it guarantee the avoidance of negative sum losses. What it does do is perform a hedging or insurance to increase the probability of these occurrences. Indeed, this is the basic premise of the insurance industry, the spreading of risk among participating entities.

The other perhaps more important point to consider is that in order for non-zero sum dynamics to occur there needs to be two foundational components that can not be avoided. These are the ability for the players to have a common method of communication and the ability for the players to establish trust with each other within the alliance or coalition. If either of these paradigms can not be met then non-zero sum dynamics can not (or at least is very unlikely to) occur.

Getting out of Jail

There is a text book example for game theory that demonstrates these concepts well. It is known as ‘the Prisoner’s Dilemma’. In this example, there are two criminals that are arrested for given crime. As they are brought into the Police station they of course are separated and interrogated. Each prisoner is given a set of similar ultimatums. The police officers tell each suspect that if they confess to the crime but the other suspects stays quiet they will be let off free while the other suspect will be put in jail for ten years. Conversely, if they remain quiet and the other suspect confesses, they will go to jail for ten years and the other will be let off free. Going further, each suspect is told that if they both end up confessing, each will receive three years in jail. Finally, if neither confesses, they will both only end up with six months in jail.

What results is a matrix of possibilities where the optimal scenario for both suspects is to keep quiet and take the six month sentence. This scenario however requires both suspects to trust that the other will keep his mouth shut. What creates the problem is that there is an individual ‘selfish’ option that appears to be more optimal to each but this option holds a risk and consequent factor of mistrust. Obviously, if they had the ability to communicate they would be able to assure one another and reinforce the joint understanding. Since they do not have the ability to communicate and being criminals they might not be the strongest in the way of trust, it is highly likely that at least one of the suspects will confess with the hopes of getting out of jail free. As we established however, both probably will. If both confess, both will receive a three year sentence which is a far cry from the six month slap on the wrist that they would have received had they kept quiet.

What the scenario illustrates is that logic alone will not lead to the optimal scenario for both individuals. The only (or the most probable) way that the optimal scenario can occur is for there to be trust and understanding between the two suspects.  Furthermore, the ability to establish that understanding needs to occur by communication either prior to or during the scenario. As we covered earlier the suspects can not communicate during the scenario, so the understanding and corresponding trust needs to be established prior. Without this, the scenario reverts to a fixed sum dynamic.

What does this have to do with evolution?

At this point, a reasonable question to ask is what this all has to do with the evolution of technology solutions and architectures. The answer is quite a lot. However, in order to validate this, we must first establish what this has to do with the general principles of evolutionary process.

Some of the latest theories on evolution propose that much of it is driven by non-zero sum dynamics. This was made quite clear by a contest that was held by Robert Axelrod, where several universities submitted modified code for the prisoner’s dilemma. The goal was to find the best and most efficient algorithm for playing the modified game. In this modified game, the six month jail sentence became evolutionary success of the code ‘species’, while all other options became equivalent to various degrees of evolutionary failure.

A particular code named ‘tit for tat’ won. Tit for tat is based on initial trust (i.e. tit for tat does not ‘rat’ on the first iteration), the algorithm will then take note of its partners actions and then act in like on the next run of the game with a different participant. So if the code is met with cheating, it will cheat in turn on the next run.

The game was run over several hundred iterations and as the game processed, the tit for tat ‘species’ became more numerous, eventually forming rudimentary coalitions that cooperated by ‘implicit’ trust. Importantly, the tit for tat species did this at the expense of the other species in the game. What this illustrates is that not only do non-zero sum dynamics seem to play a strong role in evolutionary dynamics, but zero and fixed sum dynamics do as well.

While this is not proof in and of itself, there are several other studies that validate these findings. There are also several studies that allude to a transitional relationship between chaotic determinism (which is deemed as the mathematical engine for non-animate matter to gain higher orders of complexity, examples are star formation and planetary accretion) and game theory dynamics, which would seem to be a higher mathematical order in turn that allows for animate matter (life) to further leverage this proto-deterministic foundation. While these tenets are far beyond the scope of this paper, they are important concepts. References are provided at the end of this paper that will allow the reader to further investigate according to their level of interest.

There are a few very important points that need to be brought out before we proceed on. First, multiple iterations became a form of pseudo-communication, while history (even a short term history such as a single past iteration) can serve as a form of pseudo-trust. It is interesting to note that by modifying the tit for tat code to have a longer memory (each player kept track of its actions and then advertised them to the other player upon each iteration) the coalitions formed quicker and grew larger, thereby allowing for the tit for tat code to gain quicker dominance over the other code species. The second point to consider is that tit for tat is initially altruistic in the way it behaves and very quickly reverts to altruism when given the indication to do so. All of this points to a degree of success that can be garnered from cooperative behaviors. The third and perhaps the most important point is that direct explicit communication and trust is not a requirement. Implicit methods will suffice. This is obviously true in that we do not ‘know’ everyone we do business with and we do not communicate or trust everyone on an explicit basis only. If that were the case we would have never moved beyond simple village economics. As an example, if I buy a car from a Japanese auto manufacturer, literally all communications and trust are implicit with the possible exception of the local car dealer. Although the car could be bought on line, thereby remove all explicit trust and communication.  Hence, it could even be said that the whole basis for e-commerce could not occur without an implicit trust model and all evidence validates this conclusion. This also gives us our first insight into the evolution of technology and solutions. Communications and trust (the ability to establish identity and corresponding factors of trust) are of extreme importance.

Now let’s add a Barrel full of Monkeys

At this point we will now begin to address the aspects of human culture and its evolution. It should be noted that many researchers feel strongly that game theory dynamics is also one of the major components responsible for the evolution of human intelligence. This is actually fairly intuitive if one can consider that when groups of primates meet, they are basically set with two choices; they either attack each other, or they choose to co-exist. In this example the basic premise of game theory dynamics is; 1). Zero sum dynamics is leveraged in the attack option and, 2). Non-zero sum dynamics is leveraged in the co-existence option with the emergent benefit being increased gene pool and better awareness of potential predators.

If we take a look at the higher apes (particularly chimpanzees) we find that the range of dynamics has increased with the zero sum dynamic being on-going territorial war (with well documented conquering scenarios) and the non-zero sum dynamic moving on from simple coexistence to extended cooperative behavior (say for instance engaging in on-going territorial war to conquer another group). From here the dynamic points to the mannerisms of human tribes and the fact that when two tribes or even cultures for that matter come into contact they are basically left with the very same choices as our primate cousins, we either fight or we co-exist and with humans, co-existence means trading.

As we can see, there are not a wide range of options to choose from. It is also clear that once tribes or cultures come in close cultural contact, ignoring each other is not an option. It is important also to realize that either choice drives emergent benefits and also one choice will often drive the other. As the examples of aggression show, the best way to drive cooperative non-zero sum dynamics is for a set of groups to have a common enemy, which is in turn the fixed sum dynamic. The first and most obvious emergent benefit that derives from allegiances is larger numbers. Larger numbers also means a greater communication network and this network is established and reinforced by trust. As a result, these networks can tend to be very good conduits for ideas and inventions. These inventions were often of a war-like nature such as a better bow and arrow; but just as often would be of peaceful nature; such a more innovative farming practice.

So it is clear that non-zero sum dynamics is often found in intimate relationship with zero or fixed sum ‘drivers’. It is also clear that the ability to share ideas and innovations is the baseline hallmark of culture. As a result, we can safely conclude that game theory dynamics has played a very major role in human cultural evolution. With literally the same mix of dynamics, we move from simple prehistoric economies through to chiefdoms and empires to the eventual advent of the modern nation state and its corresponding international economies. Each successive era built on two past precedents for their foundation namely, communication and trust or the lack thereof.

There are a few salient points that we should cover at this point. The first is that each phase in cultural evolution has its established ‘personality’ and ability to establish common channels for communications and trust. The second point is that each culture created their own system of governance to maintain these frameworks and provide assurances that individuals were not cheating the system. The third point is that cultures tend to grow and assimilate other cultures with each subsequent phase. Sometimes these assimilations were peaceful other times not, but in most instances the assimilated cultures benefited from the dynamic. Indeed, history shows than in many instances cultures and peoples willingly submitted their own sovereignty in lieu of increased security or enhanced prosperity. Finally, as these cultures grew they created larger and more complex non-zero sum dynamics with larger and more complex potential for emergent benefits. In each instance, new and more complex systems of governance became required because the potential of cheating became higher, more problematic and harder to identify. As an example, if a goat shows up missing in an isolated village of one hundred people it is fairly easy to find the perpetrator. When we compare this with fraudulent representation in a trading contract with several middlemen along the ancient silk highway between China and Rome we have quite a different problem. At this point we need to consider that in order to meet these newer requirements, new technologies and methods needed to be created which would facilitate the preservation of communication and trust by identifying those who would try to subvert the system. Today we can extrapolate this to E-commerce and real time international trading. Indeed, this is the primary and most solid rationale for a communications company to be involved in Identity and Security Policy Management. They are the 21^st century equivalent of the middlemen on the silk highway.

Business as usual?

            As shown in the previous section, the evolution and development of human technology is directly and inextricably tied to human culture and commerce. This is so much so that human technology could even be described as an expression of human culture. Indeed, this is the position of many anthropologists and historians. Many, if not most of the innovations throughout human history have been created or invented to either increase the potential non-zero sum dynamic (i.e. to make more profits) or to preserve its existing scope (i.e. to protect existing profits). If this is indeed the case then we can see that by analyzing the cultural and corresponding business and commercial dynamics in terms of the requirements mentioned in the previous section we can to some degree gain better insight as to the next directions in technological evolution. In fact, it is the position of this paper that by using this mentality, better insight will provided with which to actually invent the technologies to meet these ongoing and constantly changing requirements.

            It is also important to consider that in larger technology companies, both external and internal dynamics need to be considered. Up until now, most of the focus has been on the external dynamics, or rather what this all means to a company and its customers who will work within the system of governance for E-commerce. Consider also though that there is a game theory dynamic that also occurs within a company itself and that by paying attention to this dynamic an increase in emergent benefits can in turn be realized.

            In order to qualify these statements we will now take a look at the different forms in which game theory is manifested and why. While this could range into a deep philosophical discussion, we will avoid that by limiting the discourse to the perspectives of a company or enterprise and its business focus, specifically communications technology. It is the position of this paper that a company can vastly optimize both its internal and external processes by tuning them to better realize the non-zero sum dynamic that has shown to be a fairly dependable guideline over the millennia of human history.

The Prisoners Dilemma revisited

            If we will recall the text book game theory example that we covered earlier, there is an optimal non-zero sum option that can only be reached by mutual trust and communication. Without this, it is highly likely that the optimal option will not be realized. Instead, some lower fixed sum option instead will likely be selected. In the example, we used logical deduction to illustrate that the most probable scenario is that both suspects will get three years in jail. Now one might argue that only one of the suspects might confess (note that this is ‘cheating’ against the system of governance). In that instance, one of the suspects will reach a higher optimal option, but they will do so at the expense of the other. Now one might ask, what is the problem with that? Clearly, one of the suspects has benefited significantly. 

The answer is that yes, this is a great approach and most definitely the best option if the other entity is your competition. However, the approach is flawed and self negating if the other entity is a partner, a colleague or another department. This is a critical differentiator that needs closer scrutiny as it largely dictates a successful versus an unsuccessful strategy, realizing that strategies can be competitive, cooperative, external or internal.

When we look at how many telecommunications companies have operated in the past, there were two major divisions, Service Provider and Enterprise. Traditionally, these two divisions were kept largely separate in both agenda and process. As we can see by the prisoner’s dilemma however, isolation is highly unlikely to yield the optimal benefit. Each division is likely to make their own decisions based on their own requirements and motivations. While these decisions will likely benefit each respective division (as we (hopefully) have very competent individuals and groups making these decisions), it is not likely that the other division will reap those benefits. Worse yet, it is possible that the decision of one division could adversely affect the other.

Recent activities in many Chief Technology Offices have focused on cross communication and trust between these divisions which will better allow for technology reuse and more optimal benefits from research and development. Two pertinent examples are the recent activities around Identity Management and Video. In Identity Management specifically, the Enterprise IdM strategy will reuse technology out of the Service Provider 3GPP/IMS project while the 3GPP/IMS project will benefit by the work that has come out of Enterprise in the areas of federated security policy frameworks. It should be noted that these emergent benefits are most often facilitated by establishing clear communication and trust between the divisions and that only by this are these types of benefits realized.

            We need to also realize that most often each division is broken into product ‘silos’ that are largely run as separate P&L organizations. In the past, decisions regarding product and solution directions were made within these silos with little or no communication between them. Obviously, this takes the two dimensional prisoner’s dilemma into a multi-faceted ‘cut throat’ monopoly game where no coalitions are allowed and every player is in for their own interests. This is so because each silo formulates their own strategy and then goes to the investment board for funding based on their own interests. Of primary importance here is that this all occurs against a fixed amount that is defined in the budget. What results is a fixed sum dynamic. The optimal scenario can not or is highly unlikely to be realized. This means that the company will not fully realize its investment. Now it could be argued that the investment board could make the choice of which strategy receives the lion’s share of the funding and indeed it does. But this allocation is done at low modularity, somewhat like doing Bonsai with a machete. What happens in this instance is that the investment board becomes the equivalent of the ‘state of war’ and all of the silos become the equivalent of participating nation states. In order to mitigate this, communication and trust needs to be established between the different organizations and a degree of sovereignty needs to be sacrificed in order to realize any emergent non-zero sum benefits. What is communication in this example? It is the sharing and ultimately the joint-creation of strategies that cross-leverage each other. These higher level strategies are then what are presented to the investment board. What is trust in this instance? It is the ability for everyone involved in the strategy to know and trust that everyone will complete their plan of record (their ‘tribal’ obligation if you will) and thereby trust in the vision that the strategy promises. They will also trust that the ‘system of governance’ will assure that this takes place and that the appropriate allocations of development funding will be provided to them. Again, it is in the best interest for those in the system of governance to assure this otherwise trust is lost, which is the essence of their power.

            Again, the recent activities around Identity and Policy Management are defining the strategy at the high end architectural level and then creating the cross-silo communication and trust to establish concise plans of record with associated phases and timelines. The CTO groups should then work as the system of governance to assure that the levels of communications and trust are maintained as the overall strategy is realized. If this is done, then the mechanics of non-zero sum dynamics are realized and the emergent benefits of the strategy are realized. If the CTO groups fail in this regard, the trust will be lost and all assurance of the strategy along with it. The question begs whether a company can really do this. History says yes, provided that the system of governance is maintained.

Hey, let’s gang up on them!

            Up until now we have been focused on internal processes and issues. At a face value, we can make the statement that from an internal perspective everyone should work in a non-zero sum dynamic; whereas from an external perspective the company (the tribe for all intensive purposes) faces the vast fixed sum dynamic of the industry at large. In essence, all other companies have the potential of becoming enemies or more properly put competition (no matter how much we would like – we really can not throw spears at them). Obviously, it is not as simple as that. The reason is that the industry could be viewed as a vast ecosystem with many niches and broad fields. This means that some companies are less of a threat than others simply because of their areas of focus.

            In these instances companies will often create partnerships. By now, it should seem intuitive about what this means. Yes, it means that a non-zero sum dynamic emerges out of the fixed sum dynamic of the industry at large. By this, both companies should realize additional emergent benefits that would not have occurred otherwise. Obviously, there needs to be a cohesive strategy for the alliance, which requires communications at an in depth level and there needs to be established trust that each company will hold up its end of the bargain and will not act in a predacious manner. The problem with partnerships is that it is difficult to provide a firm system of governance to provide assurance. Therefore, strategies with partners require more energy from a human perspective. There needs to be the face time at the highest levels as makes sense on a continuing basis to provide this assurance. It is also critical that ongoing communications is maintained between development teams in a very similar fashion to the internal processes, with joint plans of record and associated phases and timelines. There is one key difference however in that there is no sacrifice of sovereignty to the partnership. Here we see a different non-zero sum dynamic, or rather a non-zero sum dynamic that is based on a different system of governance; a mutual system of governance. In this system the parties involved will hold their end of the bargain as long as everyone else does and (this is very important) the perceived emergent benefits of the strategy are realized. If either does not occur then the system of governance fails and the partnership fades away or worse yet becomes competitive (which can bring its own set of problems particularly if the partnership has been long and successful because this means shared installed base – but as non-zero sum logic dictates – if the system of governance is good and emergent benefits are continued to be realized then there is no reason for the partnership to fail outside of neglect).

            We have rough equivalencies in the political world with the internal non-zero sum dynamic being something like a national government or legal system and the partnership non-zero sum dynamic being more like a coalition type of arrangement similar to the United Nations or NATO. Again, the main difference between the two is that in the first instance constituents surrender some degree of sovereignty while in the second example this does not occur. It should be noted though that some degree of autonomy is always surrendered in order for any partnership to be maintained. It should also be noted and we have already shown that the major tenets of game theory and non-zero sum dynamics holds true in both instances.

            In summary, we can conclude that game theory dynamics not only adequately describes evolutionary frameworks, both biological and cultural. We can also state that practically every human thought and action is in some way associated with these dynamics. As a result, any aspect of culture, even that of directions in technology can be extrapolated and to some degree preempted by examining them against a game theory context.

It’s a Jungle out there

            The last dynamic we will speak to is the fixed sum dynamic. Here we will show that not only is it impossible to eliminate fixed sum dynamics, there are valid reasons why one would not want to do so. The reason is simple; fixed sum dynamics provide the background or fuel for non-zero sum dynamics. Put into the context of a company, it is their competition that causes them to innovate.

            History has shown that it is the fixed sum dynamics that creates the motivation to build non-zero sum dynamic systems. This motivation is always driven against a framework of adversity. Even in the case of peaceful innovations such as farming practices, there is the fixed sum dynamic of the adversity of nature and the fact that, if left to itself, entropy (or more properly put competitive elements of order) would drive the environment back into a wild state.

            History also shows that cultures tend to decline when reaching a zenith. Recent findings have shown that often the reasons are a decrease if not ceasing of the non-zero sum dynamic. As the innovations cease the society also ceases to be competitive and will eventually subside to another more dominant, more innovative culture that has a stronger set of non-zero sum dynamics from which to leverage.

            In summary it is the entity that maintains the non-zero sum dynamic that maintains dominance and perceived invulnerability. As an example, it is often said that some companies create markets. This is a very serious misconception, not even monopolies can claim that. Smart companies preempt markets and they do it by remaining tuned to the non-zero sum dynamic whether they realize it or not. If one takes the time to analyze every dominant technology company, one will find that this is the case in every instance. There is an ability to sense market trends and dynamics, (the fixed sum environment if you will) and thereby adjust their own non-zero sum dynamics (their strategies) to better align with the optimal attractor (the market direction). The end result is the emergent benefits to both the company and the market in question.

Now let’s put some gas in this sucker

            What this all comes down to is providing a win-win situation for those who participate in the non-zero sum dynamic. If this were not the case then those who are getting the short end of the stick will remove themselves from the dynamic. If we look at business ecosystems and why they succeed, it is because they drive non-zero sum dynamics in several different vectors. If take the example of a bank or financial institution for instance there is the obvious customer dynamic between a communications company and theirs. This would be termed as the northbound vector. They also have customers however (this would be termed as the southbound vector) and they realize success not by whether or not they buy from a certain communications company. They succeed because they are able to better offer their services and satisfy their customers banking requirements. As long as they are able to do this and realize the emergent benefits of increased business or reduced cost of business then the decision to purchase from a given company is deemed a good one. Note though that there are many other elements within the bank that also determine the level of success their customer’s experience. A good account manager would know these other elements and try as much as possible to influence them to better align to a positive result. All of this points to a requirement for expertise in the financial sector for that account manager.

            A more complex example is that of IP based television services. In this instance we also have the northbound ( the companies customer) and southbound (the customers’ customers) business vectors. In addition, we also have a serious set of fixed sum dynamics in the way of competition from competing companies. A wise company will provide a method of gaining inroads into this market by leveraging on partnerships against the common fixed sum dynamic of the competition. These partners are hoping to realize emergent benefits as well. By aligning a strategy to a non-zero sum dynamic, a successful business ecosystem can be driven where a company and its partners are better able to compete against the competition. This is a non-zero sum dynamic that is intended to negate or avoid negative sum losses, which would be the competition taking the emerging market. Further, if the right kind of innovation is provided, a partner ecosystem can build a different non-zero sum vector that in turn provides emergent benefits to the provider customers and finally to their customers, the actual subscribers. By creating solid non-zero sum dynamic vectors a business ecosystem can flourish and grow. Furthermore, in order to continue along with a successful strategy, the dynamics of the ecosystem would need to be continually tuned and optimized to the market requirements and demand (which provides the role of the attractor for the system).

            If we take things down to their most basic level, people (which are the market) want a series of five things. First, they want to be able to provide for themselves and their family, second they want to be able to acquire things; third they want to be provided with security to protect themselves, their loved ones and the things they acquire. Forth, provided that the other requirements are met they want to be entertained in their idle time. Lastly, there is the nebulous aspect of self fulfillment and worth that every individual needs to feel. It could be argued that this last need is met by succeeding in the other four. This is simple in that it ignores the spiritual dimension of life. However, since we are purposely avoiding any issues of abstract philosophy we will ignore this, but note that it often can be a very strong need or motivating driver.

By looking at these requirements, we can more easily begin to tease out the areas where technology can better address these needs. A good emergent example that is driven by a strong fixed sum dynamic of the recent rise in international terrorism is the area of civil communications infrastructures. Additional threats from the environment such as the 2004 tsunami and Katrina all point to the need to provide increased security for the populous.

This has created a very strong non-zero sum dynamic in the market where enhanced communication and coordination services for emergency response teams will be a big market and a lot of dollars will be spent to meet these requirements. We need to remember that there are other dimensions of non-zero sum dynamics as well. As an example, look at the amount of negative press that resulted to the Bush administration and FEMA over the bungling of the emergency response to New Orleans. As a result to the imbalance of the political non-zero sum dynamic, the federal agencies in question are making major enhancements to their infrastructures and communications systems. In essence they are reacting to a fixed sum dynamic.

            Again, companies are being preemptive to this market by aligning with some of the major integrators in the industry to put together a civil communications practice to address these new and emerging market demands. Again, we have internal as well as partner non-zero sum vectors as well as the customer vectors which would be the agencies themselves.  These vectors define the business eco-system. From there the business vectors end or more so translate into the political non-zero sum vectors. In essence, if companies succeed in the market with a civil communications practice, then partners in turn succeed. In turn the agencies are better able to respond to any new threats or emergencies. This in turn saves lives and this in turn creates a positive political non-zero sum dynamic. People feel safer because they believe that they are safer by the improvements that they see when the next emergency situation transpires.

            I do not mean to belittle people’s lives by tying them into the political dynamics, but unfortunately this is often the reality. No one wants to see human lives lost but the reality is that this will often be tolerated or ignored if the political costs are not that high. This again pits us into a different set of game theory dynamics that is outside the scope of this paper, but can be readily extrapolated from the tenets thus far established.

In Summary

It has been the intention of this paper to show that game theory dynamics is an effective method for representing evolutionary dynamics. It has also been the intention to show that almost every aspect of human behavior is in some way associated with these dynamics. These are important insights whose import can not be overstated.

By using these concepts we have shown that we can to some degree predict the requirements of the industry by looking at the associated dynamics of society. We have also shown that technologies, no matter how high or abstract, are prone to these dynamics.

It has also been the position of this paper that these dynamics also come into play within a companies business processes, its partnerships as well as how they address their competition. Indeed, game theory dynamics is about as close to a crystal ball as you can get. No, it’s not magic, but it’s pretty damn close.

Bibliography – Books for further reading interest

NonZero – The logic of Human Destiny

Robert Wright                                                  Vantage Press              ISBN 0-679-75894-1

In the wake of Chaos

Stephen H. Kellert                                            Chicago Press              ISBN 0-226-42976-8

Deep Simplicity

John Gribben                                                    Random House ISBN 1-4000-6256-X