<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ ModeShape (http://www.modeshape.org)
  ~
  ~ See the COPYRIGHT.txt file distributed with this work for information
  ~ regarding copyright ownership.  Some portions may be licensed
  ~ to Red Hat, Inc. under one or more contributor license agreements.
  ~ See the AUTHORS.txt file in the distribution for a full listing of 
  ~ individual contributors.
  ~
  ~ ModeShape is free software. Unless otherwise indicated, all code in ModeShape
  ~ is licensed to you under the terms of the GNU Lesser General Public License as
  ~ published by the Free Software Foundation; either version 2.1 of
  ~ the License, or (at your option) any later version.
  ~
  ~ ModeShape is distributed in the hope that it will be useful,
  ~ but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY
  ~ or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser General Public License
  ~ for more details.
  ~
  ~ You should have received a copy of the GNU Lesser General Public License
  ~ along with this distribution; if not, write to:
  ~ Free Software Foundation, Inc.
  ~ 51 Franklin Street, Fifth Floor
  ~ Boston, MA  02110-1301  USA
  -->
<!DOCTYPE preface PUBLIC "-//OASIS//DTD DocBook XML V4.5//EN" "http://www.oasis-open.org/docbook/xml/4.5/docbookx.dtd"	[
<!ENTITY % CustomDTD SYSTEM "../../custom.dtd">
%CustomDTD;
]>
<chapter id="federation-connector">
  <title>Federation Connector</title>
     <para>
     	The federated repository source provides a unified repository consisting of information that is dynamically federated from multiple other
			&RepositorySource; instances.  This is a very powerful repository source that appears to be a single repository, when in 
			fact the content is stored and managed in multiple other systems.  Each &FederatedRepositorySource; is typically configured
			with the name of another &RepositorySource; that should be used as the local, unified cache of the federated content.
			The &FederatedRepositorySource; then looks in the configuration repository to determine the various workspaces
			and how other sources are projected into each workspace.
     </para>
		<figure id="modeshape-connector-federation-image">
			<title>Federating multiple sources using the Federated Repository Connector</title>
			<graphic align="center" scale="100" fileref="modeshape-connector-federation.png"/>
		</figure>
		<sect1>
			<title>Projections</title>
      <para>
			 Each federated repository source provides a unified repository consisting of information that is dynamically federated
			 from multiple other &RepositorySource; instances. The connector is configured with a number of <emphasis role="bold">projections</emphasis>
			 that each describe where in the unified repository the federated connector should place the content from another source.
			 Projections consist of the name of the source containing the content and a number of <emphasis role="bold">rules</emphasis> that
			 define the path mappings, where each rule is defined as a string with this format:
      </para>
			<programlisting>pathInFederatedRepository => pathInSourceRepository</programlisting>
     	<para>
			 Here, the <code>pathInFederatedRepository</code> is the string representation of the path in the unified
			 (or federated) repository, and <code>pathInSourceRepository</code> is the string representation of the path of the
			 actual content in the underlying source.  For example:
    	</para>
			<programlisting>/ => /</programlisting>
     	<para>
			  is a trivial rule that states that all of the content in the underlying source should be mapped into the unified
			  repository such that the locations are the same.  Therefore, a node at <code>/a/b/c</code> in the source would
			  appear in the unified repository at <code>/a/b/c</code>.  This is called a <emphasis role="bold">mirror projection</emphasis>,
			  since the unified repository mirrors the underlying source repository.
    	</para>
     	<para>
			  Another example is an <emphasis role="bold">offset projection</emphasis>, which is similar to the mirror projection except that
			  the federated path includes an offset not found in the source:
    	</para>
			<programlisting>/alpha/beta => /</programlisting>
     	<para>
			  Here, a node at <code>/a/b/c</code> in the source would actually appear in the unified repository at 
			  <code>/alpha/beta/a/b/c</code>.  The offset path (<code>/alpha/beta</code> in this example) can have 1 or more segments.
			  (If there are no segments, then it reduces to a mirror projection.)
    	</para>
     	<para>
			  Often a rule will map a path in one source into another path in the unified source:
    	</para>
			<programlisting>/alpha/beta => /foo/bar</programlisting>
     	<para>
			  Here, the content at <code>/foo/bar</code> is projected in the unified repository under <code>/alpha/beta</code>,
			  meaning that the <code>/foo/bar</code> prefix never even appears in the unified repository.  So the node at
			  <code>/foo/bar/baz/raz</code> would appear in the unified repository at <code>/alpha/beta/baz/raz</code>.  Again,
			  the size of the two paths in the rule don't matter.
     	</para>
		</sect1>
		<sect1>
			<title>Multiple Projections</title>
		  <para>
		  Federated repositories that use a single projection are useful, but they aren't as interesting or powerful as
		  those that use multiple projections.  Consider a federated repository that is defined by two projections:
		  </para>
<programlisting>
/ => /                         for source "S1"
/alpha => /foo/bar             for source "S2"
</programlisting>
		  <para>
		  And consider that S1 contains the following structure:
		  </para>
<programlisting>
+- a
|  +- i
|  +- j
+- b
   +- k
   +- m
   +- n
</programlisting>
	  <para>
		  and S2 contains the following:
		</para>
<programlisting>
+- foo
   +- bar
   |  +- baz
   |  |  +- taz
   |  |  +- zaz
   |  +- raz
   +- bum
      +- bot
</programlisting>
		  <para>
		  The unified repository would then have this structure:
		  </para>
<programlisting>
+- a
|  +- i
|  +- j
+- b
|  +- k
|  +- m
|  +- n
+- alpha
   +- baz
      +- taz
      |  +- zaz
      +- raz
</programlisting>
		  <para>
		  Note how the <code>/foo/bum</code> branch does not even appear in the unified repository, since it is outside of the
		  branch being projected. Also, the <code>/alpha</code> node doesn't exist in S1 or S2; it's what is called a
		  <emphasis role="bold">placeholder</emphasis> node that exists purely so that the nodes below it have a place to exist.
		  Placeholders are somewhat special: they allow any structure below them (including other placeholder nodes or real
		  projected nodes), but they cannot be modified.
		  </para>
		  <para>
		  Even more interesting are cases that involve more projections.  Consider a federated repository that contains
		  information about different kinds of automobiles, aircraft, and spacecraft, except that the information
		  about each kind of vehicle exists in a different source (and possibly a different <emphasis>kind</emphasis> of source, such as
		  a database, or file, or web service).
			</para>
		  <para>
		  First, the sources.  The "Cars" source contains the following structure:
		  </para>
<programlisting>
+- Cars
   +- Hybrid
   |  +- Toyota Prius
   |  +- Toyota Highlander
   |  +- Nissan Altima
   +- Sports
   |  +- Aston Martin DB9
   |  +- Infinity G37
   +- Luxury
   |  +- Cadillac DTS
   |  +- Bentley Continental
   |  +- Lexus IS350
   +- Utility
      +- Land Rover LR2
      +- Land Rover LR3
      +- Hummer H3
      +- Ford F-150
</programlisting>
		  <para>
		  The "Aircraft" source contains the following structure:
		  </para>
<programlisting>
+- Aviation
   +- Business
   |  +- Gulfstream V
   |  +- Learjet 45
   +- Commercial
   |  +- Boeing 777
   |  +- Boeing 767
   |  +- Boeing 787
   |  +- Boeing 757
   |  +- Airbus A380
   |  +- Airbus A340
   |  +- Airbus A310
   |  +- Embraer RJ-175
   +- Vintage
   |  +- Fokker Trimotor
   |  +- P-38 Lightning
   |  +- A6M Zero
   |  +- Bf 109
   |  +- Wright Flyer
   +- Homebuilt
      +- Long-EZ
      +- Cirrus VK-30
      +- Van's RV-4
</programlisting>
		  <para>
		  Finally, our "Spacecraft" source contains the following structure:
		  </para>
<programlisting>
+- Space Vehicles
   +- Manned
   |  +- Space Shuttle
   |  +- Soyuz
   |  +- Skylab
   |  +- ISS
   +- Unmanned
   |  +- Sputnik
   |  +- Explorer
   |  +- Vanguard
   |  +- Pioneer
   |  +- Marsnik
   |  +- Mariner
   |  +- Mars Pathfinder
   |  +- Mars Observer
   |  +- Mars Polar Lander
   +- Launch Vehicles
   |  +- Saturn V
   |  +- Aries
   |  +- Delta
   |  +- Delta II
   |  +- Orion
   +- X-Prize
      +- SpaceShipOne
      +- WildFire
      +- Spirit of Liberty
</programlisting>
		  <para>
		  So, we can define our unified "Vehicles" source with the following projections:
		  </para>
<programlisting>
/Vehicles => /                                  for source "Cars"
/Vehicles/Aircraft => /Aviation                 for source "Aircraft"
/Vehicles/Spacecraft => /Space Vehicles         for source "Spacecraft"
</programlisting>
		  <para>
		  The result is a unified repository with the following structure:
		  </para>
<programlisting>
+- Vehicles
   +- Cars
   |  +- Hybrid
   |   |  +- Toyota Prius
   |   |  +- Toyota Highlander
   |   |  +- Nissan Altima
   |   +- Sports
   |   |  +- Aston Martin DB9
   |   |  +- Infinity G37
   |   +- Luxury
   |   |  +- Cadillac DTS
   |   |  +- Bentley Continental
   |  +- Lexus IS350
   |  +- Utility
   |     +- Land Rover LR2
   |     +- Land Rover LR3
   |     +- Hummer H3
   |     +- Ford F-150
   +- Aircraft
   |   +- Business
   |   |  +- Gulfstream V
   |   |  +- Learjet 45
   |   +- Commercial
   |   |  +- Boeing 777
   |   |  +- Boeing 767
   |   |  +- Boeing 787
   |   |  +- Boeing 757
   |   |  +- Airbus A380
   |   |  +- Airbus A340
   |   |  +- Airbus A310
   |   |  +- Embraer RJ-175
   |   +- Vintage
   |   |  +- Fokker Trimotor
   |   |  +- P-38 Lightning
   |   |  +- A6M Zero
   |   |  +- Bf 109
   |   |  +- Wright Flyer
   |   +- Homebuilt
   |      +- Long-EZ
   |      +- Cirrus VK-30
   |      +- Van's RV-4
   +- Spacecraft
      +- Manned
      |  +- Space Shuttle
      |  +- Soyuz
      |  +- Skylab
      |  +- ISS
      +- Unmanned
      |  +- Sputnik
      |  +- Explorer
      |  +- Vanguard
      |  +- Pioneer
      |  +- Marsnik
      |  +- Mariner
      |  +- Mars Pathfinder
      |  +- Mars Observer
      |  +- Mars Polar Lander
      +- Launch Vehicles
      |  +- Saturn V
      |  +- Aries
      |  +- Delta
      |  +- Delta II
      |  +- Orion
      +- X-Prize
         +- SpaceShipOne
         +- WildFire
         +- Spirit of Liberty
</programlisting>
		  <para>
		  Other combinations are of course possible.
		  </para>
		</sect1>
		<sect1>
			<title>Processing flow</title>
			<para>
		  This connector executes &Request;s against the federated repository by
		  projecting them into requests against the underlying sources that are being federated.
			</para>
		  <para>
		  One important design of the connector framework is that requests can be submitted in a batch, which may be processed more efficiently 
			than if each request was submitted one at a time.  
		  This connector design accomplishes this by projecting the incoming requests into requests against each source, then
		  submitting the batch of projected requests to each source, and then transforming the results of the projected requests back
		  into original requests.
		  </para>
		  <para>
		  This is accomplished using a three-step process:
			</para>
			<orderedlist>
				<listitem>
					<para>
		  			Process the incoming requests and for each generate the appropriate request(s) against the sources
		  			(dictated by the workspace's projections). These
		  			"projected requests" are then enqueued for each source.
					</para>
				</listitem>
				<listitem>
					<para>
		  			Submit each batch of projected requests to the appropriate source, in parallel where possible.
					  Note that the requests are still ordered correctly for each source.
					</para>
				</listitem>
				<listitem>
					<para>
					  Accumulate the results for the incoming requests by post-processing the projected requests and
					  transforming the source-specific results back into the federated workspace (again, using the workspace's projections).
					</para>
				</listitem>
			</orderedlist>
		  <para>
		  This process is a form of the <emphasis>fork-join</emphasis> divide-and-conquer algorithm, which involves splitting a problem into smaller
		  parts, forking new subtasks to execute each smaller part, joining on the subtasks (waiting until all have finished), and then
		  composing the results. Technically, Step 2 performs the fork and join operations, but this class uses &RequestProcessor;
		  implementations to do Step 1 and 3 (called &ForkRequestProcessor; and &JoinRequestProcessor;, respectively).
		  </para>
			<para>
				Such fork-join style techniques are well-suited to <emphasis>parallel processing</emphasis>.  This connector uses an &ExecutorService;
				to allow these different processors to operate concurrently.  This can greatly improve the performance as perceived
				by the clients, since indeed much of the operations on the different sources are occurring at the same time.
			</para>
			<para>
				It is also possible that not every incoming &Request; get projected to all sources. Indeed, many operations can
				effectively be mapped to a <emphasis>single projection</emphasis>.  In such cases, the overhead of the federated
				connector is quite minimal.
			</para>
			<note>
				<para>
					&Request;s that include the &Path; within the request's &Location; can be very quickly mapped to the correct projection,
					and thus such federated requests can be processed with very little overhead.  However, when requests contain &Location;s
					that only contain identification properties (e.g., UUIDs), the connector may not be able to determine the correct
					projection(s), and may have to simply forward the request to all of the projections.  This is obviously less desirable,
					so when possible ensure that the &Request; objects include the &Path;.
				</para>
			</note>
		</sect1>
		<sect1>
			<title>Update operations</title>
			<para>
				The federated connector behavior for <emphasis>read-only</emphasis> requests is fairly obvious.  In the best case, the connector determines the
				appropriate projections, forwards the request into the appropriate sources, and then combines the results.
				But what happens with <emphasis>change requests</emphasis>?
			</para>
			<para>
				Currently, the federated connector requires that each &ChangeRequest; be mapped to <emphasis>one and only one</emphasis> projection.
				However, when a single projection cannot be determined for a &ChangeRequest;, the connector throws an error.
			</para>
			<para>
				This is thought to be a minimal problem that will not actually be an issue in most uses of the federated connector.  
				If you find that your usage does indeed fall into this category, 
				please let us know via the <ulink url="&Home;lists.html">mailing lists</ulink> or log
				an enhancement request in <ulink url="&JIRA;">JIRA</ulink>.  Be sure to include as much detail as possible about the scenario,
				the problem condition, and the desired behavior.
			</para> 
		</sect1>
		<sect1>
			<title>Configuration</title>
			<para>
				The federated repository uses other &RepositorySource;s that are to be federated and a &RepositorySource; that is to be used as the
				cache of the unified contents.  These are configured in another &RepositorySource; that is treated as a configuration repository,
				which should contain information about the workspaces and how other sources are projected:
			</para>
			<programlisting role='xml'><![CDATA[<!-- Define the federation configuration. -->
<mode:workspaces>
  <mode:workspace jcr:name="default">
    <!-- Define how the content in the two sources maps to the federated/unified repository.
         This example puts the 'Cars' and 'Aircraft' content underneath '/vehicles', but the
         'Configuration' content (which is defined by this file) will appear under '/'. -->
    <mode:projections>
      <!-- Project the 'Cars' content, starting with the '/Cars' node. -->
      <mode:projection jcr:name="Cars projection" 
      				  mode:source="Cars" 
      				  mode:workspaceName="workspace1">
          <mode:projectionRules>/Vehicles/Cars => /Cars</mode:projectionRules>
      </mode:projection>
      <!-- Project the 'Aicraft' content, starting with the '/Aircraft' node. -->
      <mode:projection jcr:name="Aircarft projection" 
      				  mode:source="Aircraft" 
      				  mode:workspaceName="workspace2">
          <mode:projectionRules>/Vehicles/Aircraft => /Aircraft</mode:projectionRules>
      </mode:projection>
      <!-- Project the 'System' content. Only needed when this source is accessed through JCR. -->
      <mode:projection jcr:name="System projection" mode:source="System" mode:workspaceName="default">
          <mode:projectionRules>/jcr:system => /</mode:projectionRules>
      </mode:projection>
    </mode:projections>
  </mode:workspace>
</mode:workspaces>
]]></programlisting>
			<note>
				<para>
					We're using XML to represent a graph structure, since the two map pretty well.  Each XML element represents
					a node and XML attributes represent properties on a node.  The name of the node is defined by either the 
					<code>jcr:name</code> attribute (if it exists) or the name of the XML element.  And we use XML namespaces
					to define the namespaces used in the node and property names.  As an aside, this is exactly how the XML graph importer
					works.
				</para>
			</note>
		</sect1>
		<sect1>
			<title>Repository Source properties</title>
			<para>
				While the majority of the configuration is defined using the configuration source (as discussed above), the &FederatedRepositorySource; 
				class does have a few JavaBean properties:
			</para>
      <variablelist>
        <varlistentry>
          <term>name</term>
          <listitem>
            <para>
              Required property that specifies the name of the repository source, which is used by the &RepositoryService; 
              when obtaining a &RepositoryConnection; by name.
    				</para>
          </listitem>
        </varlistentry>
        <varlistentry>
          <term>retryLimit</term>
          <listitem>
            <para>
              Optional property that, if used, defines the number of times that any single operation on a &RepositoryConnection; to this source should be retried
							following a communication failure. The default value is '0'.
    				</para>
          </listitem>
        </varlistentry>
      </variablelist>
		</sect1>
</chapter>

