Creating Custom Connector Sending Claims with SharePoint 2013

from:http://blogs.msdn.com/b/security_trimming_in_sharepoint_2013/archive/2012/10/29/creating-custom-connector-sending-claims-to-sharepoint-2013.aspx

What You Will Learn

This blog entry describes how to take an existing custom XML connector to the Search service application, modify the connector to submit security ACLs as claims, install and deploy for SharePoint 2013 and test it. Using this blog post, you can index external sources with its security model within SharePoint itself. Keep reading and we will show you how.

Why Use Claims

Claims have the potential to simplify authentication logic for external contents in SharePoint 2013. Claims-based identity enables applications to know a few facts about the user's permission to view content. Thus, we can render legacy or different security models into custom claims in SharePoint 2013.

Modifying the XML Connector to send ACL information as claims should be helpful to demonstrate how to "light up" external content in search results, i.e. be able to search securely in any content.

Requirements

SharePoint 2013 Server
Visual Studio 2012

The Starting Point: XML Connector

With this XML Connectoras a starting point, we need to modify the connector code as it does not send security claims with its document submissions.

The next two sub-sections are "greatly copied" from Anders Fagerhaug's blog post. A huge thank-you and many kudos goes out to him for producing a great starting point for a wonderful custom connector!

Install Custom Connector (Thanks Anders!)

Download the zip archive attached to the bottom of this blog entry.
Unzip the contents of the zip archive to a folder on your computer, e.g. C:\CustomSecurityConnector
On the Start menu, choose All Programs, choose Microsoft Visual Studio 2010, and then choose Visual Studio Tools and open a Visual Studio command prompt.
To install the XmlFileConnector.dll, type the following command the command prompt
gacutil /i <ExtractedFolderPath>\XmlFileConnector\bin\Debug\XmlFileConnector.dll

This worked if the output after running gacutil is along the lines of "Assembly successfully added to the cache".
Merge the registry entries for the protocol handler by double-clicking on the registry file located at <ExtractedFolderPath>\xmldoc.reg.
On the Start menu, choose All Programs, then choose Microsoft SharePoint 2013 Products and open aSharePoint 2013 Management Shell as an administrator.
To configure the custom XML connector, at the command prompt, type the following command and run it:
$searchapp = Get-SPEnterpriseSearchServiceApplication -Identity "Search Service Application"
New-SPEnterpriseSearchCrawlCustomConnector -SearchApplication $searchapp –Protocol xmldoc -Name xmldoc –ModelFilePath "<ExtractedFolderPath>\XmlFileConnector\Model.xml"

To confirm the configuration, at the command prompt, type the following command and run it:

Get-SPEnterpriseSearchCrawlCustomConnector -SearchApplication $searchapp

The expected output from this should be a protocol "xmldoc" with a ModeFileLocation pointing to the Model.xml given above.
Finally, we need to restart the search service, type the following commands in the command prompt:
net stop osearch15
net start osearch15

Create Crawled Property for the Custom XML Connector (Thanks Anders!)

When the custom XML connector crawls content, the crawled properties discovered during crawl will have to be added to a crawled property category. You need to create this category. Note: The user that performs this operation has to be an administrator for the Search service application.

On the Start menu, choose All Programs, then choose Microsoft SharePoint 2013 Products and open a SharePoint 2013 Management Shell as an administrator.
To create a new crawled property category, at the command prompt type the following commands and run them, where: <ConnectorName> is the name you want to give the custom XML connector, for example Custom XML Connector:
$searchapp = Get-SPEnterpriseSearchServiceApplication -Identity "Search Service Application"
New-SPEnterpriseSearchMetadataCategory -Name "<ConnectorName>" -Propset "BCC9619B-BFBD-4BD6-8E51-466F9241A27A" -searchApplication $searchapp

The Propset GUID, BCC9619B-BFBD-4BD6-8E51-466F9241A27A, is hardcoded in the file XmlDocumentNamingContainer.cs and should not be changed.
To specify that if there are unknown properties in the newly created crawled property category, these should be discovered during crawl, at the command prompt, type and run the following:
$c = Get-SPEnterpriseSearchMetadataCategory -SearchApplication $searchapp -Identity "<ConnectorName>"
$c.DiscoverNewProperties = $true
$c.Update()

where <ConnectorName> is the name you want to give the custom XML connector, for example Custom XML Connector.

Create Content Source for the XML Contents (Thanks Anders!)

To specify what, when and how the XML content should be crawled, you have to create a new content source for your XML content. Note: The user that performs this operation has to be an administrator for the Search service application.

On the home page of the SharePoint Central Administration website, in the Application Management section, choose Manage service applications
On the Manage Service Applications page, choose Search service application.
On the Search Service Administration Page, in the Crawling section, choose Content Sources.
On the Manage Content Sources page, choose New Content Source.
On the Add Content Source page, in the Name section, in the Name box, type a name for the new content source, for example XML Connector.
In the Content Source Type section, select Custom repository.
In the Type of Repository section, select xmldoc.
In the Start Address section, in the Type start addresses below (one per line) box, type the address from where the crawler should being crawling the XML content. The start address syntax is different depending on where the XML content is located.
XML content is located on a local drive, use the following syntax:
xmldoc://localhost/<XMLcontentfolder>/#x=doc:id;;urielm=url;;titleelm=title#

XML content is located on a network drive, use the following syntax:
xmldoc://<SharedNetworkPath>/#x=doc:id;;urielm=url;;titleelm=title#

XML content from the supplied example of this blog entry:
xmldoc://localhost/C$/CustomSecurityConnector/#x=Product:ID;;titleelm=Title;;urlelm=Url#

where given that you extracted the zip archive of this blog post to C:\CustomSecurityConnector.
Verify that your newly created content source is shown on the Search Service Application page.

The CustomSecurityConnector folder given with the ZIP archive contains Product.xml, which is a sample of a small product catalog.

Create a Crawl Rule for Content Source

It is useful to create a crawl rule if we want to do post-Security trimming for these documents.

Go to SharePoint Central Admin, choose Search Administration.
Under the Crawling section, choose Crawl Rules and then New Crawl Rule
In the Path field, type in xmldoc://* which should match up the CrawlUrl of the documents we will crawl in our example Product.xml file. Even if the Url in our data states urls along the lines ofhttp://wfe/site/catalog, remember that these are the display urls. The SharePoint gatherer/crawler needs the access URL given in the content source setting.



<Product>
<ID>1</ID>
<Url>http://wfe/site/catalog/item1</Url>
...
</Product>
Select "Include all items in this path" and then create the rule by selecting OK

Crawl

Finally, start a full crawl of the newly created content source. Then select "View Crawl Logs" when the content source status goes back to the Idle status again. You should see 11 items successfully crawled.

Modifying the Connector to Send Custom Claims for Security

The Model.xml

First, we need to change the Model.xml of the connector. To enable sending claims in the connector, we need to submit a binary security descriptor, a boolean to say that we will be providing our own type of security and finally an optional string field (docaclmeta).

Essentially, we need to notify the connector framework of our security field type descriptors plus set a few properties to enable this in the model.

Let's start with the TypeDescriptors first. For every item that we wish to enforce custom security on, we have to set the type descriptors for the following fields:

UsesPluggableAuth as a boolean field type (if the value of this field is true = custom security claims instead of the Windows Security descriptors next)
SecurityDescriptor as a byte array for the actual encoded claims data
docaclmeta as an optional string field, which will only be displayed in the search results if populated. This field is not queryable in the index.

In the model file itself, the added lines for TypeDescriptors are encapsulated by the XML comments for Claims Security, part 1/2, like this:

    <Parameter Name="Return" Direction="Return">

      <TypeDescriptor Name="Return" TypeName=... >

        <TypeDescriptors>

          <TypeDescriptor Name="Documents" TypeName=...>

            <TypeDescriptors>

              <TypeDescriptor Name="Item" TypeName=...>

                <TypeDescriptors>

                  ...

                  ...

                  <!-- Claims Security Start, part 1/2 -->

                  <TypeDescriptor Name="UsesPluggableAuth" TypeName="System.Boolean" />

                  <TypeDescriptor Name="SecurityDescriptor" TypeName="System.Byte[]" IsCollection="true">

                    <TypeDescriptors>

                      <TypeDescriptor Name="Item" TypeName="System.Byte" />

                    </TypeDescriptors>

                  </TypeDescriptor>

                  <TypeDescriptor Name="docaclmeta" TypeName="System.String, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" />

                  <!-- Claims Security End  , part 1/2 -->

                  ...

                  ...

We are almost done with the model file. The only thing we have left is to supply the field names for the UsesPluggableAuthentication, WindowsSecurityDescriptorField and the DocaclmetaField, like this:

  <MethodInstances>

    <Association Name="GetAllDocuments_Instance" Type="AssociationNavigator" ReturnParameterName="Return" ReturnTypeDescriptorPath="Return.Documents">

      <Properties>

        ...

        ...

        <!-- Claims Security Start, part 2/2 -->

        <Property Name="UsesPluggableAuthentication" Type="System.String">UsesPluggableAuth</Property>

        <Property Name="WindowsSecurityDescriptorField" Type="System.String">SecurityDescriptor</Property>

        <Property Name="DocaclmetaField" Type="System.String">docaclmeta</Property>

        <!-- Claims Security End  , part 2/2 -->

        ...

        ...

The Entities.cs

We need to modify the corresponding C# code for the objects (i.e. documents) that the connector will submit with custom security ACLs. We have to adjust the Document class with the same fields as specified in the Model.xml:

SecurityDescriptor
UsesPluggableAuth
docaclmeta

Thus, we added these three properties in the Entities.cs file as shown next:

public class Document

{

    private DateTime lastModifiedTime = DateTime.Now;

    public string Title { get; set; }

    public string DocumentID { get; set; }

    public string Url { get; set; }

    public DateTime LastModifiedTime { get { return this.lastModifiedTime; } set { this.lastModifiedTime = value; } }

    public DocumentProperty[] DocumentProperties { get; set; }

    // Security Begin

    public Byte[] SecurityDescriptor { get; set; }

    public Boolean UsesPluggableAuth { get; set; }

    public string docaclmeta { get; set; }

    // Security End

}

The XmlFileLoader.cs

Finally, we need to modify the connector code to get the input data for the security data, translate this into a corresponding byte array of claims and set the proper field values of the Document class.

Key points to note:

The UsesPluggableAuth should only be set to true if we indeed will be supplying claim ACLs with this document.
Claims are encoded as a binary byte stream. The data type of this example is always of type string but this is no requirement of the SharePoint backend.
The encoding is done according to the protocol documentation where:
- The first byte signals an allow or deny claim
- The second byte is always 1 to indicate that this is a non-NT security ACL (i.e. it is a claim ACL type)
- The next four bytes is the size of the following claim value array.
- The claim value string follows as a Unicode byte array.
- The next four bytes following the claim value array, gives the length of the claim type
- The claim type string follows as a Unicode byte array.
- The next four bytes following the claim type array, gives the length of the claim data type
- The claim data type string follows as a Unicode byte array
- The next four bytes following the claim data type array, gives the length of the claim original issuer
- The claim issuer string finally follows as a Unicode byte array

The Input XML

We modified the input XML from the original connector, adding the claimtype, claimvalue, claimissuer and claimaclmeta field to the input:

<Product>
<ID>1</ID>
<Url>http://wfe/site/catalog/item1</Url>
<Title>Adventure Works Laptop15.4W M1548 White</Title>
<Item_x0020_Number>1010101</Item_x0020_Number>
<Group_x0020_Number>10101</Group_x0020_Number>
<ItemCategoryNumber>101</ItemCategoryNumber><ItemCategoryText>Laptops</ItemCategoryText>
<About>Laptop with ... </About>
<UnitPrice>$758,00</UnitPrice>
<Brand>Adventure Works</Brand>
<Color>White</Color>
<Weight>3.2</Weight>
<ScreenSize>15.4</ScreenSize>
<Memory>1000</Memory>
<HardDrive>160</HardDrive>
<Campaign>0</Campaign>
<OnSale>1</OnSale>
<Discount>-0.2</Discount>
<Language_x0020_Tag>en-US</Language_x0020_Tag>

<claimtype>http://surface.microsoft.com/security/acl</claimtype>
<claimvalue>user1</claimvalue>
<claimissuer>customtrimmer</claimissuer>
<claimaclmeta>access</claimaclmeta>

</Product>

With this sample input in XML, we have to modify the connector to pick up these extra XML tags.

We will create a GetXml method for this. For instance, we could illustrate one GetXml method where

the first parameter is the tag name in the XML (e.g. "claimtype")
the second parameter is the default return value for the GetXml method if the tag does not exist and finally,
the third parameter is the XML element.

    var claimType = GetXml(documentAclClaimTypeElmName, "http://demo.sharepoint.com/acl", elm);

    var claimValue = GetXml(documentAclClaimValueElmName, "user1", elm);

    var claimIssuer = GetXml(documentAclClaimIssuerElmName, "windows", elm);

    var docAclMeta = GetXml(documentAclMetaElmName, null, elm);

With these variables properly filled out from the XML data, we need a method to transform these variables into a claims byte array. We will call this method GetSecurityAcl:

    private static byte[] GetSecurityAcl(

        string claimtype,

        string claimvalue,

        string claimissuer)

    {

        Byte[] spAcl = null;

        if (!string.IsNullOrEmpty(claimtype) &&

            !string.IsNullOrEmpty(claimvalue) &&

            !string.IsNullOrEmpty(claimissuer))

        {

            using (var aclStream = new MemoryStream())

            {

                var dest = new BinaryWriter(aclStream);

                AddClaimAcl(dest, false, claimtype, claimvalue, claimissuer);

                dest.Flush();

                spAcl = aclStream.ToArray();

            }

        }

        return spAcl;

    }

We need a method AddClaimAcl to add an encoded claim to a given byte stream:

    private static void AddClaimAcl(

        BinaryWriter dest,

        bool isDeny,

        string claimtype,

        string claimvalue,

        string claimissuer)

    {

        const string datatype = @"http://www.w3.org/2001/XMLSchema#string";

        if (string.IsNullOrEmpty(claimvalue))

        {

            return;

        }

        dest.Write(isDeny ? (byte)1 : (byte)0); // Allow = 0, Deny = 1

        dest.Write((byte)1); // Indicate that this is a non-NT claim type

        //

        // Claim Value

        //

        dest.Write((Int32)claimvalue.Length);

        dest.Write(Encoding.Unicode.GetBytes(claimvalue));

        //

        // Claim Type

        //

        dest.Write((Int32)claimtype.Length);

        dest.Write(Encoding.Unicode.GetBytes(claimtype));

        //

        // Claim Data Value Type

        //

        dest.Write((Int32)datatype.Length);

        dest.Write(Encoding.Unicode.GetBytes(datatype));

        //

        // Claim Original Issuer

        //

        dest.Write((Int32)claimissuer.Length);

        dest.Write(Encoding.Unicode.GetBytes(claimissuer));

    }

Now we have all our variables filled out. Next, we will set the properties of the Document class using these variables:

    var security = GetSecurityAcl(claimType, claimValue, claimIssuer);

    var doc = new Document

    {

        DocumentID = id,

        LastModifiedTime = fileModifiedTime,

        UsesPluggableAuth = security != null,

        SecurityDescriptor = security,

        docaclmeta = docAclMeta

    };

    docs.Add(doc);

That is really it. The connector will read the XML security information, encode the claims and supply the optional docaclmeta field before sending this to the SharePoint 2013 indexing backend.

The next blog post will outline how to write a custom pre-security trimmer to unlock these documents for users who are entitled to view it.

Acknowledgements

Author: Sveinar Rasmussen (sveinar)

A huge thank-you goes out to Anders Fagerhaug (andersfa) and Armen Kirakosyan (armenk) for the original Custom XML Connector blog entry.

http://blogs.msdn.com/cfs-file.ashx/__key/communityserver-blogs-components-weblogfiles/00-00-01-55-87-Trimming/6786.Connector.zip