How to: Achieve Count(*) on a large SharePoint list

This has been a mission of mine for a while now (before I went on holiday and took a 2 week hiatus from all things SharePoint :)).

One of the clients I've been working with has been trying to replicate a pretty simple operation (by normal development standards). They have a SharePoint list with a LOT of items in it (we are talking 200,000 list items and above) and includes some Choice fields.

They want to return a count of how often each choice value is being used. Now, if you were using SQL Server you would simply do the following pseudo-SQL:

select count(*) from myList group by myChoiceField

At first look in SharePoint this is not possible:

• There is no "count" operation in CAML, nor any other kind of aggregation function
• SharePoint Search "full text query" does not support the count(*) operator (or anything similar)
• The only reference to aggregations is in the SPView.Aggregations property .. this is only used by the rendered HTML and the values are not returned in the result set.

Now .. I know that you can get count values on a list, if you create a View with a Group By then it shows you the number of items in each group, so it MUST be possible! So my mission started

List view with groups

We want to replicate this behaviour,

but programmatically!

First.. we need a test environment
The first thing I did was create a really big list. We are talking about 200,000 list items, so you can't just pull all the items out in an SPQuery (as it would be far too slow!).

I generated a simple custom list. I add a choice field (with optional values of 1-20) and then generated 200,000 list items with a randomly assigned choice value (and a bunch of them without any choice value at all .. just for laughs).

Now I could play with my code

Attempt Number 1 - Retrieve all list and programmatically calculate the counts (fail)
I kinda knew this wouldn't work .. but I needed a sounding board to know HOW bad it really was. There are 200,000 items after all, so this was never going to be fast.

• Use SPQuery to retrieve 2 fields (the ID, and my "choice" field).
• Retrieve the result set, and iterate through them, incremementing an integer value to get each "group" count value

This was a definite #fail.To retrieve all 200,000 list items in a single SPQuery took about 25 seconds to execute ... FAR too slow.

Attempt Number 2 - Execute separate query for each "group" (fail)
I was a little more positive with this one ... smaller queries execute much faster so this had some legs (and this is certainly a viable option if you only want the count for a SINGLE group).

• Create an SPQuery for each of the "choice" values we want to group by (there are 20 of them!)
• Execute each query, and use SPListItemCollection.Count to get the value

Unfortunately this was another spectacular #fail. Each query executed in around 2 seconds .. which would be fine if we didn't have to do it 20 times! :( (i.e. 40 second page load!!)

Attempt Number 3 - Use the SPView object (success!)
Ok .. so I know that the SPView can render extremely fast. With my sample list, and creating a streamlined "group by" view it was rendering in about 2 seconds (and thats on my laptop VM! I'm sure a production box would be much much quicker).

The main problem is ... how do you get these values programmatically?

The SPView class contains a "RenderAsHtml" method which returns the full HTML output of the List View (including all of the group values, javascript functions, the lot). My main question was how did it actually work? (and how on earth did it get those values so quickly!)

I started off poking into the SPView object using Reflector (tsk tsk). The chain I ended up following was this:

• SPView.RenderAsHtml() -->


• SPList.RenderAsHtml() (obfuscated ... arghhhh)

So that was a dead end .. I did some more poking around and found out that SPContext also has a view render method ...

• SPContext.RenderViewAsHtml() -->


• SPContextInternalClass.RenderViewAsHtml() -->


• COM object ! (arghhhh)

Now .. the fact that we just hit a COM object suggests that we are starting to wander towards the SQL queries that get executed to retrieve the view data .. I didn't want to go anywhere NEAR that one, so I decided to leave it there and perhaps try using the output HTML instead (nasty .. but not much of a choice left!).

using (SPSite site = new SPSite(http://myspsite))
{
SPList list = site.RootWeb.Lists["TestList"];
string strViewHtml = list.Views["GroupedView"].RenderAsHtml();
}

Having done this we now have the HTML output of our view (and this code takes about 2-3 seconds to execute ... fast enough for my laptop .. we can always cache the value if needed).
 
Looking through the DOM output in the browser, it was possible to identify the "group" element by their attributes. It is a TBody node with both an ID attribute and a "groupString" attribute (the GroupString is the important one, as it tells us the view is configured to "Group By").
 
What I needed next was a way of getting the actual values out of the HTML. For this I used the extremely awesome "HTML Agility Pack" from Codeplex. This is a set of libraries that allow you to parse DOM elements, including both plain "poorly formed" HTML as well as XHTML, and then use XPath queries to extract any values you want (much in the same way you would normally use the XML namespace for XHTML).
 
This gave me the TBODY node, and from there I could use string manipulation on the "InnerText" to pull out the group name and the count value :)

// Using HTML Agility Pack - Codeplex
// load the HTML into the HtmlDocument object

HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(strViewHtml);

// retrieve all TBODY elements which have both
// an ID and groupString attribute
HtmlNodeCollection nodes = doc.DocumentNode.SelectNodes("//tbody[@id][@groupstring]");

if (nodes != null)
{
foreach (HtmlNode node in nodes)
{
// extract the Group Name
string strGroupName = node.InnerText.Substring(node.InnerText.LastIndexOf(" ")+6);
strGroupName = strGroupName.Substring(0, strGroupName.IndexOf("&#")-1);
Console.Write ("Group: " + strGroupName + ", ");

// extract the number of items
string strValueText = node.InnerText.Substring(node.InnerText.LastIndexOf("(") + 1);
Console.WriteLine("Number of Items: " + strValueText.Substring(0, strValueText.Length - 1));
}
}

As you can see I'm doing some rather nasty SubString statements.. there may well be a quicker and cleaner way to do this using Regex .. this was more a proof of concept than anything else :)

Result!

Console output, showing group names and counts.

3 seconds isn't bad, running on a "single server" laptop VM image :)

The end result was 2-3 second bit of code, retreiving Group By, Count values for a list with 200,000 list items.

Not bad for an afternoons work :)

Attempt 4 - Do the same thing in JQuery (kudos to Jaap Vossers)
This was actually the original solution, I asked Jaap if he could look at this if he had spare time, as I knew he had a lot of JQuery experience (and he blew me away by having it all working in under 30 minutes!).

Basically it uses pretty standard JQuery to go off and retrieve the HTML content from another page, scraping the HTML and pulling back the values. Same as the C# it grabs the group TBody, then walks down the DOM to retrieve the text value that it outputs.

The speed is roughly the same as the actual view itself. I'm sure some more JQuery could be employed to pull out the specific values and do more with them, but the concept appears to be sound:

<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.4.2/jquery.min.js"></script>

<script type="text/javascript">
$(document).ready(function(){

// you will need to change this URL
var url = http://myspsite/Lists/MyList/GroupedView.aspx;

var groupings = [];

$.get(url, function(data) {
$(data).find("tbody[id^=titl][groupString] > tr > td").each(

function(index, value){
groupings.push($(this).text());
}
);

$("#placeholder").append("<ul></ul>");

$.each(groupings, function(index, value){

$("#placeholder ul").append("<li>" + value + "</li>")
});
});
});

</script>
<div id="placeholder"></div>

tada .. (thanks Jaap!)

Result from JQuery output,

dropped into a Content Editor Web Part

Summary
Well .. I know doing HTML scraping isn't pretty, but seeing as the code is MUCH faster than anything else I've seen (and is stuck in the middle of a COM object) there didn't seem to be much choice.

By all means, feel free to let me know if you have any alternatives to this.

Fixed: SPWeb.Navigation.QuickLaunch == null

This one plagued me for quite some time.. you create a new site, and some pages. The navigation menus are all there but when you look in code the SPNavigationNodeCollections are null.

This is the same for both the standard WSS navigation collection (SPWeb.Navigation.QuickLaunch) and the publishing site collection (PublishingWeb.CurrentNavigationNodes).

However ... you navigate the the "Modify Navigation" screen and make any change, click Ok and magically all the nodes appear in code!

Why does this happen?
the main thing is the role of the "SiteMap" providers in SharePoint. These are what really drive your navigation rendering, presenting an XML map of the "nodes" structure of your pages and using controls like the ASP.Net menu to render them.

The problem is, these don't always match up with the data in the content database around navigation settings. I think this follows the similar grounds of "ghosting" in that (for performance reasons) the SPNavigationNodeCollection doesn't get filled up with data until you "customise" something.

This is normally achieved by going to the "Modify Navigation Settings" page and clicking the "ok" button, at which point SharePoint conveniently goes and populates the nodes collection, and it becomes available to you in code.

What can I do about it?
Well, thankfully there is a fix for this.

You can manually create the nodes yourself, but you DO have to add some special node properties so that SharePoint recognises them as being "linked" to the existing pages and sites, otherwise you end up with duplicates all over the place!

The trick is that any node you add for pages you add a custom Property of NodeType set to the value of Page. For sub-sites you set this property to Area.

This way, SharePoint sees your custom nodes as being "Page" and "Site" nodes instead of new custom links!

Simple eh???

The code is listed below:

// use a Publishing Web object so we can access pages


PublishingWeb pWeb = PublishingWeb.GetPublishingWeb(web);

// use this to store the urls that exist in the current nav

List currentUrls = new List();

foreach (SPNavigationNode node in web.Navigation.QuickLaunch)

{

// collect all of the existing navigation nodes

// so that we don't add them twice!

currentUrls.Add(node.Url.ToLower());

}

foreach(PublishingPage page in pWeb.GetPublishingPages())

{

// check to make sure we don't add the page twice

if (!currentUrls.Contains(page.Uri.AbsolutePath.ToLower())

&& page.IncludeInCurrentNavigation)

{

// create the new node

SPNavigationNode newNode = new SPNavigationNode(page.Title, page.Url);

newNode = web.Navigation.QuickLaunch.AddAsFirst(newNode);

// IMPORTANT

// SET THE NodeType to "Page"

newNode.Properties["NodeType"] = "Page";

// save changes

newNode.Update();

}

}

foreach(SPWeb tempWeb in web.Webs)

{

// make sure we don't add the sub-site twice

if (!currentUrls.Contains(tempWeb.ServerRelativeUrl.ToLower()))

{

// create the new node

SPNavigationNode newNode = new SPNavigationNode(tempWeb.Title, tempWeb.ServerRelativeUrl);

newNode = web.Navigation.QuickLaunch.AddAsLast(newNode);

// IMPORTANT

// SET THE NodeType to "Area"

// (this is a throwback to SPS 2003 where it used

// "Portal Area" instead of sub sites)

newNode.Properties["NodeType"] = "Area";

newNode.Update();

}

}

// save changes to the Publishing Web

pWeb.Update();

SharePoint Timer Jobs and Multiple Servers

Timer jobs are wonderfully robust creatures. They effectively replace the "Windows Scheduled Task" for SharePoint servers, allowing you to run .Net code on a scheduled or "one off" basis.

However, there can be some confusion about running Timer Jobs on multiple servers, so this should clear it up a bit.

1. By default Timer Jobs will only execute on the server that they are called from (typically the Central Administration server)
2. Through code you can specify a specific box (SPServer) to run your Timer Job on.

3. It is possible to run a Timer Job on every server on the farm (although this can be dangerous!)

The crux of all this is based on the SPJobDefinition constructor (http://msdn.microsoft.com/en-us/library/microsoft.sharepoint.administration.spjobdefinition.spjobdefinition.aspx).

The constructor includes a parameter for an SPServer object, which allows you to specify which server the timer job will run on.

Example Code 1 – Add Job to single Server:

The code below is for a Web Application scoped which will register a job definition on a single server. As Web Application scoped features are activated from Central Administration, it will use the current SPServer (i.e. the Central Admin server).

This is useful for code which modified content in the database, as you only want it to execute a single time.

public override void FeatureActivated(SPFeatureReceiverProperties properties)

{

SPWebApplication webApp = properties.Feature.Parent as SPWebApplication;

SPJobDefinition job = new SPJobDefinition("CustomJobDef", webApp);

}

Example Code 2 – Add Job to All Servers:

The code below is for a Web Application scoped which will register a job definition on all web front end servers in the farm.

This is useful for code which modifies files or server settings, as you need it to execute separately for each server.

public override void FeatureActivated(SPFeatureReceiverProperties properties)

{

SPWebApplication webApp = properties.Feature.Parent as SPWebApplication;

foreach (SPServer server in SPFarm.Local.Servers)

{

if (server.Role == SPServerRole.WebFrontEnd)

{

SPJobDefinition job = new SPJobDefinition("CustomJobDef", webApp, server, SPJobLockType.None);

}

}

}

Hope this helps! Enjoy!

Resource files in SharePoint done properly ..

The message I really want to get across ... Don’t use Properties/Resources.resx

 

SharePoint resource files should be stored in one of 2 places (there are other places, but they are not commonly used):

 

·         12

o   Config

§  Resources ß Location 1 – use this for all Pages, Controls and Web Parts

o   Resources ß Location 2 – use this for all Features, Site Definitions, List Definition, Content Types (i.e. all CAML)

 

If you need to access those resources from code, then use the following:

 

C# (programmatically)

String strValue = System.Web.HttpContext.GetGlobalResourceObject(“NameOfResourceFile”, “NameOfProperty”).ToString();

 

ASPX  Markup

<%$Resources: NameOfResourceFile, NameOfProperty%>

 

XML (CAML)

$Resources: NameOfResourceFile, NameOfProperty;

Do Content Types inherit Event Handlers?

Answer ... YES

It's amazing how many people think that this is not the case. What is even more amazing is how many people blindly accept it without actually trying it out themselves.

If you attach an EventReceiver to your Content Type and then create a Child Content Type then the event receiver will still fire on child content types!

The lesson of the day?

A man who asks a question is a fool for 5 minutes ..
A man who asks no questions is a fool for life!

Delegate Controls - Why you need to delegate!

I wanted to introduce this topic before my break for the festive season, as I am going to be offline pretty much until a few weeks into 2009. And this topic is around the subject of Delegate Controls!

What are Delegate Controls?
Well .. delegates are basically a new control that can be used in SharePoint (both WSS 3.0 and MOSS 2007) which allow you to render different controls on different sites, using features to switch them on and off.

Sound interesting? Well let me whet your appetite further.

Apart from the fact that you can create your own delegates (hopefully your dev brain boggles already!) but Delegates are already in use in a large number of places in SharePoint:

• Top Navigation Menu Control
• Quick Launch Menu Control
• Search Controls
• My Links / Quick Links controls
• etc ...
  

So .. in short, if you want to modify these controls you don't need to modify the master page.
I'll say that again, just in case you missed it. You don't need to modify the master page.

The delegate controls are already in the default.master, so you just need to install and activate a feature which tells SharePoint to use your custom control instead of the standard ones!

Neat huh? (I knew you'd like it)

So what kind of controls can I use with Delegates?
You'll like this answer too .... anything.

Yep, any control (either ASCX based user controls, or DLL based server controls) can be dropped into a delegate. All you need is a feature that registers it!

So how does it all work then?
Well .. delegates have 2 parts to them:

• ASP.Net Delegate Control (effectively a placeholder)
• Feature which registers a new ASP.Net control to use in place of the delegate

The structure of a Delegate Control is very simple:

<SharePoint:DelegateControl runat="server" ControlId="SmallSearchInputBox" />

You only have 1 attribute to worry about; ControlId is basically a unique "name" for your placeholder. The example shown here is for the standard Search control that appears on every page.

After you've got your Delegate Control, you need a feature to implement your actual ASP.Net controls. Luckily the feature is quite straightforward.

Example using Server Control

<?xml version="1.0" encoding="utf-8" ?>
<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
    <Control
        Id="SmallSearchInputBox"
        Sequence="25"
        ControlClass="Microsoft.SharePoint.Portal.WebControls.SearchBoxEx"
        ControlAssembly="Microsoft.SharePoint.Portal, Version=12.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c">
            <Property Name="GoImageUrl">/_layouts/images/gosearch.gif</Property>
            <Property Name="GoImageUrlRTL">/_layouts/images/goRTL.gif</Property>
            <Property Name="GoImageActiveUrl">/_layouts/images/gosearch.gif</Property>
            <Property Name="GoImageActiveUrlRTL">/_layouts/images/goRTL.gif</Property>
            <Property Name="UseSiteDefaults">true</Property>
            <Property Name="FrameType">None</Property>
            <Property Name="ShowAdvancedSearch">true</Property>
    </Control>   
</Elements>

 

Example using User Control

<?xml version="1.0" encoding="utf-8" ?>
<Elements xmlns="http://schemas.microsoft.com/sharepoint/"> 

    <Control Id="CentralAdminLeftNavBarTop" Sequence="100" ControlSrc="~/_admin/configurationmonitor.ascx" />

</Elements>

The Id is the same as your "ControlId" property in your Delegate control. Then you either reference a set of Control Class / Control Assembly, or you point it to a relative path to the ASCX control using the ControlSrc property.

Thats it! Once your feature is activated (at whatever scope you choose ... Farm, Application, Site, Web) then your ASP.Net control will appear in place of the delegate!

But what about Sequence??
Ahhh ... well that is where the real beauty of Delegates comes through in a SharePoint environment! As the controls are installed and activated using Features, how do you account for having multiple features at multiple scopes?

The Sequence attribute allows you to offer a number value which will be used when choosing which feature has priority. Simply put, the feature with the lowest sequence number will be rendered.

This allows you to achieve 2 different things:

1. Upgrade Path
2. Different controls on different sites

In the example of an Upgrade path, take the example of the "SmallSearchInputBox" ControlId.

In WSS 3.0 this is implemented through a feature with a Sequence of 100. In MOSS 2007 Standard a replacement control was installed with a Sequence of 50. And in MOSS 2007 Enterprise another new control is implemented with a Sequence of 25!

Hopefully you can see the picture, but I'll join the dots anyway!

The same delegate, and the same master page is used in all 3 installs. But as you install a newer version, a replacement feature is activated (with a lower sequence number) which takes preference.

So .. if you wanted to replace the "SmallSearchInputBox" control in a WSS 3.0 system, but wanted the "out of the box" control to come back if they ever install MOSS 2007, then you could register your own feature with a Sequence of anything between 100 and 50.

This same approach can be used if you wanted to replace controls on a specific site. So lets say you want to replace the Quick Launch navigation control in specific web sites. You could create a Web scoped feature, and activate it on those sites that you want to replace the Quick Launch in.

No Master Page, No ASP.Net Web Form dev ... easy.

Creating "External" URLs in the Quick Launch menu

This post was prompted by the problems I've been having trying to get header nodes to be added to a site when it is provisioned, which have a custom URL pointing at a different site in the site collection.

Despite the NavBar element in the onet.xml having a URL attribute, this had no effect on my quick launch and it was always created using the current web's URL (i.e. on click it just took me straight to the current web site home page).

So, I embarked on a simple feature which I could use to create those navigation headers.

The code is pretty simple:

public override void FeatureActivated(SPFeatureReceiverProperties properties)
        {
            // get the current "Web" object
            using (SPWeb web = (SPWeb)properties.Feature.Parent)
            {
                    // create the new navigation node
                    SPNavigationNode newNode =
                        new SPNavigationNode("Go to any sub site", "/anysite/anysubsite", true);

                    // add the node to the quicklaunch menu
                    web.Navigation.QuickLaunch.AddAsLast(newNode);

                    // save the changes
                    newNode.Update();
            }
        }

Now, the main thing to note is that when you create your SPNavigationNode object to use the true argument in the constructor!

This tells SharePoint that the URL is an External URL (and by External .. I mean "not in the current web"). If you don't set this then you will get an error telling you that the URL is invalid, or that the page/file cannot be found.

Even if the URL is a relative URL to the same site collection, you still have to set this value to true! Not entirely obvious that one, so one to watch out for.

Adding FAVICONS to SharePoint

Ok .. I admit it .. this isn't really just a SharePoint thing, it's a "web" thing, but quite frankly I am astounded that this isn't something provided out of the box.

So what are these favicon things anyway?

Favicons are the icons that appear in your web browser when you are surfing. They usually appear in the address bar (depending on your browser) and were originally created for older IE environments to help identify web sites in the Favourites folder (hence the name "favicon" = "favourites icon").

Ok, thats great, so how do I get them in SharePoint?

Well, it's really quite simple. Favicons are represented by small "link" tags in your HTML. In SharePoint the most sensible place to put them is in the Master Page. The reason for this is that (generally) your Master Page is rendered for pretty much every page .. therefore you don't have to add favicon links all over the place.

All you need to do is put a "link" tag into your Master Page, within the "<head>" tags (i.e. it needs to be in the page header!)

<link rel="shortcut icon" href="/Portal Images/favicon.jpg" />

How to "SharePoint'ise" it!

I know, that's not a real word, but the best approach you can take is to place your icon image (either an icon or image file) into an Image Library. You can then place a relative URL to a specific image in that image library, and it allows the content editors to upload a new favicon whenever they want :) (with the same version control and content approval you would normally have).

Thats exactly what I did with the sample line of code above, with a file called "favicon.jpg" which is located in an Image Library called "Portal Images" at the top level of the site collection.

Now, if you want to get REALLY fancy, then you could change it to be relative to the Site Collection (so each site collection could potentially have a different favicon). Do you that, just follow my earlier post on creating "Site Collection Relative URLs in MOSS 2007".

Neat eh?

Blank "Web File Properties" when saving document from Word 2003

This is a problem that I encountered with a MOSS 2007 system that was upgraded from SPS 2003.

Specifically, we were developing a custom document library, and found that if we had libraries with multiple content types, when we tried to save documents from Microsoft Word 2003 the "Web File Properties" dialog (the pop-up window that asks you for metadata before the file hits the document library) was completely empty.

None of the fields were visible, not even the Content Type drop-down box!

Well, It seems that this problem is documented and known about, and there is a Microsoft Hotfix available (upon request).

(KB934253) The Web File Properties dialog box displays incorrect properties for a document that is saved in a Windows SharePoint Services 3.0 document library - http://support.microsoft.com/default.aspx?scid=kb;EN-US;934253

The solution to the issue above has been included in to a hotfix, which you can get the information from the following KB article:

(KB934790) Description of the Windows SharePoint Services 3.0 hotfix package: April 12, 2007 - http://support.microsoft.com/default.aspx?scid=kb;EN-US;934790

*files this one away for later*

The onet.xml order of execution (shown in 8 easy steps)

This post was prompted by a post in the MSDN Forums. Simply put … what order do things happen in the onet.xml?

This may seem like a trivial question .. but if you are writing custom features, it is important (to say vital) that you know whether or not your feature will activate before a list is created or after, or whether or not your files in modules will have been provisioned or not.

The order that I discovered is as follows:

1. *<SiteFeatures> in onet.xml
2. *Stapled Site Features (stapled using FeatureSiteTemplateAssociation)
3. <WebFeature> in onet.xml
4. Stapled Web Features (using FeatureSiteTemplateAssociation)
5. <Lists> in onet.xml
6. <Modules> in onet.xml

* note - obviously Site Features will only activate if you are creating a Site Collection, as opposed to a normal web site

How I worked this out

Well, you can copy the steps yourself, in 8 easy steps:

First off, create a new Custom Site Definition (a simple copy of STS). In my example, I called it MHSTS with a single configuration (#0) and created a WEBTEMP file to register it.

Second, create a new SPFeatureEventReceiver class (in an assembly from a class library). The code in the "FeatureActivated" should be something like this:

 

// this will let you check which feature is being activated .. so you can work out the order

string strFeature = properties.Definition.DisplayName;

 

// Add more code to check the SPWeb.Features, SPSite.Features, SPWeb.Lists and SPWeb.Files …

// this will tell you what has been created in the site, and what other features have been activated

Third, create 5 features (the scope is in [ ])

• mhOnetSiteFeature [Site]
• mhStapledSiteFeature [Site]
• mhOnetWebFeature [WEB]
• mhStapledWebFeature [WEB
• mhManualWebFeature [WEB]

Each of these needs to be attached to the same SPFeatureEventReceiver class that we referenced earlier. This allows your debugger to work out which feature is activating … so each time your code steps into breakpoints, you can check the "Feature.Definition" properties and work out which feature is being activated.

Fourth step, create 2 more features which will act as Feature Staplers (using FeatureSiteTemplateAssociation elements).

• mhWebStapler [Site]
  • Staples the mhStapledWebFeature to the MHSTS#0 template (when used to create a new Site or Site Collection).
• mhSiteStapler [WebApplication]
  • Staples the mhStapledSiteFeature to the MHSTS#0 template (when used to create a Site Collection)

Fifth.. in your custom site definition (MHSTS) add the following features into the onet.xml:

• <SiteFeatures>
  • mhOnetSiteFeature
  • mhWebStapler
• <WebFeatures>
  • mhOnetWebFeature

Sixth; Install all of the features using STSADM, and (in Central Administration) activate the "mhSiteStapler" feature within a test Web Application.

Seventh; Within that Web Application, create a new Site Collection using your new site definition (MHSTS). This should fire off all of your features in the following order;

1. mhOnetSiteFeature
2. mhStapledSiteFeature
3. mhOnetWebFeature
4. mhStapledWebFeature

(you can check this by debugging your event handler, and setting a watch on the properties.Definition.DisplayName value)

Now .. if you are keeping a close eye on your debugging window, then you can also start to look at the state of the SPWeb itself. The main thing is that None of the Lists or pages will have been created … at ANY point in this process. This means that <Lists> and <Modules> will not be executed until ALL of your features have activated.

Eighth; Now you can double-check the theory … navigate to your new Site Collection .. and activate your remaining feature manually (mhManualWebFeature).

You should still have your debugging window open .. which should let you know that now (after the site is provisioned) all of the Lists and web pages are now accessible from Code.

Ok .. that's great … errrr … so What ??

Well, this has quite a lot of importance for writing custom features. If you want to write features which automatically manipulate pages and lists then you are going to be a bit stuck, as the lists and pages won't have been created until after your feature activates! (bummer eh? But what you gonna do?)

To get around this problem you really have 2 options:

1. Make your own custom definition. Strip the lists out of the onet.xml, and create ListInstance Features. These can then be created during the feature activation stage, allowing them to be modified by other features that are being activated / stapled.
2. Put some While Loops with Thread.Sleep() statements in your code, basically waiting until the lists have been created. This is not a very elegant solution but you don't have much choice if you are trying to use staplers to modify out of the box definitions (or if you don't want to touch the onet.xml).

Thats it ... comments welcome as always :)

How to check whether a user has completed a survey or not [MSDN Forum Post]

Spotted this one on the MSDN forums.

 

Describes code you can run to check whether or not a specific user has responded to a survey (great for those corporate compliance ones!)

 

You can read the full post here.

Hiding Menu Items and Site Settings in SharePoint using Features

I have used the example of hiding the "Theme" button in Site Settings (as for corporate intranets with strict branding, this is quite a comment request. I would like to show you though how to not only hide specific menus, but how to hide whole sections of the Site Settings menu and other SharePoint menus too!

This all hinges on the HideCustomAction element which you can use in Features.

There are actually some pretty good (MSDN) references:

HideCustomAction - element

Default Custom Action Locations & IDs

How to: Hide a Menu Item in the ECB from SharePoint List Items

John Holliday - SharePoint Custom Action Identifiers

And a cross post, for completion:

How to: Add Actions to the User Interface

So What about this HideCustomAction thing then?

Ahhh yes, got carried away with MSDN references …

First off, you will need a feature (scoped at the Site Level)

Feature File (Feature.xml, don't forget to put your own GUID value in)

<?xml version="1.0" encoding="utf-8" ?>

<Feature Id="GUID" 
    Title="Hide UI Custom Actions"

    Description="This example shows how you can hide menus inside Windows SharePoint Services."

    Version="1.0.0.0"

    Scope="Site"

    xmlns="http://schemas.microsoft.com/sharepoint/">

  <ElementManifests>

    <ElementManifest Location="HideUICustomActions.xml" />

  </ElementManifests>

</Feature>



Elements File (HideUICustomActions.xml)

<?xml version="1.0" encoding="utf-8" ?>

<Elements xmlns="http://schemas.microsoft.com/sharepoint/">

<HideCustomAction

Id="HideThemeLink"

GroupId="Customization"

HideActionId="Theme"

Location="Microsoft.SharePoint.SiteSettings"

</HideCustomAction>

</Elements>

That will give you a feature which will remove the "Theme" links from the Site Settings menu. Now, let me quickly walk you through what the element file is composed of (hopefully the Feature file should be familiar enough).

HideCustomAction – This element defines a new custom action which we want to hide.

Id – This is optional (but recommended), used to identify this specific "hide" action from others.

GroupId – This is the GroupId value for the Custom Action that you want to hide.

HideActionId – This is the Id value in the Custom Action that you want to hide.

Location – This is the location value from the Custom Action that you want to hide.

If you want to find custom actions (and therefore access all of these values) then simply open up the Features folder (..\12\Templates\Features) and do a search for the phrase "CustomAction" in the files. You should find all of the CustomActions that are provided out of the box, and subsequently the ability to hide them if you want to! Exactly the same thing applies to Site Settings, so if you wanted to hide (for example) the "Themes" menu then you can find the Custom Action in the standard "SiteSettings" feature.

Enjoy, and let me know how you get on!

[Code Update] SDK and Microsoft Press Both Wrong?? Custom Fields in the schema.xml

UPDATE

For those who missed my original article (SDK and Microsoft Press Both Wrong?? Custom Fields in the schema.xml) , you can find it here!

 

I finally got round to posting the code that goes with this article ..

 

Please find below the details on code for an SPFeatureReceiver class, that will automatically push down changes from your Content Type Features

(you will need to attach the code to your feature using the ReceiverAssembly and ReceiverClass attributes in your feature.

 

The code needs to be created as a Class in a Class Library, with Microsoft.SharePoint.dll added as a reference, and then added as a Using statement in the class itself.

 

Code Sample

 

EDIT - Thanks to a quick spot from Nick's SharePoint blog, I have updated the code to match! Thanks Nick!

 

class

ContentTypeInstaller : SPFeatureReceiver

{

// CODE DESCRIPTION

// ----------------

/*

* This is a feature handler which should be paired

* with a content type feature.

*

* We have identified a problem with Content Types,

* where the content type site columns are not pushed

* down to custom list definitions, until they are

* "modified" first.

*

* This feature will interrogate all of the associated

* xml files that are attached to the feature.

* Once found, it will "modify" each of the custom fields

* that are referenced.

*

* This should allow list definitions to use the content

* type columns, without declaring them in the schema.xml

*/

public override void FeatureActivated(SPFeatureReceiverProperties properties)

{

using(SPSite site = (SPSite)properties.Feature.Parent)

{

SPWeb web = site.OpenWeb(

"");

// loop through each of the elements in the feature

foreach (SPElementDefinition element in properties.Definition.GetElementDefinitions(CultureInfo.CurrentCulture))

{

#region

Loop through feature elements

 

// retrieve the xml content for the feature element

XmlNode content = element.XmlDefinition;

// only continue if the element is a content type definition

if (content.Name == "ContentType")

{

// grab a new Content Type object

string strCTypeID = content.Attributes["ID"].Value;

SPContentType cType = web.ContentTypes[

new SPContentTypeId(strCTypeID)];

#region

Get FieldRef Order from Content type

// grab the original order, we will need this later

string[] fieldOrder = new string[cType.FieldLinks.Count];

int x = 0;

foreach (SPFieldLink fieldlink in cType.FieldLinks)

{

fieldOrder[x] = fieldlink.Name;

x++;

}

#endregion

#region

Add new columns to the content type

// loop through each sub-node in the Content Type file

foreach (XmlNode node in content.ChildNodes)

{

#region

loop through sub nodes

// only continue for

// the FieldRefs collection

if (node.Name == "FieldRefs")

{

foreach (XmlNode fieldRef in node.ChildNodes)

{

#region

Loop through FieldRefs

// only apply for FieldRef tags

if (fieldRef.Name == "FieldRef")

{

// get the FieldID and use it to

// retrieve the SPField object

string fieldID = fieldRef.Attributes["ID"].Value;

//SPFieldLink fieldLink = cType.FieldLinks[new Guid(fieldID)];

SPField field = cType.Fields[

new Guid(fieldID)];

// first we need to remove the fieldref

cType.FieldLinks.Delete(

new Guid(fieldID));

// and save, pushing this change down

cType.Update(

true);

// NOTE - this will NOT delete any content

// in existing lists!

// now add the field back in again

cType.FieldLinks.Add(

new SPFieldLink(field));

// and call an update, pushing down all changes

cType.Update(

true);

// NOTE - this is what adds the column to those

// lists who don't already have it.

}

#endregion

}

 

}

#endregion

}

#endregion

#region

Apply changes

// reset the field order

// it is possible that adding and

// removing fields would have

// affected this

cType.FieldLinks.Reorder(fieldOrder);

// force update of the content type,

// pushing down to children

cType.Update(

true);

#endregion

}

#endregion

}

}

}

public override void FeatureDeactivating(SPFeatureReceiverProperties properties)

{

}

public override void FeatureInstalled(SPFeatureReceiverProperties properties)

{

}

public override void FeatureUninstalling(SPFeatureReceiverProperties properties)

{

}

}

Enjoy, and happy coding!

SDK and Microsoft Press Both Wrong?? Custom Fields in the schema.xml

Ok, so I think I've found something here. It relates specifically to Site Columns and Content Types, and how they can be applied to Custom List Definitions.

According to the latest release of the WSS 3.0 SDK (August 2008?) and one of the "bible" textbooks for WSS 3.0 development (Inside WSS 3.0 - Ted Pattison & Daniel Larson - Microsoft Press) there are a couple of questionable "best practice" methods that I wanted some insight on.

Personally, I think I've found quite a big oversight, which could be causing problems all over the shop.

The Big Issue (not the magazine!)
The question is around building a custom List Definition, in which you want to implement a custom Content Type (with custom Site Columns).

Now, both the SDK and the Inside WSS 3.0 book (along with many many other references I've seen, including people) state that if you have custom fields in your content type, then you have to add them to the Schema XML.

This has always struck me as very odd. I mean, you have to add all of that information already to your Site Columns (fields) feature .. right? So aren't you just copy-pasting the same info?

Also, if you try to do this through the user interface, SharePoint will automatically add all of the columns from the content type when you add it to the list ... so why can't it do that through CAML too ??

Well, I wasn't happy about this, and decided to try things a different way.

The Big Experiment (not the Discovery Channel)
I wanted to try and make a set of features that proved these statements wrong:

• Site Column Feature (which declares a single, custom text field)
• Content Type Feature (which declares a single Content Type, inheriting from Document, and includes my site column as a Field Reference)
• Document Library Feature (declares a new custom document library definition. The only change is the Content Type and switching on "management of content types". The custom field is NOT added to the Fields section in the schema.xml)

TAKE 1 - As expected
I finished my CAML, installed my features and activated them in a completely new vanilla Site Collection.
My site column and content type both appeared on activation (as expected) and my new library definition popped up in the create menu (also as expected).

I then created my Document Library ... and ... <anti><climax>my columns did not appear in the library</climax></anti>.

This was, for the cynics among you, entirely expected. Microsoft themselves (through the holy scriptures of the SDK) have stated that this would happen.

But then things got interesting ...

TAKE 2 - The interesting result
Ok I thought ... what is going on behind the scenes? Somehow I need my Content Type to take preference over the schema...

I had a bit of a think (and probably scratched my chin a few times).. eventually I hit upon an idea (don't ask me how .. one of those late night coffee moments!)

I reset my environment (brand new, clean site collection) and re-installed and re-activated my features.

BUT .. this time, BEFORE I created any libraries I went to the Content Type Gallery and looked at my Content Type.
I clicked on my custom field (in the content type itself) and changed one of the values (I actually changed it into a required field) and hit the OK Button (note - the "push changes down to child content types" WAS selected).

I then backed out of those screens, and created my Document Library... and ... <surprise>my custom columns DID APPEAR!!</surprise>!

This was quite a shock .. I mean .. the Fields section in my custom document library schema is empty (apart from the fields that come out of the box).

I checked the library settings, tried uploading a file, and even checked the Document Information Panel in Word 2007. My custom field was there, with all the correct metadata.

Conclusion - (the important bit .. for those who bothered to scroll down)

My conclusion:

You do not have to declare custom fields in the schema.xml, as long as you manually update your content type before creating any libraries.

But that's not practical ... I have <insert huge number>'s of content types!
Well .. I hear ya.

I have just finished off a custom Feature Receiver which can be attached to any Content Type Feature as a Receiver Assembly / Receiver Class.

The code will (when the feature is activated) locate all of the Content Type elements in the feature, and do an initial "Update - push down changes" when the content types are first activated. (UPDATE code as been published to my Blog. See this article for details).

This is effectively the "magic bullet" I was looking for, and with this:

• Custom Fields do not have to be declared in the schema.xml
• Custom Fields can still be referenced in views, even though they are not referenced as Fields.
• All column and meta data management can be developed in the Content Types and Site Columns

Comments / Discussions / <not>Flames</not> ...

all are welcome :)

Some things to note about using "Templates"

I’ve been doing some testing, mainly prompted when I was questioned about Site Templates (I originally thought that Workflows built in SharePoint Designer would not be copied over in templates) and they proved me wrong! I have done some reasonably extensive testing, and have come to the following results:

Quick Summary

• SharePoint Designer Workflows will ONLY be copied over in a SITE TEMPLATE WITH CONTENT (any other configuration fails)
• Custom Workflows (including the “out of the box” ones) will be copied over in ALL TEMPLATES.
• If you hide the “default” content type then it will re-appear after using a Template.
• Version history is not preserved when using templates with content.

List Templates include the following items:

• Custom Workflows
• Adding more Content Types
• Re-ordering the Content Types
• Changing the default Content Types
• Adding a custom Content Type
• Versioning Settings
• Content approval Settings

Site Templates include all of the above, but if you also select “Include Content” then you will also get “SharePoint Designer Workflows”!

If is important to note that if you hide one of the “out of the box” content types from your list then saving a List Template OR Site Template will not have any effect! The content type will be back and visible when your new list / site is provisioned!

I then did some more general “what if” scenarios:

 

Query	Answer
What happens if the custom Content Type in the Site Template does not exist in the target system?	It creates a LOCAL (i.e. list level) copy of the Content Type, and links it to the parent of the original content type. This means that it works as it used to work, but the “central” copy of the content type no longer exists so maintenance has to be done manually and changes cannot be “pushed down”.
What happens if you delete a Custom Content type while it is in use normally?	Error – “The Content Type is in use”
If you save with content, does it preserve version history?	No. All items copied over will be created as version 1.0. All draft versions are removed.

 

It is also important to state the Disadvantages of using Templates and that is mainly regarding Site Templates, and it is 3 main things:

• Performance. Using lots of site templates will degrade the performance of your system.
• Maintability. Maintaining the structure going forward will be difficult, as you have no “schema” to modify. All you can do is
• Limited to the User Interface. Mainly with reference to lists, changing the default fields, default content types, and modifying some of the “back end” properties is not possible through the user interface, and therefore cannot be achieved using “List Templates”.
• Automation. There is still much more flexibility in terms of automation when using “definitions”. We can tie in event handlers and automation tools to do “almost anything”.

Correctly disposing SPWeb objects

I guess SharePoint development is a bit like riding a wave. First off you find you do almost everything wrong ... after a while you pick up some of the "best practices" (like disposing SPWeb objects) and then you find that you are still doing things wrong. Well ... thats the learning curve I'm afraid .. and it rounds down to where your SPWeb object came from, and when you should be disposing it.
There are several different ways of retrieving an SPWeb object.
Take the following 3 examples:
1) SPContext
SPWeb web = SPContext.Current.Web
2) OpenWeb
SPSite site = new Site("http://myserver/");
SPWeb web = site.OpenWeb("");
3) Site "RootWeb"
SPSite site = new Site("http://myserver/");
SPWeb web = site.RootWeb;
All 3 of them are valid SPWeb objects, but they are also all slightly different.
Lets start off with SPContext. This is basically a context object which represents the context of the current site that you are navigating. The SPContext.Current reflects where you currently are in the SharePoint site collection (so SPContext.Current.Web will return the SPWeb object that represents the site from which your code is executing ... for example .. the site that your web part is sat in).
There are 2 major issues with using SPContext.
1) It is context sensitive. Yes .. I know thats like saying the sky is blue, but you do need to consider this if you are looking at generic code that might be accessed from multiple projects. Pulling out the "SPContext" properties is fine from a web part, but you don't want the same code executing from a Console Application or a Workflow Custom Action.
2) SPContext is in use by all manor of controls and parts on the site and the page. If you dispose of the SPContext.Current.Web object then you will find that you get a nasty error message next time you try to access that object (forcing you to refresh the page). So if you've ever seen the error message below, check your disposal!

"Trying to use an SPWeb object that has been closed or disposed and is no longer valid"

Make sure you dispose of OpenWeb("") objects!

This is very important. When you are basically just creating your own SPWeb objects you will need to make sure that you dispose of your objects correctly. The best way of doing this is encompassing your web object in a "Using" statement

using(SPSite site = new Site(http://myserver/);
{

using(SPWeb web = site.OpenWeb(""))

{

// work with web object

} // web object disposed

} // site object disposed

Do NOT Dispose SPContext.Current.Web or SPSite.RootWeb
If you write a using statement such as:

// This is bad! m'kay ?
using(SPWeb web = SPContext.Current.Web)
{
}

What you are actually doing is disposing the SPContext.Current.Web object .. not just your own "web" object. THIS IS BAD. You should actually just declare them with a single, normal line, such as:

SPWeb web = SPContext.Current.Web;

And exactly the same applies to SPSite.RootWeb properties too!

Tip - Apply Branding in WSS 3.0

Here's a quick tip, cos I don't have loads of time.

We've had lots of enquiries from our clients about Branding in WSS 3.0. If you are running MOSS 2007 then you have all of the nice "Look and Feel" settings when you go to Site Settings, allowing you to not only change the Master Page and Style Sheet, but also to push those changes down through the rest of the Portal.

Well ... to put it bluntly, WSS 3.0 won't let you do any of that.

So .. how about creating a Branding Feature?

You can write custom event handlers for Features (called Feature Receivers, using the SPFeatureReceiver class), so that you can have code that executes when a Feature is Installed, Activated, DeActivated or Uninstalled.

I won't go into the details here, but there are some great articles about this on the web and MSDN.

So hows this for an idea?

Write yourself a Feature that applies Branding to WSS 3.0 sites (or any site for that matter!). You can make it set a new Master Page, change the Style Sheet, the Logo, set a custom "Site Theme" or even just inherit all those properties from the Parent site.

One great use (that we use quite regularly) is to create a Hidden Feature that just applies the Parent Site's branding settings to the new site. You can then use Feature Stapling to make sure that ANY (staple to "GLOBAL") site that gets created will be branded .. you don't need to worry about the site definition!

As for the values used for the Branding, well you can pass these in as "Properties". Every feature can have them, and they are specified as follows (in the Feature.xml file).

<?xml version="1.0" encoding="utf-8" ?>
<Feature xmlns="http://schemas.microsoft.com/sharepoint/"
...>
    <Properties>
      <Property Key="NewMasterPage" Value="/_layouts/CustomBranding/NewMasterPage.master"/>
    </Properties> 
</Feature>

So in that feature we have specified a Property called "NewMasterPage" (you can have as many of these as you like).

Then ... to get that Property value out in your Feature Receiver you can use the following line of code:

string strMasterUrl = ((SPFeatureProperty)properties.Feature.Properties["NewMasterPage"]).Value;

If the property doesn't exist, or the value is blank then it will return an empty String (so you shouldn't need to worry about null reference exceptions!)

Then, all you need to do is change the SPWeb property and call SPWeb.Update() to apply the changes! For example:

public override void FeatureActivated(SPFeatureReceiverProperties properties)
{

// apply branding
using (SPWeb web = (SPWeb)properties.Feature.Parent)
{

   web.MasterUrl = strMasterUrl;

   web.Update();

  

}

}

Hope this gives you some good starting points! You might want to extend this further; here are some interesting things you might do:

• 
  
  Add a Property Flag to reset all sub-site branding settings. This can then be activated at the top level to reset all branding Site Collection wide.
  
  
  
• 
  
  Add an "inherit from Parent" Property. Then you can use SPWeb.ParentWeb to get the parent SPWeb object (and all of it's branding settings).
  
  
  
• 
  
  Store the "original" branding settings (before you overwrite them) in the SPWeb.Properties hashtable. Then, add an "DeActivating" event, and put all the original properties back again! (so when you deactivate the feature, it puts the branding back in place.. a great "rollback" option!)
  
  
  

Good luck .. and get Branding!!!

MKeeper

Understanding Content Type IDs

This is a topic that is confusing to many developers, and this post aims to clear up some of the issues around Content Type IDs, how they are structured and how you can extend them.

Content Types are re-usable definitions for meta-data that can be stored in Lists and Libraries in SharePoint. I won't go into them in too much detail, you can find a reasonable amount of information about Content Types here and Content Type IDs here on MSDN.

Content Type IDs
As you should know, Content Types are based around inheritence (and if you didn't know that, then you should have followed the links above more carefully :P). Each content type inherits from a parent content type (apart from System which is the root of all Content Types). A child content type will automatically have the same site columns that the parent has.
You can see this structure in the "out of the box" (ootb) Content Type IDs:

0x = System
0x01 = Item (inherits from System)
0x0101 = Document (inherits from Item)
0x0120 = Folder (inherits from Item)

Basically you have a {ParentID}{ChildID} structure.
This goes for all of the ootb Content Types (and you can find a list of them all in the WSS 3.0 SDK, downloadable free from Microsoft).

Custom Content Types
If you want to customise Content Types then you are looking at a slightly different ball game. For some reason (don't ask me) you cannot use the same {ParentID}{ChildID} structure. Instead you have to use a {ParentID}{00}{GUID} structure.

So lets say you wanted to create a Custom Content Type that inherits from Document and call it "Demo Document". You would create the ID as follows:

0x010100763A775BFF7849a0A65FE1F57F56CC33 = "Demo Document"

Now lets say you want to inherit from "Demo Document" and create a further child, say "Demo Sales Document", you would create the following:

0x010100763A775BFF7849a0A65FE1F57F56CC3301 = "Demo Sales Document"
0x010100763A775BFF7849a0A65FE1F57F56CC3302 = "Demo Sales Document"

Note that here we already have a custom Content Type ID, so we can just go back to {ParentID}{ChildID} again.

So each "child"  of a custom content type simply adds a unique 2-digit  id on the end of the Content Type ID. If you wanted more child types you would add "02", "03" and so on.

And you can go one step further, if you wanted to create another "child" section, for example:

0x010100763A775BFF7849a0A65FE1F57F56CC3301 = "Demo Sales Document" (inherits from "Demo Document")
0x010100763A775BFF7849a0A65FE1F57F56CC330101 = "Demo Sales Order Form" (inherits from "Demo Sales Document")
0x010100763A775BFF7849a0A65FE1F57F56CC330102 = "Demo Sales Invoice Document" (inherits from "Demo Sales Document")

So from this you can create simple and effective "Parent - Child" relationships throughout your Content Types which allow you to manage your meta-data quickly and effectively.

In the above structure, if you want to add a custom column to all custom documents? You just modify "Demo Documents" and it will also update all of the child content types too!

Closing Thoughts ...
One final thing to be careful of. If you are using CAML (Collaborative Application Markup Language) then you need to be aware of the following:
Modifications to a Content Type applied through a CAML based feature will NOT update child content types!!!!!
If you want to make sure that child content types are updated then you either have to make the change through Site Settings --> Content Type Gallery, or make your changes through managed code (such as C# using the SharePoint API).
Both of those give you the option of updating all child Content Types .. but CAML does NOT (unless I am mistaken, please let me know if I am! :))