Now an MCPD in SharePoint 2010!

Well, I had a few days spare as my new contract hadn't started so was pondering what to do, and decided to sit the two developer based SharePoint 2010 Exams.

There are two of them in the new development based certification track for SharePoint:

• 70-573: TS: Microsoft SharePoint 2010, Application Development
• 70-576: PRO: Designing and Developing Microsoft SharePoint 2010 Applications

The first exam will grant you an Microsoft Certified Technology Specialist and the second one the new Microsoft Certified Professional Developer.

I am very pleased to report that I passed both exams (yay me!) and can now proudly sport my new certification logos:

Microsoft Community Contributor Award 2011

Well, I got a very nice surprise today as I have been awarded the Microsoft Community Contributor Award for 2011.

This is an award given to people to "recognize notable contributions to Microsoft online community forums such as TechNet, MSDN^®, and Answers".

For those not aware there are fantastic SharePoint forums on MSDN (http://social.technet.microsoft.com/Forums/en-US/category/sharepoint2010) and I can typically be found browsing the Development forum, answering the odd question here and there, so I am well chuffed to be recognised in such a way.

As well as my funky new logo / badge (below) I have also been awarded a Safari Books subscription for over 400 online Microsoft Press books! :) Which is extremely awesome.

So thank you to whomever put me forward for this, thanks to Microsoft for the recognition (and the prize) and thanks to the SharePoint community in general for making this product (and the community itself) so brilliant in itself!

Blank Ribbon? Check your Locale Settings!

We recently had this problem on a corporate Intranet. We setup the SharePoint farm and (being good boys) setup the regional settings to match the corporate culture (in this case .. we set it to United Kingdom).

Now, the United Kingdom locale happens to be a bit of a funny one .. apart from the fact that English people spend half their IT lives changing settings from "US" to "UK" (or bemoaning the fact that there IS no UK setting) and removing the Z in organisation.

It does seem though that SharePoint 2010 includes a few things to trip you up too! The following screenshot shows what happens you configure your Site Collection Regional Settings to Locale "United Kingdom":

As you can see, a whole raft of the ribbon buttons have disappeared. Why?

Well, looking in the DOM we can find a reference to the PNG Image which used to render the ribbon buttons:

/_layouts/2057/images/formatmap32x32.png

There is only one problem ... There is no Layouts/2057!

You see .. SharePoint is taking the local regional settings locale (United Kingdom) and is getting the language ID (en-GB) which is 2057 ... and then looks in the Layouts folder.
Note - On a default install this is setup as en-US, which is 1033

Normally you would install a Language Pack to support additional languages ... but United Kingdom is the same language (English) so it doesn't have one (2057 is in fact a sub-set of 1033).

Solution
There really is only one solution ... make sure your "Locale" is set to Enlgish (United States) but set the Time Zone to London (this will adjust the Date/Time formats).

Took a while to work this one out (kudos to Vikash Daya .. the Infrastructure Architect on the project who spotted it)

Code Solution - Import AD Photos into SharePoint User Profiles

[Update: Code download files updated 18/03/2011 - see below]

This is in relation to a previous post I made last week;

Active Directory Images are not imported by the SP2010 User Profile Import

So, the source code is finally ready and uploaded for your enjoyment :)

Now - I must first off give credit to the sources of inspiration. A lot of the code in this solution is copied / borrowed / inspired by the following posts:

• MSDN Blog - Background information on why you cannot import the property from AD directly
• Blog Post - How to upload a user profile photo programmatically
• Technet Blog - Programmatically using the jpegPhoto attribute in Active Directory

All I have done is brought their code and samples together and packaged it into a WSP that runs from SharePoint Timer jobs, so you have the convenience of a SharePoint 2010 farm solution :)

Also it should be understood that both AD and the User Profile database are quite critical parts of anyone's SharePoint farm, so sorry, but first I need to make a ....

... Disclaimer - All code and solutions are provided "as is" and should be used at your own risk! It is highly recommended that you test these in an isolated environment, and I confer no responsibility for any loss or damage from using the code, advice or solutions provided on this blog, or any related content.

Ok, now that is out of the way we can get on with business :)

[Update - 18/03/2011 - I have updated both sets of files so that it now uses the "distinguishedName" attribute to identify users in AD .. as this is a more reliable method and was in response to a reported bug]

I have uploaded the files to my Sky Drive including:

• Full Visual Studio 2010 Solution
• Installable WSP Package

When you roll out the WSP you will find that it includes the following functionality:

Farm Scoped Feature
The WSP package includes a farm scoped feature called:

Hatch Solutions - Import Photos from AD (Timer Job)

When activated this will automatically identify the default MySite host application and create a custom Timer Job (see below) attached to that web application.

My Site Timer Job
The Timer Job (installed by the Farm Feature) is designed to run on the My Site Host web application, and is pre-configured to run once-per hour. It is called:

Hatch Solutions - Import Photos from AD

This will do the following:

• Automatically identify all AD accounts in the current User Profile Database
• If the AD account has a "jpegPhoto" attribute, then this is extracted
• The photo is converted to three thumbnail images, and uploaded to the My Site Host profile photo asset library
• The photo for that user profile is updated to point at their newly uploaded photo

Hope you enjoy, the source code is there for all to see, and good luck!

Tip - CAML Query Retrieve tasks assigned to user (including both AD and SP Groups)

This is something I have seen so many people struggle with, but it really is very easy, with the help of the "<Membership>" element.

The Membership element allows you to basically check to see if the AssignedTo field is assigned to any group which the current user is a member of.

Of course, you still need to use it in conjunction with a standard FieldRef check against the user's ID (which you can get using the <UserID/> element.

Below is the CAML query to return All Tasks Assigned to the Current User, including specific assignments, and where the task is assigned to a group that contains the current user (both AD Groups and SharePoint Groups).

I suppose theoretically this should also work with groups in custom Membership Providers .. but haven't tried it.

<Where>
  <Or>
    <Eq>
      <FieldRef ID='Assigned To' />
      <Value Type='Integer'><UserID/></Value>
    </Eq>
    <Membership Type='CurrentUserGroups'>
        <FieldRef Name='AssignedTo' />
    </Membership>";
  </Or>
</Where>

Active Directory Images are not imported by the SP2010 User Profile Import

This is something that I found when starting a new SharePoint 2010 (SP2010) Intranet project. You see, my client has quite a simple requirement, at least it seems that way at first.

They have an HR database which they use to store everything from employee details to security card information. For this reason they manage all of the employee photos here, and they are pushed into Active Directory (using the LDAP "jpegPhoto" attribute) to make them available in applications like Outlook.

So ... to put it quite simply .. they don't want to use SharePoint to upload their photos. In fact, they already have their photos in AD .. they just want to pull them into SharePoint 2010.

Can't we just map the Profile Property to AD and Import the value?
So here is where we hit the roadblock.. The Active Directory "jpegPhoto" attribute is of type "Binary Data" and the SharePoint 2010 User Profile property is of type HyperLink (as it typically links to an image in an Asset Library in the My Site Host site collection).

As a result, you cannot import it using SharePoint 2010 functionality (although it may be possible if you have also purchased the more sophisticated "ForeFront Identity Manager Synchronisation Server" product).

Options ??
Well .. SharePoint IS used as a development platform so there are some options. Obviously there is a full API for reading and writing in and out of the User Profile database (some more accessible than others).

There are a few good blog articles that you can follow if you want to build your own import function. If you are happy to wait a while longer then I already have my own solution which I will be posting up with the following features:

UPDATE - Full Source code and WSP now published

* WSP Package
* Farm scoped feature, which installs a Timer Job attached to the MySite web application
* Iterates through all User Profiles, and finds and extracts Binary image data from Active Directory
* Automatically creates thumbnailed images in the My Site Host Asset Library
* Automatically updates User Profiles to point to those new images

During my research Glyn Clough also pointed me in the direction of another solution which allows you to do the same using a Console Application... personally I prefer the more "SharePoint" route of Timer Job and WSP package :)

In the meantime, you can feel free to avail yourselves of these posts:

• MSDN Blog - Background information on why you cannot import the property from AD directly
• Blog Post - How to upload a user profile photo programmatically
• Technet Blog - Programmatically using the jpegPhoto attribute in Active Directory

SUGUK London, 14th October Sandbox Development and jQuery

Well it was a great night last night. We had a really good turn out (around the 90 mark I think in the end) and it was a cracking dev-focused session.

Firstly, hats off to Jaap Vossers (@jvossers / blog.vossers.com) who stepped up for his first community event and gave a really good presentation on jQuery and SharePoint.

My personal favourites were the Developer Dashboard Visualiser (which created a graphical time-line of how long each page component took to load, which you could drill-into) and the Site Settings snap-in (which allowed you to drop in the "Site Settings" page for the current site by pressing Ctrl-S from any page!). Lots of content in there, including using Web Services and utilising some of the Client Object Model.

Finally Jaap ran a prize draw giving away 5 free copies of a jQuery development book! Good stuff!

My turn came next, presenting on SharePoint 2010 Sandbox Development. For those of you who follow me on Twitter (@MartinHatch) you will know that I accidentally deleted my virtual machines last week so had to rebuild my farm and re-code all my examples.

Well .. not one to try making things easier for myself I decided to code all of them from scratch (Sandbox Web Part, Full Trust Proxy and a Solution Validator).

The only one which didn't work was the Full Trust Proxy (I then found out I had a note in my slide deck on the "demo" slide which read - "make sure you re-start the Sandbox Service after you install the full-trust proxy" .. I have updated my slide deck now to include that VERY important line!) but overall I think it went very well and muchos thanks to all the feedback on Twitter (make sure you use the #SUGUK hashtag!)

My own slide decks and code samples can all be found on my Sky Drive account here:
http://cid-60f12a60288e5607.skydrive.live.com/redir.aspx?resid=60F12A60288E5607!621

Finally we had an impromtu "Ask the Experts Session" .. with the awesome team of Eric Shupps, Chris O'Brien, Andrew Woodward, Steve Smith, Jaap Vossers and myself!

The evening ended up with a SharePint (although I had to head off to get the last train out into the sussex wilderness I call home!).

If you were there .. hope you enjoyed it (all feedback welcome!). If not ... I hope to see you at the next SUGUK meeting! :)

Presenting at SUGUK London - October 14th

Yep, the next SharePoint User Group UK (SUGUK) London session has been announced, and I'm going to be presenting on Sandbox Development. (yay!)

The event is on October 14th 2010 at at Mostyn Hotel (Marble Arch, Bryanston Street, London, W1H 7BY) and the evening starts at 18:00.

I have the second slot and will be talking about Sandbox Development, a subject which I've been keenly interested in what with Content and Code's products and the expectation of SharePoint 2010 suposedly coming to BPOS (Microsoft's cloud services).

"This development focused session will walk through the new SharePoint 2010 Sandbox Solutions framework, including the architecture, configuration and development of Sandboxed solutions. Although covering off a few administrative functions do not be fooled .. we will be opening Visual Studio 2010, stepping through code and debugging processes!"

The session kicks off with Jaap Vossers talking about SharePoint and JQuery. I've been working with Jaap for several months now and some of the JQuery stuff he's done is fantastic, so this is well worth coming along for that session alone!

"This session will cover everything you need to know about harnessing the power of jQuery in your SharePoint sites. The introduction will look at getting started, the syntax and the plugins before we look at jQuery in a SharePoint context; the benefits, examples, calling web services, loading scripts and deployment. We will then cover the various frameworks and utilities (SPServices, jQueryLoader) before rounding off with a look at further integration opportunities in SP2010 and specifically the Client OM and REST. This is a technical session."

If you want to register then please sign up at the SUGUK Forum Post (http://suguk.org/forums/thread/25091.aspx).

I'm really looking forward to it, so please come along, and hopefully I'll see you there!

Forays into SharePoint 2010 Performance Testing with Visual Studio 2010

Over the past six months I have increasingly become an evangelist of Performance Testing. It has always previously been an area that I was aware of but I never really got massively involved in, but recently I've seen it as an increasingly important part of my work, especially on the larger scale projects with load balanced web front ends (for performance, not just redundancy) and you start hitting I/O limits on SQL. I suppose this may have been triggered by the SharePoint Conference 2009, and one of my follow up blog posts "Load Testing SharePoint 2010 with Visual Studio Team Test".

So in this post I firstly wanted to look at why you should do Performance Testing?

It sounds like a bit of a stupid question (with an obvious answer) but it really is surprising how many people don't do it. How many of you have ever asked the following questions on a project?

"How many users can the production system support?"
"What would be the impact of doubling the number of users?"
"What impact with backups have on performance?"
"How fast will the solution perform during peak hours?"
"What is the most cost-effective way of improving performance?"

All of these are questions that you absolutely HAVE to be able to answer. The client (whether it is your organisation, or another organisation who you are running a project for) deserves to know the answers to these, and without them how can you have any idea whether your solution is going to be fit for purpose?

Sure, you can read up on Estimating Performance and Capacity Planning in SharePoint, but all that gives you is some rough guidelines.. we need to be able to apply some science to the process!

The last question is probable the most compelling. Re-configuring farms and buying new hardware is an expensive process, the consultancy alone can cost thousands of pounds, and you don't want to have your client coming back asking why they just spent tens of thousands of pounds on a new state of the art iSCSI SAN array, to have zero impact on performance ("hey .. we thought it would help .. but we didn't really know!") because the bottleneck was actually the CPU on the Web Front End (WFE).

The story often gets even worse when things do start going wrong. If you have ever been in the unfortunate position where you are troubleshooting a system that is performing badly, these kinds of questions are quite common:

"What is causing the poor performance?"
"How can we fix this?"
"Why did you not notice this during development?"

Again, the last two questions is the killer.. if you don't do any Performance Testing then you won't know that you have a problem until it is too late. The earlier you can get some metrics on this, the faster you will be able to react to performance issues (in some cases finding them and fixing them before the client even knows about it!)

Equally, without performance testing you won't know WHY the problems are occuring. If you don't know why then you can't know HOW the best way is to fix them!

So the key messages are this:

• Early Warning .. catch problems early on and they will be easier to fix. There is no point waiting until users are hitting the system to find out the solution can't cope with the load!
• Knowledge ... what is causing the problems, and how do you fix them?
• Confidence ... not just that you know what you are doing, but you can prove it. This instils confidence in your sales, confidence in your delivery, and confidence from your clients too!

Performance Testing with Visual Studio 2010
I've been using Visual Studio 2010 Ultimate edition. It is the only "2010" product that incorporates Web Performance Tests and Load Tests, the two critical pieces that you will use to test the performance on SharePoint 2010 (or any other web based system). It also integrates tightly with Team Foundation Server and provides "Lab Management" capability, but that is out of the scope of this blog post.

In order to do comprehensive testing you really need 4 different software packages:

1. Visual Studio 2010 Ultimate: This is where you create your tests and control the execution of them.
2. Visual Studio 2010 Test Controller: Part of the Visual Studio Agents 2010 ISO, this allows you to co-ordinate tests executed by several "agents", as well as collecting results and storing all of the test results (and performance counters) in a database. The license for this is included in Visual Studio 2010 Ultimate.
3. Visual Studio 2010 Test Agent: Part of the Visual Studio Agents 2010 ISO, this can be installed on machines that will simulate load and execute tests. They are connected to a "Controller" which gives them instructions. The license for this is included in Visual Studio 2010 Ultimate.
4. Visual Studio 2010 Virtual User Pack: This is a license that allows you to increase the number of virtual "users" you can simulate by 1,000 (for each pack that you purchase). This is a separate license that must be purchased separately (there is no trial version!)

If you need any help installing these and getting them running then there is a great MSDN article which you should read: Installing and Configuring Visual Studio Agents and Test and Build Controllers or the equally awesome article from Visual Studio Magazine: Load Testing with Visual Studio 2010.

So what about actually creating the tests?

Well, the interface is pretty simple. You can create your "Web Performance Tests" using a simple Browser Recorder (literally using a Web Browser which records all of your actions, and then click "stop" when you are finished). This works great, but there are a few caveats:

• You might want to use the "Generate Code" option if you are adding documents or list items. This converts your recorded web test into a code file, allowing you to programmatically change document names, or field values .. useful to make sure you are not just overwriting the same document over and over again
• Web Service tests require a bit more "knowledge" of how they work, needing the SOAP envelope (in XML) and the SOAPAction header.

It is worth noting that there is an excellent Code Plex project available: "SharePoint Performance Tests". Although this was written for Visual Studio 2008 (you can convert it to 2010 if you want) it contains a number of configurable tests (via XML) that allow you to dynamically create tests for generic SharePoint platforms .. well worth a look!

You can then very easily create a "Load Test" which allows you to pick'n'mix tests, and a distribution of which tests you want to run.

My personal favourite is the "Tests Per User Per Hour". For this you would sit down with your client and work out "what would a typical user do in an hour of using the system.." one such activity resulted in this kind of activity distribution:

• Hit the site home page 50 times
• Execute 10 searches
• Upload 5 documents
• Respond to 20 workflow tasks

This kind of valuable information allows you to build your tests and then distribute them using the Load Test. All you do then is plug in how many users you want to simulate and away you go!

Counting the Counters?
All of this so far is great stuff, but without the performance counters you really aren't going to get much legs from Visual Studio. You might get the WHAT is going on (i.e. do the tests complete very quickly?) but you certainly won't get the WHY information which is oh-so important (i.e. is it the CPU, RAM or Disk?)

For this you need to add Performance Counters... thankfully this is rediculously simple. You have something called "Counter Sets" which you can configure to collect from the computers that operate in your farm.
There are a bunch of pre-defined counter-sets you can choose from:

• Application
• ASP.Net (I pick this for my WFE Servers)
• .Net Application (I pick this for my Application Servers)
• IIS
• SQL (I pick this for my SQL Servers)

I won't go into any more detail than that. A step-by-step walkthrough of the options (including screenshots) can be found at the Load Testing with Visual Studio 2010 article at Visual Studio Magazine.

What about the Results?
Well, there isn't a really simple answer to this. You really need to have a good understanding on how the different hardware components interact, and what limits you should be looking for.

The big hardware counters (CPU usage, Available Memory) are the obvious ones. Any server which exceeds 80% CPU usage for any sustained period is going to be in trouble and is close to a bottleneck. Equally any server which starts to run out of memory (or more importantly .. slowly loses memory, suggesting a memory leak!) should be identified.

But it's the deeper, more granular analysis that proves most useful. On a recent client project I was looking at a Proof of Concept environment. We knew that we had a bottleneck in our WFE (CPU was averaging around 90%) and it was extremely workflow heavy, but the page performance was far too bad to put down to just the CPU.

On closer inspection we found a direct correlation between ther Page Response Time and the Disk Queue Length in SQL Server:

The top-left corner is the Disk Queue Length in SQL Server, and the Top Right is the Page Response Time for the Document Upload operation (bottom right is the overall Test Response time), clearly the spikes happened at the same time.

This is the true power of using Visual Studio. All of the tests and performance counters are time-stamped, allowing you to drill into any specific instance and see exactly what was happening at that moment in time!

Looking closer at the SQL Disk usage, the Write Time (%) and Read Time (%) show us even more interesting results:

The top of the graph shows the Disk Write Usage (%) and the bottom half shows the Disk Read Usage (%). Clearly, the disk is very busy writing (often being at 100%) while it does very little reading. This fits perfectly with our test results as most of the "read" operations (like viewing the home page, or executing a search result) were extremely fast ... but most of the "write" operations (like uploading a document) were much slower.

So the WHAT is slow write performance (uploading of documents).
The WHY is now very simple, the disks on the SQL Server need looking at (possibly upgrading to faster disks, or some optimisation in the configuration of the databases).

Conclusion
To be honest I could talk about this subject all day, but hopefully this gives you some indication of just how crucial Performance Testing is .. and how powerful Visual Studio can be as a testing tool.

The ease of creating test scripts, the vast flexibility and power of the enormous performance counters available, and the ability to drill into a single second of activity and see (simultaneously) what was going on in all of the other servers .. its an awesome combination.

I'll probably be posting more blog posts on this in the future, but for now good luck, and hope you get as much of a kick out of VS2010 as I have :)

HTC Twitter "Peep" and OAuth

[UPDATE - 27/09/2010]
HTC Peep on my HTC HD2 is now working!
I found that I needed to login using my email address instead of my username, but it is now working again!
(hurrah)
[/UPDATE]

If you are using a Windows Mobile device from HTC (like me .. I have an HTC) then you've probably run into the same Twitter issue that I have.

Twitter recently shutdown their Basic Authentication method for Twitter and this hosed a whole range of Twitter applications which were not appropriately using the "OAuth" method that Twitter preferred.

One of those applications is the HTC "Peep" application. Now, I'm a quite avid consumer of Twitter, and although Twitter did recently release an announcement that it was working  it seems that HTC Windows Mobile clients (such as the HD2) are still not working.

Well, I submitted a question to HTC Support and they very kindly sent me a response back (in under 1 hour, very impressive). Their email response was as follows:

"We are currently investigating an issue with our Peep/Twitter/Friend Stream client that has stopped working and hope to have a solution soon. Please monitor the support pages for updates, or if you prefer we can record your details and contact you again once a solution is available"

So hopefully it will all be working again soon. Fingers crossed ... It's not exactly a mission-critical application for me, but I do hate it when things just "stop working" that I was a user of.