Lync 2013 is here – planned your upgrade yet?

Lync 2013 is here, and it’s action-packed full of fantastic new features that will quite honestly change your life. If, like many organisations, you drank the kool-aid a while back and deployed OCS or Lync 2010, you’ll be asking yourself whether you should be upgrading, and how you go about it.

We’ll do a whistle-stop tour through the pick of the new features – hopefully helping you build your business case to upgrade – and run step-by-step through a Lync 2010-2013 upgrade, including a few lessons from the field to save you some pain.

Registration is required for this event – please do so here if you’d like to attend. (did I mention there’s free beer and pizza?)

Failed to create external configuration stare at <path>: General access denied error (Virtual machine ID 0x80070005)

User "domain\user" failed to create external configuration store at <path>: General access denied error (virtual machine ID 0x80070005)"

There’s no shortage of suggested workarounds on the web, but most relate to permissions or disk format (NTFS/FAT32), and none worked for me. After exploring a bunch of the suggested fixes, I stumbled on this superbly technical post that raised the excellent point that not all USB devices mount in the same way, and therefore don’t all present to Hyper-V in a way that is compatible.

So on a hunch, I tried an alternative mounting method in Disk Management. Rather than the normal approach of creating a volume and assigning it a drive letter, I mounted it as an NTFS volume under an existing drive – my existing HDD was partitioned with C:\ and D:\ partitions, so I mounted the SSD volume as D:\SSD\.

And just like that, everything worked perfectly. I had no further issues using this storage media with Hyper-V. Problem solved.

Now, there’s a distinct likelihood this has a performance impact (I’m genuinely unsure if there is a tradeoff involved in using the NTFS mount method) – though in my ad-hoc testing I saw no discernible difference in performance of the SSD-based data. If anything, the only performance hit seemed to be when accessing other data on the rest of the D:\ partition – but I haven’t managed to quantify that (and may well be imagining it).

Bare in mind also that because the drive is mounted under your local file system, your drive is no longer strictly portable, in as much as there is no file system reference data on the USB drive itself, so if you plug the drive into a different computer, that computer wont be able to see whats on the external disk (and will report it as empty).

All users in this particular deployment were subject to the same client policy, same client version, same voice routing.. generally, everything was the same from one user to the next.

After running some S4/SIPStack trace logs on the gateway, and analysing the SIP Options packet that was being sent to the SIP gateway, I could see that in a working request, the SIP Invite that got sent included a=inactive (which is normal), whereas in a failing request, the Invite sent a=sendonly instead. What I couldn’t figure out was why two clients with the same settings/policy/routes would send two different hold methods. What made this particularly odd was that the user that was having the issue could log on to a different computer, and placing a call on hold would work fine.

So, issue had to be client-side.

One of the things I looked at during the debug process was the Lync registry entries. After painstakingly comparing registry keys line by line, I found that only one of the clients had the MusicOnHold registry keys listed – and it was the one that wasn’t working. Looking at the client options (Tools > Options > Alerts) showed that both had the same options selected (in this case, ‘Enable Music on Hold’ was ticked and greyed out (managed by policy), and the hold music file was populated with the default wma file), yet on the client that was working fine, the two MusicOnHold registry keys (below) were completely missing.

[HKEY_CURRENT_USER\Software\Microsoft\Communicator]

“MusicOnHoldDisabled”=dword:00000000

“MusicOnHoldAudioFile” =”C:\Program Files (x86)\Microsoft Lync\Media\DefaultHold.wma”

After backing up the keys, I deleted the MusicOnHoldAudioFile key, and rebooted. Bingo. Problem solved.

Tried reinstating the key again, and sure enough, problem returned immediately.

I have come across one other customer having the same issue, and they apparently found the problem disappeared when they apply CU5, however I haven’t been able to verify that completely with them, and when I tried the CU5 update, the issue persisted.

Why exactly this happens is soon to be the subject of a support ticket with Microsoft. When I get an outcome from that, I’ll be sure to update this post.

However..

We struck an issue after completing the process whereby calling voicemail, or trying to dial the UM dial-in numbers failed. Checking the logs revealed an error along the following lines:

ms-diagnostics: 1036; reason=”Previous hop shared address space peer did not report diagnostic information”; source=”<fe-server>”; dialplan=”Hosted__exap.um.outlook.com__<multipleSMTPdomains>”; umserver=”exap.um.outlook.com”;responsecode=”503″; msexchDomain=”<primarySMTPdomain>”; msexchPeerServer=”exap.um.outlook.com”; msexchsource=”<edgeaccessfqdn>”; appName=”ExumRouting”

Followed by:

ms-diagnostics: 15030; reason=”Failed to route to Exchange Server”; source=”<fe-server>“;dialplan=”Hosted__exap.um.outlook.com__<multipleSMTPdomains>“; appName=”ExumRouting”

Turns out we (and by that I mean me) had made an error when running the command:

New-CsHostedVoicemailPolicy -Identity Office365UM -Destination exap.um.outlook.com -Description “Hosted voice mail policy for O365 users.” -Organization “domain.com”

In my desire to validate blog posts before blindly following them (crazy right!), I’d checked the UCGuys’ NewCsHostedVoicemailPolicy  syntax against the Technet cmdlet library for New-CsHostedVoicemailPolicy, which states for the Organisation field..

This parameter contains a comma-separated list of the Exchange tenants that contain Lync Server 2010 users. Each tenant must be specified as an FQDN of the tenant on the hosted Exchange Service.

Which I duly interpreted as meaning all SMTP domains associated with the Exchange Online tenant – of which we had three. Especially as the example syntax at the bottom of the article does exactly that.

Turns out, that aint gonna fly.

For Exchange Online UM, you must specify one domain only in the Organisation field. And that domain must be one that Exchange Online is authorative for. If you’ve done a cutover migration, that will mean you can probably use your primary SMTP domain, as by dint of the cutover, Exchange Online will be authorative for that domain. However if you’ve done a hybrid migration, chances are good that your on-premise Exchange platform is still authorative for your primary SMTP domain. So best option here is to use your <customer>.onmicrosoft.com domain, as Exchange Online will always be authorative for that one.

This is briefly outlined at the end of the Connect Lync Server 2010 to Exchange Online UM Checklist from Microsoft.

Microsoft have done you a huge favour here.

I personally think Lync is hands-down the most superbly well documented product ever to roll out the doors of Redmond. Sure there’s stuff that could be better documented, but the planning, implementation, support, and (yup) training material available is beyond compare.

There is a huge list of pre-canned resources available at http://lync.microsoft.com/Adoption-and-Training-Kit/rollout-and-adoption/Pages/Resources/Intro-to-Resources.aspx that includes the likes of reference sheets, training decks, and videos, but what I really want to flag for attention is the ‘How To Guide’. Quite simply – it’s brilliant.

The guide is a Silverlight and/or HTML web application that contains a huge range of common (and not so common) Lync user tasks, presented in a sensibly structured manner that any user should have no issue following. It even comes with a handy set of instructions that outlines how to easily add your own items to the list – useful if you perhaps have a custom app integrated with your Lync platform.

The package can be downloaded from http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=5735

Setup is a peice of cake, and should take less than five minutes. Here’s the basics for IIS7/7.5

• Create a folder on your webserver
• Extract contents of the zip file
• Create an IIS site and point it at the folder you created
• Define your default document as either rolodex.html (for Silverlight version) or jQueryRolodex.html (for the HTML version).
• Configure your host headers and DNS entries
• Done

The only difference if you’re doing this on older versions of IIS is you need to manually create a MIME type for .xap extensions, defined as application/x-silverlight-app

So here’s a shortlist of the more pertinant features that you don’t get with Lync Online.

• PSTN calling (incoming or outgoing)
• PBX integration
• Advanced call handling (hold, redirection, park)
• IP Phone support (USB only)
• Analog line support (eg. fax)
• Response groups (ie. Direct inbound call to a recipient group)
• Persistent group chat
• Skill search from SharePoint (either on-premise or online)
• Client-side recording
• Dial-in conferencing
• Interop with on-premise video conferencing systems (eg. Polycom suites)
• QoS
• Quality of Experience Reporting

This usually leads to a question like, “ok, so what do they have in common then?”

So for completeness, here’s some of the more popular things you can do with both versions of the product.

• PC-to-PC audio/video
• Address book search
• IMPresence
• Office application integration (click-to-chat)
• Federation with Lync Online, Lync On-Premise, and OCS On-Premise
• Application/Desktop/Whiteboard/Presentation sharing
• Online Meetings
• Guest attendees (via rich client and web client)
• Roundtable support
• Meeting lobby

Hope that’s helpful.  Might add SharePoint Online and Exchange Online comparisons too.

NOTE: This post has been superceded by a more comprehensive global federation list, available here.

The following is a list of companies and organisations in New Zealand (or at least with a presence in NZ) that are using Lync/OCS federation…

[table id=1 /]

Edit: Matt Landis at windowspbx.blogspot.com has started compiling a similar list for US-based organisations. Check it out.

I would like to make this as comprehensive a list as possible, so if you are aware of other organisations that are federating, please drop the details in a comment and I’ll add them to the list.

If you’d like to connect with me via federation, you’ll find my contact details here.

Disclaimer

This list is not intended to breach privacy or publish sensitive information. All of this information is publicly available via DNS lookups, however if you or your company object to your details being listed here, please contact me and I will be more than happy to remove it.

Rather handily, Microsoft made this really simple to do.

Open the Event Viewer on one of your Edge servers and filter for Event ID 14601. You should find one of these logged every hour. This will contain the following:

Report of discovered partners that the Access Edge Server is currently monitoring.
There are 1 discovered partners, identified by the common name of their certificate.Name: accessurl.domain.com; Domains: domain.com

You can use these details to populate the Domain and AccessEdge fields in the Federation Domains section of your Lync Control Panel.

Easy.

You apply SP1, reboot the server, and open Central Admin to see all the bright shiny bits in SP1.

But alas, Central Admin returns a 503 error. In services.msc you find most of the SharePoint services are not started – admin, timer, user code host, search. Oh no!

Fear not, it’s just a case of being too keen for the shiny. The last step of the upgrade is to run the SharePoint Config Wizard. This completes the upgrade process, starts your services, and brings you all that is shiny and new.

Enjoy

The OCS Admin Tools are not listed in the removal order, and unfortunately will prevent you removing the Core role. So, where it says this….

If you are removing an Edge Server, a Mediation Server, an Archiving Server, or a Monitoring Server, remove the Office Communications Server 2007 R2 components in the following sequence:

• Microsoft Office Communications Server 2007 R2, Edge Server
• Microsoft Office Communications Server 2007 R2, Mediation Server
• Microsoft Office Communications Server 2007 R2, Archiving Server
• Microsoft Office Communications Server 2007 R2, Monitoring Server
• Microsoft Office Communications Server 2007 R2, Core Components
• Microsoft Office Communications Server 2007 R2, Unified Communications Managed API 2.0 Core Redistribution package

..it should actually read…

If you are removing an Edge Server, a Mediation Server, an Archiving Server, or a Monitoring Server, remove the Office Communications Server 2007 R2 components in the following sequence:

• Microsoft Office Communications Server 2007 R2, Edge Server
• Microsoft Office Communications Server 2007 R2, Mediation Server
• Microsoft Office Communications Server 2007 R2, Archiving Server
• Microsoft Office Communications Server 2007 R2, Monitoring Server
• Microsoft Office Communications Server 2007 R2, Administrative Tools
• Microsoft Office Communications Server 2007 R2, Core Components
• Microsoft Office Communications Server 2007 R2, Unified Communications Managed API 2.0 Core Redistribution package

Just sayin.

As you see in the diagram, the topology is fairly straight foward. Two PSTN gateways on 10.10.10.1 and 10.10.20.1, listening on TCP:5066 and 5060 respectively – one talking (hopefully) to a Lync mediation server (10.10.10.4) on TCP:5066 and the other still talking to an old OCS 2007 R2 mediation server (10.10.20.4) on TCP:5060. For now, forget the 10.10.20.x devices.

Steps I followed to get to this point were essentially as outlined in this Technet library (and the equivalent downloadable word version) http://technet.microsoft.com/en-us/library/gg398092.aspx. In short, define a PSTN gateway in the Lync topology, set listening ports on PSTN and Mediation objects, define and commit a voice route, publish topology, done.

What isn’t well documented here is that if for any reason you change the listening ports for the mediation server after initial publish (which I did, mostly just to bring the port allocations in line with the new Lync default ports as the old OCS deployment was using weird ports), you have to restart the Lync Mediation service, otherwise it wont pick up your port changes, even if you re-run the deployment tool.

So at this point, outbound calls were working. No issue. Happy admin.

But incoming calls from the PSTN would fail with “VoIP Transport Failure” on the Dialogic.

What I had already done on the Dialogic was change the routing table properties, specifically the VoIP Host Group settings, to use the new gateway (as the old OCS R2 mediation server was on 10.10.10.3). This is done in the Dialogic UI under Configuration > Routing Table > VoIP Host Groups (radio button at the top). My config here is pretty straightfoward – a single host group, with a single host entry of the old mediation server. Removed the old host entry, added a new one for 10.10.10.4, and expected everything to just work. It didn’t.

When calling in, the line would never ring, you’d see the call appear in the Dialogic Call Log, but after about 15sec it would fail giving the caller a ‘number not connected’ tone, and the Call Log would display “VoIP: Transport Failure”.

Spun up two sets of logging – first I started the Trace Logger on the Dialogic, and second started a set of Lync logs on the mediation server (S4, SIPStack, MediationServer) – then tried another inbound call. Logging on the mediation server showed zero entries, so clearly nothing was getting past the Dialogic. Looking at the trace logs on the Dialogic, I found the following:

[RouteTable] Code outbound device: VOIP (user@host:port) @10.10.10.4:0

So my incoming call was being routed to the mediation server IP ok, but no port was being passed, so logically I would expect the mediation server to reject it at the network layer, as it didn’t match a listening port.

So, on a hunch, I went back to the VoIP Host Groups setting, and changed the host value from 10.10.10.4 to 10.10.10.4:5066. Instant success. Just for consistency, checked another trace log, and sure enough it looks like this:

[RouteTable] Code outbound device: VOIP (user@host:port) @10.10.10.4:5066

What made this unexpected was that the old host value didn’t include the port, just the IP, but for some reason it worked fine.

Usefully, at this point I’d only migrated one of our two Dialogics/Mediation Servers, so ran the same trace log on the un-modified Dialogic (10.10.20.1). Sure enough, the same thing happens – no port is passed in that RouteTable call. Yet it works. Keen to see more, I ran a “netstat -an 1 | findstr 10.10.20.1” on the OCS mediation server to start an endless netstat search for open connections to the Dialogic (refreshing every second). As soon as the call comes in, a connection opens between the two devices on 10.10.20.1:5060 and the call connects. So somewhere in between that routing call to 10.10.20.1:0 becomes a call to 10.10.20.1:5060. I confess, I don’t know how this manages to happen.

Hopefully this helps someone else avoid the frustration this has caused me. And if anyone has any ideas how OCS manages this port wizardry where Lync fails, I’d love to hear your thoughts.

Let me paint you a picture..

You have a SharePoint list that contains information about a customer entity (yes it should probably be in CRM, but lets assume you don’t have one) – fields like contact names/numbers, addresses, unique systems, notes, etc. Some of these fields are single lines of text, pull-down menus, yes/no radio buttons, multiple lines of text, you name it.

You have another list that relates to sales of products to customers. Unsurprisingly, you want to link a sale to a customer, and you want to leverage the power of lookup columns to make that a simple and seamless process.

Not an unrealistic scenario. Sure there are better ways of doing it with the likes of webservices into CRM or BCS connections into LOB databases, but they all involve additional systems, coding skills, and generally more effort. All things that aren’t always readily available.

By adding a lookup column type to the sales list you can allow a customer entity to be selected from your customer list. Where this gets handier is you can have the sales list pull other values from the customer list without adding extra columns. Awesome.

But… not all the columns from your customer list are available. Why not?

SharePoint can only perform a lookup of values from columns that contain a ‘text’ value, and then only if it contains a single line of normal text (ie. “Single line of text”, a “number”, or “date”). Any field that contains multiple lines of text, other lookups, or multi-select items won’t be available to you, as SharePoint will automatically hide any columns that it knows it can’t return.

This same restriction applies to using these column types in calculated columns, and there is a great post by Dessie Lunsford on getting around this limitation in terms of calculated columns which you’ll find here – http://www.endusersharepoint.com/2009/06/17/taming-the-elusive-%E2%80%9Ccalculated-column%E2%80%9D-referencing-multiple-lines-of-text-column/

The workaround involves creating your problem field as a “single line of text” column, then creating a second calculated column that references the first column name – eg. [=ColumnName]. You then delete the first column and recreate it with the exact same name but this time selecting your column type of choice.

While Dessie’s post deals specifically with referencing these columns via calculated fields, by dint of good fortune and SharePoint consistency, the same workaround fixes the lookup problem as well. Thanks Dessie!

This issue applies to all versions of SharePoint since 2007, including SharePoint Online (BPOS/Office365)

Out of the box, Lync wont.

Any outbound call needs a voice route to determine its routing path and permissions – without one it simply cant go anywhere. In short when the RGS tries to dial out it will default to your global voice policy which (unless you’ve changed it – and you shouldn’t) wont route.

Your first task is to therefore create a voice policy that includes the number (or number pattern) you want to call and define a gateway device.

• You can do this via the Lync Control Panel or Powershell.
• Make sure the voice policy is of type ”User” otherwise you wont be able to apply it to your RGS object
• Make sure you commit the new policy otherwise it wont be available for use (you’ll get a policy is not a user policy error).

Then you need to bind that policy to your RGS object. You definitely need Powershell for this bit.

Grant-CSVoicePolicy -identity “RGSWorkflowObject” -PolicyName VoicePolicyYouCreated

For identity, use the display name of your RGS Workflow object.

And you’re done. Your RGS can now dial out.

Last tip – make sure the number you’re trying to dial out to is entered fully normalised in the format +<countrycode><areacode><number>@<sipdomain>.

eg. +6491234567@sipdomain.com

http://support.microsoft.com/kb/2496325

Download Links

32b Package – http://go.microsoft.com/fwlink/?LinkId=208565 (16Mb)

64b Package – http://go.microsoft.com/fwlink/?LinkId=208566 (16Mb)

I recently came across this post from Tobias Zimmergren that does a superb job of showing you several quick and easy ways to make these little GUID’s work for you.

In short…

get-splogevent | ?{$_.Correlation -eq "<GUID>"} | select Area, Category, Level, EventID, Message | Format-List

…will return the detailed log chain relating to that ID, in a human readable format. Handy.

And this..

SELECT	
  [RowCreatedTime],
  [ProcessName],
  [Area],   		
  [Category],  
   EventID,  
  [Message]  
FROM [WSS_UsageApplication].[dbo].[ULSTraceLog]
WHERE CorrelationId='<GUID>'

…in a SQL query will return much the same detail. I particularly like the suggestion of inserting this in a data-query web-part in Central Admin web-part. Super handy.

Out of the box, SP2010 will force you to save PDFs from SharePoint, not open them. This is to prevent XSS which is pretty easy to do in PDFs. Good solid security principal that one – I like it.

For most users however, this comes as a jarring change to what they’re used to, so queue complaints from users, and an SP Admin looking for a quick fix. Google will quickly point you at hundreds of suggestions to change the Browser File Handling setting from Strict to Permissive (set per web-application, under General Settings).

This is a purely evil approach, as it immediately relaxes file handling security for ALL file types, not just PDF.

The better way of doing this, is setting an ‘Inline Download’ exclusion just for PDF files. There’s a good post at pdfsharepoint.com by Dmitry that covers this in detail, but here’s the important bit..

Via PowerShell, run the following script to create a MIME type exclusion for PDF files in your web application. The only value you need to change here is the http://webapp.domain bit – set it to your web application hostname.

$webApp = Get-SPWebApplication http://webapp.domain
 If ($webApp.AllowedInlineDownloadedMimeTypes -notcontains "application/pdf")
 {
   Write-Host -ForegroundColor White "Adding Pdf MIME Type..."
   $webApp.AllowedInlineDownloadedMimeTypes.Add("application/pdf")
   $webApp.Update()
   Write-Host -ForegroundColor White "Added and saved."
 } Else {
   Write-Host -ForegroundColor White "Pdf MIME type is already added."
 }

“Vendor design must follow best practice”

or..

“Is this solution best practice?”

And it drive me nuts. Because if we were being really brutally honest, the answer is usually no. Best practice is not a boolean value. Its a moving target based on a combination of the things that are important to the business. Cost. Process. Performance. Manageability. Skillsets. I could go on. I tend to summarise them as ‘the real world’.

For the sake of argument (and there should be plenty of it), let’s define best practise as the set of configuration, management, and operational ‘rules’ that the vendor suggests should always be followed for a given piece of technology. With me so far?

More often than not, any given solution or platform has dependencies on dozens of other products, applications, systems and business processes. Every single one of these has its own set of ‘best practice’ parameters. In many cases, they will share similar or identical parameters, but in potentially just as many cases, they wont.

And they’ll conflict with the real world that they live in.

Lets take a real basic SharePoint example…

SharePoint runs on top of IIS.

IIS best practice is to have a separate application pool and identity for every web application on the server to ensure dedicated memory space for each application and keep a layer of isolation between sites. Simple stuff.

SharePoint 2010 involves a number of service applications that support a given web application (the website you actually set the thing up to host in the first place)

Even in a reasonably small deployment, there may well be dozens of these, each of which is its own web application, and each has an application pool.

If you were to put each of these in its own pool, you’d need a massive amount of memory for even the most simple webserver hosting a single published website.

While this isn’t impossible – you can put tonnes of memory in servers these days – in the real world its just not realistic.

So you end up grouping similar applications into common pools in a sensible manner.

So is it best practice? No. But is it what the client actually wants, can afford, and will therefore sign-off? Yes.

Now in this particular example, the core issue is the cost of memory – which while not super expensive, does start adding up when you have multiple webservers, and look to replicate this in test, dev, and DR environments. Tomorrow memory will be cheaper. It will be even cheaper the day after tomorrow. So the precise point at which you hit the ‘too much’ point will change over time, and your proximity to the mythical ‘best practice’ point will change too. The actual best practice hasn’t changed – just how acheivable it is relative to your budget.

What I would really love to start hearing from people is things like this..

“Vendor design must incorporate best practices where they align with business drivers”

or..

“How aligned are our business goals with best practice requirements?”

The point I’m trying to make is that while its great to use the general concept of best practice as a goal, it doesn’t work to simply expect everything to comply at all levels. Simply saying ‘it should follow best practice’ isn’t a silver bullet - and chances are you can’t afford best practice. Instead, know that its something you can aim for, but don’t get uppity if you can’t justify the expense or effort of achieving it.

If you’re in a situation where you need to remove a server from a SharePoint farm, then re-join it to the farm again (perhaps to resolve some sort of local corruption of the site config), make sure you manually check that all remnants of SharePoint farm membership are removed before you try adding it again. Normally SharePoint keeps a fairly tidy house and cleans up after itself well – but occassionally it leaves something behind, and this can wreak havoc on your environment if not picked up. Depending on which server role you’re dealing with in the farm, this could mean websites and/or application pools in IIS, databases, web application folders on the filesystem, or a bunch of other things.

This tripped up a colleague of mine recently – in this particular case the Security Token Service was for some reason not removed when the server was removed from the farm. When the server was re-joined to the farm, there were no errors, or issues that suggested there was a problem, but things started falling apart shortly afterwards.

How we located the issue was that the search functionality within a SharePoint site on the farm started returning those lovely generic SharePoint errors, and digging through the logs we found that old chestnut “Object reference not set to an instance of an object”.

Removing the server from the farm again, manually removing STS from IIS (including its application pool), then re-adding the server to the farm solved the problem immediately. Presumably there was some GUID under the hood that binds the STS to the farm, and as a result of the remove/join this GUID got out of sync with reality.

With nothing in the SQL logs to suggest a SQL error, and nothing additional in the client trace logs, I poked the haystack a couple of times to see if any needles fell out, but alas nothing did.

Came across a single post that suggested a solution – that being the following hotfix -http://support.microsoft.com/kb/949935/en-us. However while the error was identical, the symptom wasn’t. All OCS services were starting perfectly fine – to the point there were no errors logged on the OCS box at all when the services started. That said, tried the hotfix – didn’t help (no surprises there).

At this point a closer look was taken at the OCS databases for anything amiss. The only thing out of the ordinary was that within the rtcdyn database, both the RTCHSUniversalAdmins and RTCUniversalService logins were listed within the security section as being disabled (down-arrow icon on their user icon). A little more digging found that these accounts were not disabled at the server level within SSMS, and infact for the two other databases in which they have permissions, they were showing as enabled there as normal.

Immediate thought was to simply overwrite the permissions to resolve the issue, however trying to make any changes to these users against the rtcdyn database resulted in a SQL error along the lines of “User does not exist or you do not have sufficient permissions”. Given I was a sysadmin on the box, the latter seemed unlikely, so clearly there were some issues with the accounts.

In the end, the resolution here was to delete the logins from within the rtcdyn database (not from SQL entirely), then re-add the users to the rtcdyn database – making sure all database permissions were replicated on re-addd.

Note | Before being able to delete the users from the database, you need to change the ownership of a couple of schema objects (SQL will quickly tell you which ones by way of a handy error message). Make sure you change them back afterwards!

Quick OCS service restart once done and all systems back to normal.

My particular server is a VPS hosted by the awesome folks at KickAssVPS (they seriously do btw), and I recently decided it was overdue for an upgrade from Server 2003 to Server 2008. No worries – just a case of migrating some websites from IIS6 to IIS7 (sadly not IIS7.5 – R2 not available at KickAss yet). What I hadn’t planned on was finding out my mailserver software was 3 major versions old – SmarterMail 5.5 to be precise – and version 8 had just dropped that week. Too good to pass that up. Shiny!! Must have!

So, now I’m migrating from Server 2003 to Server 2008, IIS6 to IIS7, .NET2 to .NET4, and SmarterMail itself from 5.5 to 8.0. I’ll admit I started to wonder if I’d just made a foolish mistake.

Migration attempts 1 and 2 were a nightmare. Mail service refused to start, webmail couldn’t connect to services, and general hair-pulling ensued. Google unfortunately didn’t have the answers – seemed nobody had tried this particular array of upgrades in one go before. I was in no-mans land. It’s lonely there.

But eventually, I figured it out. If you’re curious, here’s how…

• Make sure there is beer in the fridge.
• Install SmarterMail 8.0 on new server
• Follow the steps here – http://portal.smartertools.com/KB/a1485/set-up-smartermail-as-a-site-in-iis-70.aspx – to convert the webmail interface to IIS7.
• Stop the SmarterMail service on both servers

• Stop the SmarterMail service on both servers
• Copy the following XML files from the SmarterMail program folder on the old server to the same place on the new server: (assumes x64 OS)

• Copy the contents of the entire mailbox folder located (by default) at C:SmarterMailDomains to the same place on the new server.
• Recursively delete all copies of mailbox.cfg from the C:SmarterMailDomains folder (“del /S mailbox.cfg” will do the trick). Restart the SmarterMail service on the new server.
• This resolves a problem whereby large numbers of emails show in webmail as ‘email no longer exists on server’. The mailbox.cfg file gets rebuilt automatically when you login to webmail, so don’t panic about deleting it.
• Restart the SmarterMail service on the new server.
• Retrieve beer from fridge
• Open beer and enjoy.

Bare in mind this doesn’t take into account your need to plan your MX record cutover process. If you’re managing a mail server, I’d expect you to understand how that bit works without having to explain it.

SmarterMail 8 is a massive improvement by the way. Faster, much sexier to look at, and just generally betterer.