Why bother with BizTalk?

 Uncategorized  Comments Off on Why bother with BizTalk?
Jul 022014

So in discussions with a lot of executives about the situations of integration, illness often times the question is asked: Why use BizTalk, it seems like a lot of work to get integration done when I could just use tools that Microsoft provides already.

I have struggled with coming up with a good answer to this question, because, yes Microsoft provides other integration tools packaged with other server products that solve the same problems that BizTalk Server solves.

Lets take SQL Server Integration Services (SSIS). It transforms data from one data type to another. You don’t even need SQL Server to be the source or destination. From outward appearances it can do all of the things that BizTalk can.

SSIS is great, but SSIS is akin to a machete, whereas BizTalk is akin to a Swiss army knife

Machete vs.swiss-army-knife 


They both have their uses, if I need to cut down a swath of weeds, or to clear a trail of underbrush, a machete is what I would use. If I needed to whittle away a piece of wood, I would use the Swiss army knife. Could I accomplish the same thing with the other tool? YES! Clearing under brush with a Swiss army knife, possible, but not the best, carving a wooden sculpture with a machete, I guess it can be done.

Can a Swiss army knife deal with a screw? Yes! Can I cut paper, can I open up a can, can I file my fingernails? All yes! Is it the best tool for the job? Probably not, but it is far more comprehensive than a lot of other tools.

So also is SSIS compared to BizTalk. If I wanted to do a mass update, without a lot of moving part SSIS is great, if I need to bulk move data from one place to another, SSIS is the job. If I need to design a workflow process, where there are multiple stops (along with different types of end points), BizTalk is the way to go.

Are there better screw drivers than the one provided with the Swiss army knife, how about can openers, how about scissors, yes, yes, and yes.

WCF exposed C# interfaces are much faster, and operate at a much granular level. However, you lose some of the functionality that comes out of the box with BizTalk, namely tracking, exception handling, etc,

Food for thought.

Trigger file using the sFTP adapter

 Uncategorized  Comments Off on Trigger file using the sFTP adapter
Jul 012014

So I am on a project that takes existing BTS 2004 and converting it to BTS 2013.

They have a UNIX ftp server that has processes can’t lock files while being written. This causes a little challenge with the FTP adapter. The process writes the file, and then writes a trigger file.


Payload file: eric.samplefile1.txt

Trigger file: eric.samplefile1.txt.trg

What they did back in 2004, is create the standard ftp adapter that would look for the *.trg, and then the pipeline would re-connect and swap the payload of the (nearly) empty trigger file and replace the data with the actual payload of the file.

This is a ‘tricky’ way, but one I would never champion: it is getting the payload via .net code.

Now we are connecting via sFTP, and there is no publically available sFTP code to ‘backdoor’ connect the sFTP server. I needed to find a different way.

What I did was

  • create a trigger receive location, and the pipeline
  • which uses ExplorerOM to create a non primary receive location based on the trigger file name being picked up
  • The payload receive location knows it is not a trigger file based on the receive location name in combination of the filename extension
  • When the pipeline (which is the same one as the trigger file) gets the payload, it goes and deletes itself.

This allows for multiple files to be processed, all ports are always on, and it ‘self’ cleanses.

Here is what the receive location looks like:


When the trigger is picked up eric.samplefile1.txt.trg, the pipeline creates a new receive location with the same pipeline component


When the file payload file is picked up eric.samplefile1.txt, the pipeline runs and deletes itself, leaving the receive location looking like this…



Here is the pipeline code that accomplishes this…

using System; using System.Xml; using System.ComponentModel; using System.Collections; using Microsoft.BizTalk.Message.Interop; using Microsoft.BizTalk.Component.Interop; using Microsoft.Win32; using Microsoft.BizTalk.ExplorerOM; using System.Xml.Linq; using System.Web; using System.Diagnostics; using System.IO; using System.Threading; using System.Linq; using SSOUtility; namespace StottCreations.PipelineComponents.Trigger { [ComponentCategory(CategoryTypes.CATID_PipelineComponent)] [ComponentCategory(CategoryTypes.CATID_Decoder)] [System.Runtime.InteropServices.Guid("7F3DF154-3267-4154-ABC4-E163D0B79E39")] public class ReceiveLocationConfiguration : Microsoft.BizTalk.Component.Interop.IBaseComponent, Microsoft.BizTalk.Component.Interop.IComponent, Microsoft.BizTalk.Component.Interop.IPersistPropertyBag, Microsoft.BizTalk.Component.Interop.IComponentUI { #region Variables string mgmtDb, port, name, realFileName, transportDefinition, ftpServer, ftpFolderPath, originalFileName; XDocument doc = new XDocument(); #endregion #region PipelineProperties private string ssoAppName = null; public string SSOAppName { get { return ssoAppName; } set { ssoAppName = value; } } #endregion #region IBaseComponent [Browsable(false)] public string Name { get { return "Receive Location Configuration Component"; } } [Browsable(false)] public string Version { get { return "1.0"; } } [Browsable(false)] public string Description { get { return "Modifies an available receive location for the SFTP adapter based on the trigger file"; } } #endregion #region IComponent public IBaseMessage Execute(IPipelineContext pc, IBaseMessage inmsg) { getConnectionString(); getPortName(inmsg); getTransportDefinition(inmsg); getLocationName(inmsg); getServer(); getPath(); if (isTrigger(Convert.ToString(inmsg.Context.Read("ReceiveLocationName", "http://schemas.microsoft.com/BizTalk/2003/system-properties")))) { createLocation(); inmsg = null; } else { destroyLocation(); } return inmsg; } #endregion #region IPersistPropertyBag public void GetClassID(out Guid classid) { classid = new System.Guid("7F3DF154-3267-4154-ABC4-E163D0B79E39"); } public void InitNew() { } public void Load(Microsoft.BizTalk.Component.Interop.IPropertyBag pb, Int32 errlog) { string val = (string)ReadPropertyBag(pb, "SSOAppName"); if (val != null) ssoAppName = val; } public void Save(Microsoft.BizTalk.Component.Interop.IPropertyBag pb, Boolean fClearDirty, Boolean fSaveAllProperties) { object val = (object)ssoAppName; WritePropertyBag(pb, "SSOAppName", val); } private static object ReadPropertyBag(Microsoft.BizTalk.Component.Interop.IPropertyBag pb, string propName) { object val = null; try { pb.Read(propName, out val, 0); } catch (System.ArgumentException) { return val; } catch (Exception ex) { throw new ApplicationException(ex.Message); } return val; } private static void WritePropertyBag(Microsoft.BizTalk.Component.Interop.IPropertyBag pb, string propName, object val) { try { pb.Write(propName, ref val); } catch (Exception ex) { throw new ApplicationException(ex.Message); } } #endregion #region IComponentUI [Browsable(false)] public IntPtr Icon { get { return IntPtr.Zero; } } public IEnumerator Validate(object projectSystem) { if (projectSystem == null) throw new System.ArgumentNullException("No project system"); IEnumerator enumerator = null; ArrayList strList = new ArrayList(); try { } catch (Exception e) { strList.Add(e.Message); enumerator = strList.GetEnumerator(); } return enumerator; } #endregion #region Helper private bool isTrigger(string locationName) { bool returnValue = false; BtsCatalogExplorer root = new BtsCatalogExplorer(); try { root.ConnectionString = mgmtDb; ReceivePort receivePort = root.ReceivePorts[port]; returnValue = (receivePort.PrimaryReceiveLocation.Name == locationName) ? true : false; } catch (Exception e) { root.DiscardChanges(); throw e; } return returnValue; } private void createLocation() { string password = SSOClientHelper.Read(SSOAppName, "FTP_Password"); int iteration = 0; while (true) { BtsCatalogExplorer root = new BtsCatalogExplorer(); try { root.ConnectionString = mgmtDb; ReceivePort receivePort = root.ReceivePorts[port]; int nextPort = receivePort.ReceiveLocations.Count; receivePort.AddNewReceiveLocation(); XmlDocument transportData = new XmlDocument(); transportData.LoadXml(HttpUtility.HtmlDecode(receivePort.PrimaryReceiveLocation.TransportTypeData.Replace("<Password vt=\"1\" />", "<Password vt=\"8\"></Password>"))); transportData.SelectSingleNode("//FileMask").InnerText = realFileName; transportData.SelectSingleNode("//Password").InnerText = password; receivePort.ReceiveLocations[nextPort].Name = name; receivePort.ReceiveLocations[nextPort].Address = String.Format("sftp://{0}:22{1}/{2}", ftpServer, ftpFolderPath, realFileName); receivePort.ReceiveLocations[nextPort].TransportTypeData = transportData.OuterXml; receivePort.ReceiveLocations[nextPort].TransportType = receivePort.PrimaryReceiveLocation.TransportType; receivePort.ReceiveLocations[nextPort].ReceivePipeline = receivePort.PrimaryReceiveLocation.ReceivePipeline; receivePort.ReceiveLocations[nextPort].ReceivePipelineData = receivePort.PrimaryReceiveLocation.ReceivePipelineData; receivePort.ReceiveLocations[nextPort].ReceiveHandler = receivePort.PrimaryReceiveLocation.ReceiveHandler; receivePort.ReceiveLocations[nextPort].Enable = true; root.SaveChanges(); break; } catch (Exception e) { iteration++; root.DiscardChanges(); if (iteration < 30) { Random rd = new Random(); Thread.Sleep(rd.Next(100, 1000)); } else { throw e; } } } } private void destroyLocation() { BtsCatalogExplorer root = new BtsCatalogExplorer(); try { root.ConnectionString = mgmtDb; ReceivePort receivePort = root.ReceivePorts[port]; ReceiveLocation deleteableLocation = null; foreach (ReceiveLocation location in receivePort.ReceiveLocations) { if (Path.GetFileName(location.Address).EndsWith(originalFileName)) { deleteableLocation = location; break; } } receivePort.RemoveReceiveLocation(deleteableLocation); root.SaveChanges(); } catch (Exception e) { root.DiscardChanges(); throw e; } } private void getConnectionString() { string regEntry = @"HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\BizTalk Server\3.0\Administration"; string server = Registry.GetValue(regEntry, "MgmtDBServer", "BTSServer").ToString(); string database = Registry.GetValue(regEntry, "MgmtDBName", "BizTalkMgmtDb").ToString(); mgmtDb = String.Format("Server={0};Initial Catalog={1};Integrated Security=SSPI;", server, database); } private void getPortName(IBaseMessage inmsg) { port = (System.String)inmsg.Context.Read("ReceivePortName", "http://schemas.microsoft.com/BizTalk/2003/system-properties"); } private void getLocationName(IBaseMessage inmsg) { string initialLocation = (System.String)inmsg.Context.Read("ReceiveLocationName", "http://schemas.microsoft.com/BizTalk/2003/system-properties"); string triggerFileName = (System.String)inmsg.Context.Read("ReceivedFileName", "http://schemas.microsoft.com/BizTalk/2003/file-properties"); originalFileName = Path.GetFileName(triggerFileName); realFileName = Path.GetFileNameWithoutExtension(triggerFileName); name = String.Format("{0} - {1}", initialLocation, realFileName); } private void getTransportDefinition(IBaseMessage inmsg) { transportDefinition = (System.String)inmsg.Context.Read("InboundTransportLocation", "http://schemas.microsoft.com/BizTalk/2003/system-properties"); } private void getServer() { // sftp://mysftpserver:22/dev//eric.*.trg ftpServer = transportDefinition.Replace("sftp://", String.Empty); string[] splitTransport = ftpServer.Split(':'); ftpServer = splitTransport[0].ToString(); } private void getPath() { // sftp://mysftpserver:22/dev//eric.*.trg char[] splitting = ":22".ToCharArray(); string[] segments = transportDefinition.Split(splitting); string[] elements = segments[segments.Length - 1].Split('/'); string fileMask = elements[elements.Length - 1]; ftpFolderPath = segments[segments.Length - 1].Replace("/" + fileMask, string.Empty); } #endregion } }