Apr 152009
 

I’ve added some functionality to the PGP Pipeline component to enable it to Sign and Encrypt files.

image

Properties Explained:

ASCIIArmorFlag – Writes out file in ASCII or Binary

Extension – Final File’s extension

Operation – Decrypt, pilule Encrypt, and now Sign and Encrypt

Passphrase – Private Key’s password for decrypting and signing

PrivateKeyFile – Absolute path to private key file

PublicKeyFile – Absolute path to public key file.

TempDirectory – Temporary directory used for file processing.

Email me if you could use this.

Apr 152009
 

I decided to create a new BizTalk FTP/FTPS/SFTP adapter that would be robust and allow me to connect to both FTP,  FTPS (FTP over SSL or FTP-SSL), and SFTP (SSH File Transfer Protocol) servers. It also doesn’t have the limitation on server OSs that the MS FTP adapter has.

Here are the features:

FTP Adapter

FTPS Adapter

SFTP Adapter

PGP Pipeline component

Send and receive from FTP, FTPS, and SFTP servers. Send dynamically from orchestrations.  Ability to not delete file after retrieval from SFTP or FTP(S) server.  Proxy support. Use temporary remote directory to ensure full files are written to final folder.  Ability to check best authentication method to FTP(S) servers.

Here are the Receive Location Properties.

image

Explanation of properties:

FTP Type – FTP, FTPS (FTP over SSL or FTP-SSL), or SFTP (SSH File Transfer Protocol).

CRLF Mode – The CRLF Mode  property applies when downloading files in ASCII mode. If CRLF Mode is set to No Alteration the transfer happens normally without alteration. A value of CRLF converts all line endings to CR+ LF. A value of LF Only converts all line endings to LF-only. A value of CR Only converts all line endings to CR-only.

After Download – You may delete, move, or rename the file after downloading.

After Download File Name – The file name to rename the file if not deleting.  %SourceFileName% and %SourceFileNameNoExtension% are supported.

After Download Move To Directory – The remote directory to move the file to after downloading.  Uses same absolute/relative paths as Remote Directory.  %SourceFileName% and %SourceFileNameNoExtension% are supported.

FTP Trace Mode – Send a trace of the FTP session and any errors to either a File, designated by the FTP Trace path and FileName, Event Log, Both, or None.

FTP Mode – Active, Passive, or EPSV.

Temporary Remote Directory – Use a temporary remote directory to transfer files then move to the Remote Directory.  This ensures that no partial files will be picked up by another process on your Remote Directory.  The Temporary Remote Directory will automatically have trailing forward slashes “/” added to it if you don’t add them.  For SFTP –> Preceding slash (/)  = Absolute Path, Current directory = “./”, No preceding slash (/) = Relative Path from current directory.

Remote Directory – The remote directory of the FTP(S)/SFTP server.  The Remote Directory will automatically have trailing forward slashes “/” added to it if you don’t add them.  For SFTP –> Preceding slash “/”  = Absolute Path, Current directory = “./”, No preceding slash  = Relative Path from current directory.

Transfer Mode – Binary or ASCII

Use Passive Host Address – Some FTP servers need this option for passive data transfers. In passive mode, the data connection is initiated by the client sending a PASV command to the FTP server, and the FTP server responds with the IP address and port number where it is listening for the client’s connection request. When the Use Passive Host Address property is set to Yes, the IP address in the PASV response is discarded and the IP address of the remote endpoint of the existing control connection is used instead.

Authentication Mode – By setting the Authentication Mode Property to AuthTls , a secure FTP connection can be established using either SSL 3.0 or TLS 1.0. The FTP_FTPS Adapter will automatically choose whichever is supported by the FTP server during the secure channel establishment. The FTP control port remains at the default (21). Upon connection, the channel is converted to a secure channel automatically. All control messages and data transfers are encrypted. By choosing Implicit SSL, the FTP_FTPS Adapter connects using SSL on port 990, which is the de-facto standard FTP SSL port.

Client Certificate – The FTP_FTPS_SFTP Adapter provides the ability to use a client certificate with secure FTP (implicit or explicit SSL/TLS).

Clear Control Channel – Reverts the FTP control channel from SSL/TLS to an unencrypted channel. This may be required when using FTPS with AUTH TLS where the FTP client is behind a DSL or cable-modem router that performs NAT (network address translation). If the control channel is encrypted, the router is unable to translate the IP address sent in the PORT command for data transfers. By clearing the control channel, the data transfers will remain encrypted, but the FTP commands are passed unencrypted.

Private Key File – The FTP_FTPS_SFTP Adapter provides the ability to use a client certificate with secure FTP (implicit or explicit SSL/TLS). You may load a certificate from separate .crt (or .cer) and .pvk files and use it as the client-side SSL cert. The .pvk contains the private key. The .crt/.cer file contains the PEM or DER encoded digital certificate. Note: Client-side certificates are only needed in situations where the server demands one.

Invoice VAN FTP/SSL – By choosing yes, the FTP_FTPS_SFTP Adapter sets all the properties correctly to connect to an Inovis VAN FTP/SSL.

Tumbleweed Certificate Common Name– The FTP_FTPS_SFTP Adapter  can connect, authenticate, transfer files to a Tumbleweed Secure Transport SSL FTP Server. Instead of providing a login name and password, you pass the string “site-auth” for the username, and an empty string for the password. You must also provide a client-side digital certificate — as the certificate’s credentials and validity are used to authenticate.

MODE Z – The FTP/FTPS/SFTP Adapter automatically detects if the FTP server supports MODE Z.  It allows for files to be uploaded and downloaded using compressed streams.

SOCKS Version – Both SSL/TLS and non-secure FTP communications may use SOCKS4 and SOCKS5 proxies. Choose which version of SOCKS and provide the SOCKS Proxy Host Name, SOCKS Proxy Password, SOCKS Proxy Port, and SOCKS Proxy User Name.

Active Port End Range – When Active FTP Mode is used, the client-side is responsible for choosing a random port for each data connection. (Note: In the FTP protocol, each data transfer occurs on a separate TCP/IP connection. Commands are sent over the control channel (port 21 for non-SSL, port 990 for SSL).)

Active Port Start Range – This property, along with Active Port End Range, allows the client to specify a range of ports for data connections when in Active mode.

Proxy Mode – The proxy scheme used by your FTP proxy server. Valid values are 0 to 8. Supported proxy methods are as follows:

Note: The Proxy Host Name is the hostname of the firewall, if the proxy is a firewall. Also, the Proxy User Name and Proxy Password are the firewall username/password (if the proxy is a firewall).

ProxyMethod = 1 (SITE site)

USER ProxyUsername
PASS ProxyPassword
SITE Hostname
USER Username
PASS Password

ProxyMethod = 2 (USER user@site)

USER Username@Hostname:Port
PASS Password

ProxyMethod = 3 (USER with login)

USER ProxyUsername
PASS ProxyPassword
USER Username@Hostname:Port
PASS Password

ProxyMethod = 4 (USER/PASS/ACCT)

USER Username@Hostname:Port ProxyUsername
PASS Password
ACCT ProxyPassword

ProxyMethod = 5 (OPEN site)

USER ProxyUsername
PASS ProxyPassword
OPEN Hostname
USER Username
PASS Password

ProxyMethod = 6 (firewallId@site)

USER ProxyUsername@Hostname
USER Username
PASS Password

ProxyMethod = 7

USER ProxyUsername
USER ProxyPassword
SITE Hostname:Port USER Username
PASS Password

ProxyMethod = 8

USER Username@ProxyUsername@Hostname
PASS Password@ProxyPassword

Detect FTP Authentication – Determines what combinations of FTP/FTPS/SFTP property settings result in successful data transfers. The FTP_FTPS_SFTP Adapter tries 13 different combinations of these properties: Ssl, AuthTls, AuthSsl, Port, Passive, and Use Passive Host Address. Within the FTP protocol, the process of fetching a directory listing is also considered a “data transfer”. The FTP_FTPS Adapter method works by checking to see which combinations result in a successful directory listing download. The FTP_FTPS_SFTP Adapter requires the Host Name, Username, Password, and Port and returns a string containing an XML report of the results. It is a blocking call that may take approximately a minute to run.  It is executed via the following screen that opens when the ellipses is pressed.

image 

SSH Private Key Path and File Name – Authenticates with the SSH server using public-key authentication. The corresponding public key must have been installed on the SSH server for the Username. Authentication will succeed if the matching SSH Private Key Path and File Name is provided.  Load a private key from a PEM file. Private keys may be loaded from OpenSSH or Putty formats. Both encrypted and unencrypted private key file formats are supported.

SSH Private Key Passphrase – Passphrase for an encrypted SSH private key file.

SSH Create Disposition – SSH Create Disposition is a way to provide more control over how the file is opened or created.  The following are the list and meanings of the keywords. createNew = A new file is created; if the file already exists the method fails. createTruncate = A new file is created; if the file already exists, it is opened and truncated. openExisting = An existing file is opened. If the file does not exist the method fails. openOrCreate = If the file exists, it is opened. If the file does not exist, it is created. truncateExisting = An existing file is opened and truncated. If the file does not exist the method fails.

SSH Create Disposition Additional – SSH Create Disposition Additional allows additional control over how the file is opened or created by using a comma delimited set of keywords listed below.  appendData = Data is always written at the end of the file. Data is not required to be appended atomically. This means that if multiple writers attempt to append data simultaneously, data from the first may be lost. appendDataAtomic = Data is always written at the end of the file. Data MUST be written atomically so that there is no chance that multiple appenders can collide and result in data being lost. textMode = Indicates that the server should treat the file as text and convert it to the canonical newline convention in use. When a file is opened with this flag, data is always appended to the end of the file. Servers MUST process multiple, parallel reads and writes correctly in this mode. blockRead = The server MUST guarantee that no other handle has been opened with read access, and that no other handle will be opened with read access until the client closes the handle. (This MUST apply both to other clients and to other processes on the server.) In a nutshell, this opens the file in non-sharing mode. blockWrite = The server MUST guarantee that no other handle has been opened with write access, and that no other handle will be opened with write access until the client closes the handle. (This MUST apply both to other clients and to other processes on the server.) In a nutshell, this opens the file in non-sharing mode. blockDelete = The server MUST guarantee that the file itself is not deleted in any other way until the client closes the handle. No other client or process is allowed to open the file with delete access. blockAdvisory = If set, the above “block” modes are advisory. In advisory mode, only other accesses that specify a “block” mode need be considered when determining whether the “block” can be granted, and the server need not prevent I/O operations that violate the block mode. The server MAY perform mandatory locking even if the blockAdvisory flag is set. noFollow = If the final component of the path is a symlink, then the open MUST fail. deleteOnClose = The file should be deleted when the last handle to it is closed. (The last handle may not be an sftp-handle.) This MAY be emulated by a server if the OS doesn’t support it by deleting the file when this handle is closed. accessAuditAlarmInfo = The client wishes the server to enable any privileges or extra capabilities that the user may have in to allow the reading and writing of AUDIT or ALARM access control entries. accessBackup = The client wishes the server to enable any privileges or extra capabilities that the user may have in order to bypass normal access checks for the purpose of backing up or restoring files. backupStream = This flag indicates that the client wishes to read or write a backup stream. A backup stream is a system dependent structured data stream that encodes all the information that must be preserved in order to restore the file from backup medium. The only well defined use for backup stream data read in this fashion is to write it to the same server to a file also opened using the backupStream flag. However, if the server has a well defined backup stream format, there may be other uses for this data outside the scope of this protocol.

The FTP/FTPS/SFTP Adapter supports dynamic sends.

Here is example code needed for a dynamic send:

msg_FF = msg_FFTemp;
port_Dynamic(Microsoft.XLANGs.BaseTypes.Address) = @”FTP_FTPS_SFTP://TEST@localhost:21///”;
port_Dynamic(Microsoft.XLANGs.BaseTypes.TransportType) = “FTP_FTPS_SFTP”;
msg_FF(FTP_FTPS_SFTP_DynamicTransport.Username) = “TEST”;
msg_FF(FTP_FTPS_SFTP_DynamicTransport.Hostname) = “localhost”;
msg_FF(FTP_FTPS_SFTP_DynamicTransport.Port) = 990;
msg_FF(FTP_FTPS_SFTP_DynamicTransport.RemoteDirectory) = “/New Directory”;
msg_FF(FTP_FTPS_SFTP_DynamicTransport.AuthenticationMode) = “Implicit SSL”;
msg_FF(FTP_FTPS_SFTP_DynamicTransport.Password) = “test”;
msg_FF(FTP_FTPS_SFTP_DynamicTransport.RemoteFileName) = “%MessageID%”;
msg_FF(FTP_FTPS_SFTP_DynamicTransport.FTPType) = “FTPS”;
msg_FF(FTP_FTPS_SFTP_DynamicTransport.FTPMode) = “Passive”;
msg_FF(FTP_FTPS_SFTP_DynamicTransport.LogFTPSession) = “YES”;
msg_FF(FTP_FTPS_SFTP_DynamicTransport.FTPTraceMode) = “Event Log”;

Please email me if you need another setti
ng for your particular FTP(S) or SFTP server.

You can buy the FTP/FTPS/SFTP adapter by emailing me here..  It is pretty reasonably priced given the many hours I spent developing and testing it.  The price includes email and phone support and free upgrades for the life of the product.

You can download the 32 bit trial version by clicking here.

You can download the 64 bit trial version by clicking here.

The trial version will write out partial files after 45 days.

I tested FTP, FTPS connections using FileZilla Server.  I tested SFTP connections using COPSSH and Sysax MultiServer.

PGP Pipeline Component Explained:

image

ASCIIArmorFlag – Writes out file in ASCII or Binary
Extension – Final File’s extension
Operation – Decrypt, Encrypt, and Sign and Encrypt
Passphrase – Private Key’s password for decrypting and signing
PrivateKeyFile – Absolute path to private key file
PublicKeyFile – Absolute path to public key file.
TempDirectory – Temporary directory used for file processing.

Copy the PGP Pipeline Component and Bouncy Castle .dlls in the .zip to your Program Files/BizTalk Server 20XX/Pipeline Components/ folder.

Mar 252009
 

I’ve many times needed to find a value within an entire database.

Here’s a stored procedure that accomplishes that.

CREATE PROCEDURE [dbo].[FindMatchingValueInAnyTable]
(
    @Value VARCHAR(64)
)
AS
BEGIN
    DECLARE @sql VARCHAR(MAX)
    DECLARE @TableName VARCHAR(64)
    DECLARE @ColumnName VARCHAR(64)
    CREATE TABLE #Results
    (
        TableName VARCHAR(64)
    ,    ColumnName VARCHAR(64)
    )
    DECLARE TABLES CURSOR
    FOR
    SELECT
        o.name
    ,    c.name
    FROM syscolumns c
    INNER JOIN sysobjects o
    ON c.id = o.id
    WHERE o.type = ‘U’
    AND c.xtype IN (167, 175, 231, 239)
    ORDER BY
        o.name
    ,    c.name
    OPEN TABLES
    FETCH NEXT FROM TABLES
    INTO @TableName
    ,    @ColumnName
    WHILE @@FETCH_STATUS = 0
    BEGIN
        SET @sql = ‘IF EXISTS(SELECT NULL FROM [‘ + @TableName + ‘] ‘
        SET @sql = @sql + ‘WHERE RTRIM(LTRIM([‘ + @ColumnName + ‘])) LIKE ”%’ + @value + ‘%”) ‘
        SET @sql = @sql + ‘INSERT INTO #Results ( TableName, ColumnName ) VALUES (”’ + @TableName + ”’, ”’
        SET @sql = @sql + @ColumnName + ”’)’
        EXEC(@sql)
    FETCH NEXT FROM TABLES
    INTO @TableName
    ,    @ColumnName
    END
    CLOSE TABLES
    DEALLOCATE TABLES
    SELECT *
    FROM #Results
    DROP TABLE #Results
END

Mar 172009
 
^ The pattern has to appear at the beginning of a string. ^cat matches any string that begins with cat
$ The pattern has to appear at the end of a string. cat$ matches any string that ends with cat
. Matches any character. cat. matches catT and cat2 but not catty
[] Bracket expression. Matches one of any characters enclosed. gr[ae]y matches gray or grey
[^] Negates a bracket expression. Matches one of any characters EXCEPT those enclosed. 1[^02] matches 13 but not 10 or 12
[-] Range. Matches any characters within the range. [1-9] matches any single digit EXCEPT 0
? Preceeding item must match one or zero times. colou?r matches color or colour but not colouur
+ Preceeding item must match one or more times. be+ matches be or bee but not b
* Preceeding item must match zero or more times. be* matches b or be or beeeeeeeeee
() Parentheses. Creates a substring or item that metacharacters can be applied to a(bee)?t matches at or abeet but not abet
{n} Bound. Specifies exact number of times for the preceeding item to match. [0-9]{3} matches any three digits
{n, clinic } Bound. Specifies minimum number of times for the preceeding item to match. [0-9]{3,} matches any three or more digits
{n,m} Bound. Specifies minimum and maximum number of times for the preceeding item to match. [0-9]{3,5} matches any three, four, or five digits
| Alternation. One of the alternatives has to match. July (first|1st|1) will match July 1st but not July 2

POSIX Character Classes

[:alnum:] alphanumeric character [[:alnum:]]{3} matches any three letters or numbers, like 7Ds
[:alpha:] alphabetic character, any case [[:alpha:]]{5} matches five alphabetic characters, any case, like aBcDe
[:blank:] space and tab [[:blank:]]{3,5} matches any three, four, or five spaces and tabs
[:digit:] digits [[:digit:]]{3,5} matches any three, four, or five digits, like 3, 05, 489
[:lower:] lowercase alphabetics [[:lower:]] matches a but not A
[:punct:] punctuation characters [[:punct:]] matches ! or . or , but not a or 3
[:space:] all whitespace characters, including newline and carriage return [[:space:]] matches any space, tab, newline, or carriage return
[:upper:] uppercase alphabetics [[:upper:]] matches A but not a

Perl-Style Metacharacters

// Default delimiters for pattern /colou?r/ matches color or colour
i Append to pattern to specify a case insensitive match /colou?r/i matches COLOR or Colour
 A word boundary, the spot between word (w) and non-word (W) characters /fred/i matches Fred but not Alfred or Frederick
B A non-word boundary /fredB/i matches Frederick but not Fred
d A single digit character /adb/i matches a2b but not acb
D A single non-digit character /aDb/i matches aCb but not a2b

The newline character. (ASCII 10) /
/
matches a newline

The carriage return character. (ASCII 13) /
/
matches a carriage return
s A single whitespace character /asb/ matches a b but not ab
S A single non-whitespace character /aSb/ matches a2b but not a b
The tab character. (ASCII 9) / / matches a tab.
w A single word character – alphanumeric and underscore /w/ matches 1 or _ but not ?
W A single non-word character /aWb/i matches a!b but not a2b
Mar 122009
 

I wanted a way to not show the MS-DOS prompt while I was working and the Event Log Emailer was running.  So I found two methods to have the Event Log Emailer or any MS-DOS prompt not show up while a batch is executing.

Shortcut Method:  To have an MS-DOS Batch file run in a minimized window Create a shortcut to the Batch file. Right-click the shortcut and click properties. In the properties dialogue, symptoms choose run “Minimized” in the pull-down box. Run the Batch file from the shortcut.

image

VB Script Method:  To have an MS-DOS Batch file run in a hidden window with no MS-DOS prompt nor any corresponding Taskbar button, pill begin your batch with @ECHO OFF and end with CLS and/or EXIT, for sale or the process may stall invisibly.

If my batch was called C:BizTalkEvent Log EmailerTEST.bat , create the following .VBS file:

CreateObject(“Wscript.Shell”).Run “””C:\BizTalk\Event Log Emailer\TEST.bat”””,0

Notice the extra pair of quotes around the path/filename and escape “\” to ensure the vb script can find the file even with the spaces in the path/filename.

This method requires Windows Script Host to be installed. Double-click or execute the .VBS file. For example, with the START command: start MyFile.VBS to run the Batch file invisibly.

Mar 102009
 

The easiest way I’ve found to do HL numbering is to use a global variable inside of a map and increment it for every HL segment you need to produce.

Here’s an example:

image

This is the 2nd HL segment, so here I’m needing an incrementing HL01 and the previous HL number in the HL02.

The inline C# script for the HL01 would  be:

int HL01;
public int increaseHL01()
{
HL01++;
return HL01;
}

The inline C# script for the HL02 would be:

public int HL02()
{
return HL01 -1;
}

Mar 072009
 

After thinking about using a Windows Service instead of a console application to send emails on specific Event Log entries I wanted to monitor, I decided to stick with the console application just because it’s so easy to implement.  It requires no installation, no background process is running, and you can execute it from many batches for each type of message you’re wanting to email.  I’ve considered doing the alerts via MSN Messenger, or RSS feed, but feel that email is still the most requested way that people want to receive their alerts.

I did add the capability of looking back as little as one minute using the -NUMBEROFMINUTES: argument.  I also added the -EMAILSUBJECT: argument for custom email subjects.

Now you can use any combination of -NUMBEROFMINTUES, -NUMBEROFHOURS, and -NUMBEROFDAYS to look back the time you need to search the event log.

Here again is a complete list of the arguments:

-TOEMAIL: = ToEmail (Required)

-FROMEMAIL: = FromEmail (Required)

-EMAILSUBJECT: = Email Subject (Optional)

-SMTPSERVER: = SmtpServer (Required)

-NUMBEROFDAYS: = Number of Days to look back (Optional) if not set will use 0

-NUMBEROFHOURS: = Number of Hours to look back (Optional) if not set will use 0

-NUMBEROFMINUTES: = Number of Minutes to look back (Optional) if not set will use 0

/n = Send Email if there are no events (Optional)

/e = Send Errors (Optional)

/w = Send Warnings (Optional)

/i = Send Information (Optional)

-EVENTSOURCE: = Event Source (Optional)

-EVENTCATEGORY: = Event Category (Optional)

-EVENTMESSAGESEARCHSTRING: = Event Message Search String (Optional)

Here’s an example of the resultant email:

image

Please send any comments or suggestions on its usefulness.

To get a copy of this, follow the following link: Services

Feb 262009
 

I developed a custom functoid and found the weird behavior that wouldn’t allow me to link the output to another functoid.  I found the reason here:

http://msdn.microsoft.com/en-us/library/microsoft.biztalk.basefunctoids.functoidcategory.aspx

It states:

FunctoidCategory also determines the functionality restrictions and behaviors possible with the custom functoid. For example, a custom functoid that uses the Logical FunctoidCategory cannot output a string value to the map’s destination schema, but will instead determine whether the destination record is created based on the Boolean value as mentioned at: Logical Functoids.

So using FunctoidCategory.Unknown to display the functoid in the Advanced Functoid category in the VS toolbox modified its behavior to not allow its output to link to other functoids.