3
votes

There is a pre-existing SSIS package which performs multiple file manipulations based on a source file and often fails when that file is not found at the expected directory. I just want to build some smarts into it so that it will email notification of the missing file, rather than fail.

I have attempted MANY different Script Tasks, using VB, C and SQL, but NOTHING is controlling the process flow consistently. It works sometimes, and when it doesn't work others. I have listed my variables are below --- the top three are all I expected to use, but I added FullPath to simplify things. Unfortunately, that made no change.

My tests: I remove the source files from the directory and execute the package in VS, it DOES call the send email task letting me know the file does not exist. Then I put the files into place and execute the package, it calls the send email task again, as though the file is not there. I am not very good with breakpoints and watch windows, so I put two message boxes into place for Filepath and FileExists -- the Filepath returned IS correct, with the filename, yet the FileExists message box value returned immediately thereafter returns a 0. Note, at the very same time this is telling me it doesn't see the file, I have checked the disk and can see it is there.

Here's the kicker: I've been on this for days and was testing yesterday -- suddenly it works! I put the files into the source directory, it ran the whole process correctly. I removed the files from the source directory, it called the send mail task and completed successfully. I tested both conditions multiple times successfully -- and now it is failing again. I do not understand and have no desire or time to keep testing this file existence check script task that only works intermittently. I even tried today to get the File Properties task that I am hearing so much about (https://archive.codeplex.com/?p=filepropertiestask) but it is incompatible with the current versions of the software. I've tried VS 2019 and SSDT 2017, File Properties is incompatible/unsupported in either. Or, I just don't know how to install it.

Can anyone advise?

Variables -

  • FileName string, fileName.txt
  • FilePath string, C:\directory path\
  • FileExists boolean, False (though I've tried int32, even char N/Y)
  • FullPath string, C:\Directory path\filename.txt

C Script Task attempts -

public void Main()
{
    // TODO: Add your code here
    String Filepath = Dts.Variables["User::FilePath"].Value.ToString() + Dts.Variables["User::FileName"].Value.ToString();
    if (
    File.Exists(Filepath))
    {
        Dts.Variables["User::FileExists"].Value = 1;
    }
    else
        Dts.Variables["User::FileExists"].Value = 0;

    Dts.TaskResult = (int)ScriptResults.Success;
}

OR

//TODO: Add your code here
String Filepath = Dts.Variables["User::FilePath"].Value.ToString() + Dts.Variables["User::FileName"].Value.ToString();
if (
FileExists(Filepath))
{
    Dts.Variables["User::FileExists"].Value = 1;
}

MessageBox.Show(Filepath);
MessageBox.Show(Dts.Variables{"User::FileExists"].Value.ToString());

Dts.TaskResult = (int)ScriptResults.Success;
}

or even just as simple as this:

Dts.Variables("FileExists").Value = File.Exists(Dts.Variables("FilePath").Value).ToString
Dts.TaskResult = (int)ScriptResults.Success;

VB Script Task -

Public Sub Main()
    ' Fill WriteVariable with value from ReadVariable 
    Dts.Variables("User::FileExists").Value = Dts.Variables("User::FullPath").Value
 
    Dts.TaskResult = ScriptResults.Success
End Sub

Exec SQL Task -

    DECLARE
      @FilesExist BIT = 0,
      @FolderPath VARCHAR(100) = 'C:\directory path\'

DECLARE @Files TABLE ([FileName] VARCHAR(100), Depth INT, [File] INT)
INSERT INTO @Files
EXEC master.sys.xp_dirtree @FolderPath,1,1;

IF EXISTS(
      SELECT 1 FROM @Files
        WHERE [FileName] = 'fileName.txt'
      AND Depth = 1
      AND [File] = 1
  )
            SET @FilesExist = 1
            RETURN;

Script Task Precedent constraints:

  • Evaluation Operation: Expression and Constraint
  • Value: Success
  • Expression: @[User::FileExists]==1
  • Logical AND

Evaluation Operation: Expression and Constraint

  • Value: Success
  • Expression: @[User::FileExists]==0
  • Logical AND

This is a dummied screenshot of my control flow. Where the script task file existence check is the 7th item in the flow. The filename has no date in it. It is always 'filename.txt'. This file is created by the 4th task in the flow by merging other files, and I have just learned that I need to add a similar check here -- but there are multiple files, so I will need to do a wildcard check before the 3rd task in the package as well. enter image description here

1
Pop a screenshot of your Control Flow into the question if you would. I'd like to verify how the tasks are laid outbillinkc
And then for resolution, do you have a preferred route you'd like to see this go? Code, existing componetry? At the core of it, you would like to check if a file exists. If not, notify. If so, continue on some other line of logic, yeah?billinkc
Terribly sorry, I did not see these questions until now. Give me a moment to genericize it, an I will post my control flow.sqldba

1 Answers

0
votes

While I like the elegance of Script Tasks, given the current set of tasks, I think you can get by with the out of the box tooling.

Foreach Loop Container

This is the work horse in the solution. It has a built in file enumerator so if your source file is actually SourceFile_YYYYMMDD.txt or something like that, you can just use sourcefile*.txt and it'll find it no problem.

The reason I like this, is that you can put all of your logic inside this container and if the file is found, it's just going to do the expected work. No precursor/successor stuff has to be defined to make it go.

I created another variable called CurrentFile and initialized it to the empty string. This, empty string starting value, is crucial for the success of the package. When the Foreach Loop finds a file, it is going to put the full file name into this variable.

When the package runs and all the file manipulation work is done, if a file was found, the current value of @[User::CurrentFile] is not going to be the empty string. If no file is found, then the value of CurrentFile remains the starting value.

I modified your existing FileExists boolean SSIS Variable to be Expression driven. In the properties, I used the following expression @[User::CurrentFile] != "" If the value of CurrentFile isn't our starting value, then it evaluates to true. Otherwise, it remains false.

Using those two "tricks", that leads us to an SSIS package like the following

enter image description here

I have two paths leading out of the Foreach loop container. Both specify success as the constraint and then we use @[User::FileExists] for the happy path (file found) and the inversion of that !@[User::FileExists] for the not exists path. Put your notification logic inside the container that says "No file found path"

Because I love me some Biml, I am attaching the Biml to create this solution.

Slightly less useful as you need to patch this into an existing package, but you should be able to create a minimum viable package that handles checking and alerting if file not found. And then you can compare a working example to your current implmentation.

<Biml xmlns="http://schemas.varigence.com/biml.xsd">
    <Packages>
        <Package Name="so_62505561">
            <Variables>
                <Variable Name="FileExists" DataType="Boolean" EvaluateAsExpression="true">@[User::CurrentFile] != ""</Variable>
                <Variable Name="FileName" DataType="String">so_62505561.txt</Variable>
                <Variable Name="FilePath" DataType="String">C:\ssisdata\input</Variable>
                <Variable Name="FileSpecification" DataType="String">so_62505561*.txt</Variable>
                <Variable Name="FullPath" DataType="String"></Variable>
                <Variable Name="CurrentFile" DataType="String"></Variable>
            </Variables>
            <Tasks>
                <ForEachFileLoop Name="FELC Do File Work" FileSpecification="*.txt" ConstraintMode="LinearOnSuccess" Folder="C:\tmp">
                    <Expressions>
                        <Expression ExternalProperty="FileSpec">@[User::FileSpecification]</Expression>
                        <Expression ExternalProperty="Directory">@[User::FilePath]</Expression>
                    </Expressions>
                    <VariableMappings>
                        <VariableMapping Name="0" VariableName="User.CurrentFile" />
                    </VariableMappings>
                    <Tasks>
                        <Container Name="Placeholder for work">
                        </Container>
                    </Tasks>
                </ForEachFileLoop>
                <!-- this is the unhappy path -->
                <Container Name="No file found path">
                    <PrecedenceConstraints>
                        <Inputs>
                            <Input EvaluationOperation="ExpressionAndConstraint" EvaluationValue="Success" Expression="!@[User::FileExists]" OutputPathName="FELC Do File Work.Output" />
                        </Inputs>
                    </PrecedenceConstraints>
                </Container>
                <Container Name="File found path">
                    <PrecedenceConstraints>
                        <Inputs>
                            <Input EvaluationOperation="ExpressionAndConstraint" EvaluationValue="Success" Expression="@[User::FileExists]" OutputPathName="FELC Do File Work.Output" />
                        </Inputs>
                    </PrecedenceConstraints>
                </Container>
            </Tasks>
        </Package>
    </Packages>
</Biml>

Using the script task to test for file existence

Given your assumptions

  • FileName string, fileName.txt
  • FilePath string, C:\directory path\
  • FileExists boolean, False

The Script Task should have FileName and FilePath as read only variables, FileExists is a read/write variable.

// I favor System.IO.Path.Combine for path manipulation as it figures out the correct separator to use
string filepath = System.IO.Path.Combine(Dts.Variables["User::FilePath"].Value.ToString(),  Dts.Variables["User::FileName"].Value.ToString());
if (System.IO.File.Exists(filepath))
{
    Dts.Variables["User::FileExists"].Value = true;
}

// I favor emitting to the log I can audit it. Also, GUI events are not allowed when run from jobs
// Log the state of all our SSIS variables
bool fireAgain = false;
string message = "{0}::{1} : {2}";
foreach (var item in Dts.Variables)
{
    Dts.Events.FireInformation(0, "SCR Echo Back", string.Format(message, item.Namespace, item.Name, item.Value), string.Empty, 0, ref fireAgain);
}

// Log the file path that was built
    Dts.Events.FireInformation(0, "SCR Echo Back", string.Format(message, "local", "filepath", filepath), string.Empty, 0, ref fireAgain);

Dts.TaskResult = (int)ScriptResults.Success;

At this point, when the script runs, you'll have 4 information events in your log: the state of our 3 SSIS scoped variables and the state of our built path. In the event the downstream bits that depend on @[User::FileExists] are not working as expected, you have all your values in one convenient place.

At this point, the file check script task has exited and the precedent constraints should be flagged as Expression and Constraint using the following combos

  • Success + @[User::FileExists] (File exists path)
  • Success + !@[User::FileExists] (Alert path)

Logging note

Firing information events results in the information being emitted in two different locations for the a package run in Visual Studio.

The Output Window and the Progress tab.

Output Window would show something like

SSIS package "C:\Users\bfellows\source\repos\Integration Services Project1\Integration Services Project1\NewPackage.dtsx" starting.
Information: 0x0 at Script Task 1, SCR Echo Back: User::FileExists : False
Information: 0x0 at Script Task 1, SCR Echo Back: User::FileName : fileName.txt
Information: 0x0 at Script Task 1, SCR Echo Back: User::FilePath : C:\directory path\
Information: 0x0 at Script Task 1, SCR Echo Back: local::filepath : C:\directory path\fileName.txt
SSIS package "C:\Users\bfellows\source\repos\Integration Services Project1\Integration Services Project1\NewPackage.dtsx" finished: Success.

whereas the progress tab is a gui element

Progress tab in VS. Shows the same events as shown above plus Progress events

Final resolution

@sqldba discovered that in the 4th step the VBA was using a local variable FilesExist and not FileExists and since VB is forgiving of non-declared variables, that's where things got "weird"