Monitor spawned process for shell errors in Expect script

Question

I am trying to figure out how to continuously monitor an automated SSH session I spawned in my Expect script for shell errors. Is there a way to have a background process that continuously searches the output of my SSH session for "-ksh:"?

In KSH, I would simply trap ERR signals, but ERR is not a valid signal type in Expect. The point is that I'd like to monitor for shell errors so I can go through my exit handler rather than the Expect script continuing execution.

Below is a snippet from one of my shell functions/expect scripts. The specialized Expect procedures have been excluded, as they are not pertinent to the question. This particular Expect script spawns and SSH session to a partner server and sources/runs several user defined shell functions. When the init_log function is sourced and executed, log files are created and STDOUT/STDERR are redirected to those logs - hence the need for the tee /dev/tty command so that I can see STDOUT from the functions I am running on the partner server. I am not sure if I should temporarily redirect STDERR to the screen and expect "-ksh:" at each of my expect blocks, or if there is an easier and hopefully more robust method to monitor if a shell error occurs at any point during my SSH session.

function exp_run_funcs {
# Check for DEBUG mode
[[ -n $DEBUG ]] && typeset PS4="$0: " && set -x  

# Log status message
log_stdout $0 "Running functions on the $rsvr server."

# Run functions on remote host
expect -c "set logdir $sai_logdir; set dlogfl $detailflnm;\
set ulogfl $usrflnm; set prgmfunc \"$program $0\"; set remsvr $rsvr;\
set user $usr; set host ${ip_ad[$rind]}; set pwd $pswd;\
set cfgfl $sai_cfgflnm; set sai_chksum \"$chksum\"" -c '

    ########## INITIALIZATION ##########
    ### Expect environment initialization ###
    set timeout 15
    log_user 0
    set prompt "(%|#|>|\\\$) $"
    set status 1 ; # Initialize status to failure

    ########## ESTABLISH SSH CONNECTION TO REMOTE SERVER ##########
    spawn -noecho ssh $user@$host
    if [catch "spawn ssh $user@$host" reason] {
        puts "Failed to establish SSH connection\
        to $remsvr server ($host): $reason.\n"
        set status 3
        exit $status
    }

    ### Set responses for expected login prompts ###
    expect {
        -re "(.*)assword:" { 
            send "$pwd\r"               
        }
        "you sure you want to continue connecting" {
            send -s "yes\r"
            exp_continue
        }
        timeout {
            set status 4
            exit $status
        }
        eof {
            set status 5
            exit $status
        }
    }

    expect -re $prompt {
        set timeout -1

        ### Source the configuration file ###
        puts "Sourcing the configuration file on the\
        $remsvr server.\n"

        send ". $cfgfl\r"
        if [catch "send \". \$cfgfl\r\"" reason] {
            puts "Failed to source the configuration file:\
            $reason.\n"
            set status 9
            exit $status
        }
        expect -re "(.*)config.ini\r\n(.*)$prompt"

        ### Run individual functions ###
        set timeout 15
        set tdir "$logdir/tmp"
        set fcnlst "init_log init_env install_func\
            srv_setup" 

        foreach fcn $fcnlst {
            puts "Sourcing and running function $fcn.\n"

            if {[run_function "$fcn"] != 0} {
                exit $status
            }
            if {$fcn != "init_log"} {puts "\n"}
            merge_logs
        }

        expect -re $ { set status 0 }
    }
' | tee /dev/tty
}

@GlennJackman I have added one of my expect scripts. This is one of many functions/expect scripts that work together to automatically handle the installation of software on two servers in a paired configuration. Prior to this function running, a configuration file for the 'primary' server has been SCP'ed to the partner. The configuration file is to be modified on the partner for use in the installation, then sourced to set the environment. After the configuration file has been sourced, individual functions are run. — legendmac
@GlennJackman I saw in my search for similar issues that you had recommended to one user using lassign [wait] to detect if an error occurred. I know that I have an old enough version of expect that lassign isn't an option. I'd have to use foreach {pid spawnid os_error_flag value} [wait] break and check if os_error_flag returns 0. However, I'm new enough to programming in Expect that I'm not sure if that would work in this case for catching any shell errors and exiting through the exit handler, or how I would go about implementing it. — legendmac
Before you start to run_functions, define an expect_after pattern to look for shell error strings. Something like expect_after "err" { call my cleanup code }. expect_after basically appends the pattern and action (or more than one) to each expect command that comes after. — glenn jackman
@GlennJackman Would you define the expect_after just before calling run_functions, or would it make sense to do it after I have successfully connected to the partner server prior to the expect -re $prompt? Then it would append the pattern and action to all expect commands that occur during the SSH session, right? — legendmac

legendmac legendmac · Accepted Answer · 2015-10-13T21:17:11

The inherent redirection in my scripts that is mentioned in the comments required more work that is irrelevant to this question, so I will simply post the answer to expecting KSH errors.

Prior to the ### Source the configuration file ### part, I added the following:

### Expect any KSH or function errors ###
expect_before -re "-ksh(.*): | (.*):(.*): " {set status 12; exit $status}

This essentially adds an expect -re block to all expect blocks following this code snippet, including those in procedures. It will expect the syntax that is typically seen for KSH or shell function errors when they arise.

Monitor spawned process for shell errors in Expect script

1 Answers