As part of the Data Archive product, the System Coordinator works with it to allow SMH to serve as the admin console. I was able to run the activation process to install under OMVS (z/OS), but SMH shows:
Response: 101 Sub Code: 0x274DA001
NET_R_IP [10061] No connection could be made because the target machine actively refused it.
Suspicious about the reason it can’t talk to the mainframe under port 53376, that it was not a firewall blocking a port but that a daemon was not executing to listen under it, I logged in as saguser and went to the directory path where cor.sh resides and issued:
cor.sh start
What I got back is: FSUM7351 not found
The other odd thing, when I type more cor.sh, I see funny Ys in there:
#!/bin/sh
get pid
pid_exe () {
PID_EXE=ps -ef | grep $1 | grep -v grep | awk '{ print $2 }'
}
corenv script
if Ý expr /$0 : // -eq 0 ¨ ; then
CORENV=echo \pwd`/$0 | sed ‘s|/./|/|g’ | sed ‘s|cor.sh|corenv|g’ else CORENV=echo $0 | sed ‘s|cor.sh|corenv|g’`
fi
clear $SAG
SAG=“”
add-ons enabled
ADABAS_ADDONS=ON
source corenv
. $CORENV >> /dev/null
cor launch control
Ýcor.sh¨(14%)
===>
Could my problem be how the files transferred to the mainframe?
-Brian
Will try again here…
I have a COR process running under OMVS:
$ ps -ef
UID PID PPID C STIME TTY TIME CMD
SAGUSER 50331838 1 - Nov 24 ? 0:00
SAGUSER 16777411 1 - Nov 24 ? 0:12 /opt/softwareag/sag/co
r/v826/bin/corlc
SAGUSER 33555073 50332346 - 16:03:37 ttyp0000 0:00 ps -ef
SAGUSER 50332346 50332413 - 15:03:18 ttyp0000 0:01 -sh
SAGUSER 50332413 1 - 15:03:18 ? 0:01 OMVS
However, it doesn’t seem to be listening under the group services port (and nothing else is):
$ netstat -a | grep 53376
$
I am thinking that this is why SMH can’t display branches under Adabas System Coordinator 8.2.6 and Data Archiving for Adabas 1.6.1. I tried adding the perspective with host:portname for both the daemon repository and specific repository but it changes nothing. The only difference is with a specific repository it times out when I try to discover perspective on it and makes me log into SMH again.
I have an SR open but now going into US Thanksgiving I am concerned I won’t hear back until Monday and am losing too many days on this installation and activation of the product.
Any help is appreciated. Thanks!
-Brian
The other odd thing… when I run cor.sh start, it comes back with:
$ cor.sh start
cor.sh: FSUM7351 not found
Something isn’t right here.
Hi Brian,
With regards to ‘cor.sh: FSUM7351 not found’ message, it might just be that the current directory ‘.’ is not in the $PATH environment variable. Does it work with an explicit path is used?
$ cd /opt/softwareag/cor/v826/INSTALL
$ ./cor.sh restart
From the process list I can see the Adabas System Coordinator Launch Control process ‘corlc’ running but it does not look like corlc has started the Adabas System Coordinator Daemon process ‘cord’.
To start with, I would check the activation log file for any errors. It will be in the C:\SoftwareAG\cor\v826\INSTALL directory on the machine used for the installation and is named activation.-.log. There should be an ‘Activation complete’ message at the end if it was successful.
The next step is to check the Adabas System Coordinator log files. These can be found in the /opt/softwareag/cor/v826/log directory and are named cor.log. There should be some messages from the Launch Control process and then some from the Daemon process.
LC0001I Started DA3F.eur.ad.sag
COR1001I DA3F Launch Control started
LC0004I Configuration da3f.eur.ad.sag:53376/17061,16?timeout=30&ascii=8&ttl=0
LC0009I Starting DEFAULT:DA3F:DAEMON
CD0001I DEFAULT:DA3F:DAEMON started (da3f.eur.ad.sag:53376/16,500?ttl=0&timeout=30&ascii=8)
COR1003I DEFAULT:DA3F:DAEMON Daemon started
Regards,
Rob.
Hi Rob,
Thanks much for the very helpful reply.
I followed what you said after killing the existing PID.
$ ./cor.sh start
Starting Adabas System Coordinator Launch Control …
PID 16777439
Done.
Then I checked the current day’s log file:
26/11/2015 14:11:03 LC0001I Started vista.tcc.etn.com
26/11/2015 14:11:03 COR1001I VISTA Launch Control started
26/11/2015 14:11:05 LC0004I Configuration vista.tcc.etn.com:53376/120,2?timeout
=30&ascii=8&ttl=16
26/11/2015 14:11:05 LC0021W Network error (NET_R_IP Ý1128¨ EDC8128I Connection
refused.), retrying …
Ýcor20151126.log¨(EOF)
This must be why it doesn’t work. I’ve been trying to figure out why SMH won’t talk to mainframe COR, but it seems it’s mainframe COR on OMVS that can’t talk to z/OS.
Based on this, can you advise how I can fix it?
Thanks,
Brian
Hi Brian,
When the COR Launch Control starts, it needs to know whether the configuration file is on the local machine or should be accessed via another COR Daemon.
From your log it appears that the Launch Control thinks the configuration file is not on the local machine, so it is attempting to connect to a COR Daemon on ‘vista.tcc.etn.com:53376’ rather than issuing local Adabas calls.
If ‘vista.tcc.etn.com’ is the correct hostname, then there may an issue with the DNS/name resolution in your OMVS environment. The Launch Control compares lookups of the computer name, in your case VISTA, with the hostname specified in the configuration string to determine whether they are the same machine.
In OMVS, try an ‘onslookup VISTA’ and check that it comes back with the ‘VISTA.tcc.etn.com’ for the hostname and that the IP address is correct.
$ onslookup VISTA
Running nslookup version 9
Allocated socket 5, type udp
Server: 2001:920:1809::f:20
Address: 2001:920:1809::f:20#53
Non-authoritative answer:
Name: VISTA.tcc.etn.com
Address: 111.222.333.444
Then try a ‘nslookup vista.tcc.etn.com’ and check that you get the same ‘Name:’ and ‘Address:’ back.
Regards,
Rob.
Hi Rob,
I was afraid this was going to bite us. I remember going through the pre-install validation checklist and this is the first one on the list. Our machine name is CH40 based on the prior result, and when I do the onslookup on CH40, it returns:
$ uname -a
OS/390 CH40 24.00 04 2828
$ onslookup CH40
Running nslookup version 9
Note: nslookup is deprecated and may be removed from future releases.
Consider using the dig' or host’ programs instead. Run nslookup with
the `-silÝent¨’ option to prevent this message from appearing.
Allocated socket 5, type udp
Allocated socket 6, type udp
Allocated socket 5, type udp
Server: 151.110.231.240
Address: 151.110.231.240#53
** server can’t find CH40: NXDOMAIN
When I do as you suggest I seem to get better results:
$ onslookup vista
Running nslookup version 9
Note: nslookup is deprecated and may be removed from future releases.
Consider using the dig' or host’ programs instead. Run nslookup with
the `-silÝent¨’ option to prevent this message from appearing.
Allocated socket 5, type udp
Server: 151.110.126.240
Address: 151.110.126.240#53
Non-authoritative answer:
Name: vista.TCC.ETN.COM
Address: 151.110.7.69
Same with the full domain name:
$ nslookup vista.tcc.etn.com
Running nslookup version 9
Note: nslookup is deprecated and may be removed from future releases.
Consider using the dig' or host’ programs instead. Run nslookup with
the `-silÝent¨’ option to prevent this message from appearing.
Allocated socket 5, type udp
Server: 151.110.126.240
Address: 151.110.126.240#53
Non-authoritative answer:
Name: vista.tcc.etn.com
Address: 151.110.7.69
Based on the fact it finds vista and vista.tcc.etn.com, it sounds like it ought to work?
-Brian
If corlc is the system coordinator launch control and cord is the daemon it is looking for, in looking at cor.sh, I am not sure how the daemon gets started. No where does it issue:
pid_exe “$CORD”
…except when stopping the launch control.
Here is the content of cor.sh:
#!/bin/sh
# get pid
pid_exe () {
PID_EXE=`ps -ef | grep $1 | grep -v grep | awk '{ print $2 }'`
}
# corenv script
if `expr /$0 : //` -eq 0 ; then
CORENV=`echo \`pwd\`/$0 | sed 's|/\./|/|g' | sed 's|cor.sh|corenv|g'
else
CORENV=`echo $0 | sed 's|cor.sh|corenv|g'`
fi
# clear $SAG
SAG=""
# add-ons enabled
ADABAS_ADDONS=ON
# source corenv
. $CORENV >> /dev/null
# cor launch control
CORLC=$CORDIR/$CORVERS/bin/corlc
# cor daemon
CORD=$CORDIR/$CORVERS/bin/cord
# start/stop
case "$1" in
start|Start|START)
# start corlc
pid_exe "$CORLC"
if "$PID_EXE" = "" ; then
echo "Starting Adabas System Coordinator Launch Control ..."
# don't redirect stdout/stderr when started from batch
if "`uname -s`" = "OS/390" && "$SHELL" = "" ; then
$CORLC
else
$CORLC 1>$CORDIR/$CORVERS/log/cor.stdout 2>$CORDIR/$CORV
fi
sleep 3
pid_exe "$CORLC"
echo "PID $PID_EXE"
echo "Done."
fi
;;
stop|Stop|STOP)
# stop corlc
pid_exe "$CORLC"
if "$PID_EXE" != "" ; then
echo "Stopping Adabas System Coordinator Launch Control ..."
echo "SIGTERM $PID_EXE"
kill -TERM $PID_EXE
COUNTER=0 ; TIMEOUT=60
while -n "$PID_EXE" && $COUNTER -lt $TIMEOUT ; do
COUNTER=`expr $COUNTER + 1`
pid_exe "$CORLC"
sleep 1
done
if -n "$PID_EXE" ; then
echo "SIGKILL $PID_EXE"
kill -KILL $PID_EXE
fi
echo "Done."
fi
# stop cord
pid_exe "$CORD"
if "$PID_EXE" != "" ; then
echo "Stopping Adabas System Coordinator Daemons ..."
for PID in $PID_EXE
do
echo "SIGTERM $PID"
kill -TERM $PID
done
COUNTER=0 ; TIMEOUT=60
while -n "$PID_EXE" && $COUNTER -lt $TIMEOUT ; do
COUNTER=`expr $COUNTER + 1`
pid_exe "$CORD"
sleep 1
done
for PID in $PID_EXE
do
echo "SIGKILL $PID"
kill -KILL $PID
done
echo "Done."
fi
;;
restart|Restart|RESTART)
$0 stop
$0 start
;;
*)
echo "Usage: cor.sh {start|stop|restart}"
;;
esac
Hi Brian,
The ‘cor.sh start’ option runs the corlc process, and it’s corlc that starts the cord processes.
Could you run the following commands in turn and provide the output so that we can see which names are coming from where :-
uname -a
hostname -d –c
hostname -d -g
hostname -d –r
Regards,
Rob.
The results of the commands:
$ uname -a
OS/390 CH40 24.00 04 2828
$ hostname -d -c
IT_catopen()->1405D020
IT_snapout(): reg->0
IT_snapout(): dereg->0
IT_snapout()->0
getViaTcpipData(0,1402A2FC,2000,1)
getViaTcpipData()->0
VISTA2.TCC.ETN.COM
$ hostname -d -g
IT_catopen()->1405D020
IT_snapout(): reg->0
IT_snapout(): dereg->0
IT_snapout()->0
getViaGethostname(1402A2FC,2000,1)
getViaGethostname()->0
NODENAME
$ hostname -d -r
IT_catopen()->1405D020
IT_snapout(): reg->0
IT_snapout(): dereg->0
IT_snapout()->0
getViaGethostname(1402A2FC,2000,1)
getViaGethostname()->0
getViaDNS(1402A2FC,2000,1)
EZZ8342I gethostbyname(NODENAME): Unknown host
retcode set to 2 at SYS12212.T050954.RA000.EZAHOHNM.SRCCMS.H01(EZAHOHNM)/400: ge
thostbyname() failed
getViaDNS()->2
Please advise what this tells you and what we need to change to make it work.
Thanks much for your quick replies!
-Brian
And an ‘onslookup VISTA2.TCC.ETN.COM’ please …
$ onslookup vista2.tcc.etn.com
Running nslookup version 9
Note: nslookup is deprecated and may be removed from future releases.
Consider using the dig' or host’ programs instead. Run nslookup with
the `-silÝent¨’ option to prevent this message from appearing.
Allocated socket 5, type udp
Server: 151.110.126.240
Address: 151.110.126.240#53
Non-authoritative answer:
Name: vista2.tcc.etn.com
Address: 151.110.7.102
Hmm… different IP address than vista.tcc.etn.com. I know vista.tcc.etn.com is the normal z/OS system… is vista2.tcc.etn.com the DNS for the OMVS segment?
Hi Brian,
The configuration is site-specific, but the OMVS segment name/address is often the same as the host z/OS.
I’m still not sure where the VISTA in your log file has come from, it does not seem to match the results from above.
26/11/2015 14:11:03 COR1001I VISTA Launch Control started
Could you try a ‘hostname -d -s’ please and see what that returns.
If you could also look in the activation log file in the C:\SoftwareAG\cor\v826\INSTALL directory on the installation machine and post the top part of the file. The lines I am interested in are at the top in the ‘pre-activation’ checks, it looks like this:
# # Adabas Add-On Products # #
# # COR pre-activation checks # #
Using configuration parameters:
#IBMOMVS.CONFIGURATION
IBMOMVS.SagInstallDir=/FS/fs3304/sag/IBMOMVS
IBMOMVS.DataSet=RDU.RJE.ADR17
IBMOMVS.GrpSvcPort=53376
IBMOMVS.DynamicPortRange=53377-53399
IBMOMVS.ConfigHostname=da3f.eur.ad.sag
Validating node name and hostname …OK
Node name: DA3F IP Address: DNS name: DA3F.eur.ad.sag
Checking memory access…OK
Checking path to the Software AG product directory…OK
Checking free space…OK
Checking ADALNKR & LNKGBLS dataset name…OK
Checking group services port…OK
Checking group services port availability…OK
Checking dynamic port range…OK
Checking port availability…OK
Checking hostname of the machine sharing the config file…OK
Pre activation checks completed OK.
Copying files…done
Uncompressing files…done
Copying activation configuration file…done
Tidying up files…done
This will be my last reply today, I’ll get back to you tomorrow.
Regards,
Rob.
Hi Rob,
The results of hostname -d -s is:
$ hostname -d -s
IT_catopen()->1405D020
IT_snapout(): reg->0
IT_snapout(): dereg->0
IT_snapout()->0
getViaTcpipData(1,1402A2FC,2000,1)
getViaTcpipData()->0
VISTA2
The activation log shows this:
# # Adabas Add-On Products # #
# # COR pre-activation checks # #
Using configuration parameters:
#IBMOMVS.CONFIGURATION
IBMOMVS.SagInstallDir=/opt/softwareag/sag
IBMOMVS.DataSet=VSTA.DBA.ADR161.LOAD.NEW
IBMOMVS.GrpSvcPort=53376
IBMOMVS.DynamicPortRange=53377-53399
IBMOMVS.ConfigHostname=vista.tcc.etn.com
Validating node name and hostname … FAILED
ACTIVATION_WARNING: Uknown host, the node name cannot be pinged!
.EZZ8342I gethostbyname(NODENAME): Unknown host
… FAILED
ACTIVATION_WARNING: The node name does not appear to match the DNS name.
ACTIVATION_WARNING: Node name: CH40 IP Address: DNS name:
Checking memory access…OK
Checking path to the Software AG product directory…OK
Checking free space…OK
Checking ADALNKR & LNKGBLS dataset name…OK
Checking group services port…OK
Checking group services port availability…OK (commented out)
Checking dynamic port range…OK
Checking port availability (commented out)…OK
Checking hostname of the machine sharing the config file…OK
Pre activation checks completed OK.
Copying files…done
Uncompressing files…done
Copying activation configuration file…done
Tidying up files…done
Note: the port availability checks were failing if the port number was anywhere in the result of netstat -a | grep port# . It didn’t matter if it was the port number on the other side or part of the address/process number of the ftp connection. I verified none of the hits were from the port # being used on vista.tcc.etn.com before commenting that part out.
Thanks so much for your help today… I feel like we’re close to getting to the root of the problem here.
-Brian
Hi Brian,
Unfortunately the check on the port numbers in the activation for ADR 1.6 does not differentiate between local/remote endpoints, so commenting the step out after a manual check on the port numbers is fine.
I’m still not sure where name ‘VISTA’ in the “26/11/2015 14:11:03 COR1001I VISTA Launch Control started” message has come from. Has the LCOMP= line in the /opt/software/cor/v826/cfg/cor.cfg file been modified? If not, we’ll have to turn some tracing on in corlc to get some diagnostics.
Could you try an nslookup/ping of ‘vista.tcc.etn.com’ on the PC used to install and see what the IP address comes back as please.
Regards,
Rob.
Hi Rob,
The cor.cfg was not modified after installation, and the LCOMP= line says:
LCOMP=vista.tcc.etn.com
From the Windows server SMH was installed on, ping vista.tcc.etn.com returns the expected 151.110.7.69.
From my perspective it is not odd at all that it uses VISTA as the local machine name. Up until yesterday I never heard of VISTA2. When I activated the product, I specified vista.tcc.etn.com as the local machine name, the override machine name, and set it in the ftp settings (which worked). As you said, the whole z/OS box including the OMVS segment is vista.tcc.etn.com, so I guess I am confused what exactly is VISTA2.tcc.etn.com.
Could this be the source of all my problems? Should I have specified vista2.tcc.etn.com as the override machine name, for example? Or should I now change LCOMP to vista2? Or will that still not work since the uname -a returns CH40 and not vista or vista2?
-Brian
Hi Rob,
After looking through what is contained in the cor.cfg file, I decided to start playing around with different options such as changing LCOMP to
LCOMP=vista2.tcc.etn.com
and changing DAEMON= to point to vista2.tcc.etn.com.
After stopping and starting under different combinations, I tried using:
LCOMP=
DAEMON=
Now when I start it, I get this:
27/11/2015 16:10:53 LC0001I Started NODENAME
27/11/2015 16:10:53 COR1001I NODENAME Launch Control started
27/11/2015 16:10:56 LC0004I Configuration 120,2?ttl=0&timeout=30&ascii=8
27/11/2015 16:10:56 LC0006E Adabas error (response 101 subcode 00000065), termi
nating …
27/11/2015 16:10:56 LC0002I Stopped NODENAME
27/11/2015 16:10:56 COR1002I NODENAME Launch Control stopped
Adabas response 101 / subcode 65 says:
An error has occurred using one of the client-based Adabas add-on products such as Adabas System Coordinator or Adabas Fastpath.
Mainframe Systems
On mainframe systems, one of the following subcodes may also be included with this response code:
Subcodes of 20 or less are produced by Adabas Fastpath when an error occurs during the processing of an Adabas command..
Subcodes of 21 or greater are produced by Adabas System Coordinator.
Subcode 65:
Net-Work Clustering: Message segment protocol error. This is an internal error in Adabas System Coordinator.
Action: Contact your Software AG technical support representative.
-Brian
Hi Brian,
It may be that the LCOMP= line has been set as a result of selecting the ‘Override default computer node name’ option in the Activation, but at least that explains where the Launch Control is getting the name ‘VISTA’ from.
As you say, having specified vista.tcc.etn.com in the FTP settings the software ended up in the right place, so I think we can assume that is the correct name for the machine.
The good news is that the DNS appears okay, ‘onslookup vista’ returns the fully qualified hostname ‘vista.tcc.etn.com’ and the same IP address as ‘onslookup vista.tcc.etn.com’.
However, you have to resolve the issues with the host/node names in the OMVS environment before you can proceed. The ‘uname -a’ and ‘hostname -g’ commands should return ‘VISTA’.
Name mismatches aside, the call to the gethostname() function (hostname -g) returns a name (in this case ‘NODENAME’) which cannot be resolved by a call to the gethostbyname() function (hostname -r), which is ‘broken’ behaviour (this is the standard mechanism applications use to determine the local address).
I think there’s a TCPIP.DATA system file that contains HOSTNAME (returned by ‘hostname -g’) and the nodename (returned by ‘uname -a’) is set by the SYSNAME sysparm specified at IPL, but your operators should know how to make the changes.
After that, re-run the activation and check that the warnings relating to the host/node names have gone. The LFILE= parameter should be blank and DAEMON= parameter should be DAEMON:53376=vista.tcc.etn.com/120,2?timeout=30&ascii=8&ttl=16.
If you are still having problems after the host/node names have been fixed, then you should pursue them with Support via the SR.
Regards,
Rob.
Hi Rob,
This is much appreciated you taking the time you have these last couple days.
I now have the action plan which can begin one all the US employees are back next week.
- Find hostname and nodename in the TCPIP.DATA system file and have it correctly bring back vista.tcc.etn.com.
- IPL specifying SYSNAME sysparm correctly to make #1 changes take effect.
- Confirm correct hostname -g and uname -a results.
- Start cor.sh and check log and ps -ef to ensure corlc started cord and no other issues are present.
I’ve changed cor.cfg already now in preparation of this by setting:
LCOMP=vista.tcc.etn.com
LFILE=
DAEMON:53376=vista.tcc.etn.com/120,2?timeout=30&ascii=8&ttl=16
Unfortunately we don’t even IPL every week, but when we do it’s on Sunday. The best we can hope is these changes will be staged next week for an IPL on 6 Dec. I should be able to validate and start cor.sh on 7 Dec.
Please let me know if it appears to you that I misunderstood something. Otherwise, we shall see in a week and a half how we are doing.
Regds,
Brian
Hi Brian,
Apologies, there was a typo in my last reply, I should have said that LCOMP= should be blank rather than LFILE.
After the issues with the host/node name have been corrected, I would tidy up the machine and do the Activation again to make sure it runs through without any errors and that you have clean install.
So, just to elaborate on your plan a little:-
(1) Make sure the nodename returned by ‘uname -a’ is ‘VISTA’ not ‘CH40’.
SYSNAME sysparm
https://www-01.ibm.com/support/knowledgecenter/SSLTBW_1.13.0/com.ibm.zos.r13.bpxa500/uname.htm%23uname
(2) Correct the host/domain names returned by the ‘hostname’ command.
TCPIP.DATA
https://www-01.ibm.com/support/knowledgecenter/SSLTBW_1.13.0/com.ibm.zos.r13.halz001/dxec.htm%23dxec?lang=en
‘hostname -g’ should return ‘vista’ (currently returns ‘NODENAME’)
‘hostname -r’ should return ‘vista.tcc.etn.com’ (currently fails)
‘hostname -c’ should return ‘vista.tcc.etn.com’ (currently returns ‘VISTA2.TCC.ETN.COM’)
‘hostname -s’ should return ‘vista’ (not tried but I suspect this currently returns ‘VISTA2’)
(3) Check the DNS still returns the correct results.
‘onslookup vista’ should return ‘vista.tcc.etn.com’ and the same IP address as ‘onslookup vista.tcc.etn.com’.
(3) Tidy up and re-run the Activation.
-
Make sure there are no active corlc or cord processes.
-
Remove the /opt/softwareag/cor and /opt/softwareag/adr directories.
-
Run the Activation, de-select the ‘override default computer node name’ option.
-
Check C:\SoftwareAG\cor\v826\INSTALL\activation*.log to make sure there are no errors.
-
The Activation should have set entries in the /opt/softwareag/cor/v826/cfg/cor.cfg file as follows:-
LCOMP=
LFILE=120,2
DAEMON:53376=vista.tcc.etn.com/120,2?timeout=30&ascii=8&ttl=16
-
When the Activation has been completed, the corlc and cord processes should be running.
-
Check the log files in /opt/softwareag/cor/v826/log for errors, the COR Daemon gives ‘started’ and ‘loaded plug-in’ messages when it comes up:-
CD0001I DEFAULT:VISTA:DAEMON started (vista.tcc.etn.com:53376/120,2?timeout=30&ascii=8&ttl=16)
CD0026I DEFAULT:VISTA:DAEMON loaded plug-in library for Data Archiving for Adabas
-
Check the log files in /opt/softwareag/adr/v161/log for errors, the ADR Service gives ‘started’ message when it is loaded:-
ADR0001I DEFAULT:VISTA:DAEMON started (vista.tcc.etn.com:53376/120,2?timeout=30&ascii=8&ttl=16)
If you’re still having problems after the changes then Support will be able to help.
Regards,
Rob.