one troubleshooting case about em access issue

Today when trying to open my Oracle EM url, I can not open it. So I thought may be the network is having issue. I use the tenlet to try to identifiy if the network is ok. The command is telnet racnode1 1158 and it shows the network is ok. So I go to the the rac server to check.

First I use emctl status dbconsole. The output is wired because it stuck here

[oracle@racnode1 ~]$ emctl status dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.
Copyright (c) ,  Oracle Corporation.  All rights reserved.
https://racnode1:1158/em/console/aboutApplication

after I issue ctrl+c to stop the command. The who output came out.

^COracle Enterprise Manager 11g is running.
------------------------------------------------------------------
Logs are generated in directory /u01/app/oracle/product/11.2./dbhome_1/racnode1_orcl/sysman/log

As it said the dbconsole is running. So I think I might need to restart it first.

[oracle@racnode1 log]$ emctl stop dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.
Copyright (c) ,  Oracle Corporation.  All rights reserved.
https://racnode1:1158/em/console/aboutApplication
Stopping Oracle Enterprise Manager 11g Database Control ...

 ...  Stopped.

[oracle@racnode1 log]$ emctl start dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.
Copyright (c) ,  Oracle Corporation.  All rights reserved.
https://racnode1:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 11g Database Control .....

.....^C started.
------------------------------------------------------------------

As you can see, the stop command workds fine but start command stuck.

I try to read the logs to find out why. So I go to the log directory

/u01/app/oracle/product/11.2.0/dbhome_1/racnode1_orcl/sysman/log

There are logs below

[oracle@racnode1 log]$ ls -ltr
total
-rw-r-----.  oracle oinstall        Sep  : emdctl.log
-rw-r-----.  oracle oinstall        Sep  : patchAgtStPlugin.log
-rw-r-----.  oracle oinstall        Sep  : nfsPatchPlug.log
-rw-r-----.  oracle oinstall        Sep  : emagentfetchlet.log
-rw-r-----.  oracle oinstall        Sep  : emagentfetchlet.trc
drwxr-----.  oracle oinstall     Sep  : pafLogs
-rw-r-----.  oracle oinstall    Sep  : secure.log
-rw-r-----.  oracle oinstall  Sep  : emoms.trc.
-rw-r-----.  oracle oinstall      Sep  : emctl.msg
-rw-r-----.  oracle oinstall   Sep  : emdctl.trc
-rw-r-----.  oracle oinstall   Sep  : emdb.nohup
-rw-r-----.  oracle oinstall   Sep  : emagent_perl.trc
-rw-r-----.  oracle oinstall    Sep  : emagent.log
-rw-r-----.  oracle oinstall   Sep  : emoms.trc
-rw-r-----.  oracle oinstall  Sep  : emoms.log
-rw-r-----.  oracle oinstall  Sep  : emagent.trc

I tail -100f emdb.nohup found below information.

----- Mon Sep  :: ::Checking status of DBConsole :  -----
----- Mon Sep  :: ::DBConsole exited at Mon Sep  ::  with return value . -----
----- Mon Sep  :: ::Restarting DBConsole. -----
----- Mon Sep  :: ::Console Launched with PID  at time Mon Sep  ::  -----
----- Mon Sep  :: ::osname is linux -----
----- Mon Sep  :: ::heap size used is - -----
----- Mon Sep  :: ::perm size used is - -----
----- Mon Sep  :: ::joptions used are -server -Xmx192M -XX:MaxPermSize=200M -XX:MinHeapFreeRatio= -XX:MaxHeapFreeRatio=  -----
// :: Error starting ORMI server racnode1 port ,: Address already in use
// :: Shutting down OC4J...

It said the 5,520 port are being used. So I try to use netstat to find out what is the process using this port.

[oracle@racnode1 log]$ netstat -ap | grep
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp               *:                      *:*                         LISTEN      /java
udp               169.254.166.63:        *:*                                     /asm_psp0_+ASM1
[oracle@racnode1 log]$ ps -ef | grep
oracle           Sep21 ?        :: /u01/app/oracle/product/11.2./dbhome_1/jdk/bin/java -server -Xmx192M -XX:MaxPermSize=200M -XX:MinHeapFreeRatio= -XX:MaxHeapFreeRatio= -DORACLE_HOME=/u01/app/oracle/product/11.2./dbhome_1 -Doracle.home=/u01/app/oracle/product/11.2./dbhome_1/oc4j -Doracle.oc4j.localhome=/u01/app/oracle/product/11.2./dbhome_1/racnode1_orcl/sysman -DEMSTATE=/u01/app/oracle/product/11.2./dbhome_1/racnode1_orcl -Doracle.j2ee.dont.use.memory.archive=true -Djava.protocol.handler.pkgs=HTTPClient -Doracle.security.jazn.config=/u01/app/oracle/product/11.2./dbhome_1/oc4j/j2ee/OC4J_DBConsole_racnode1_orcl/config/jazn.xml -Djava.security.policy=/u01/app/oracle/product/11.2./dbhome_1/oc4j/j2ee/OC4J_DBConsole_racnode1_orcl/config/java2.policy -Djavax.net.ssl.KeyStore=/u01/app/oracle/product/11.2./dbhome_1/sysman/config/OCMTrustedCerts.txt-Djava.security.properties=/u01/app/oracle/product/11.2./dbhome_1/oc4j/j2ee/home/config/jazn.security.props -DEMDROOT=/u01/app/oracle/product/11.2./dbhome_1/racnode1_orcl -Dsysman.md5password=true -Drepapi.oracle.home=/u01/app/oracle/product/11.2./dbhome_1 -Ddisable.checkForUpdate=true -Doracle.
sysman.ccr.ocmSDK.websvc.keystore=/u01/app/oracle/product/11.2./dbhome_1/jlib/emocmclnt.ks -Dice.pilots.html4.ignoreNonGenericFonts=true -Djava.awt.headless=true -jar /u01/app/oracle/product/11.2./dbhome_1/oc4j/j2ee/home/oc4j.jar -config /u01/app/oracle/product/11.2.0/dbhome_1/oc4j/j2ee/OC4J_DBConsole_racnode1_orcl/config/server.xml

There do have one process using this port. One thing you need to be careful about is that if you are not root. The netstat may not return you all the result. Ok I checked the process found it is an OC4J process.  And the parent process id is 1. Means this process should have be exited. I killed it. Try to start em again.

[oracle@racnode1 log]$ emctl start dbconsole
Oracle Enterprise Manager 11g Database Control Release 11.2.0.3.
Copyright (c) ,  Oracle Corporation.  All rights reserved.
https://racnode1:1158/em/console/aboutApplication
Starting Oracle Enterprise Manager 11g Database Control ............ started.
------------------------------------------------------------------
Logs are generated in directory /u01/app/oracle/product/11.2./dbhome_1/racnode1_orcl/sysman/log

Very smoothly. I started it without any problem.