How to solve common DG broker issues?

Jephe Wu - http://linuxtechres.blogspot.com

Objective: to make dgmgrl to work, show configuration should display 'SUCCESS'.
Environment:  RHEL4.8 64bit, Oracle 10.2.0.3 64bit (RAC+Dataguard, RAC as primary, standalone DB as standby).

Error messages:
When running dgmgrl, connect /, show configuration, Get the following errors:
1.   In the alert log: /u01/app/oracle/admin/LIVEDB/bdump/drcLIVEDB1.log

DG 2011-07-22-16:55:32        0 2 756506724 DMON: Automatic health check timed out
DG 2011-07-22-16:55:32        0 2 756506724 Operation CTL_GET_STATUS continuing with warning, status = ORA-16619
DG 2011-07-22-16:55:32        0 2 0 INSV: cancelling message publish for opcode CTL_GET_STATUS
DG 2011-07-22-16:55:32        0 2 0       request ID = 1.2.756506724

2  When run 'show configuration', got 'Warning: DMON: health check timeout'

3. in the dataguard log file on one of RAC DB server, got error:
DG 2011-07-22-15:35:07        0 2 0 DMON: found read, but unconsumed message
DG 2011-07-22-15:35:07        0 2 0 DMON: req_id = 1.2.756506586


Solution:
1. On one of the RAC DB, run command 'ALTER SYSTEM SET DG_BROKER_START=FALSE' to shutdown dg broker for RAC, then run 'ALTER SYSTEM SET DG_BROKER_START=TRUE' to start it again.


Other useful commands:
select * from v$archive_gap;
show parameter dg_broker_start;  # should be true on dataguard
http://www.ritzyblogs.com/OraTalk/PostID/105/How-to-setup-DGMGRL-broker-with-example 
SQL> set line 32767;
SQL> select * from v$logfile;

Startup standby db and redo apply:
sql>STARTUP MOUNT;
sql> alter database recover managed standby database cancel; 
sql>ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
or
sql> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE using current logfile DISCONNECT FROM SESSION;
note: cancel first like above, otherwise, you will get error such as "an incompatible media recovery is active".