34 Replies Latest reply: May 24, 2012 2:56 PM by mkrauz

OML 9.10 integration - unable to validate credentials

mkrauz Novice

We have an existing OMU8.x integration configured and working with AlarmPoint 4.10 with the latest patch.

 

Recently we installed the Integration Agent (latest patch) and OML9 integration package on the new OML9.1 server.

 

Both old OMU8.x and OML9.1 Integration Agents connect to the same AlarmPoint server.

 

When we test to validate credentials, we get the message:

 

AP-OMU-Bridge -u AlarmPointOMU -p Password1 --validate

ValidCredentials:false

 

OML is configured to authenticate with LDAP and even with LDAP authentication disabled, we get the same result.

 

We can login to OML Java Console using that same user account and we can see all the messages.

 

Has anyone ran into this issue?

 

When we generate the test OML message, Integration Agent sends this message:

 

Integration Agent a01111/127.0.0.1:8079 was processing an Integration Service Request for Integration Service (hp_operations_manager_unix, HPOMU) at 2012-05-04 16:48:21,012 when an unhandled exception occurred.

 

The reason given was: The Integration Service (hp_operations_manager_unix,HPOMU) terminated the request because of an unhandled exception.. Message payload is of type: ActiveMQObjectMessage

 

Stack trace:

 

  • org.mule.umo.MessagingException: The Integration Service (hp_operations_manager_unix,HPOMU) terminated the request because of an unhandled exception.. Message payload is of type: ActiveMQObjectMessage

      at com.alarmpoint.integrationagent.services.ServiceInterceptor.intercept(ServiceInterceptor.java:60)

      at org.mule.impl.InterceptorsInvoker.execute(InterceptorsInvoker.java:47)

      at com.alarmpoint.integrationagent.interceptors.AuthenticationInterceptor.intercept(AuthenticationInterceptor.java:105)

      at org.mule.impl.InterceptorsInvoker.execute(InterceptorsInvoker.java:47)

      at org.mule.interceptors.EnvelopeInterceptor.intercept(EnvelopeInterceptor.java:44)

      at org.mule.impl.InterceptorsInvoker.execute(InterceptorsInvoker.java:47)

      at org.mule.impl.model.DefaultMuleProxy.onCall(DefaultMuleProxy.java:258)

      at org.mule.impl.model.seda.SedaComponent.doSend(SedaComponent.java:393)

      at org.mule.impl.model.AbstractComponent.sendEvent(AbstractComponent.java:418)

      at org.mule.impl.MuleSession.sendEvent(MuleSession.java:349)

      at org.mule.routing.inbound.InboundRouterCollection.send(InboundRouterCollection.java:197)

      at org.mule.routing.inbound.InboundRouterCollection.route(InboundRouterCollection.java:163)

      at org.mule.providers.AbstractMessageReceiver$DefaultInternalMessageListener.onMessage(AbstractMessageReceiver.java:581)

      at org.mule.providers.AbstractMessageReceiver.routeMessage(AbstractMessageReceiver.java:322)

      at org.mule.providers.AbstractMessageReceiver.routeMessage(AbstractMessageReceiver.java:251)

      at com.alarmpoint.integrationagent.receivers.JmsGroupMessageReceiver$ReceiverThread.process(JmsGroupMessageReceiver.java:369)

      at com.alarmpoint.integrationagent.receivers.JmsGroupMessageReceiver$ReceiverThread.processWhenReady(JmsGroupMessageReceiver.java:291)

      at com.alarmpoint.integrationagent.receivers.JmsGroupMessageReceiver$ReceiverThread.run(JmsGroupMessageReceiver.java:219)

Caused by: com.alarmpoint.integrationagent.exceptions.ServiceInvocationException: The Integration Service (hp_operations_manager_unix,HPOMU) terminated the request because of an unhandled exception.

      at com.alarmpoint.integrationagent.services.ServiceProxyImpl.processRequest(ServiceProxyImpl.java:511)

      at com.alarmpoint.integrationagent.services.ServiceInterceptor.intercept(ServiceInterceptor.java:56)

      ... 17 more

Caused by: com.alarmpoint.integration.hpomu.OMUException: Invalid Credentials

      at com.alarmpoint.integration.hpomu.HPOMUInteractionWrapper.validateCredentials(HPOMUInteractionWrapper.java:318)

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

      at java.lang.reflect.Method.invoke(Unknown Source)

      at org.mozilla.javascript.MemberBox.invoke(MemberBox.java:145)

      at org.mozilla.javascript.NativeJavaMethod.call(NativeJavaMethod.java:204)

      at org.mozilla.javascript.optimizer.OptRuntime.callProp0(OptRuntime.java:119)

      at org.mozilla.javascript.gen.c5._c7(/opt/alarmpointsystems/integrationagent/integrationservices/hpomu/hpomu.js:345)

      at org.mozilla.javascript.gen.c5._c1(/opt/alarmpointsystems/integrationagent/integrationservices/hpomu/hpomu.js:89)

      at org.mozilla.javascript.gen.c5.call(/opt/alarmpointsystems/integrationagent/integrationservices/hpomu/hpomu.js)

      at org.mozilla.javascript.ContextFactory.doTopCall(ContextFactory.java:340)

      at org.mozilla.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:2758)

      at org.mozilla.javascript.gen.c5.call(/opt/alarmpointsystems/integrationagent/integrationservices/hpomu/hpomu.js)

      at com.alarmpoint.integrationagent.script.JavaScript.callFunction(JavaScript.java:191)

      at com.alarmpoint.integrationagent.GenericService.callJSFunc(GenericService.java:542)

      at com.alarmpoint.integrationagent.GenericService.processModernAPXML(GenericService.java:488)

      at com.alarmpoint.integrationagent.GenericService.doProcessAPXML(GenericService.java:469)

      at com.alarmpoint.integrationagent.GenericService.processInboundQueue(GenericService.java:178)

      at com.alarmpoint.integrationagent.GenericService.onCall(GenericService.java:652)

      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

      at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)

      at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

      at java.lang.reflect.Method.invoke(Unknown Source)

      at org.mule.impl.model.resolvers.DynamicEntryPoint.invokeMethod(DynamicEntryPoint.java:312)

      at org.mule.impl.model.resolvers.DynamicEntryPoint.invoke(DynamicEntryPoint.java:259)

      at org.mule.impl.DefaultLifecycleAdapter.intercept(DefaultLifecycleAdapter.java:194)

      at org.mule.impl.InterceptorsInvoker.execute(InterceptorsInvoker.java:47)

      at com.alarmpoint.integrationagent.services.ServiceProxyImpl$1.call(ServiceProxyImpl.java:679)

      at com.alarmpoint.integrationagent.services.ServiceProxyImpl$1.call(ServiceProxyImpl.java:667)

      at com.alarmpoint.integrationagent.util.ThreadUtils.executeWithTimeout(ThreadUtils.java:109)

      at com.alarmpoint.integrationagent.services.ServiceProxyImpl.executeRequest(ServiceProxyImpl.java:666)

      at com.alarmpoint.integrationagent.services.ServiceProxyImpl.processRequest(ServiceProxyImpl.java:494)

      ... 18 more

  • Re: OML 9.10 integration - unable to validate credentials
    rodney_au Hero

    Hi Marek,

     

    The stack trace is indicating an authentication error. I did notice on the integration page for theOML product that it mentions

     

    "If you are using a user other than AlarmPointoOMU, you will find a mistake in the documentation on Pg. 32.   The name of the configuration file that needs to be changed is hpomu.js, not hp_operations_manager_unix_int.xml.  A new version of the integration manual will be released shortly with this update"

     

    Is the password set in the hpomu.js file?

     

    Rod

     

     

    • Re: OML 9.10 integration - unable to validate credentials
      mkrauz Novice

      Thank you Rod for your response.

       

      Yes, I have updated hpomu.js with correct user name and password.

       

      This fails even before hpomu.js is used when running AP-OMU-Bridge with the --validate parameter.

       

      OML9.10 the database is remote.

       

      Any idea if that affects the AlarmPoint integration somehow?

       

      Thanks

      • Re: OML 9.10 integration - unable to validate credentials
        amagi Novice

        Hi Marek,

         

        Can you also run the other test listed in the documentation and let us know what that returns.

         

        <IAHOME>/bin/AP-OMU-Bridge -u AlarmPointOMU -p AlarmPointOMU -e /opt/OV/bin/OpC/call_sqlplus.sh --count-active-messages

         

        You mentioned that you have LDAP configured for OML, did you have it configured for OMU authentication when you were using AlarmPoint on that system?

         

        Thanks

        • Re: OML 9.10 integration - unable to validate credentials
          mkrauz Novice

          Hi Aaron,

           

          Thanks for looking into this.

           

          I did run the other test and it is successful no matter what password I use. I think it's because call_sqlplus.sh calls opcdbpwd, which uses opc_op user's password.

           

          I tried using AlarmPoint with LDAP enabled for OMU authentication and without it.

           

          Thanks,

          • Re: OML 9.10 integration - unable to validate credentials
            amagi Novice

            Hi Marek,

             

            That is right that it does not use the user credentials, I wanted to verify it was not an issue with the binary.

             

            In hpomu.js can you find

             

            function annotateOnSubmission(annotation, incidentId)

             

            Can you comment out the following line:

              //wrapper.validateCredentials();

             

            After that please try submiting a new message using OML. 

             

            You test message is it going into xMatters even with this error in the IA logs?  The annotate on submission is sending a annotation back to OML but trying to validate the credentials first.  This is not necessarily needed as I believe the 2way part of the integration does not utalize the omu user and uses the opc_op user.

             

            Aaron

            • Re: OML 9.10 integration - unable to validate credentials
              mkrauz Novice

              There are 2  lines with "wrapper.validateCredentials();"; one under

              function annotateOnSubmission(annotation, incidentId) and one under function handleSend(apxml)

               

              I have commented both lines and now I get "Access denied" error:

               

              Caused by: com.alarmpoint.integration.hpomu.OMUException: Error running program

              Output:

                  opcanno_add() = -44: Access denied

              Exit Value: 255

                    at com.alarmpoint.integration.hpomu.HPOMUInteractionWrapper.addAnnotation(HPOMUInteractionWrapper.java:252)

               

              The alert doesn't get to AlarmPoint.

              • Re: OML 9.10 integration - unable to validate credentials
                mkrauz Novice

                I also noticed that I had commented out this section in hp_operations_manager_unix.xml. I have done that when troublehooting this issue.

                 

                <classpath>

                      <path>libs/hpomu-ia.jar</path>

                   </classpath>

                 

                 

                It was causing an error: 

                 

                ServiceNotFoundException: There are no Integration Services in the Event Domain hp_operations_manager_unix.

                 

                I have now removed the comments around that section and now I get these errors:

                 

                 

                 

                Caused by: com.alarmpoint.integrationagent.exceptions.ServiceInvocationException: The Integration Service (hp_operations_manager_unix,HPOMU) terminated the request because of an unhandled exception.

                      at com.alarmpoint.integrationagent.services.ServiceProxyImpl.processRequest(ServiceProxyImpl.java:511)

                      at com.alarmpoint.integrationagent.services.ServiceInterceptor.intercept(ServiceInterceptor.java:56)

                      ... 17 more

                Caused by: com.alarmpoint.integration.hpomu.OMUException: Error running program

                Output:

                    Cannot connect to OM-U

                    = -53: Cannot connect to database

                Exit Value: 5

                      at com.alarmpoint.integration.hpomu.HPOMUInteractionWrapper.addAnnotation(HPOMUInteractionWrapper.java:252)

                 

                 

                 

                 

                Caused by: com.alarmpoint.integrationagent.exceptions.ServiceInvocationException: The Integration Service (hp_operations_manager_unix,HPOMU) terminated the request because of an unhandled exception.

                      at com.alarmpoint.integrationagent.services.ServiceProxyImpl.processRequest(ServiceProxyImpl.java:511)

                      at com.alarmpoint.integrationagent.services.ServiceInterceptor.intercept(ServiceInterceptor.java:56)

                      ... 17 more

                Caused by: com.alarmpoint.integration.hpomu.OMUException: Error running program

                Output:

                    opcanno_add() = -44: Access denied

                Exit Value: 255

                      at com.alarmpoint.integration.hpomu.HPOMUInteractionWrapper.addAnnotation(HPOMUInteractionWrapper.java:252)

          • Re: OML 9.10 integration - unable to validate credentials
            amagi Novice

            Marek,

             

            Also, are you able to use another OM user for the validate credentials command line test that works?  I wonder if the script that creates the user in OML has the AlarmPointOMU user missconfigured.

             

            Aaron

            • Re: OML 9.10 integration - unable to validate credentials
              mkrauz Novice

              Yes, I have first renamed the AlarmPointOMU user account in OMU to the account name that is in LDAP and then I created completely new AlarmPointOMU user account in OMU.

               

              Both user accounts are configured to give access to all messages.

               

              Both fail with LDAP authentication enabled and disabled.

               

              Thanks,

              • Re: OML 9.10 integration - unable to validate credentials
                amagi Novice

                Hi Marek,

                 

                Can you get the validation to work with any of the OM admins users?

                 

                Thanks,

                 

                Aaron

                • Re: OML 9.10 integration - unable to validate credentials
                  mkrauz Novice

                  I tried different omu users even the admin users.

                  • Re: OML 9.10 integration - unable to validate credentials
                    amagi Novice

                    This may be an issue with the permission on the files assocaited to run these commands.  I know the install script does some changes to the file permissions so it may be useful to make sure these were completed on Linux. 

                    • Re: OML 9.10 integration - unable to validate credentials
                      mkrauz Novice

                      These are the lines from the install script relating to file ownership and permissions:

                       

                      chmod ug+rx ${OPC_CONF_DIR}/mgmt_sv/reports/C/*_messages.sql
                      chown ${AP_USER}:${AP_GROUP} ${OPC_CONF_DIR}/mgmt_sv/reports/C/*_messages.sql

                      # Adjust permissions on Integration Agent installation
                      chown -R ${AP_USER}:${AP_GROUP} ${AP_IA_DIR}
                      find ${AP_IA_DIR}/bin/ -type d | xargs chmod a+rx
                      find ${AP_IA_DIR}/bin/ -name 'APClient.bin' | xargs chmod a+rx
                      find ${AP_IA_DIR}/bin/ -name 'AP-OMU-Bridge' | xargs chmod a+rx
                      find ${AP_IA_DIR}/integrationservices/hpomu/ -name 'omu_alarmpoint.sh' | xargs chmod a+rx
                      chmod 775 ${AP_IA_DIR}/log

                       

                      I have verified the permissions based on the commands in the install script and they are all good.

                       

                      One thing I noticed is that when running "AP-OMU-Bridge -u AlarmPointOMU -p AlarmPointOMU -e /opt/OV/bin/OpC/call_sqlplus.sh --count-active-messages" under user xmatters, I get this error after I changed the permission on sqlplus utility to allow execution by others:

                       

                      ERROR:   Error occurred calling SQL*Plus.

                      ERROR:   Report /etc/opt/OV/share/conf/OpC/mgmt_sv/reports/C/count_active_messages.sql failed.

                      Total:ERROR:

                       

                      If I run the same command under user root, it works fine.

                       

                      Any other binaries that would need additional permissions?

                      • Re: OML 9.10 integration - unable to validate credentials
                        amagi Novice

                        Marek when you do the validation via the Bridge under root does that also work?

                         

                        Thanks,

                         

                        Aaron

                        • Re: OML 9.10 integration - unable to validate credentials
                          mkrauz Novice

                          Good point...

                           

                          The command executes without any errors under both xmatters and root users and it gives ValidCredentials:true when running without LDAP authentication under user root.

                           

                          When LDAP authentication is enabled it runs but doens't generate any output under user root.

                           

                          It always gives ValidCredentials:false when running under user xmatters with or without LDAP authentication.

                          • Re: OML 9.10 integration - unable to validate credentials
                            mkrauz Novice

                            Thank you for your help so far,

                             

                            It looks like we are getting closer to narrow down the root cause.

                             

                            I have re-compiled the AP-OMU-Bridge using the provided source files and the result is the same as described in the previous post.

                             

                            It appears there are 2 issues here. One relating to permissions and one to LDAP.

                             

                            Any idea which files to check for permissions and if the bridge supports OMU PAM (HP's implementation of LDAP integration)?

                          • Re: OML 9.10 integration - unable to validate credentials
                            amagi Novice

                            Hi Marek,

                             

                            I am a little unclear, you said "The command executes without any errors under both xmatters and root users" but then you said "It always gives ValidCredentials:false when running under user xmatters with or without LDAP authentication."  By the command are you refering to calling the Bridge with --count-active-messages?

                             

                            Can you try using your personal user or creating a new user in OMU and try to use them for ValidCredentials.  I want to know if the script just incorrectly created the xMatters user within OMU.

                             

                            Thanks

                      • Re: OML 9.10 integration - unable to validate credentials
                        dhowell xMatters Employee

                        Marek,

                         

                        One thing I noticed is that when running "AP-OMU-Bridge -u AlarmPointOMU -p AlarmPointOMU -e /opt/OV/bin/OpC/call_sqlplus.sh --count-active-messages" under user xmatters, I get this error after I changed the permission on sqlplus utility to allow execution by others:

                         

                        ERROR:   Error occurred calling SQL*Plus.

                        ERROR:   Report /etc/opt/OV/share/conf/OpC/mgmt_sv/reports/C/count_active_messages.sql failed.

                        Total:ERROR:


                         

                        In my copy of call_sqlplus.sh, the "Error occurred calling SQL*Plus" message is only generated here:

                         

                        call_sqlplus_report()

                        {

                          # Create report, remove trailing blanks

                          RET_FILE=/tmp/ret.$$

                          (${OPCDBPWD} -r "$1" -s; echo $? > ${RET_FILE})|expand|sed -e 's/ *$//g'

                          sleep 1

                          if [ "`cat ${RET_FILE}`" -ne 0 ]

                          then

                            print_msg E2 "Error occurred calling SQL*Plus."

                            rm -f ${RET_FILE}

                            return 1

                          fi

                         

                          rm -f ${RET_FILE}

                          return 0

                        }

                         

                        Will the xmatters OS user have permission to create a file in /tmp?

                         

                        Dave

                        • Re: OML 9.10 integration - unable to validate credentials
                          dhowell xMatters Employee

                          Also, it looks to me as thought the call_sqlplus.sh script is actually using /opt/OV/bin/OpC/opcdbpwd to run the report, so the xmatters OS user would need execute permission on it as well.

                          • Re: OML 9.10 integration - unable to validate credentials
                            mkrauz Novice

                            xmatters user can create files in /tmp and does have execute permission to opcdbpwd through user group opcgrp.

                             

                            When I tried running opcdbpwd as user xmatters, I got error:

                             

                            Error opcdbpwd (OpC DB password tool)(16813) : You must be superuser  to run this service. (OpC20-20)

                             

                             

                            I also tried executing call_sqlplush.sh directly:

                             

                            /opt/OV/bin/OpC/call_sqlplus.sh count_active_messages AlarmPointOMU

                             

                            I got these results under user xmatters:

                             

                            ERROR:
                            ORA-12154: TNS:could not resolve the connect identifier specified


                            SP2-0306: Invalid option.
                            Usage: CONN[ECT] [{logon|/|proxy} [AS {SYSDBA|SYSOPER|SYSASM}] [edition=value]]
                            where <logon> ::= <username>[/<password>][@<connect_identifier>]
                                  <proxy> ::= <proxyuser>[<username>][/<password>][@<connect_identifier>]
                            SP2-0306: Invalid option.
                            Usage: CONN[ECT] [{logon|/|proxy} [AS {SYSDBA|SYSOPER|SYSASM}] [edition=value]]
                            where <logon> ::= <username>[/<password>][@<connect_identifier>]
                                  <proxy> ::= <proxyuser>[<username>][/<password>][@<connect_identifier>]
                            SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus
                            opcdbpwd: sqlplus exited with exit code 1.
                            ERROR:   Error occurred calling SQL*Plus.
                            ERROR:   Report /etc/opt/OV/share/conf/OpC/mgmt_sv/reports/C/count_active_messages.sql failed.

                             

                             

                            I gave read permission to others on tnsnames.ora and now call_sqlplus.sh works fine standalone and through AP-OMU-Bridge

                             

                            --validate also works under user xmatters now, but only if LDAP is disabled.

                             

                            We made a great progress. Now we need to find out why it doesn't validate when LDAP is enabled.

  • Re: OML 9.10 integration - unable to validate credentials
    denali Zero

    We have the integration working with LDAP enabled.  A couple of things we've found:

     

    - the AlarmPointOMU user must be a system account on Linux (UID < 500)

    - the opcanno_add() error occurs if the AlarmPointOMU user does not have the message group for that particular message assigned in the OML responsibilities matrix

    • Re: OML 9.10 integration - unable to validate credentials
      mkrauz Novice

      Thank Alan for your post.

       

      I just got the bridge to validate the credentials with LDAP authentication enabled by recompiling AP-OMU-Bridge with these additional parameters in Makef.AP-OMU-Bridge.rhel55: "-lkrb5 -lldap -llber"

       

      e.g:

       

      OPCLIB=-lopcsv_r -lopcdb -lnspsv -lkrb5 -lldap -llber

       

      To compile I executed:

      # make -f Makef.AP-OMU-Bridge.rhel55

       

       

      Our AlarmPointOMU account is in AD and we do not used PAM for system authentication.

       

      In regards to opcanno_add(), I have seen it in the past on our OMU 8 integration for a message from an external node. We had to add that node manually to OMU and assign it to a nodegroup.

       

      In this case the error is coming from our OMU 8 integration agent when OML9 message with notification flag is forwarded to OMU8. It works fine when OMU8 message is forwarded to OML9 (both servers forward a notification to AlarmPoint). We are in a process of migrating tom omu8 to oml9 and thus need to have both servers integrated with AlarmPoint. Now,both servers are set as MSGCONTROLLINGMGR in msgforw.

    • Re: OML 9.10 integration - unable to validate credentials
      mkrauz Novice

      Hi Alan,

       

      I did revisit the responsibility matrix of user AlarmPointOMU and it was missing some node group/message group assignments. I no longer have the Access denided errors.

       

      Thanks.

Actions

More Like This

Retrieving data ...

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points