Setting Alarm by using Autosys job and tool handling for alarm

By -

Alarms are automated notifications triggered when a system detects an abnormal condition that requires human attention to resolve. Unlike automated recovery processes, alarms highlight scenarios where:

Manual intervention is necessary (e.g., decision-making, troubleshooting).

Automated fixes are not possible (e.g., missing dependencies, corrupted data).

Timely action is critical to prevent cascading failures.

Key Characteristics of Such Alarm

Event-Driven

Triggered by specific conditions (e.g., a file not arriving on time, a job failing repeatedly).

Example: A scheduled ETL job fails because an expected input file is missing.

Requires Human Judgment

The system cannot auto-resolve (e.g., deciding whether to proceed with incomplete data or wait).

Example: A payment processing system flags a transaction for fraud review.

Escalation Mechanisms

If unacknowledged, alarms escalate (e.g., email → SMS → phone call → on-call engineer).

Example: A server disk reaching 95% capacity triggers an alert to IT ops.

Prioritized by Severity

Critical (Red): Immediate action needed (e.g., production outage).


Warning (Yellow): Needs investigation but not urgent (e.g., delayed file arrival).


Informational (Blue): Logged for awareness (e.g., job completed with warnings).


Example Scenario: Missing File Alarm

Situation

A scheduled batch job depends on an input file expected by 2:00 AM.

By 2:30 AM, the file hasn’t arrived, and the job fails to start.

Alarm Behavior

Detection: Monitoring tool identifies the file is overdue.
Notification: Sends an alert (email/SMS) to the support team.

Manual Intervention Required:

  • Investigate: Check if the file was delayed or lost.

  • Decide: Proceed with a backup file, rerun the job later, or abort.

  • Resolve: Manually trigger the job or fix the upstream issue.

Why Automation Isn’t Enough

The system doesn’t know whether to:
Wait longer (network delay?).
Use an older file (is it acceptable?).
Skip the job (will downstream processes break?).
A human must assess and decide.

Best Practices for Such Alarms

Clear Alert Messaging

Include: 

What failed, Why it matters, Possible actions.

Avoid Alarm Fatigue

Only alert for true exceptions (not every minor delay).

Runbook Integration

Link alarms to troubleshooting guides (e.g., "Steps if a file is missing").

Post-Incident Review

Analyze if the alarm could be automated or prevented in the future.

Tools Handling Such Alarms:

IT Operations: PagerDuty, Opsgenie, Nagios
Data Pipelines: Apache Airflow (alerts on DAG failures)
Cloud Monitoring: AWS CloudWatch, Azure Monitor


Comments

Post a Comment

Popular posts from this blog

SOA Introduction

By -
Oracle SOA Suite Introduction Oracle SOA suite is a middleware tool that is used for integration, this tool helps two system to exchange information between each other. This tool is based upon Service Oriented Architecture. Versions till date We have three versions of this tool as mentioned below. 1. 10g 2. 11g 3. 12c 10g was the first version of Oracle SOA Suite tool when it was launched by Oracle and then few years later Oracle launched 11g version, there are few sub versions under 11g version also. Recently in 2014 Oracle has launched 12c version also. This tutorial is based on Oracle SOA Suite 11g version. Prerequisites You should know about below concepts for developers 1. XML 2. XSD (Schema) 3. WSDL 4. SOAP The following components comprise an Oracle SOA Suite installation: Service Infrastructure Oracle Mediator Oracle Adapters Business Events and Events Delivery Network Oracle Metadata Repository Oracle Business Rules  Oracle WSM Policy Mana...
Read more

Interview question for File and FTP Adapter

By -
1)  Why File Adapter is used in SOA Suite tool ?.  File adapter is used to read and write file from and to local system. 2)  Why FTP Adapter is used in SOA Suite tool ?. FTP adapter is similar to File adapter with one difference that this adapter is used to read/write the file from/to remote system. 3)  What all differences we have between File and FTP adapters.  Following are the major differences between File and FTP adapter. File Adapter 1. It is used to deal with local systems only. 2. Do not require any configuration to make this adapter work. FTP Adapter 1. It is used to deal with remote systems. 2. Outbound Connection Pool needs to be configured with FTP details to make this adapter work. 4)  File & FTP adapters are known and Transactional or Non-Transactional adapters?.  These adapters are known as non-transactional adapters as these adapters does not support transactions. 5)  Can we read flat files using these adapter...
Read more

SSL Exceptions in Admin Server and Node Manager.

By -
javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from oracle.test.com – xx.xxx.xx.xx. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected. <WARNING> <Uncaught exception in server handlerjavax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake>javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:849) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170) The above exceptions are  the most common exceptions encountered during the setup of Weblogic Server in an environment. The stack does suggest what could be the reasons but the diagnostics are not mentioned. To debug this issue, first we need...
Read more