Setting Alarm by using Autosys job and tool handling for alarm

By -

Alarms are automated notifications triggered when a system detects an abnormal condition that requires human attention to resolve. Unlike automated recovery processes, alarms highlight scenarios where:

Manual intervention is necessary (e.g., decision-making, troubleshooting).

Automated fixes are not possible (e.g., missing dependencies, corrupted data).

Timely action is critical to prevent cascading failures.

Key Characteristics of Such Alarm

Event-Driven

Triggered by specific conditions (e.g., a file not arriving on time, a job failing repeatedly).

Example: A scheduled ETL job fails because an expected input file is missing.

Requires Human Judgment

The system cannot auto-resolve (e.g., deciding whether to proceed with incomplete data or wait).

Example: A payment processing system flags a transaction for fraud review.

Escalation Mechanisms

If unacknowledged, alarms escalate (e.g., email → SMS → phone call → on-call engineer).

Example: A server disk reaching 95% capacity triggers an alert to IT ops.

Prioritized by Severity

Critical (Red): Immediate action needed (e.g., production outage).


Warning (Yellow): Needs investigation but not urgent (e.g., delayed file arrival).


Informational (Blue): Logged for awareness (e.g., job completed with warnings).


Example Scenario: Missing File Alarm

Situation

A scheduled batch job depends on an input file expected by 2:00 AM.

By 2:30 AM, the file hasn’t arrived, and the job fails to start.

Alarm Behavior

Detection: Monitoring tool identifies the file is overdue.
Notification: Sends an alert (email/SMS) to the support team.

Manual Intervention Required:

  • Investigate: Check if the file was delayed or lost.

  • Decide: Proceed with a backup file, rerun the job later, or abort.

  • Resolve: Manually trigger the job or fix the upstream issue.

Why Automation Isn’t Enough

The system doesn’t know whether to:
Wait longer (network delay?).
Use an older file (is it acceptable?).
Skip the job (will downstream processes break?).
A human must assess and decide.

Best Practices for Such Alarms

Clear Alert Messaging

Include: 

What failed, Why it matters, Possible actions.

Avoid Alarm Fatigue

Only alert for true exceptions (not every minor delay).

Runbook Integration

Link alarms to troubleshooting guides (e.g., "Steps if a file is missing").

Post-Incident Review

Analyze if the alarm could be automated or prevented in the future.

Tools Handling Such Alarms:

IT Operations: PagerDuty, Opsgenie, Nagios
Data Pipelines: Apache Airflow (alerts on DAG failures)
Cloud Monitoring: AWS CloudWatch, Azure Monitor


Comments

Post a Comment

Popular posts from this blog

Interview question for File and FTP Adapter

By -
1)  Why File Adapter is used in SOA Suite tool ?.  File adapter is used to read and write file from and to local system. 2)  Why FTP Adapter is used in SOA Suite tool ?. FTP adapter is similar to File adapter with one difference that this adapter is used to read/write the file from/to remote system. 3)  What all differences we have between File and FTP adapters.  Following are the major differences between File and FTP adapter. File Adapter 1. It is used to deal with local systems only. 2. Do not require any configuration to make this adapter work. FTP Adapter 1. It is used to deal with remote systems. 2. Outbound Connection Pool needs to be configured with FTP details to make this adapter work. 4)  File & FTP adapters are known and Transactional or Non-Transactional adapters?.  These adapters are known as non-transactional adapters as these adapters does not support transactions. 5)  Can we read flat files using these adapter...
Read more

What is boot.properties file and how to create

By -
Boot.properties is a file, it containing  weblogic server username and password . If your server is in production mode then only you have to create manually this boot.properties file. If your server is in Development then you no need to create this file because server it self create this file while creating Domain. A boot identity file contains the user credentials for starting and stopping an instance of WebLogic Server. An administration server or managed server can refer to this file for user credentials instead of prompting at the command line to provide them. To Create boot.properties: Go to below location Win :   C:\Oracle\Middleware\user_projects\domains\test_domain\servers\AdminServer\ Linux : Oracle\Middleware\user_projects\domains\test_domain\servers\AdminServer\ 1.First you have to create security folder under AdminServer 2.Create boot.properties file under security folder. 3.Open notepad and write the below two lines username=********** password=*****...
Read more

SSL Exceptions in Admin Server and Node Manager.

By -
javax.net.ssl.SSLKeyException: [Security:090482]BAD_CERTIFICATE alert was received from oracle.test.com – xx.xxx.xx.xx. Check the peer to determine why it rejected the certificate chain (trusted CA configuration, hostname verification). SSL debug tracing may be required to determine the exact reason the certificate was rejected. <WARNING> <Uncaught exception in server handlerjavax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake>javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:849) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1170) The above exceptions are  the most common exceptions encountered during the setup of Weblogic Server in an environment. The stack does suggest what could be the reasons but the diagnostics are not mentioned. To debug this issue, first we need...
Read more