stuck threads

Hogging threads are candidates for stuck threads.

Threads that “might” get stuck. These threads will be declared “stuck” after StuckThreadMaxTimeout seconds which usually is 600secs.

If the thread gets released by the request before this timeout, it wont be called hogging thread anymore and will be released to the thread pool.

Hogging Threads that have taken the too much time and we can assume that they are never going to come back.
Hogging threads help us take some decisions, lets say many threads are hogging, we may take a decision to create new threads for next cycle.

Understanding about Thread States in WebLogic Server:
ACTIVE
STUCK
STANDBY

A live thread which is ready to process the request, which is known as ACTIVE state. That is indicated when thread newly created. WebLogic Server start the server instance with 1 ACTIVE thread and the thread count grows as per the min size if specified other wise it will do self-tune as per the request.

Threads might wait for other thread to release resource. This might happen due to application variables. The variables are 2 types thread-safe other is risk for thread. All local variables in the methods are thread-safe. The variable defined in class level are unsafe. which causes memory leak, this state of threads are known as hogging. WebLogic identify a thread as hog by the time interval. If thread is waiting more than 600 sec will be treated as hog. STUCK thread interval we can tune as per the project need.
If the number of HoggingThreadCount increases then the server health is in dangerous. That time you can take the ThreadDump
After Threads increase to a max utilization then the thread will be in STANDBY state.

WebLogic Server Health Status can be one of the following:

HEALTH_OK
HEALTH_WARN
HEALTH_FAILED
HEALTH_CRITICAL
LOW_MEMORY_REASON
HEALTH_OVERLOADED

OK is indicates everything fine, no worries!!

WARN raised when there is few stuck threads in the server instance.

LOW_MEMORY_REASON is going to tell you about JVM crash expected. You can configure to ‘Exit’ the managed server on low memory conditions with the help of NodeManager and WorkManager.

CRITICAL when multiple number of stuck threads happening and the threadpool count reaching unsual number. This case you need to suspect Network, JDBC or back-end connectivity has trouble.

FAILED happen when the new deployments fails. The NodeManager should not restart this managed server.

OVERLOADED Change the server health state to OVERLOADED on overload. The Nodemanager need to work at this state and bounce such WebLogic instance. This is a new feature of WebLogic 9.x and later versions, for detecting, avoiding and recovering from an overload condition of a WebLogic managed server. Overload protection can be used to throttle Work Managers and thread pools for performance. You can configure Shutdown the Work Manager or application on stuck threads when it crosss more than 5 or you can set threshold.

Server Monitoring script

def reportDomainHealth(usern, passw, url):
print ""
print "===================================================================================="
connect(usern,passw,url)
domainRuntime()

print "Found Servers: "
serverList=ls('ServerRuntimes');
serverList=serverList.split()
print "%15s %15s %20s %15s %15s %40s" % ("Server","Threads","HoggingThreads","ServerState","Heap_Free","HealthState ")
print "----------------------------------------------------------------------------------------------------------------------------"
for i in range(len(serverList)):
if serverList[i] != 'dr--':
server_st=get('ServerRuntimes/' + serverList[i] + '/HealthState')
server_tc=get('ServerRuntimes/' + serverList[i] + '/ThreadPoolRuntime/ThreadPoolRuntime/ExecuteThreadTotalCount')
server_hog=get('ServerRuntimes/' + serverList[i] + '/ThreadPoolRuntime/ThreadPoolRuntime/HoggingThreadCount')
server_ql=get('ServerRuntimes/' + serverList[i] + '/State')
server_hpfp=get('ServerRuntimes/' + serverList[i] + '/JVMRuntime/' + serverList[i] + '/HeapFreePercent')
print "%15s %15s %20s %15s %15s %40s" % (serverList[i],str(server_tc),str(server_hog),str(server_ql),str(server_hpfp)+"%",str(server_st))
print "===================================================================================="


reportDomainHealth('weblogic','weblogic1','t3://localhost:7001')

delete files before n days in the folder

find * -mtime +n -exec rm {} \;

n=1 before 1 day

n=2 before 2 days

n=7 before 7 days

Vi Search and Replace Commands

Vi: Search and Replace

Change to normal mode with <ESC>.

Search (Wrapped around at end of file):

  Search STRING forward :   / STRING.

  Search STRING backward:   ? STRING.

  Repeat search:   n

  Repeat search in opposite direction:  N  (SHIFT-n)

Replace: Same as with sed, Replace OLD with NEW:

 First occurrence on current line:      :s/OLD/NEW

  

 Globally (all) on current line:        :s/OLD/NEW/g 

 Between two lines #,#:                 :#,#s/OLD/NEW/g

  

 Every occurrence in file:              :%s/OLD/NEW/g 

Time out while waiting for a managed process to stop HTTP_Server

when some times we found OHS server in stopped mode.

opmnctl startall: starting opmn and all managed processes...

oracle@localhost [/l01/apps/oracle/middleware/Oracle_WT1/instances/ohs1/bin] opmnctl status

Processes in Instance: ohs1

---------------------------------+--------------------+---------+---------

ias-component                    | process-type       |     pid | status

---------------------------------+--------------------+---------+---------

ohs1                             | OHS                |   26508 | Stop

When we normally start this process we will get 

oracle@localhost [/l01/apps/oracle/middleware/Oracle_WT1/instances/ohs1/bin] opmnctl startproc ias-component=ohs1

opmnctl startproc: starting opmn managed processes...

================================================================================

opmn id=uslx148:6701

  0 of 0 processes started.

  Processes are already started: ohs1~ohs1~OHS~OHS

When we have face these problems we need to follow the below process 

step1:

ps -ef | grep ohs

Step2:

kill all realted processes 

kill -9 process id 

Step3:

oracle@localhost [/l01/apps/oracle/middleware/Oracle_WT1/instances/ohs1/bin] opmnctl startall

opmnctl startall: starting opmn and all managed processes...

oracle@localhost [/l01/apps/oracle/middleware/Oracle_WT1/instances/ohs1/bin] opmnctl status

Find the Port number with process id

oracle@uslx296 [/home/oracle] netstat -pl | grep 14452

tcp        0      0 10.0.0.6:8011 *:*                         LISTEN      14452/java

netstat -pl | grep NAME_OR_PID

RSA in security

RSA is one of the first practicable public-key cryptosystems and is widely used for secure data transmission. In such a cryptosystem, the encryption keyis public and differs from the decryption key which is kept secret. In RSA, this asymmetry is based on the practical difficulty of factoring the product of two large prime numbers

remove last 70 characters in each line in the file

 rev qa1_test.txt | cut -c 70- | rev > newqa1_.txt

Weblogic 12c New JDBC future

WebLogic 12.1.3 will come with the JDBC 12c driver, this driver will gives us better integration and reliability between WebLogic 12c and Oracle Database 12c (Application Continuity).

When we take a look at the administration side of WebLogic we can see that the Enterprise Manager will be more important, in WebLogic 12.1.3 we can also do Application and JMS administration from the EM application. Also 12.1.3 will contain a Restful management APIs for additional monitoring, operations, Datasource and deployment support.  Plus a global OWSM policy to protect all your Web and Rest services with one server policy.

WebLogic 12.1.4 will have a new feature called Multi-Tenant Applications, this way you can define an WebLogic template for an application, so one or more customers of this application which will have its own Cluster, Managed Servers, Application and (not shared) resources plus it will support Oracle Database 12c pluggable databases.

Weblogic 12c Dynamic Clustering

WebLogic 12.1.3 will be the first version for many FMW 12c products like Oracle SOA Suite 12c and probably come in one big jar. 12.1.3 & 12.1.4 will add extra features and improvements to Elastic JMS & Dynamic Clusters. Elastic JMS in 12.1.3 will support Server Migration so you can’t lose any JMS messages.

In 12.1.4, Dynamic Clusters will have support for auto-scaling based on thresholds based on user-defined metrics. WebLogic 12.1.4 will also have an API to control the Dynamic Clusters, this way we can easily program when to stop, start or remove nodes from a dynamic cluster.

Weblogic Full RoadMap

Weblogic 12C history

WebLogic 10.3.6 is still the latest 11g version but Oracle will support 10.3.6 till 2018 and extended support till 2021. So Oracle’s Fusion Apps and we have enough time to migrate to WebLogic 12.1.X. Oracle also promised that the upgrade should be easy. That being said we can take look at the WebLogic 12.1.X features. 

Jun 2013 Oracle already released WebLogic 12.1.2 which has since WebLogic 12.1.1 been certified for Java EE 6 and it looks like the Java EE 7 certification is still far away, so Oracle updated the 12.1.2 version with some badly needed frameworks like WebSockets. To make the developer experience more complete Oracle added more support for Maven and it comes with a utility to synchronize a Maven repository with all the needed WebLogic libraries.

12.1.2 is also the first release, which comes with Fusion Middleware infrastructure components. For now FMW 12.1.2 contains ADF & OWSM and comes with Enterprise Manager & MDS.

WebLogic 12.1.2 replaced the BEA installer and the BSU patching utilities with the Oracle Universal Installer and the OPatch utilities for applying patches.  To make it even more easier it just comes in one taste ( no Java included or a specific Operating System installer ).  Just one big jar file for WebLogic with Coherence or one for WebLogic, Coherence and FMW.

WebLogic 12.1.2 introduced Dynamic Clusters, Elastic JMS and integrated Coherence configuration and management. Dynamic Clusters is a great feature to extend the WebLogic Cluster with Managed Servers based on a Server Template. For this we only need to change a parameter on the cluster and the new Managed Servers are distributed over the NodeManagers. To use this feature your blades need to have enough free resources to handle the extra Managed Servers.

Elastic JMS can be combined with Dynamic Clusters, this way we only need to create 1 JMS server, target this to the cluster and every Cluster node will have its own JMS server.

When we look at the announced features of WebLogic 12.1.3 and 12.1.4 we can see that Oracle continues on this road.

WebLogic 12.1.2 Futures

WebLogic 12.1.2 Futures

1. Dynamic Cluster Support:

It is one of the most striking features of WebLogic 12.1.2. This release introduces use of the dynamic cluster for a highly scalable systems, high availability environment.  Dynamic cluster allows the managed servers to be automatically added based on a server template.

2. Support with Oracle Database 12c.

3. JMS enhancements:

Supports clustered targeted JMS Servers for providing high availability  eliminating the need to configure many JMS resources for every single server.

4. Enhanced maven support.

Support for POMs, Maven 3.0.4.

5. WebSocket support.

Supports for WebSocket Protocol (RFC 6455),which provides two-way, full-duplex communication over a single TCP connection between clients and servers, where each side can send data independently from the other. The WebSockets communication model occurs in real-time and promotes user interaction.

find out top 10 largest file/directories

To find  top 10 largest file/directories

du -a /opt/middleware | sort -n -r | head -n 10

 more human readable output try:
$ du -hsx * | sort -rh | head -10

delete 100 lines forward from (including) the current one

delete 100 lines forward from (including) the current one

repeat dd (delete current line) 100 times:

100dd
delete from current line to 99 lines forward

d99j
delete 100 lines backwards from (including) the current one

d99k
delete lines in a specific range by line number

:1,100d
delete lines in a range beginning with the current lin

Alert for diskspace in Weblogic Environemnts

#!/bin/bash

#admin email account

ADMIN="madhuchilipi@gmail.com"

# set usage alert threshold

THRESHOLD=10

#hostname

HOSTNAME=$(hostname)

#mail client

MAIL=/usr/bin/mail

# store all disk info here

EMAIL="/l01/apps/oracle"

for line in $(df -hP | egrep '^/l01/apps/oracle' | awk '{ print $6 "_:_" $5 }')

do

part=$(echo "$line" | awk -F"_:_" '{ print $1 }')

part_usage=$(echo "$line" | awk -F"_:_" '{ print $2 }' | cut -d'%' -f1 )

if [ $part_usage -ge $THRESHOLD -a -z "$EMAIL" ];

then

EMAIL="$(date): Running out of diskspace on $HOSTNAME\n"

EMAIL="$EMAIL\n$part ($part_usage%) >= (Threshold = $THRESHOLD%)"

elif [ $part_usage -ge $THRESHOLD ];

then

EMAIL="$EMAIL\n$part ($part_usage%) >= (Threshold = $THRESHOLD%)"

fi

done 

if [ -n "$EMAIL" ];

then 

echo -e "$EMAIL" | $MAIL -s "Alert: Partition(s) almost out of diskspace on $HOSTNAME" "$ADMIN"

fi

em console access denied Access denied. Required roles: Admin, Operator, Monitor, executing subject: principals=[].

em console access denied Access denied. Required roles: Admin, Operator, Monitor, executing subject: principals=[].

Not able to see all composites in em console

Fix for EM account issues

Open em Navigate to Security à Application Roles

 Click on Search Button.

  All groups will be displayed. Select SOAMointer group and click on Edit button.

  

 Click on Add Button in the following screen.

 Select Type as “Group” and Click on Search buttion.All groups will be displayed like below.

Add AddTesters group for SOAMoniter role.

 Repeat the same steps and add IntegrationMoniters group to SOAMoniter role.

Get Connection Pool Information through WLST

The below script will get information about JNDI,JDBC URL 

###################****##############****########################  
# Generic script applicable on any Operating Environments (Unix, Windows)  
# ScriptName    : togetConnectionpoolURL.py  
# Author        : Madhusudhana Chilipi  
###############     Connecting to Start     #################################  

connect('weblogic','welcome1','t3://192.168.1.2:7001')


###############     Changing the Datasources     #################################  

allJDBCResources = cmo.getJDBCSystemResources()
for jdbcResource in allJDBCResources:
   dsname = jdbcResource.getName()
   print dsname, jdbcResource.getJDBCResource().getJDBCDataSourceParams().getJNDINames()[0], jdbcResource.getJDBCResource().getJDBCDriverParams().getUrl() 

###############     End    ################################# 

Spotfire Introduction

What is TIBCO Spotfire?

Spotfire originally a business intelligence company based in Somerville, Massachusetts was brought by TIBCO in 2007.

Tibco Spotfire allows customers to analyze the data using predictive and complex statistics in the analysis. It has customers from a variety of industries ranging from Life sciences, Energy, Financial Services, Government, Healthcare and many others

Why using Spotfire.

Spotfire comes at Business Intelligence through visualization. It uses in-memory processing and good user interface design to develop highly interactive displays of data. Business intelligence reporting is intended to improve business and financial analytics. Spotfire's interactive and highly visual analytical environment helps achieve this with a self-configuring visual data analysis environment that lets users query, visualize and explore data in real time. 

Users can choose and manipulate a wide variety of visual representations, including map-based data displays to review business intelligence reporting results geographically, multidimensional scatter plots to view data statistically, and bar charts, pie charts, line graphs, profile charts and more.

SPOTFIRE ARCHITECTURE

OBIEE Introuduction

What is OBIEE?

Oracle Business Intelligence Enterprise Edition 11g (OBIEE) is a comprehensive business intelligence platform that delivers a full range of capabilities - including interactive dashboards, ad hoc queries, notifications and alerts, enterprise and financial reporting, scorecard and strategy management, business process invocation, search and collaboration, mobile, integrated systems management and more.

OBIEE (Oracle Business Intelligence Enterprise Edition)

Business Intelligence? (BI) is an application for gathering, storing, analyzing, and providing access to data to help enterprise

users make better business decisions

its includes the decision support system, activities of OLAP(online analytical processing),analytics, data mining, datawarehose, reporting

Business Intelligence is a tool used for querying the analysis, reporting using the data, developing reports and dashboards for decision support system.

Why  we are using  OBIEE?

1.      Consolidate and standardize BI tools :Consolidated view of 'Projects' including data and CEIM which is Common Enterprise Information Model that enables user self-service and ensures consistent, accurate information by allowing you to model once and deploy anywhere.

2.      Unified BI metadata and infrastructure

3.      User Interact with simplified Model

4.      Seamless BI user experience

5.      Real-Time and Historical data

6.      Faster Development

7.      Role based Dash board Development

8.      Schedule & distribute enterprise-class reports

OBIEE 11g Architecture