Weblogic Dynamic Cluster configuration

This is a new feature available in Weblogic 12c (12.1.2).
Dynamic cluster contains one or more dynamic servers. It allows you to add a new server dynamically at peak loads.  This would eliminate the need of adding the managed server manually to a cluster.  The businesses can define number of servers that needs to be added dynamically anticipating the peak loads.
The concept of dynamic cluster is based on a single shared server template.
For further reading on server template please refer the below post.
Prerequisites:

1. Configured machine .

You can create the setup in two ways.

1. From Administration console
2. Using WLST

I. Using Administration console:
1. Login into the administration console  – >  Cluster  – > Click on new — > Select Dynamic Cluster from the drop down.

Weblogic Dynamic Cluster

2. Select the messaging mode and define number of servers you would need at the peak load.
You can either create a new server template or use an already existing one.

Dynamic Cluster-NoOfServers

3. Associate dynamic servers with machines.
Note: Associating dynamic servers with machines is essential if you intend to use the Node Manager and the Administration Console (or WLST) to start servers.

Weblogic Dynamic Cluster-Machine

4. Specify Listen port bindings:
You would need to define port bindings for only one dynamic server, and each subsequent server will be given an incremental port.

Weblogic Dynamic Cluster

weblogic Dynamic Cluster

Weblogic Dynamic Cluster Details

This would create a server template based on the dynamic cluster configuration, which can be reused.

Dynamic Cluster Server Template

Introducing Dynamic Clusters in WebLogic Server 12.1.2

Introducing Dynamic Clusters in WebLogic Server 12.1.2 

In WebLogic 12.1.2, we added Dynamic Clusters, which is a set of new cluster configuration options that make it really easy to initially roll out a new cluster, and then make it really easy to scale out that cluster.
With Dynamic Clusters, you define how many servers you want in the cluster and you define a server template to use as the base configuration for servers in the cluster. Machine mappings, listen addresses, and port mappings can be mapped auto-magically. You can also use the WebLogic Diagnostics Framework to drive automation around expanding a cluster based on any metric you choose.

These features rely on some new and extended MBeans and configuration objects:

As shown in this diagram, the Cluster MBean has a new child: DynamicServers. That MBean handles the configuration aspects of the cluster that enable the cluster to be expanded dynamically, including a link to the Server Template, machine mapping settings, Managed Server Name prefix, the number of Dynamic Servers to create, and so forth.
The number of Dynamic Servers that you define in the Dynamic Servers configuration spurs the system to create a ServerLifeCycleRuntime MBean, which is a lightweight bean that reserves a Managed Server name and provides methods for starting and stopping the servers with the Node Manager.
The Server Template is a new configuration object that can store the same configuration attributes that a Server configuration contains. A Server Template is used as the basis for other Managed Servers, both configured servers and Dynamic Servers. And when you make a change to a Server Template, all servers that are based on that template inherit the changes dynamically. That makes changing configuration much easier when you have a bunch of servers to update.
 At runtime, a cluster is still a cluster: the runtime behavior for clusters has not changed. It is only the configuration and scaling that we have made easier. And just to be clear: configured clusters are still fully available and supported, so your existing WLST scripts that you use to configure a cluster are still valid.

http://docs.oracle.com/middleware/1212/wls/CLUST/dynamic_clusters.htm

 

WLST script for Dynamic Cluster creation in weblgoic

# This example demonstrates the WLST commands needed to create a dynamic cluster#
(dynamic-cluster). The dynamic cluster utilizes a server template
# dynamic-cluster-server-template. To keep this example simple, error handling# was omitted.#
connect()
edit()
startEdit()
## Create the server template for the dynamic servers and set the attributes for# the dynamic servers. Setting the cluster is not required.#
dynamicServerTemplate=cmo.createServerTemplate("dynamic-cluster-server-template")
dynamicServerTemplate.setAcceptBacklog(2000)
dynamicServerTemplate.setAutoRestart(true)
dynamicServerTemplate.setRestartMax(10)
dynamicServerTemplate.setStartupTimeout(600)
## Create the dynamic cluster and set the dynamicservers.#
dynCluster=cmo.createCluster("dynamic-cluster")
dynServers=dynCluster.getDynamicServers()
dynServers.setMaximumDynamicServerCount(10)
dynServers.setServerTemplate(dynamicServerTemplate)
## Dynamic server names will be dynamic-server-1, dynamic-server-2, ...,# dynamic-server-10.#
dynServers.setServerNamePrefix("dynamic-server-")
## Listen ports and machines assignments will be calculated. Using a round-robin#
algorithm, servers will be assigned to machines with names that start with#dyn-machine.#
dynServers.setCalculatedMachineNames(true)dynServers.setMachineNameMatchExpression("dyn-machine*")
## activate the changes#
activate()

WebLogic Server 12c - PerDomain Node Manager Configuration Model

Let's start with giving a brief definition of Node Manager. Server instances in a WebLogic Server production environment are often distributed across multiple domains, machines, and geographic locations. Node Manager is a WebLogic Server utility that enables you to start, shut down, and restart server instances from a remote location. Node Manager is optional but it is recommended if your WebLogic Server environment hosts applications with high availability requirements.

WebLogic Server 12c has over 200 new features. One of these features is a new Node Manager configuration model. In WebLogic 12c, the Java version of Node Manager has a simplified, out-of-the-box configuration. Node Manager is moved from PerHost configuration to PerDomain configuration model. The Install Windows Service screen, which provided the option to install Node Manager as a service on Windows systems, is no longer included.

In WebLogic 12c for each domain you create, an instance of domain-specific Node Manager is created/configured by default. With this default configuration, using the security credentials supplied for the Administration Server, nm_password.properties is created in DOMAIN_HOME\config\nodemanager. We can use Configuration Wizard or WLST tools to create Domain/Node Manager configuration. We can select a PerDomain or CustomLocationJava-based Node Manager configuration model. The PerDomain configuration is exactly as described for the default configuration, except that it allows you to provide unique Node Manager credentials. If you want a seperate location for NodeManagerHome, select CustomLocation and specify an empty directory or select to create one. The resulting Node Manager also runs as a per domain process.

After the default installation, the nodemanager.properties and nodemanager.domains files are created for you under DOMAIN_HOME\nodemanager and domain-specific scripts to start, stop, install and uninstall Node Manager as a Windows service, are created under DOMAIN_HOME\bin. With the default Node Manager configuration, you cannot edit the NodeManagerHome location, in this case it will be DOMAIN_HOME\nodemanager.

Let's see the process step by step. I will use the configuration wizard tool to create a sample domain. After starting the wizard and proceeding for the first few steps you will be prompted to choose the components to modify/create. Here you can select Node Manager option to configure some of it's properties like Node Manager type, Node Manager username and password, and Node Manager location.

On the next screen we can modify the following properties;

Node Manager Type:

Per Domain: If you select this option, the Node Manager home is predefined within the domain as <domain_name>/nodemanager and you cannot edit the Node Manager home. 

Custom Location: Select this option if you want the Node Manager configuration files to be created in a specific location for this domain. Specify the directory in the Node Manager Home field, or click Browse to navigate to the location. The specified directory must be empty. The nodemanager.properties and nodemanager.domains files will be created in this directory. 

Manual Node Manager Setup: If you select this option, creation of the Node Manager configuration for the domain is skipped, and you must manually create and update the Node Manager configuration for the domain.

Node Manager Credentials: Here you enter the username and password for this Node Manager instance.

After the wizard exits, we will have the following folder structure and the Node Manager related configuration files. 

Please remember you can still use Per Host Node Manager (scripts are located in WL_HOME\server\bin). However there is extra step you must take to use this model. Below is the summary of this steps;

1. Create a nodemanager.domains file that specifies the domains that you want this Node Manager instance to control, under     ORACLE_HOME\oracle_common\common\nodemanager, the per host NodeManagerHome location.

2. Generate a certificate and private key to communicate with Node manager over SSL

analysing GC log using GC Viewer

This article is about analysing GC log using GC Viewer.
We can download the software from this link
http://www.tagtraum.com/gcviewer-download.html
In order to capture the GC information we need to start Weblogic Server by adding the following JAVA Option
set MEM_ARGS=-Xms1024m -Xmx1024m -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:gc.log
If you see Out Of Memory errors in you server logs, you should be ready with GC Viewer to anylyze the gc log file.
Out Of Memory occurs due to the exhaustion of the heap. This can be due to an overload condition or programming negligence.If its due to an overload condition, we need to increase the heap size depending on the load. If its due to programming errors, we need to take a deeper look. Take heapdumps and analyze using Memory Analyzer tools.
This has been described in the following article
TROUBLESHOOTING OUT OF MEMORY ERRORS IN WEBLOGIC USING ECLIPSE MEMORY ANANLYSER
There are two types of garbage collection, major and minor. In minor garbage collection only young generation is cleared where as major garbage collection cleans the tenured and the young generation.
e.g.
[GC [PSYoungGen: 40951K->3360K(42432K)] 564334K->526886K(566720K), 0.0720524 secs]
2274304.031: [GC [PSYoungGen: 41632K->3584K(42496K)] 565158K->527274K(566784K), 0.0665242 secs]
2274663.654: [GC [PSYoungGen: 41920K->3296K(42560K)] 565610K->527134K(566848K), 0.0636812 secs]
2275023.726: [GC [PSYoungGen: 41696K->3746K(42560K)] 565534K->527728K(566848K), 0.0642125 secs]
2275428.250: [GC [PSYoungGen: 42210K->3424K(42752K)] 566192K->527574K(567040K), 0.0637819 secs]
2275428.314: [Full GC [PSYoungGen: 3424K->2828K(42752K)] [PSOldGen: 524150K->524287K(524288K)] 527574K->527116K(567040K) [PSPermGen: 88758K->88758K(90112K)], 11.1667328 secs]
2275743.907: [Full GC [PSYoungGen: 38528K->5936K(42752K)] [PSOldGen: 524287K->524287K(524288K)] 562815K->530224K(567040K) [PSPermGen: 88891K->88891K(90112K)], 11.0188252 secs]
2276067.615: [Full GC [PSYoungGen: 38528K->5349K(42752K)] [PSOldGen: 524287K->524287K(524288K)] 562815K->529637K(567040K) [PSPermGen: 88956K->88956K(90112K)], 11.0896643 secs]
2276404.096: [Full GC [PSYoungGen: 38420K->0K(42752K)] [PSOldGen: 524287K->517865K(524288K)] 562707K->517865K(567040K) [PSPermGen: 88987K->88611K(90112K)], 14.6099121 secs]
2276764.208: [GC [PSYoungGen: 38350K->1890K(42752K)] 556215K->519755K(567040K), 0.0890990 secs]
2277126.468: [GC [PSYoungGen: 40482K->1932K(40640K)] 558347K->519797K(564928K), 0.0915124 secs]
Line starting with GC is minor GC and the line starting with Full GC is major GC.
If we take a closer look at a single Full GC
2275428.314: [Full GC [PSYoungGen: 3424K->2828K(42752K)] [PSOldGen: 524150K->524287K(524288K)] 527574K->527116K(567040K) [PSPermGen: 88758K->88758K(90112K)], 11.1667328 secs]
Heap in use in young generation befor GC is 3424K and after is 2828K, whereas the total heap is 42752K.
Similarly for tenured generation heap usage before GC is 88758K and after GC it is 88758K, and the time taken is 11.1667328 secs. The timestamp helps us keep an eye on the frequency of Garbage collection. From the frequency we can deduce whether its a leak or not.
If we see FUll GC happening continuously, its a leak and you can expect an Out Of Memory.
4364435.019: [Full GC [PSYoungGen: 43776K->42364K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->566652K(572544K) [PSPermGen: 88381K->88381K(90112K)], 12.2471219 secs]
4371857.025: [Full GC [PSYoungGen: 43776K->43776K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->568064K(572544K) [PSPermGen: 88381K->88381K(90112K)], 12.5911727 secs]
4371869.617: [Full GC [PSYoungGen: 43776K->42364K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->566652K(572544K) [PSPermGen: 88381K->88381K(90112K)], 12.9294566 secs]
4379280.165: [Full GC [PSYoungGen: 43776K->43776K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->568064K(572544K) [PSPermGen: 88384K->88384K(90112K)], 13.1517479 secs]
4379293.318: [Full GC [PSYoungGen: 43776K->42364K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->566652K(572544K) [PSPermGen: 88384K->88384K(90112K)], 13.0984445 secs]
4386653.185: [Full GC [PSYoungGen: 43776K->43776K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->568064K(572544K) [PSPermGen: 88384K->88384K(90112K)], 13.1889873 secs]
4386666.375: [Full GC [PSYoungGen: 43776K->42364K(48256K)] [PSOldGen: 524288K->524288K(524288K)] 568064K->566652K(572544K) [PSPermGen: 88384K->88384K(90112K)], 13.3019379 secs]
If the heap usage keeps on increasing with time even after FUll GC,it is a memory leak.
In GCViewer it should look something like this

The other thing that we look for is Throughput.(Click the summary tab)
It should ideally be above 97%.

WebLogic Server: "TOO MANY OPEN FILES" Exception

WebLogic Server: "TOO MANY OPEN FILES" Exception

In Solaris and Linux, the file descriptors parameter specifies the number of open files permitted per process. If this value is too low or your application requires too many files to be opened or there is a huge number of concurrent users then WebLogic may throw the "TOO MANY OPEN FILES" exception. Note that on UNIX systems, each socket connection to WebLogic Server consumes a file descriptor.

…
Caused by: java.io.IOException: java.io.IOException: error=24, Too many open files
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:148)
    at java.lang.ProcessImpl.start(ProcessImpl.java:65)
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453)
    ... 40 more

If you face this error, then you must increase the limit beyond the current value for the WebLogic. We must also ensure that the operating system is able to handle the increase in the number of open files.

On OS level we can use the following to check or increase the file descriptor limit;

$ ps –ef |grep java                     (Get  the WebLogic process id)
$ pfiles processId | grep rlimit        (Display the current file descriptor limit)
$ ulimit -H -n                             (Display hard limit, cannot be exceeded)
$ ulimit -S -n                             (Display soft limit may be increased up to hard limit value)
$ ulimit -S -n 4096                    (Increase the soft limit to 4096)

By default, WebLogic Server configures 1024 file descriptors. WebLogic limits the number of open file descriptors by checking the OS limit in a shell script. You can configure WebLogic specific number from WEBLOGIC_HOME/common/bin/commEnv.sh script (note that this script is used on each domain created, if you want the domain specific limits you can use DOMAIN_HOME/bin/setDomainEnv.sh).

Note that when you tune the number of file descriptors for WebLogic Server, your changes should be in balance with any changes made to the complete message timeout parameter. A higher complete message timeout setting results in a socket not closing until the message timeout occurs, which therefore results in a longer hold on the file descriptor. So if the complete message timeout setting is high, the file descriptor limit should also be set high

WebLogic Server Performance and Tuning: Part II - Thread Management

WebLogic Server Performance and Tuning: Part II - Thread Management

WebLogic Server, like any other java application server, provides resources so that your applications use them to provide services. Unfortunately none of these resources are unlimited and they must be managed carefully. One of these resources is threads which are pooled to provide better throughput and performance along with the fast response time and to avoid deadlocks. Threads are execution points that WebLogic Server delivers its power and execute work. Managing threads is very important because it may affect the overall performance of the entire system. In previous releases of WebLogic Server 9.0 we had multiple execute queues and user defined thread pools. There were different queues for different type of work which had fixed number of execute threads.  Tuning of this thread pools and finding the proper number of threads was time consuming which required many trials.

WebLogic Server 9.0 and the following releases use a single thread pool and a single priority-based execute queue. All type of work is executed in this single thread pool. Its size (thread count) is automatically decreased or increased (self-tuned). The new “self-tuning” system simplifies getting the proper number of threads and utilizing them.

Work manager allows your applications to run concurrently in multiple threads. Work manager is a mechanism that allows you to manage and utilize threads and create rules/guidelines to follow when assigning requests to threads. We can set a scheduling guideline or priority a request with a work manager and then associate this work manager with one or more applications. At run-time, WebLogic Server uses these guidelines to assign pending work/requests to execution threads. The position of a request in the execute queue is determined by its priority. There is a default work manager that is provided. The default work manager should be sufficient for most applications. However there can be cases you want to change this default configuration. Your application(s) may be providing services that need mixture of fast response time and long running processes like batch updates. However wrong configuration of work managers can lead a performance penalty while expecting improvement.

We can define/configure work managers at;

•    Domain Level: config.xml
•    Application Level: weblogic-application.xml
•    Component Level: weblogic-ejb-jar.xml or weblogic.xml
(For a specific web application use weblogic.xml)

We can use the following predefined rules/constraints to manage the work;

•    Fair Share Request Class: Specifies the average thread-use time required to process requests. The default is 50.
•    Response Time Request Class: Specifies a response time goal in milliseconds.
•    Context Request Class: Assigns request classes to requests based on context information.
•    Min Threads Constraint: Limits the number of concurrent threads executing requests.
•    Max Threads Constraint: Guarantees the number of threads the server will allocate to requests.
•    Capacity Constraint: Causes the server to reject requests only when it has reached its capacity.


Let’s create a work manager for our application for a long running work.
Go to WebLogic console and select Environment | Work Managers from the domain structure tree. Click New button and select Work manager and click next.


Enter the name for the work manager and click next. Then select the managed server instances(s) or clusters from available targets (the one that your long running application is deployed) and finish.




Click on MyWorkManager, and open the Configuration tab and check Ignore Stuck Threads and save. This will prevent WebLogic to tread long running processes (that is taking more than a specified time) as stuck and enable to finish the process.

Troubleshooting Out Of Memory Errors in Weblogic using Eclipse Memory Ananlyser

Eclipse Memory Ananlyser is a very useful tool to analyze heap dumps. It has a lot of features such as Memory Leak detection where it runs an automated test to determine the suspected leaks.
Here is the Procedure
Step 1). Start the Weblogic Server, with the application in active state which causes memory leak.
Step 2). Get the process id of the server using jps

Step 3). Acess the application that causes memory leak
Step 4).Take heap dump at regular interval using jmap.
jmap -dump:format=b,file=dump1.bin 5004
Step 5).Open the Heap Dump in Eclipse Memory Ananlyzer (U can download it from http://www.eclipse.org/mat/downloads.php) Approximate size is 42 MB..Just u need to extract this Zip then u can directly start (no Installation needed)

Observe the heap usage of Objects in the heap dumps. If the object instance keeps on increasing in the subequent heap dumps, force a garbage collection from the Weblogic Server console

Take heap dumps again and open in the Eclipse Memory Analyzer. If the number of instances still don’t go down for those objects, you can expect to see this

<Jul 16, 2010 10:49:15 AM IST> <Critical> <Health> <BEA-310003> <Free memory in
the server is 47,856 bytes. There is danger of OutOfMemoryError>
Exception in thread “Thread-12″ java.lang.OutOfMemoryError: Java heap space
at demo.MemoryLeakTest.runTest(MemoryLeakTest.java:14)
at jsp_servlet.__memoryleak$1.run(__memoryleak.java:86)
at java.lang.Thread.run(Thread.java:619)
.
If the leak is happening due to a Weblogic Class, it can be a known issue or an undiscovered BUG. You need to get in touch with Oracle Support. If it’s an Application Class, you need to contact the developers. Out of Memory can also happen dude to third party codes such as database drivers.

WebLogic Server Performance and Tuning: Part I - Tuning JVM

WebLogic Server Performance and Tuning: Part I - Tuning JVM

Each WebLogic Server instance runs in its own dedicated Java Virtual Machine (JVM) which is their runtime environment. Every Admin Server in any domain executes within a JVM. The same also applies for Managed Servers. WebLogic Server can be used for a wide variety of applications and services which uses the same runtime environment and resources. Oracle WebLogic ships with 2 different JVM, HotSpot and JRocket but you can choose which JVM you want to use.

JVM is designed to optimize itself however it also provides some startup options to make small changes. There are default values for its memory and garbage collection. In real world, you will not want to stick with the default values provided by the JVM rather want to customize these values based on your applications which can produce large gains in performance by making small changes with the JVM parameters. We can tell the garbage collector how to delete garbage and we can also tell JVM how much space to allocate for each generation (of java Objects) or for heap. Remember during the garbage collection no other process is executed within the JVM or runtime, which is called STOP THE WORLD which can affect the overall throughput.


Each JVM has its own memory segment called Heap Memory which is the storage for java Objects. These objects can be grouped based on their age like young generation (recently created objects) or old generation (surviving objects that have lived to some extent), etc. A java object is considered garbage when it can no longer be reached from anywhere in the running program. Each generation has its own memory segment within the heap. When this segment gets full, garbage collector deletes all the objects that are marked as garbage to create space. When the old generation space gets full, the JVM performs a major collection to remove the unused objects and reclaim their space. A major garbage collect takes a significant amount of time and can affect system performance.

When we create a managed server either on the same machine or on remote machine it gets its initial startup parameters from $DOMAIN_HOME/bin/setDomainEnv.sh/cmd file.
By default two parameters are set:

    Xms: The initial heapsize
    Xmx: The max heapsize


Try to set equal initial and max heapsize. The startup time can be a little longer but for long running applications it will provide a better performance.

When we set -Xms512m -Xmx1024m, the physical heap size will be 512m. This means that there are pages of memory (in the state of the 512m) that the JVM does not explicitly control. It will be controlled by OS which could be reserve for the other tasks. In this case, it is an advantage if the JVM claims the entire memory at once and try not to spend time to extend when more memory is needed. Also you can use -XX:MaxPermSize (Maximum size of the permanent generation) option for Sun JVM. You should adjust the size accordingly if your application dynamically load and unload a lot of classes in order to optimize the performance.

You can set the JVM options/heap size from the following places:

    Through the Admin console, in the Server start tab (Please note this feature is enabled ONLY if managed server is started via nodemanage/Admin Console and StartScriptEnabled is set to false in the nodemanager.properties which is default)



    In the startManagedWeblogic script for the managed servers
    $DOMAIN_HOME/bin/startManagedWebLogic.sh/cmd
    JAVA_OPTIONS="-Xms1024m -Xmx1024m" ${JAVA_OPTIONS}


    In the setDomainEnv script for the managed servers and admin server (domain wide)
    USER_MEM_ARGS="-Xms1024m -Xmx1024m"

When there is free memory available in the heap but it is too fragmented and not contiguously located to store the object or when there is actually insufficient memory we can get java.lang.OutOfMemoryError. We should create Thread Dump and analyze if that is possible in case of such error.


The second option we can use to produce higher throughput is to garbage collection. We can roughly divide GC algorithms into 2 categories: parallel and concurrent. Parallel GC stops the execution of all the application and performs the full GC, this generally provides better throughput but also high latency using all the CPU resources during GC. Concurrent GC on the other hand, produces low latency but also low throughput since it performs GC while application executes. The JRockit JVM provides some useful command-line parameters that to control of its GC scheme like -XgcPrio command-line parameter which takes the following options;

XgcPrio:pausetime (To minimize latency, parallel GC)
XgcPrio:throughput (To minimize throughput, concurrent GC )
XgcPrio:deterministic (To guarantee maximum pause time, for real time systems)


Sun JVM has similar parameters (like  -XX:UseParallelGC or -XX:+UseConcMarkSweepGC) to control its GC scheme. We can add -verbosegc -XX:+PrintGCDetails to monitor indications of a problem with garbage collection.

Try configuring JVM’s of all managed servers to execute in -server mode to ensure that it is optimized for a server-side production environment.

starting an Administration Server using Node Manager

The steps for starting an Administration Server using WLST and Node Manager are :

i)  Setting up your environment.
Add WebLogic Server classes to the CLASSPATH environment variable and WL_HOME\server\bin to the PATH environment variable.  OR
You can use a  C:\Oracle\Middleware\wlserver_12.1\server\bin\setWLSEnv.cmd  script to set both variables.


ii)  Start WLST Session using command : java weblogic.WLST

iii)  Start Node Manager using command :
                         startNodeManager(verbose=’true’, NodeManagerHome=’C:\\Oracle\\Middleware\\wlserver_12.1\\common\\nodemanager’, ListenPort=’5556′, ListenAddress=’localhost’)


iv) Connect WLST to a Node Manager by entering the nmConnect command.
nmConnect(‘weblogic’, ‘weblogic123′, ‘localhost’, ’5556′, ‘base_domain’ , ‘C:\\Oracle\\Middleware\\user_projects\\domains\\base_domain’ , ‘ssl’)

note: Node Manager security relies on a one-way SSL connection between the client and server.If you are establishing a command line connection to the Java Node Manager using the WebLogic Server Scripting Tool (WLST) nmConnect command, you provide the Node Manager user name and password(here weblogic and weblogic123 respectively). Node Manager verifies the username and password against the domain’s nm_password.properties file.Node Manager credentials are located on the Security>General>Advanced Options of Console page.Administration Console users do not need to explicitly provide credentials to connect to Node Manager—the Node Manager user name and password are available in the domain configuration and are provided automatically.

v) Use the nmStart command to start a server.
nmStart(‘AdminServer’)

Monitor the status of the Administration Server by entering the nmServerStatus command.
nmServerStatus(‘AdminServer’)
vi) Connect WLST to a running WebLogic Administration Server instance using the connect command.
connect(‘weblogic’,'weblogic123′)

vii) Stop the AdminServer by entering the nmKill command.
nmKill(‘AdminServer‘)

viii) Disconnect the node manager :     nmDisconnect()   and   Exit the WLST:     exit()

opatch commands

Opatch Commands

http://docs.oracle.com/cd/B16240_01/doc/em.102/e15294/options.htm

apply	Installs an interim patch. Refer to "apply Command" for more information.
napply	Installs n number of patches (hence napply). Refer to "napply Command" for more information.
auto	Applies Oracle Clusterware patches. Refer to "auto Command" for more information.
lsinventory	Lists what is currently installed on the system. Refer to "lsinventory Command" for more information.
query	Queries a given patch for specific details. Refer to "query Command" for more information.
rollback	Removes an interim patch. Refer to "rollback Command" for more information.
version	Prints the current version of the patch tool. Refer to "version Command" for more information.

weblogic application Mointering Script in WLST

###################****##############****######################## 
# Generic script applicable on any Operating Environments (Unix, Windows) 
# ScriptName    : appmointer.py 
# Author        : Madhusudhana Chilipi 
###############     Connecting to Start     ################################# 


connect('weblogic','welcome1','t3://usadc-vswlst01:7001')
domainRuntime()
cd('AppRuntimeStateRuntime/AppRuntimeStateRuntime')
#cd('ServerRuntimes')
#ls()
AppList = cmo.getApplicationIds()
print '*******Application State*******'
print '***********************************************\n'
for App in AppList:

    print '****',App ,'****', cmo.getIntendedState(App)

print '***********************************************\n'

Setting up Redirection from Oracle HTTP Server Load Balancer to Internet Information Services

Generally, in the absence of a standard load balancer, you can configure an Oracle HTTP Server to load balance web applications deployed on multiple IIS Servers. Complete this procedure to redirect requests from the Oracle HTTP Server that is used as the load balancer to the IIS Servers that the following components use:

• Performance Management Architect
• Financial Management
• FDM
• Strategic Finance
• Oracle Hyperion Data Relationship Management, Fusion Edition

The steps in the following procedure assume the following:

• EPM System component, for example, Financial Management, is deployed on two IIS Servers.
• Some EPM System components, for example, Shared Services and EPM Workspace, are deployed on WebLogic Server.
• EPM System components are configured to use Oracle HTTP Server as the web server.
  The logical web application for EPM System components deployed on IIS Server is defined as http://epm.myCompany.com:19000.

  To configure redirection from Oracle HTTP Server:

1. Using a text editor, open EPM_ORACLE_INSTANCE/httpConfig/ohs/config/OHS/ohs_component/httpd.conf.
2. Ensure that the following entry is not commented out. If the entry is missing, add it.
   

   LoadModule proxy_balancer_module “${ORACLE_HOME}/ohs/modules/mod_proxy_balancer.so”
   

3. Ensure that the mod_header module is loaded by including or uncommenting the following directive.
   

   LoadModule headers_module “${ORACLE_HOME}/ohs/modules/mod_headers.so”
   

4. Create proxy balancer definition for EPM System components (Financial Management, Performance Management Architect, FDM, and Strategic Finance) hosted on IIS Servers by adding definitions similar to the following:
   You must create a Proxy balancer definition for each EPM System component deployed on the IIS Server. Within the Proxy balancer definition, you must add a BalanceMember directive for each IIS Server that hosts EPM System components.
   

   <Proxy balancer://iisappshfm>
       BalancerMember http://myIISserver1.mycompany.com:80/hfm 
          loadfactor=1 route=1 
       BalancerMember http://myIISserver2.mycompany.com:80/hfm 
          loadfactor=1 route=2
       ProxySet lbmethod=bytraffic
   </Proxy>
   <Proxy balancer://iisappshfmlcmserver>
       BalancerMember http://myIISserver1.mycompany.com:80/hfmlcmserver 
          loadfactor=1 route=1 
       BalancerMember http://myIISserver2.mycompany.com:80/hfmlcmserver 
          loadfactor=1 route=2
       ProxySet lbmethod=bytraffic
   </Proxy> 
   <Proxy balancer://iisappshfmsmartviewprovider>
       BalancerMember http://myIISserver1.mycompany.com:80/hfmsmartviewprovider 
          loadfactor=1 route=1 
       BalancerMember http://myIISserver2.mycompany.com:80/hfmsmartviewprovider 
          loadfactor=1 route=2
       ProxySet lbmethod=bytraffic
   </Proxy> 
   <Proxy balancer://iisappshfmapplicationservice>
       BalancerMember http://myIISserver1.mycompany.com:80/hfmapplicationservice 
          loadfactor=1 route=1 
       BalancerMember http://myIISserver2.mycompany.com:80/hfmapplicationservice 
          loadfactor=1 route=2
       ProxySet lbmethod=bytraffic
   </Proxy>
   <Proxy balancer://iisappshfmlcmservice>
       BalancerMember http://myIISserver1.mycompany.com:80/hfmlcmservice 
          loadfactor=1 route=1 
       BalancerMember http://myIISserver2.mycompany.com:80/hfmlcmservice 
          loadfactor=1 route=2
       ProxySet lbmethod=bytraffic
   </Proxy>
   <Proxy balancer://iisappsdrm>
       BalancerMember http://myIISserver1.mycompany.com:80/drm-web-client
       loadfactor=1 route=1
       BalancerMember http://myIISserver2.mycompany.com:80/drm-web-client
       loadfactor=1 route=2
   ProxySet lbmethod=bytraffic
   </Proxy>
   

   In the preceding example, values such as iisappshfm and iisappshfmlcmserver are references and not URLs. myIISserver1.myCompany.com and myIISserver2.myCompany.com are the names of the IIS Server host machines.
5. Enable sticky load balancing by adding directives similar to the following. These sample directives instruct Oracle HTTP Server to insert a cookie that keeps track of the route for sticky load balancing of the proxy balancers you defined in step 4.
   

   Header add Set-Cookie "ORA_EPM_IIShfm_ROUTE_ID=iisappshfm .%{BALANCER_WORKER_ROUTE}e; path=/hfm ;" env=BALANCER_ROUTE_CHANGED
   Header add Set-Cookie "ORA_EPM_IIShfmofficeprovider_ROUTE_ID=iisappshfmofficeprovider.%{BALANCER_WORKER_ROUTE}e; path=/hfmofficeprovider;" env=BALANCER_ROUTE_CHANGED
   Header add Set-Cookie "ORA_EPM_IIShfmlcmserver_ROUTE_ID=iisappshfmlcmserver.%{BALANCER_WORKER_ROUTE}e; path=/hfmlcmserver;" env=BALANCER_ROUTE_CHANGED
   Header add Set-Cookie "ORA_EPM_IIShfmsmartviewprovider_ROUTE_ID=iisappshfmsmartviewprovider.%{BALANCER_WORKER_ROUTE}e; path=/hfmsmartviewprovider;" env=BALANCER_ROUTE_CHANGED
   Header add Set-Cookie "ORA_EPM_IIShfmapplicationservice_ROUTE_ID=iisappshfmapplicationservice.%{BALANCER_WORKER_ROUTE}e; path=/hfmapplicationservice;" env=BALANCER_ROUTE_CHANGED
   Header add Set-Cookie "ORA_EPM_IIShfmlcmservice_ROUTE_ID=iisappshfmlcmservice.%{BALANCER_WORKER_ROUTE}e; path=/hfmlcmservice;" env=BALANCER_ROUTE_CHANGED
   Header add Set-Cookie "ORA_EPM_IIS_ROUTE_ID=iisapps.%{BALANCER_WORKER_ROUTE}e; path=/;" env=BALANCER_ROUTE_CHANGED
   Header add Set-Cookie "BALANCEID= iisappsdrm.%
   {BALANCER_WORKER_ROUTE}e; path=/drm-web-client;" env=BALANCER_ROUTE_CHANGED
   

6. Replace the existing ProxyPass directive for EPM System components (Financial Management, Performance Management Architect, FDM, Oracle Hyperion Data Relationship Management, Fusion Edition, and Strategic Finance) deployed on IIS Server with definitions similar to the following:
   

   ProxyPass /hfm balancer://iisappshfm/ stickysession=ORA_EPM_IIShfm_ROUTE_ID nofailover=On
   ProxyPassReverse /hfm http://epm.myCompany.com:19000/hfm 
   ProxyPreserveHost ON
   
   ProxyPass /hfmofficeprovider balancer://iisappshfmofficeprovider/ stickysession=ORA_EPM_IIShfmofficeprovider_ROUTE_ID nofailover=On
   ProxyPassReverse /hfmofficeprovider http://epm.myCompany.com:19000/hfmofficeprovider
   ProxyPreserveHost ON
   
   ProxyPass /hfmlcmserver balancer://iisappshfmlcmserver/ stickysession=ORA_EPM_IIShfmlcmserver_ROUTE_ID nofailover=On
   ProxyPassReverse /hfmlcmserver http://epm.myCompany.com:19000/hfmlcmserver 
   ProxyPreserveHost ON
   
   ProxyPass /hfmsmartviewprovider balancer://iisappshfmsmartviewprovider/ stickysession=ORA_EPM_IIShfmsmartviewprovider_ROUTE_ID nofailover=On
   ProxyPassReverse /hfmsmartviewprovider http://epm.myCompany.com:19000/hfmsmartviewprovider 
   ProxyPreserveHost ON
   
   ProxyPass /hfmapplicationservice balancer://iisappshfmapplicationservice/ stickysession=ORA_EPM_IIShfmapplicationservice_ROUTE_ID nofailover=On
   ProxyPassReverse /hfmapplicationservice http://epm.mycompany.com:19000/hfmapplicationservice 
   ProxyPreserveHost ON
   
   ProxyPass /hfmlcmservice balancer://iisappshfmlcmservice/ stickysession=ORA_EPM_IIShfmlcmservice_ROUTE_ID nofailover=On
   ProxyPassReverse /hfmlcmservice http://epm.mycompany.com:19000/hfmlcmservice 
   ProxyPreserveHost ON
   
   ProxyPass /drm-web-client balancer://iisappsdrm stickysession=BALANCEID nofailover=Off
   ProxyPassReverse /drm-web-client http://epm.mycompany.com:19000/drm-web-client
   ProxyPassReverse /drm-web-client http://epm.mycompany1.com:19000/drm-web-client
   

7. Save and close httpd.conf.

Commands for Retrieving Information about WebLogic Server

Overview of Commands for Retrieving Information about WebLogic Server 

Command	Description
CONNECT	Makes the specified number of connections to a WebLogic Server instance and returns two numbers representing the total time for each round trip and the average amount of time (in milliseconds) that each connection is maintained. See CONNECT.
GETSTATE	Returns the current state of the specified WebLogic Server instance. See GETSTATE.
HELP	Provides syntax and usage information for all WebLogic Server commands (by default) or for a single command if a command value is specified on the HELP command line. See HELP.
LICENSES	Lists the licenses for all WebLogic Server instances that are installed on a specific server. See LICENSES.
LIST	Lists the bindings of a node in a server's JNDI naming tree. See LIST.
PING	Sends a message to verify that a WebLogic Server instance is listening on a port and is ready to accept client requests. See PING. For a similar command that returns information about all servers in a cluster, see CLUSTERSTATE.
SERVERLOG	Displays the server log file generated on a specific server instance. See SERVERLOG.
THREAD_DUMP	Provides a real-time snapshot of the WebLogic Server threads that are currently running on a particular instance. See THREAD_DUMP.
VERSION	Displays the version of the WebLogic Server software that is running on the machine specified by the value of URL. See VERSION.

UNLOCK weblogic password

In the following example, an administrator named adminuser with a password of gumby1234 requests the unlocking of the WebLogic Server listening on port 7001 on machine localhost:

java weblogic.Admin -url localhost:7001 -username adminuser 
   -password gumby1234 UNLOCK

JDBC: IO Error: Got minus one from a read calloracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)

IO Error: Got minus one from a read call<br/>oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:458)<br/>oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)<br/>oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)<br/>oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)<br/>oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)<br/>oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:280)<br/>oracle.jdbc.xa.client.OracleXADataSource.getPooledConnection(OracleXADataSource.java:469)<br/>oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:156)<br/>oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:101)<br/>weblogic.jdbc.common.internal.DataSourceUtil.testConnection(DataSourceUtil.java:314)<br/>com.bea.console.utils.jdbc.JDBCUtils.testConnection(JDBCUtils.java:734)<br/>com.bea.console.actions.jdbc.datasources.createjdbcdatasource.CreateJDBCDataSource.testConnectionConfiguration(CreateJDBCDataSource.java:474)<br/>sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)<br/>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)<br/>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)<br/>java.lang.reflect.Method.invoke(Method.java:597)<br/>org.apache.beehive.netui.pageflow.FlowController.invokeActionMethod(FlowController.java:870)<br/>org.apache.beehive.netui.pageflow.FlowController.getActionMethodForward(FlowController.java:809)<br/>org.apache.beehive.netui.pageflow.FlowController.internalExecute(FlowController.java:478)<br/>org.apache.beehive.netui.pageflow.PageFlowController.internalExecute(PageFlowController.java:306)<br/>... 

Solutions:

Degugging the root Cause of above Issue:
The error “Caused by: java.sql.SQLException: Io exception: Got minus one from a read call” clearly indicates the root cause of this issue. “We get the following error “Got minus one from a read call” if the Database goes under Maintenance…..or Database is in inconsistent State (Due to any reason like Database Maintenance or Database unmounting…etc ).
WLS is asking a JDBC driver for a connection and getting that exception, which means the DBMS or network dropped the socket during the driver-DBMS handshake.
The above error ensures that there is Nothing Wrong from WebLogic Side …we need to contact the Database Administrator only.
Step1). Open the Server Log and check the timestamp of the first occurance of that Error in the Server log.
Step2). Confirm with the Database Administrator…. What was the activity happening on the Database side at that time (90% cases u will find that the Database might be under Maintenance that time)

Setting the Audit Level at the BPEL Process Service Component Level

Setting the Audit Level at the BPEL Process Service Component Level

You can set the audit level for a BPEL process service component. This setting takes precedence over audit level settings at the SOA Infrastructure, service engine, and SOA composite application levels. The service component level setting is only available for BPEL processes and is not supported for the mediator, human workflow, and business rule service components.
There are two ways to set the audit level for BPEL process service components. Supported values are Off, Minimal, Inherit, Development, and Production.

• In the System MBean Browser of Oracle Enterprise Manager Fusion Middleware Control Console:
  
  1. In the navigation tree, expand the SOA folder.
     
  2. Right-click soa-infra, and select Administration > System MBean Browser.
     
  3. Select Application Defined MBeans > oracle.soa.config > Server: server_name > SCAComposite > Composite_Name > SCAComposite.SCAComponent > BPEL_Service_Component > Properties.
     
  4. Click the Add icon.
     
  5. Expand the Element_number folder.
     
  6. From the many list, select false.
     
  7. In the name field, enter bpel.config.auditlevel.
     
  8. In the value field, enter a value.
     
  9. Click Apply.
     

Configuring BPEL Process Service Engine Properties

Configuring BPEL Process Service Engine Properties

You can configure BPEL process service engine properties. The properties are used by the BPEL process service engine during processing of BPEL service components.
To configure BPEL process service engine properties:

1. Access this page through one of the following options:
   

   From the SOA Infrastructure Menu...	From the SOA Folder in the Navigator...
   Select SOA Administration > BPEL Properties.	Right-click soa-infra. Select SOA Administration > BPEL Properties.

   The BPEL Service Engine Properties page displays properties for setting audit trail and large document thresholds, setting dispatcher thread properties, validating payload schema, and setting the audit trail level.
   
   Description of the illustration soaadmin_bpel_props.gif
   
2. Make changes to the service engine properties that are appropriate to your environment.
   

   Property	Description
   Audit Level	Select one of the following options: Off: Composite instance tracking and payload tracking information is not collected. Inherit: Logging equals the SOA Infrastructure audit level. This setting enables the BPEL audit level to automatically change when the global setting is changed. Setting a different audit level tracking in this page overrides the tracking set at the SOA Infrastructure level. Minimal: The BPEL service engine does not capture any audit details. Therefore, they are not available in the flow audit trails. All other events are logged. Production: The BPEL service engine does not capture the payload. The payload details are not available in the flow audit trails. Payload details for other BPEL activities are collected, except for assign activities. This level is optimal for most normal operations and testing. Development: Allows both composite instance tracking and payload tracking. All events are logged. However, it may impact performance. This level is useful mostly for debugging purposes.
   Audit Trail Threshold	Enter the maximum size in bytes of an instance audit trail before it is chunked and saved in a dehydration store table separate from the audit trail. If the threshold is exceeded, the View XML link is shown in the audit trail instead of the payload.
   Large Document Threshold	Enter the maximum size of a generated document within a BPEL process component instance before it is stored in a separate table in the dehydration store.
   Dispatcher System Threads	Specify the total number of threads allocated to process system dispatcher messages. System dispatcher messages are general clean-up tasks that are typically processed quickly by the server (for example, releasing stateful message beans back to the pool). Typically, only a small number of threads are required to handle the number of system dispatch messages generated during run time. The default value is 2 threads. Any value less than 1 thread is changed to the default.
   Dispatcher Invoke Threads	Specify the total number of threads allocated to process invocation dispatcher messages. Invocation dispatcher messages are generated for each payload received and are meant to instantiate a new instance. If the majority of requests processed by the engine are instance invocations (as opposed to instance callbacks), greater performance may be achieved by increasing the number of invocation threads. Higher thread counts may cause greater CPU utilization due to higher context switching costs. The default value is 20 threads. Any value less than 1 thread is changed to the default.
   Dispatcher Engine Threads	Specify the total number of threads allocated to process engine dispatcher messages. Engine dispatcher messages are generated whenever an activity must be processed asynchronously. If the majority of processes deployed are durable with a large number of dehydration points (midprocess receive, onMessage, onAlarm, and wait activities), greater performance may be achieved by increasing the number of engine threads. Note that higher thread counts can cause greater CPU utilization due to higher context switching costs. The default value is 30 threads. Any value less than 1 thread is changed to the default.
   Payload Validation	Select to enable validation of inbound and outbound messages. Nonschema-compliant payload data is intercepted and displayed as a fault. Note: This setting is independent of the SOA composite application and SOA Infrastructure payload validation level settings. If payload validation is enabled at both the service engine and SOA Infrastructure levels, data is checked twice: once when it enters the SOA Infrastructure, and again when it enters the service engine.
   Disable BPEL Monitors and Sensors	Select this check box to disable all BPEL monitors and sensors defined for all BPEL components across all deployed SOA composite applications.

3. Click Apply.
   
4. If you want to configure advanced BPEL properties in the System MBean Browser, click More BPEL Configuration Properties. Properties that display include, but are not limited to, the following. Descriptions are provided for each property.
   
   • BpelcClasspath — The extra BPEL class path to include when compiling BPEL-generated Java sources.
     
   • DisableAsserts — Disables the execution of assertions in BPEL, including the bpelx:assert activity.
     
   • DisableSensors — Disables all calls to sensors.
     
   • ExpirationMaxRetry — The maximum number of times a failed expiration call (wait/onAlarm) is retried before failing.
     
   • ExpirationRetryDelay — The delay between expiration retries.
     
   • InstanceKeyBlockSize — The size of the block of instance IDs to allocate from the dehydration store during each fetch.
     
   • MaximumNumberOfInvokeMessagesInCache — The number of invoke messages stored in in-memory cache.
     
   • OneWayDeliveryPolicy — Changes whether one-way invocation messages are delivered.
     
   • StatsLastN — The size of the most recently processed request list.
     
   • SyncMaxWaitTime — The maximum time a request and response operation takes before timing out.
     
5. Make changes appropriate to your environment.