Pacific Gas and Electric
Company
Ishaq Mohammed / Anil Saka
Online and Middleware Application Operations
Agenda
• Overview
• Challenges
• Solution
• Middleware Provisioning Use Cases
• PG&E’s Provisioning Portal
• Benefits Gained
• Future Plans
• Summary
• Q/A
2
Agenda
• Overview
• Challenges
• Solution
• Middleware Provisioning Use Cases
• PG&E’s Provisioning Portal
• Benefits Gained
• Future Plans
• Summary
• Q/A
3
Company Profile
• Headquarters in San Francisco,
California
• Provides energy to approximately
15 million people
• 70,000 square mile service territory
• Total Assets: $50 billion
• 2012 Net Income: $816 million
• 2012 Capital Expenditures: ~$4.5
billion
PG&E SERVICE AREA
IN CALIFORNIA
4
IT Overview - Infrastructure
• 280+ WebLogic Domains (versions 9.2 through 12.1.1)
• SOA Suite & Oracle Service Bus (OSB)
• WebLogic Portal & Integration
• 970+ WebLogic Servers
• 205 Databases (versions 10g through 11.2.0.3)
• 500+ host servers running Linux and AIX
• Microsoft SQL Server, Apache HTTP Server 2.2 and Sun One.
• 2,000 IT employees – Team of 8 in middleware management
5
IT Overview – MyEnergy Business Application
• MyEnergy application most
critical from business
perspective
• Account management (e.g.
customers can monitor /manage
energy use, pay bills online)
• Promotion of energy-savings
programs
• Scheduled outages of PG&E
services
• 2.2 million active users
• 6,000 users logged in hourly
during peak times
6
IT Overview – MyEnergy Architecture Diagram
7
Agenda
• Overview
• Challenges
• Solution
• Middleware Provisioning Use Cases
• PG&E’s Provisioning Portal
• Benefits Gained
• Future Plans
• Summary
• Q/A
8
Challenges
• High cost in manually building 30+ middleware environments each
year (~$2.5 million)
• 20% increase in WebLogic landscape past 2 years
• What worked in pre-production sometimes failed to work in
production
9
Agenda
• Overview
• Challenges
• Solution
• Middleware Provisioning Use Cases
• PG&E’s Provisioning Portal
• Benefits Gained
• Future Plans
• Summary
• Q/A
10
Solution: Oracle Enterprise Manager to
Provision Middleware Environments
• Modernizes the build process
• Guarantees speed to market delivery
• Eliminates errors
• Reduces labor cost and time spent
• Provides out-of-box monitoring
11
Oracle Enterprise Manager Cloud Control 12c
Deployment
• Four environments for Cloud Control 12c – dev, test, stage and
production
• Production environment for Cloud Control 12c:
 Management Service 12.1.0.3 Release 3 on Redhat Linux with
Active-Standby high availability configured
 Management Repository in two node RAC DB 11.2.0.3 on Redhat
Linux
 Management Agents 12.1.0.3 Release 3 on Redhat Linux and AIX
 Fusion Middleware Plug-in 12.1.0.5
12
Agenda
• Overview
• Challenges
• Solution
• Middleware Provisioning Use Cases
• PG&E’s Provisioning Portal
• Benefits Gained
• Future Plans
• Summary
• Q/A
13
Middleware Use Cases
WebLogic Server 12c (WLS) version 12.1.1
Oracle Service Bus 11g (OSB) version 11.1.1.6
WebLogic Portal 10g (WLP) version 10.3.4
Small, Medium Or Large Deployment
14
Middleware Provisioning
•
Discover reference Oracle Home/domain
•
Create provisioning profile in software library
•
Create directives in software library
•
Edit predefined deployment procedure
•
Test deployment procedure from console to capture input parameters
•
Create wrapper script which calls emcli “submit_procedure”
•
Enable end users to make requests using PG&E’s Provisioning Portal
15
Prerequisites
• Configure Software Library on OEM Server
• Enable passwordless (ssh keys) connection between Oracle
Management Service and new VM
• Update the /etc/sudoers with: weblogic
ALL=(root)/u01/app/Oracle/product/agent12c/sbin/nmosudo
• Setup named credentials and privilege delegation.
• Install required packages on the OS, create OS user and
group(webLogic)
• Install and start Oracle Management Agent on new server
16
Sample: Provisioning Profiles
17
Directives
• Provisioning Profiles Capture “Standard” Files and Artifacts of
WebLogic Domain[Gold Image].
• Additional Files Specific to WL Domain must be moved to Target
Host(s) with File Transfer Step or Directive.
• With Directives you upload required Files to Software Library along
with script to move and/or Manipulate Files, called Main file
• Be sure to check “Use Most Recent Version” check box, else your
procedure keeps calling older version
18
Sample Directive
19
Customized Deployment Procedure
Oracle-provided deployment procedure named "Provision
Middleware" customized to accommodate PG&E requirements





Custom Directives (Described Earlier), or
Shell Script that Execute Shell Commands and/or call WLST
We Generally Added One Directive and 3 to 4 Shell Scripts to Customized Procedures
Shell Script Cannot Exceed 4096 Characters.
Can Use Runtime Variables in Scripts, e.g.
${target.DOMAIN_HOME_DEST_ADMIN_HOST}
20
Sample Deployment Procedure
21
Options to Provision
Provision using console:
• Several clicks and repetitive
process
Provision using EMCLI:
• Capture once and execute many
22
EMCLI Wrapper Script
•
Generated Properties Files Made Generic by Substituting Actual Values with
Specific Keywords:
<Hostname> -> MyHostName
<Domain Name> -> MyDomainName
<DBSid> -> MyDBSid
•
•
•
Wrapper Scripts Creates Copy of Properties Files and Substitutes New Values
Passed on Command Line
Removes Copy When Completed
Provision New WebLogic Portal Domain Example:
sh ./emcliWrapperDB.sh "DB_DATA" "S" “database-host“ "Linux" "WLPDBS"
sh ./emcliWrapperWL.sh WLP S EIWLP wlp-mw-host database-host WLPDBS
23
Agenda
• Overview
• Challenges
• Solution
• Middleware Provisioning Use Cases
• PG&E’s Provisioning Portal
• Benefits Gained
• Future Plans
• Summary
• Q/A
24
PG&E Provisioning Portal
Capable of Provisioning:
Provision
Requested
Operating System
Artifact
• Operating Systems
Provision
Requested
SAP Artifact
• SAP Environments
• Oracle Databases
• Oracle Fusion Middleware
Environments
System Provisioning
Artifact
Provisioner
Provision
Requested
Database
Artifact
Provision
Requested
Fusion
Middleware
Artifact
25
PG&E Provisioning Portal Processes
•
PG&E has built a portal for request intake and complete orchestration of
the provisioning process
•
Provides central interface to end users
•
Makes REST calls to Pureflex System for OS provisioning
•
Installs Management Agent
•
Invokes EMCLI scripts to provision middleware environments
26
PG&E Provisioning Portal Features
•
•
•
•
•
•
Authentication against PG&E Active
Directory / SSO
Ability to notify Requestor/Approver
via email
System provisioning clients pay for
the resources
System provisioning clients are not
responsible for the maintenance of
any part of the shared infrastructure
Ability to monitor performance of their
workloads to ensure SLAs are met
Provides at least three deployment
options for each artifact in terms of
sizing: small, medium, large
27
One Click!
• Provision Operating System
• Install Middleware
• Configure Domain
• Discover Domain to obtain
out-of-box monitoring
28
Agenda
• Overview
• Challenges
• Solution
• Middleware Provisioning Use Cases
• PG&E’s Provisioning Portal
• Benefits Gained
• Future Plans
• Summary
• Q/A
29
Achieved Results and Improved Business
• Standardized OS, middleware and database images
• Reduced overall provisioning time by 75%
• Saved ~$1 million since rollout
30
Agenda
• Overview
• Challenges
• Solution
• Middleware Provisioning Use Cases
• PG&E’s Provisioning Portal
• Benefits Gained
• Future Plans
• Summary
• Q/A
31
Future Plans
Extend provisioning to more use cases:
• Oracle SOA Suite
• Oracle Business Process Management Suite
• Web Servers
32
Agenda
• Overview
• Challenges
• Solution
• Middleware Provisioning Use Cases
• PG&E’s Provisioning Portal
• Benefits Gained
• Future Plans
• Summary
• Q/A
33
Summary
• PG&E uses Oracle Enterprise Manager Cloud Control 12c
deployment procedures with custom steps and scripts to
provision all Oracle WebLogic Server, Portal and Service Bus
environments
• PG&E has built an end-user portal to provision middleware
infrastructure with a single click
• Provisioning brings PG&E a 75% reduction in time to market
• Many lines of businesses are enjoying the benefits of lower costs,
faster delivery, and standardized deployments
34
Agenda
• Overview
• Challenges
• Solution
• Middleware Provisioning Use Cases
• PG&E’s Provisioning Portal
• Benefits Gained
• Future Plans
• Summary
• Q/A
35
Thank you for your
participation!
Q/A
36