Using Atlassian Products in
an Enterprise Environment
Mark McCormack
British Telecom
Inlife Support Lead
Development Services Organisation
Outline
•
•
•
•
•
•
•
•
•
•
•
Using Atlassian products in an Enterprise
Atlassian Product adoption
Size of Confluence and Jira in our Enterprise
Some Atlassian Use Cases
Customisations
Application Monitoring
Platform
What’s Next
Product Wishlist
Some Lessons Learnt
Q&A
© British Telecommunications plc
About
•
•
•
•
•
•
•
Network is available in 173 countries.
Around 3,000 new customer sites every month
Carry 80% of all credit card transactions in the UK.
More than 90% of the UK’s financial institutions use our secure data networks
Rolling out super-fast broadband to 40% of the UK by 2012.
FTTC downloads of 40Mb/s and uploads of 2 Mb/s.
FTTP downloads of 100 Mb/s
London 2012 Olympics
© British Telecommunications plc
Development Services Organisation
•
•
We advise on best practices for all tools
Inlife team looks after all hosted applications:
– We are an Atlassian shop
– Subversion, Jive Forums, Nexus
– Other in-house tools
• Authz, Teams, IPPI, Provisioning, Static Data & PALMS
• SDK Catalogue (will soon become BT Enterprise Repository)
© British Telecommunications plc
Product History
Dec 2003
•
•
5 people created a team (called code red) but there was no way of coordinating their efforts
Bought Confluence and Jira and installed it on a PC and put it in a meeting room
June 2004
•
•
Usage spread to a few development teams (200 users)
Bought new hardware and installed it in a server room (engineering.intra.bt.com)
April 2006
•
Another group installed a separate instance of Jira and Confluence to manage service
improvements
February 2007
•
•
•
Management Services and Engineering decided joining forces would be a good idea
Userbase reached 300 and 2000 respectively
http://collaborate.intra.bt.com was born and hosted by VS Group on VMWare.
July 2007
•
•
A Virtual Administrator crew formed out of 35 volunteers from the userbase to perform end
user support and administer the platform
Two users in particular did most of the Admin work. Userbase grows to over 9000 people.
© British Telecommunications plc
Product Maturity
August 2007
• Sponsors found to pay for new hardware & uplift plan created
• Team split started to disrupt progress to achieve this and ownership was contested
September 2007
• Userbase grows to 10,000
October 2007
• I/O Capacity of the disks in the VSP environment started to be exceeded.
• Platform started to have outages with huge traffic increase
January 2009:
• Moved to an enterprise hardware infrastructure, Oracle 10G RAC & 24x7 support
• Collaborate Wiki was formed, Confluence 2.10 and integrated with Crowd
February 2009:
• Introduced Storm & Jira Issues
July 2009:
• Started to assimilate other wikis (like twiki, rdrwiki, ProfComms)
© British Telecommunications plc
Confluence Numbers
Current,
100,000
•
•
•
•
•
•
800,000 pages
102,000 attachments
3000 spaces
25,000 display requests per day
30,000 actions per day
(eg login, edit, comments, rss feeds etc)
44,000
Pages
Old,
27,000
Old,
700,000
Attachments
Current,
75,000
13,000
Users
© British Telecommunications plc
Contributers
Numbers
4,500
83,000
3 flavours of Jira
ACF – Architecture Conformance Framework
Issue Tracking (100 projects)
Story Management Operating Model
BT is adding > 20,000 stories every month
300
•
•
•
•
•
5,500 Users
This
Week
1,000
110,000 Stories
© British Telecommunications plc
2,000
7,800
8,000
8,250
2,000
1
20
20
60
400 Spaces
Confluence Use Cases
•
•
•
•
•
•
•
•
•
Tool Guides on how to get started
Tool Access Request Forms
– We have a teams application we encourage before requesting a Space
– Early stages of a Provision Application
Application Deployment Schedule
Collaboration between Dev and Inlife Teams (geographically dispersed)
Don’t send emails but rather send Confluence links
Trying to have an FAQ before raising an issue for help
Application Usage Statistics
MyBT has a dashboard interface and makes use of the Confluence remote API and
render as html inside a portlet.
Most of the Ajax related actions don’t work
© British Telecommunications plc
Confluence Customisations
•
•
•
•
Implemented the functionality of security classification requirements in the scaffolding
plugin.
To implement this functionality the edit page / do edit page actions of the confluence
needs to be overridden. Scaffolding plugin was already overriding the edit actions of
confluence.
Confluence doesn't allow two plugins to override a particular action.
Extended the edit actions to include classification
© British Telecommunications plc
More Confluence Customisations
•
•
•
•
https://collaborate.bt.com/staticdata plugin to provide a LOB picker from LDAP
Static Data Macro that retrieves platforms
BT Space Admin Confluence plugin makes use of our eorg DB in order to get the Line
of Business functionality associated with a Space
Modified the Excel Plugin as it was hanging Confluence a lot
– We had one user with a page that had 48 references to the excel macro
– Every reference was an API call that forced a GC, so we disabled explicit GC
– Page went from 14 minutes to load to about 4 seconds
© British Telecommunications plc
Customisations
•
•
•
•
•
•
Hierarchy plugin
Static Data plugin to strip LOB info out from LDAP
Loads of constraints on Story creation and workflow
– All Stories must have at least one child and at least on Acceptance Criteria
– Organogram
Added Active User Report under Users, Groups and Roles
Roles for Storm
Loads of different workflows (put issues On Hold whilst with Dev team or Vendor)
© British Telecommunications plc
More Customisations
•
•
•
•
https://collaborate.bt.com/fisheye plugin to allow the authorisation of SVN repositories
(handled by hasPermissionToAccess method which is overrided to authorise against
our Authz).
Use https://collaborate.bt.com/crowd client libraries for apps like
https://collaborate.bt.com/forum
We have a Jira Project that helps us keeps track of Subversion Projects ensuring
everyone has a Salsa ID.
Unified Rolling Calendar Plugin
© British Telecommunications plc
Application Monitoring
•
•
•
•
•
•
•
•
•
•
ZXTM
Nagios alerts via email for slow (or dead) apps.
JMX is enabled for most of our java apps
Disk space (e.g shell script to alert > 80%)
Backup successes or failures uploaded to our wiki
Access logs with IP tracking
Usage statistics to figure out what our users are doing
Perl script we run for snapshots of ugly apps
YourKit Profiling from within IntelliJ
kSar
© British Telecommunications plc
Platform
•
•
•
•
•
•
•
•
Currently 18 HP Blades mostly BL460s with 2 Quad cores and 32GB of RAM for our
existing hosted application servers
RHEL 5 (soon OEL)
Were using Weblogic 10.3 but now Tomcat 6
Java 6.0.18
ZXTM 5.0
Oracle 10g in a Cluster hosted by another team on HP Blade 680s with 2 Quad cores
and 64GB of RAM
Crowd is our only clustered application (currently)
We use Capistrano to help automate our deployments and allows easy rollback
© British Telecommunications plc
What’s Next?
•
•
•
•
•
•
•
Clustering of Confluence, Jira and SVN
Triple our hardware for Disaster Recovery
Archiving & stats analysis plugins (customised)
Improve provisioning
All the Atlassian 2s (Bamboo, Crowd, Fisheye, Crucible) although upgrading to
Fisheye 2.0 will be expensive on disk space.
Start charting long-term trends with better monitoring
Assimilate all wikis (like LimsWiki for one)
© British Telecommunications plc
Wishlist
Confluence Wishlist
• Tighter integration with Jira
• Allow Jira Issues to be raised via Confluence and vice versa
• Allow profiles to have attributes (like team, group or project)
• Confluence attachment storage API. We want to store attachments away from
Confluence and write our own adaptors to do this.
• Improve the edit permissions cascading security
• Customisations of actions (eg. Overrides to allow > 1 plugin)
Jira Wishlist
• Allow two columns for attributes
• Allow picklists to show more values without scrolling
• Smart forms that can include wiki macros
• How can we hide Administrators from assignee list?
• Can Greenhopper scale? How compatible is it with existing 3.13.2
plugins/customisations?
© British Telecommunications plc
More Wishes
Jira …
• Would like to use our Issues as a source for FAQ before users raise an issue?
• Ought to be Confluence tightly coupled with Jira Issue labels
• Timebooking to allow resource Management?
Others
• Fisheye 2 IE6 browser support
• Make Crucible a viable app for the enterprise (we have some open Atlassian issues for
10k+ users)
• Can we turn on “remember my login on this computer” checkbox during login?
© British Telecommunications plc
Lessons Learnt
•
•
•
•
•
Be wary of any plugin
– Excel Macro plugin
– We turned off the usage tracking plugin
– We trialled the stats analysis plugin but its currently off since we got hammered
two lunchtimes in a row shortly after going live
WebLogic seems to be a lot slower than tomcat
Confluence 3.x is much better than 2.x
We had problems with Crowd 1.6 client and currently use Crowd 1.3 client
Tune the JVM heap
– Confluence: we use java –server –Xms:8G –Xmx:8G PermSize=512m
MaxPermSize=512m +UseParallelGC +AggressiveHeap
LargePageSizeInBytes=4M -DisableExplicitGC
– Jira: we use –server –Xms4G –Xmx4G PermSize=1G MaxPermSize=1G
+UseParallelOldGC +AggressiveOpts
– Crowd: we use java –server –Xms:4G –Xmx:4G PermSize=512m
MaxPermSize=512m
– Fisheye: not really tuned yet –Xms:4G –Xmx:4G MaxPermSize=512m
© British Telecommunications plc
Questions
[email protected]
British Telecom
Inlife Support Lead
Development Services Organisation