Exadata and iPhone

I never thought one day I will find a similarity between Oracle Exadata and iPhone. But it was before Oracle announced X4-2C, the colorful Exadata:)

X4 2C


However, I don't think that X4-2C is cheaper as iPhone 5C is, since it's a limited edition...

Is Database 12c Supported on Exadata?

Answer is yes, even you can see 12c is listed under supported versions in document [888828.1].

However, there is a couple of buts (you should see this coming). Since latest  Exadata Storage Software (version 11.2.3.2.1) does not have new offload libraries, smart scan offload filtering and storage indexes become disabled with 12c. Smart scans are initiated but nodes get back blocks instead of rows and columns in the projection list. Also IORM plans are not enforced and inter-database plans are disabled on cells. And finally, cell metrics display 12c databases under OTHER_DATABASE.

So, it seems better to wait until next Exadata update and probably 12c patch set update to upgrade your Exadata. At least, to be able upgrade to a fully functional state. Meanwhile, you'd better upgrade to 11.2.0.3 if you haven't yet because upgrading from 11.2.0.1 is not supported.

Data Pump Options

Data pump is really a handy tool. It has many options allowing you to try different ways and provides flexibility. In this entry, I'll use some of these options and try to explain how to export a remote database to the local ASM disk group.

Let's jump in. Start with setting the environment at the local server:

1. Create a directory in ASM disk group:

SQL> alter diskgroup RECOVERY add directory '+RECOVERY/DPUMP';

2. Create a directory in the database pointing to the ASM directory:

SQL> create directory ASM_DIR as '+RECOVERY/DPUMP';

3. Grant write on directory created above to the user you'll use while exporting:

SQL> grant read, write on ASM_DIR to <export_user>;

3. Create database a link:

SQL> create public database link target_db connect to <remote_user> identified by <password> using '<tns_entry>';

It's required that both the local and remote users are granted to the EXP_FULL_DATABASE role.

Exporting

Now it's time to export. At local server, run:

$ expdp NETWORK_LINK=target_db DUMPFILE=ASM_DIR:dump_file.dmp LOGFILE=DATA_PUMP_DIR:export.log FULL=Y EXCLUDE=STATISTICS COMPRESSION=ALL PARALLEL=8

To briefly explain parameters given:
  • NETWOK_LINK is the one pointing to remote database. This is how we export remote database.
  • As you may see, dump file is being created under directory ASM_DIR pointing the directory in ASM disk group.
  • Notice that log file is not being created under ASM directory because it's not supported. That's why I've used default data pump directory. Also you may choose not to log by using parameter NOLOGFILE=Y instead.
  • We're having a full database dump by using FULL=Y parameter. Instead you may use SCHEMAS, TABLES or TABLESPACES parameters to export/import only given schemas/tables or tablespaces.
  • Our dump will not include statistics because of the parameter EXCLUDE=STATISTICS. You can use this option to exclude other objects such as tables or indexes, too. I choose to exclude statistics because I'd rather gather statistics at the database I'm going to import the dump.
  • By means of the parameter COMPRESSION we'll have a smaller sized dump file. To give a hint how much could it be reduced in size, here is a few statistics I've had with different characteristics of data:
EstimatedActualGain Percentage
687.7 MB188.13 MB72.64%
154.3 GB56.44 GB63.42%
287.2 GB207.93 GB27.6%

The reason of low compression ratio of the last one is that it is a table space with Hybrid Columnar Compression (HCC) enabled in a Exadata machine. It's already compressed quite a bit. It's possible to set COMPRESSION parameter with; ALL, METADATA_ONLY, DATA_ONLY or NONE.
  • Lastly, to improve performance we use the parameter PARALLEL. Data pump is a tool you can get full advantage of parallelism. Therefore it's good to use as much as possible.
Checking Up

After exporting you can query and check dump files. To query files exported under ASM:

SQL> select a.name, f.bytes/1024/1024 mb, f.creation_date 
from v$asm_file f, v$asm_alias a
where f.file_number=a.file_number and a.system_created='N' and f.type='DUMPSET';

To check dump file, best tool to use is again data pump, use parameter SHOW:

$ impdp DUMPFILE=ASM_DIR:dump_file.dmp NOLOGFILE=Y FULL=Y SHOW=Y

Finally, if you need to copy the dump file to somewhere else, it could be achieved by asmcmd tool:

$ asmcmd -p cp RECOVERY/DPUMP/dump_file.dmp /destination_directory

Retrieving Index Usage Statistics

Indexes are old friends of every DBA. To check out what your friends are up to, in Oracle, basic way is enable monitoring for the index by:

alter index IX_FOO monitoring usage;

Then you can query V$OBJECT_USAGE to see if it is being used. After a period of time meaningful to you, you can stop monitoring till the next time by:

alter index IX_FOO nomonitoring usage;

However what is lacking in this way of monitoring is statistics. You can see whether index is used but cannot see how many times or how it is used. Fortunately Oracle provides many V$ views. The query below returns basic statistics about your indexes. How it is used (range or full scan etc.), how many times and when was the last time it was used:

select sp.object_name as index_name,
sp.options,
sum(sa.executions) as executions#,
max(timestamp) as last_used
from v$sql_plan sp, v$sqlarea sa
where sa.address = sp.address
and sa.hash_value =sp.hash_value
and sp. operation = 'INDEX'
and object_owner in ('SCOTT')
group by object_name, options
order by 3 desc;

With Exadata, we've started to use less indexes and even dropped some existing ones. Because we want to use Exadata features like storage indexes and smart scan instead, we let the cells do their job. On Exadata we prefer to see fast full scans instead of range scans. To see Exadata specific statistics related to indexes, you can add columns with prefix "io_cell" in V$SQLSTATS or V$SQLAREA views. Such as:

select sp.object_name as index_name,
sp.options,
sum(sa.executions) as executions#,
max(timestamp) as last_used,
sum(sa.disk_reads) as disk_reads,
sum(sa.direct_writes) as direct_writes,
sum(sa.io_cell_offload_eligible_bytes)/1024/1024 as offloadable_mb,
sum(sa.io_cell_offload_returned_bytes)/1024/1024 as offloaded_mb,
sum(sa.rows_processed) as rows_processed
from v$sql_plan sp, v$sqlarea sa
where sa.address = sp.address
and sa.hash_value =sp.hash_value
and sp. operation = 'INDEX'
and object_owner in ('SCOTT')
group by object_name, options
order by 3 desc;

There are many other statistics you can find in V$ views, so you can improve the query according to your needs.

Upgrading Oracle Enterprise Manager 11g Grid Control to 12c Cloud Control with Exadata Plug-ins

As you might know OEM 12c Cloud Control is out there for some while, even its bundle patch 1 has been released. 12c comes with a fully re-designed user interface and new Exadata features which takes my attention the most. To take advantage of these new features I've decided to upgrade my 11g box to 12c. What I have is a 64-bit Oracle Linux 5.7 with 11gR2 database and 11g Grid Control with Exadata plugins installed.

There are three upgrade approaches suggested by Oracle. I'll follow the 1-System Upgrade Approach which is in-place upgrade with minimal downtime.  After upgrading to 12c, I'll discover my database machine through 12c console and implement Exadata 12c integration.

Upgrading 11g Grid Control to 12c Cloud Control with 1-System Upgrade Approach:

1. As in every upgrade and patching, first thing to do is upgrading OPatch. Since we'll patch OMS 11g during upgrade process, we need to upgrade OPatch under OMS_HOME. Download latest OPatch from MOS 6880880 for release 11.1 and:

$ unzip p6880880_111000_Linux-x86-64.zip -d <OMS_HOME>/

2. As a prerequisite if you haven't installed OMS 11g PSU 1 (10065631), you should install it. Actually this is the prerequisite for installing preupgrade console. Download the PSU then:

$ unzip p10065631_111010_Generic.zip
$ <OMS_HOME>/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./10065631

If no conflict is found
$ <OMS_HOME>/bin/emctl stop oms
$ cd 10065631
$ <OMS_HOME>/OPatch/opatch apply
$ export ORACLE_HOME=<OMS_HOME>
$ <OMS_HOME>/bin/rcuJDBCEngine sys/<Password>@<host_name>:<db_sid> JDBC_SCRIPT post_install_script.sql $PWD $ORACLE_HOME
$ <OMS_HOME>/bin/rcuJDBCEngine sys/<Password>@s<host_name>:<db_sid> JDBC_SCRIPT post_install_pactht.sql $PWD $ORACLE_HOME
$ <OMS_HOME>/bin/emctl start oms   

3. Now we can install preupgrade patch. You can download it from OTN.

$ unzip p13597150_111010_Generic.zip
$ <OMS_HOME>/bin/emctl stop oms
$ <OMS_HOME>/bin/emctl start oms
$ cd 13597150
$ <OMS_HOME>/OPatch/opatch apply   
$ <OMS_HOME>/bin/rcuJDBCEngine sys/<Password>@<host_name>:<db_sid> JDBC_SCRIPT $ORACLE_HOME/sysman/preupgc/puc_dblink_pkgdef.sql
$ <OMS_HOME>/bin/rcuJDBCEngine sys/<Password>@<host_name>:<db_sid> JDBC_SCRIPT $ORACLE_HOME/sysman/preupgc/puc_dblink_pkgbody.sql
$ <OMS_HOME>/bin/rcuJDBCEngine sysman/<Password>@<host_name>:<db_sid> JDBC_SCRIPT $ORACLE_HOME/sysman/preupgc/pre_upg_console.sql   
$ <OMS_HOME>/bin/emctl start oms   

4. Now we are ready to start upgrading. Upgrade starts with agents. Log on to 11g console, at the Deployment tab click on the new link "Enterprise Manager 12c Upgrade Console" then "Manage Software". At the Software Location area write the directory of new plugins and click validate. 

To have plugins you can unzip all three disks of 12c those you have downloaded from OTN or you can just extract the plugins directory from the third disk. You also need to put the new version of agent(s) you have. You can download the one(s) from OTN too. 

There are two things to notice here. First, you must not extract the agent software zip file while putting under plugins directory. And the second is that there is no agent for Windows, not yet for 64-bit version and never ever going to be for 32-bit. So if you have a 32-bit Windows agent running you will not be able to use it with 12c.

When you click on validate a job will run and analyze your agent(s). Your agent(s) might fall into one of these groups:
  • Completely Upgradable - A non-windows agent with no or upgradable plugins.
  • Missing Plug-Ins Software -  A non-windows agent with not upgradable plugins. My agents falls in this category since it is a Linux agent with Exadata plugins. Starting from 12c, Exadata features are included in OEM, so no plugin is required.
  • Missing Agent Software - Agent software is not available
  • Not Supported - A Windows agent 
If what you have is a fully green pie chart, move on to next step. You can also move on with a yellow one which is due to not upgradable plugins. Otherwise, if you have not supported windows agent give up upgrading or uninstall the agent. And if you have not upgradable plugins, remove those plugins.

5. After verifying agent(s), upgrade them by following steps below described in Oracle documentation. First three steps will create a job, you can find job output screen in each step:

a) Deploy and configure the software binaries of Oracle Management Agent 12c


b) Generate a health report and check the readiness of the predeployed Management Agents


c) Verify and sign off the health check report
 
d) Switch over the old Management Agents to the newly deployed ones so that they can communicate with Enterprise Manager Cloud Control


Check the status of upgraded agent:
$ <AGENT12c_HOME>/agent_inst/bin/emctl status agent  

Out put should be like:

Copyright (c) 1996, 2012 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version     : 12.1.0.1.0
OMS Version       : (unknown)
Protocol Version  : 12.1.0.1.0
Agent Home        : <AGENT12c_HOME>/agent_inst
Agent Binaries    : <AGENT12c_HOME>/core/12.1.0.1.0
Agent Process ID  : 19089
Parent Process ID : 18963  
Agent URL         : https://<host>:3872/emd/main/
Repository URL    : https://<host>:4900/empbs/upload
Started at        : 
Started by user   : oracle
Last Reload       : (none)
Last successful upload                       : (none)
Last attempted upload                        : (none)
Total Megabytes of XML files uploaded so far : 0
Number of XML files pending upload           : 69
Size of XML files pending upload(MB)         : 0.07  
Available disk space on upload filesystem    : 21.46%
Collection Status                            : Collections enabled
Last attempted heartbeat to OMS              : 
Last successful heartbeat to OMS             : (none)

---------------------------------------------------------------
Agent is Running and Ready


6. It's time to upgrade OMS. Copy emkey from the OMS to the Management Repository and stop OMS service as prerequisite and then run the graphical installer from the directory you have unzipped the 12c CDs.

$ <OMS_HOME>/bin/emctl config emkey -copy_to_repos_from_file -repos_host <host_name> -repos_port 1521 -repos_sid <db_sid> -repos_user sysman -emkey_file <OMS_HOME>/sysman/config/emkey.ora
$ <OMS_HOME>/bin/emctl stop oms
$ ./runInstaller






During prerequisite check if anything found an error screen will appear as below, click yes to make the installer fix the issue

 Also some best practices and performance recommendations may appear in a warning window. You can implement them after installation





 Run the root script when prompted for 

You may now log on to new OMS Console.

Discovering Exadata Through 12c

1. First thing to do before start monitoring Exadata through 12c is installing 12c agents to database nodes. As you know, it is strictly disallowed to install anything on storage cells, therefore all Exadata components are monitored by agents on database nodes. Download agent software from MOS, bug number 12960610. In the zip file there is file named CloudControl_Standalone_12.1.0.1.0_LINUX.X64.tar.Z, copy it to first node under /tmp/emkit directory.

An important point here to pay attention is em.param file. The file is generated by Oracle Exadata Database Machine Configurator Spreadsheet and it should be located as /opt/oracle.SupportTools/onecommand/em.param. If not, you need to generate one. Once you have one, you have to add or modify the following lines properly:

OMS_LOCATION = MY_GC
EM_BASE = / u01/app/embase

OMS_HOST = oemserver.company.com
OMS_PORT = 4900


EM_USER=oracle
EM_PASSWORD=<password>


Do check the port number by running following command on OEM server:
$ emctl status oms -details
Look for the  "Upload HTTPS Port" in the output.

To install agent, as root:
$ tar xBvpf CloudControl_Standalone_12.1.0.1.0_LINUX.X64.tar.Z
$ cd /tmp/emkit/common
$ (sh setupem.sh -emOnly -emUser <em_user> -emBase <em_base> | tee /tmp/setupem.log) 3>&1 1>&2 2>&3 | tee /tmp/setupem.err

2. After installation of OEM 12c Agent for Exadata, it is time to discover database machine. Though, before starting do check the Exadata Database Machine Schematic file located as /opt/oracle.SupportTools/onecommand/databasemachine.xml. Its version must be at least 502 and well-formed as XML (In my case, I had problems, while discovery process, due to some invalid characters in the file). Instead of trying to fix errors later the hard way,  fix simply in the beginning.

Log on to 12c console. From the Setup menu, Add Target --> Add Targets Manually. Choose "Add Non-Host Targets Using Guided Process (Also Adds Related Targets)", select  Oracle Exadata Database Machine from the Target Types and click "Add Using Guided Discover".  To start the discovery wizard select "Discover a new Database Machine and its hardware components as targets" and click Discover Targets.

Go till the end of the wizard, process takes a little while to complete. At the end, you got your Exadata manageable via 12c Cloud Control.


Installing Oracle R Enterprise

Oracle has integrated popular statistical package R as a component of database's Advanced Analytics Option. Oracle R Enterprise comes with Big Data integrated too. It also can be installed separately to 11 R2 databases on Oracle Linux 5 64-bit and naturally to Exadata. I'll try to briefly explain how you can install it.

Prerequites:

1. If not already enabled, enable yum repos for Oracle. You can find out how from Oracle public yum site.
2. Install R

$ yum list  R
$ yum install R.x86_64
3. Patch your database if its version is not 11.2.0.3. Instal patch number 12598677 for 11.2.0.1 and  129765544 for 11.2.0.2.

Installing Client on Windows:

1. Next thing to be done is installing client tool. Download R for Windows from R Project and install it.
2. Download Oracle R Enterprise Client Packages and Client Supporting Packages for Windows from OTN and unzip them.
3. To install packages; run R from All Programs as administrator, from the menu: Packages -> Install package(s) from local zip files:
   a. Navigate to <unzip_dir>\ore-supporting-windows-1.0\bin\windows\contrib\2.13
   b. Select; DBI_0.2-5.zip, png_0.1-4.zip and ROracle_1.1-1.zip
   c. Click open:
        package 'DBI' successfully unpacked and MD5 sums checked
        package 'png' successfully unpacked and MD5 sums checked
        package 'ROracle' successfully unpacked and MD5 sums checked

   d. Repeat the step for client packages. Navigate to <unzip_dir>\ore-windows-1.0\bin\windows\contrib\2.13
   e. Select; ORE_1.0.zip, OREbase_1.0.zip, OREeda_1.0.zip, OREgraphics_1.0.zip, OREstats_1.0.zip, ORExml_1.0.zip
   f. Click open:
        package 'ORE' successfully unpacked and MD5 sums checked
        package 'OREbase' successfully unpacked and MD5 sums checked
        package 'OREeda' successfully unpacked and MD5 sums checked
        package 'OREgraphics' successfully unpacked and MD5 sums checked
        package 'OREstats' successfully unpacked and MD5 sums checked
        package 'ORExml' successfully unpacked and MD5 sums checked


Installing Server:

1. To install server download ore-server-linux-x86-64-1.0.zip from OTN.
2. Extract files
$ unzip ore-server-linux-x86-64-1.0.zip
$ tar xzf ore-server-linux-x86-64-1.0.tar.gz
3. Set environment variabeles
$ export R_HOME=/usr/lib64/R
$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
4. Install
$ cd ore-server-linux-x86-64-1.0/
$ ./install.sh
5. Enable R for the user:
SQL> grant rqrole to <user_name>

To connect your database through R run R client:
R> library(ORE)
R> ore.connect(user="<user_name>", sid="<DB_SID>", host="<host_name>", password="<password>", port = 1521)
R> ore.sync
R> ore.attach()
R> ore.ls()
R> help(Startup)


Backing up Exadata Storage Cell

As a DMA, you'd like to backup Stroge Cells as well as a part of your backup process. Though, when it comes to Cell backup necessary things to be backed up are pretty much limited. Because Exadata does most of the work automatically for you using an embedded USB called CELLBOOT USB flash drive. Latest successfully working system image is stored in CELLBOOT USB. You can find versions of both active system and CELLBOOT USB by running the command as seen below:

$ imageinfo

Kernel version: 2.6.18-194.3.1.0.4.el5 #1 SMP Sat Feb 19 03:38:37 EST 2011 x86_64
Cell version: OSS_11.2.0.3.0_LINUX.X64_110520
Cell rpm version: cell-11.2.2.3.2_LINUX.X64_110520-1

Active image version: 11.2.2.3.2.110520
Active image activated: 2011-06-24 00:04:28 -0700
Active image status: success
Active system partition on device: /dev/md5
Active software partition on device: /dev/md7

In partition rollback: Impossible

Cell boot usb partition: /dev/sdm1
Cell boot usb version: 11.2.2.3.2.110520

Inactive image version: undefined
Rollback to the inactive partitions: Impossible

What is left for you to back up is just a couple files listed below:

1. /etc/hosts
2. /etc/modprobe.conf
3. /etc/sysconfig/network
4. /etc/sysconfig/network-scripts/*

Exadata also provides a way for you to create your own CELLBOOT USB. All you have to is plug-in a USB disk which should be at least 1 GB and run the tool:

$ /opt/oracle.SupportTools/make_cellboot_usb

Exadata turns your USB disk into a bootable disk of active image of your system.

Backing up Infiniband Swithes

If you're an Exadata admin, your title becomes DMA (Database Machine Administrator) instead of DBA. Since backups are routine tasks of every DBA, scope of what to backup extends a little bit when it comes to Exadata. Beyond the databases, you need to backup hardware also.

So, let's start with infiband swithes. Before configurations changes or updating its software it's a good practice to backup them. There are 3 things to backup and actually it's a really simple thing to do.

Here are the steps for firwware version 1.3.3:

1. Login to switch's ILOM interface. There should be an https address for that
2. Click on the "Maintenance" tab an then "Backup/Restore"
3. Select "Backup" as operation and "Browser" as transfer method from the drop downs
4. Write a passphrase
5. Click "Run" button and save the XML output.
6. For the next item to backup logon to switch through ssh
7. Take a backup of the file opensm.conf under /etc/opensm/ directory to somewhere else. 
8. Last to backup is the output of the commad: version

Don't forget to repeat steps for other infiband switches too:)