Skip to content

Moving to amin.bitbucket.org

WordPress! Good old friend. It’s time for us to go on our own ways.

I’m moving to my own jekyll/disqus based blog at http://amin.bitbucket.org/

I won’t need hours to fix Java/Perl syntax  and planning to write heaps on SmartOS/Telstra/DevOps/Java.

SmartOS on VMWare and undefined VNIC

After I upgraded from Lion to Mountain Lion the MAC address of VMWare Fusion adaptor had changed for no good reason. That’s why SmartOS (illumos) was failing to plumb the network and dladm complains about unknown status.

root$ dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
e1000g0 Ethernet unknown 1000 half e1000g0

view raw
gistfile1.sh
hosted with ❤ by GitHub

To fix this check the newly generated MAC and update the /usbkey/config accordingly.

grep ^admin_nic /usbkey/config
admin_nic=0:c:29:fb:7f:64

view raw
gistfile1.sh
hosted with ❤ by GitHub

Check the MAC in OS after reboot. Now interfaces should come up automatically at boot time. 

[root@00-0c-29-fb-7f-64 ~]# dladm show-phys -m
LINK SLOT ADDRESS INUSE CLIENT
e1000g0 primary 0:c:29:fb:7f:64 yes e1000g0

view raw
gistfile1.sh
hosted with ❤ by GitHub

dtrace: failed to grab pid ?!

There seems to be a reasonable security feature in Solaris that limits usage of diagnostics tools based on privileges(5) of source and target PIDs. Saying that, tools  such as dtrace(1M) and pstack(1) should have equal or more privileges(5) than the target PID they want to observe. Otherwise the process owner can use the target PID to run instrumented instructions with higher privileges which is obviously a security hole.

But this fair statement can cause some headache especially when processes start from sources other than a shell such as SMF.

Let’s examine this scenario. What prevents the owner of this process to look inside even after setting all dtrace permissions to zone and user.

$ ppriv $$
29763: -bash
flags = <none>
E: basic,dtrace_proc,dtrace_user,proc_owner
I: basic,dtrace_proc,dtrace_user,proc_owner
P: basic,dtrace_proc,dtrace_user,proc_owner
L: basic,contract_event,contract_observer,dtrace_proc,dtrace_user,file_chown,file_chown_self,
file_dac_execute,file_dac_read,file_dac_search,file_dac_write,file_owner,file_setid,ipc_dac_read,
ipc_dac_write,ipc_owner,net_bindmlp,net_icmpaccess,net_mac_aware,net_privaddr,net_rawaccess,
proc_audit,proc_chroot,proc_lock_memory,proc_owner,proc_setid,proc_taskid,sys_acct,sys_admin,
sys_audit,sys_mount,sys_nfs,sys_resource
$ /usr/ucb/ps auxwww | grep java | fgrep "XX:+ExtendedDTraceProbes" | awk '{print $1,$2}'
weblogic 1745
weblogic 27317
$ pfexec dtrace -n 'hotspot$target:::object-alloc{ @ = quantize(arg1) }' -p 1745
dtrace: failed to grab pid 1745: permission denied
$ pstack 1745
pstack: cannot examine 1745: permission denied

view raw
gistfile1.sh
hosted with ❤ by GitHub

What’s up? The reason is hiding somewhere in the SMF service manifest. Let’s have a look:

<exec_method name='start' type='method' exec=…>
<method_context>
<method_credential user='weblogic' privileges='basic,sys_resource,…'/>
</method_context>
</exec_method>

view raw
gistfile1.xml
hosted with ❤ by GitHub

Looking at the PID of the service we notice it has extra sys_resource privilege assigned via SMF that we don’t have in our bash PID ($$).

$ ppriv 1745
flags = <none>
E: basic,sys_resource,…
I: basic,sys_resource,…
P: basic,sys_resource,…
…..

view raw
gistfile1.sh
hosted with ❤ by GitHub

That’s preventing bash PID to access SMF started service PID although they are owned by the same user. So what is this extra privilege?

$ ppriv -lv sys_resource
sys_resource
Allows a process to modify the resource limits specified
by setrlimit(2) and setrctl(2) without restriction.
Allows a process to exceed the per-user maximum number of processes.
Allows a process to extend or create files on a filesystem that
has less than minfree space in reserve.

view raw
gistfile1.txt
hosted with ❤ by GitHub

That’s a necessary  for this service. So have to add the missing to the user:

usermod -K defaultpriv=basic,sys_resource,… weblogic

view raw
add_priv.sh
hosted with ❤ by GitHub

Next time bash has sys_resource privilege and executes dtrace/pstack against SMF service successfully:

$ ppriv $$
8630: -bash
flags = <none>
E: basic,dtrace_proc,dtrace_user,proc_owner,sys_resource
I: basic,dtrace_proc,dtrace_user,proc_owner,sys_resource
P: basic,dtrace_proc,dtrace_user,proc_owner,sys_resource
L: basic,contract_event,contract_observer,dtrace_proc,dtrace_user,file_chown,file_chown_self,file_dac_execute,file_dac_read,file_dac_search,file_dac_write,file_owner,file_setid,ipc_dac_read,ipc_dac_write,ipc_owner,net_bindmlp,net_icmpaccess,net_mac_aware,net_privaddr,net_rawaccess,proc_audit,proc_chroot,proc_lock_memory,proc_owner,proc_setid,proc_taskid,sys_acct,sys_admin,sys_audit,sys_mount,sys_nfs,sys_resource
$ pfexec dtrace -n 'hotspot$target:::object-alloc{ @ = quantize(arg1) }' -p 1745
dtrace: description 'hotspot$target:::object-alloc' matched 1 probe
^C
value ————- Distribution ————- count
536870912 | 0
1073741824 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ 8945
2147483648 | 0

view raw
gistfile1.sh
hosted with ❤ by GitHub

Quote

VirtualBox Import Appliance; VT-x is not available

When importing a virtual appliance from a VT-x machine into a non VT-x one, you may end up with this error:

VT-x is not available: verr_vmx_no_vmx

Well the key to fix this is to disable VT-x (i.e. hardware virtualization instructions) but the problem is that the new host machine’s virtualbox settings window is all disabled:

So how can I disable VT-x then? Then solution I found it to open the appliance vbox XML file and manually disable HardwareVirtualEx.

vim ~/VirtualBox\ VMs/windows/windows.vbox

Then search for HardwareVirtEx and set all enabled items to false.

Mount NTFS in Solaris 11

1. Grab these two files: FSWpart.tar.gz and FSWfsmisc.tar.gz

2. Rename files:

# mv  fswpart-tar-gz-remove.png FSWpart.tar.gz
# mv  fswfsmisc-tar-gz-remove.png FSWfsmisc.tar.gz

3. Install the packages.

# tar -zxvf FSWpart.tar.gz
# tar -zxvf FSWfsmisc.tar.gz
# pkgadd -d . FSWpart
# pkgadd -d . FSWfsmisc
Select “y” if prompted for conflict in files:
The following files are already installed on the system and are being used by another package:
* /etc/gnome-vfs-2.0 <attribute change only>
* /etc/gnome-vfs-2.0/modules <attribute change only>
* /usr/lib/gnome-vfs-2.0 <attribute change only>
* /usr/lib/gnome-vfs-2.0/modules <attribute change only>
* /usr/sbin/mkntfs
* /usr/sbin/ntfsclone
* /usr/sbin/ntfscp
* /usr/sbin/ntfslabel
* /usr/sbin/ntfsresize
* /usr/sbin/ntfsundelete
* - conflict with a file which does not belong to any package.
Do you want to install these conflicting files [y,n,?,q] y

4. Find the target partition

# iostat -En

5. Mount partitions in the devices into folders you like:

# mkdir /mnt/d
# mount -F ntfs /dev/dsk/c5t0d0p3 /mnt/d

6. Add mount targets to /etc/vfstab

/dev/dsk/c5t0d0p2 - /mnt/c ntfs - yes -
/dev/dsk/c5t0d0p3 - /mnt/d ntfs - yes -

Mac OS X “Host is Down” with Curl, Python, PIP,etc

If you’re connected to internet and still some CLI tools such as curl, python, easy_install and pip fail with errors such as:

curl: (7) Failed to connect to IP: Host is down

error: [Errno 64] Host is down

This is while host is ping from command line and browser connects too. First check you don’t have any proxy settings. (echo $http_proxy)

If you don’t have any proxy settings, and browser accesses the site, well then it might be because of some strange rules in LittleSnitch in the case you have one running. Try to stop LittleSnitch network monitor and give it a go.

Why two Monitors is so good for Productivity?

Just until recently I believed having multiple desktops could work same as two monitors but I proved myself wrong last week.

For most of us, daily work is a combination of routine stuff (emails, maintenance, phone-calls, meetings) and development. The problem is when this flood of routine tasks take most all of our time and attention.

By having two monitors one can easily partition this two stream and keep and eye on both. Here is how my desk looks like these days.

In one monitor I have my outlook, communicator, GTD and in the second one all dev tools (iTerm, IntelliJ, emacs, sql*plus, etc) in hand.

I also noticed how good x86 is for development rather than SPARC. Having so many cores on one SPARC is great for scalable production deployment but not necessarily suitable enough for development. I decided to run Solaris in my PC and connect monitor/keyboard to this darling little beast; MacBook Air.

Running Oracle Repository Creation Utility (RCU) on Mac OS X

1. Grab RCU for Linux and extract. Go to rcuHome/bin folder.

2. Disable platform check in ./rcu script:

3. Create a Link from JDK to RCU:


4. Change JRE Path in ./rcu script:

What Makes Solaris Shine

Solaris has so many features and the more you use it, the more you find. Before moving to Telstra my (painful) experience with Solaris dated back almost 10 years ago with Solaris 7 and 8. Mostly because at that days it was among the limited options of a reliable platform for running mission critical Oracle RDBMS instances (I guess it is still).

But then starting again with Solaris 10 in Telstra I found the huge progress it had made. I always thank Oracle for saving Solaris and do hope that the continuation the good work although unfortunately we hear news that many brains behind this master-piece have left the Oracle.

IMHO here’s the list some of the most beloved features of Solaris 10. These are also the feature that don’t have a proper equivalent in other competitors.

  • Containers: real virtualization not buzzword
  • ZFS: nothing comes even close to this File system. What on earth they had in mind.
  • DTrace: keep tuning/monitoring live applications without major overhead
  • SMF: far better that init.d especially with contracts

Some may complain the Linux has OpenVZ/LXC or Mac OS X has ZFS and DTrace. but the question remains: are they all production ready? Even if they are, no doubt they got the idea from Solaris and tried to reimplement.

Oracle Entitlement Server PD Client

Enrollment Issue

I spent couple of hours working on a problem that prevents some of our WLSM instances to fire up correctly. The issue originated from a NullPoniterException in PDClient. PDClient is required for WLSM instances running in controlled (pull/push) mode. NPE was something like this:

<Mar 28, 2012 3:26:10 PM GMT+10:00> <Error> <HTTP> <BEA-101216> <Servlet: “PDClientServiceServlethttp” failed to preload on startup in Web application  : “pd-client.war”.javax.xml.ws.WebServiceException: java.lang.NullPointerExceptionat weblogic.wsee.jaxws.WLSInstanceResolver.getSingleton(WLSInstanceResolver.java:36)
at weblogic.wsee.jaxws.WLSInstanceResolver.start(WLSInstanceResolver.java:55)
Caused By: java.lang.NullPointerException
at oracle.security.jps.soap.pd.client.PDClient.<init>(PDClient.java:46)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
Truncated. see log file for complete stacktrace

In short, although not much vivid but if you also encounter this error it’s possibly due to immature enrolment of the SM instance.

For Server based SMs (e.g. Weblogic, WebSphere, and JBoss), OESSM also creates a config inside server’s smconfig folder besides the original one under $OES_CLIENT_HOME/oes_sm_instances.

For WLSM, the configuration is under WLS_DOMAIN/config/oeswlssmconfig folder. The JPS configuration in this folder is almost independent (see jps-config.xml ) but has a small link back to original config regarding enrolment wallet.  See:

<serviceInstance location=”/oracle/Middleware/oes_client/oes_sm_instances/<SM name>/config/enroll” provider=”credstoressp” name=”credstore.enroll”/>

If the enrolment of the SM has failed, then the cwallet.sso will be still there but in an incomplete mode which prevents the correct start up of the PDClient.

How to check if Wallet is correct?

OES server comes with ORAPKI a handy tool to inspect cwallet files. You can find it under $ORACLE_MIDDLEWARE_HOME/oracle_common/bin/

Here is how to display the contents of a binary wallet file:

/oracle/Middleware/oracle_common/bin/orapki  wallet display -wallet <cwallet.sso>

For a correctly enrolled file, the contents include two user certificate entries for Oracle Secret Store.

-bash-3.00$ /oracle/Middleware/oracle_common/bin/orapki  wallet display -wallet ../../<SM Name>/config/enroll/cwallet.sso
Oracle PKI Tool : Version 11.1.1.5.0
Copyright (c) 2004, 2011, Oracle and/or its affiliates. All rights reserved.
Requested Certificates:
User Certificates:
Oracle Secret Store entries:
OES_SYMMETRIC_KEY_MAP@#3#@OES_IV_PARAMETER_alias
OES_SYMMETRIC_KEY_MAP@#3#@OES_SYMMETRIC_KEY_alias
Trusted Certificates:
Subject:        OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE
Corporation,C=US
Subject:        OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

While an immature one lacks these two:

-bash-3.00$ /oracle/Middleware/oracle_common/bin/orapki  wallet display -wallet ../../<SM name>/config/enroll/cwallet.ssoOracle PKI Tool : Version 11.1.1.5.0
Copyright (c) 2004, 2011, Oracle and/or its affiliates. All rights reserved.
Requested Certificates:
User Certificates:
Trusted Certificates:
Subject:        OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US
Subject:        CN=GTE CyberTrust Global Root,OU=GTE CyberTrust Solutions\, Inc.,O=GTE Corporation,C=US
Subject:        OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

How to (Re) Enroll an SM?

There are couple of scripts generated by OESSM tool to initialize key and perform enrolment (under bin folder of SM instance) but there is my approach:

# prepare WLST env
.  /oracle/Middleware/wlserver_10.3/server/bin/setWLSEnv.sh

# prepare OES env
.  /oracle/Middleware/oes_client/oes_sm_instances/<name>/bin/setOesEnv.shjava -cp /oracle/Middleware/oes_client/modules/oracle.oes.sm_11.1.1/oes-client.jar: \ /oracle/Middleware/wlserver_10.3/server/lib/weblogic.jar \
-Doracle.security.jps.config=<sm instance folder>/jps-config.xml \
-Doracle.security.oes.tools.KeyStorePassword=<passwd> oracle.security.oes.tools.SMConfigTool \
-initEnrollment –smConfigId <instance_name> -prpFileName smconfig.wls.prp \
-port <WLSM HTTP Port> -sslport <WLSM HTTPS Port> -serverLocation /oracle/Middleware/wlserver_10.3 \
-wlsPassword <weblogic password> -domainLocation /oracle/Middleware/user_projects/domains/<WLSM domain name>

PS1: InitEnrolment is the only step required for controlled-pull SMs, because they directly talk to Policy Store DB. For controlled-push SMs, need an extra step of DoEnrolment (see config.sh)

PS2: Try to use JDK 1.6.26+