Tuesday, August 22, 2017

Git tag and github release

Git Tag

Git tag is nothing but a marker or symbol for a point in time snapshot of your git repository.

There are two types of tags in git:

Lightweight Tag

A pointer to a specific commit. It only has a tag name.

Annotated Tag  

In addition to putting a marker on a specific commit, git physically create the snapshot of all files and keep it internally with a tag message, email, and a date.

To create a tag for a given commit, the command syntax is:

git tag tag_name commit_id

To push the tag to the remote repository

git push remote_name tag_name

Github Release

Release is a concept of github, it internally uses git tag to create a static point of time snapshot. The details can be found at https://help.github.com/articles/creating-releases/.

Releases can be created on:

  1. Current head of master or branches

  2. Existing tags

  3. Recent past commits.




Wednesday, February 17, 2016

Yahoo Closes Burbank Office



Thursday, November 05, 2015

Null Handling in Hadoop Pig Latin

For chararray type, when you load a dataset, PigStorage will convert empty fields to null. So in any relations, you won't find any empty string but only nulls.

However, in the pig script, if you have a constant as '', it is not treated as null.

So '' is not null return true.
'' is null return not true.

If A is a relation immediately after a load, A.$0 == '' will never be true.

If you compose something manually with GENERATE, it will keep the origin.

B = FOREACH A GENERATE $0, $1, ''; -- Will keep the value as empty string
C = FOREACH A GENERATE $0, $2, (chararry) null; -- Will keep the value as null

Sorting for NULLs

NULL is always treated as smallest value, if you do ORDER BY DESC, it will come last. If you do ASC, it comes first.

Friday, November 15, 2013

hadoop 2.2.0 installation resource

General steps:

http://milinda.pathirage.org/hadoop/yarn/2013/09/29/how-to-setup-multi-node-hadoop-20xyarn-cluster.html#references<br />

It refers to:


should be:

How to build 64bit native library or simply download from:

It has a detailed process on how to build from hadoop source. This is the only reliable/working steps I have found so far.
Hadoop build is already moved to maven but the hadoop document still shows old steps with ant (of course it won't work).

Really sucks.

Another error you may have:
$ hadoop fs -ls
ls: `.': No such file or directory

How to solve:
hadoop fs -mkdir $USER

RHEL6 mail attachment with uuencode won't work anymore

The old trick of sending out attachement with
uuencode /tmp/myfile myfile | mail -s "Attachemnt" user1@xyz.nowhere
won't work any more after migrated to RHEL6. All the uuencoded message will display as body contents rather than an attachment.

If you have a script like above, for sure you will get impacted:

Redhat seems to have some kind of workaround but it only opens to its registered users.

The fix is pretty simply but you'll still need to make changes to your so-far-working-well scripts.
mail -a /tmp/myfile -s "Attachement" user1@xyz.nowhere.

The problem seems coming from the headers the new version introduced.
something like:
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: 7bit

Old version headers just have:
MIME-Version: 1.0

There might be options to disable the new headers, let me know if anybody find them out.

Tuesday, November 12, 2013

Oracle Byte Length of CLOB type

All functions in DB_LOB for CLOB type operates at character level. This causes inconvenience when you want to do a substrb to limit the string length (especially oracle SQL only takes 4000 bytes for varchar2 type).

Here are 2 pure SQL queres may help you to a bit.

Query to get all complete multi-bytes characters to fill in a SQL varchar2 type
It uses a recursive query to do binary search to get the maximum character length:

with b4000 (id, e, s, len)
(select rowid as id, 666 e, 666 s, lengthb(dbms_lob.substr(text, 666, 1)) len from clob_src
union all
select b1.id,
case when b1.len + lengthb(dbms_lob.substr(text, s, e)) <= 4000 then e + s else e end,
case when b1.len + lengthb(dbms_lob.substr(text, s, e)) <= 4000 then s else trunc(s/2) end,
case when b1.len + lengthb(dbms_lob.substr(text, s, e)) <= 4000 then len + lengthb(dbms_lob.substr(text, s, e)) else len end
 from clob_src s1, b4000 b1
where s1.rowid = b1.id and case when b1.len + lengthb(dbms_lob.substr(text, s, e)) <= 4000 then s else trunc(s/2) end >= 1
) select * from b4000

* Oracle only holds 4000 bytes and a UTF-8 character can take up to 6 bytes, that comes the magic number 4000/6 ~= 666. It will be the number of characters we pull out from CLOB each time. When it reaches the 4000 bytes limits, we cut the step by half recursively until the the step reaches to 1.

Query to get bytes length of your CLOB column:

with blen(id, i, len, lenb)
as (
select rowid as id, 0, length(dbms_lob.substr(text, 666, 1)), lengthb(dbms_lob.substr(text, 666, 1)) from clob_tgt
union all
select b.id, i+ 1, len + length(dbms_lob.substr(text, 666, (i+1)*666 + 1)), lenb + lengthb(dbms_lob.substr(text, 666, (i+1)*666 + 1))
from clob_tgt s, blen b
where s.rowid = b.id and len + length(dbms_lob.substr(text, 666, (i+1)*666 + 1)) <= dbms_lob.getlength(text)
) select * from blen

* This should be much simpler to understand, just cut the clob into segments with character length 666 and summary up all of the byte lengths of each segment.

Monday, April 08, 2013

Easy way to generate Fibonacci number with oracle sql

You could use model but there is an easy way

With FB(k1, k2)
select 0 k1, 1 k2 from dual
union all
select k2 as k1, k1 + k2 as k2 from fb where greatest(k1, k2) < 100
select k1 from fb


The strange thing is that Oracle needs both columns in the where clause. If you only have one (like where k1 < 100), you will get error

Monday, January 23, 2012

simplest cgo program

Have been struggling to make a simple cgo sample to get familiar on how to link c code and go library.

Finally made it working.

package cgo_test


void out() {
printf("Hello world from c\n");
import "C"

func Out() {

package main

import "cgo_test"

func main() {

include ../go/src/Make.inc


include ../go/src/Make.pkg

main: install main.go
$(GC) main.go
$(LD) -o $@ main.$O

The most annoy error message I got is something like:
gomake main
main.go:6: cannot refer to unexported name cgo_test.out
main.go:6: undefined: cgo_test.out

It seems that GO only export Initcap names by default.