Thrift installation

In this tutorial, I am going to explain how to use python and thrift to access HBase. Here is the summary of steps you will need to follow:
1) Download thrift
2) Install thrift dependencies
3) Compile and install thrift
4) Generate HBase thrift python module
5) Add HBase thrift python module to pythonpath
6) Start HBase thrift server
7) Use the client!
Following is the detailed explanation of the steps. I am assuming that you will be using ubuntu as your development environment. That’s what I use. I am also assuming that HBase is installed and you have HBASE_HOME defined in the environment.
1) Download thrift
Download thrift by clicking on the link embedded in this sentence.
Unzip the tar.gz file using tar -xvzf  thrift-0.3.0.tar.gz. Let’s say you unzipped it in /home/horcrux/Software/thrift-0.3.0/
2) Install thrift dependencies
Thrift requires many packages for compilation. It requires boost c++ libraries, flex, mkmf and other build essentials. You can install all the dependencies by executing the following commands. ruby1.8-dev is to get mkmf installed.
sudo apt-get install build-essential
sudo apt-get install libboost1.40-dev
sudo apt-get install flex
sudo apt-get install ruby1.8-dev
3) Compile and install thrift
Execute the following commands to compile and install thrift
cd /home/horcrux/Software/thrift-0.3.0/
./configure
make
sudo make install
Now let’s install thrift python. The following command will make sure that the thrift module is in your pythonpath.
cd /home/horcrux/Software/thrift-0.3.0/lib/py
sudo python setup.py install
4) Generate HBase thrift python module
Once this is done, you should have thrift in your path. You should be able to execute thrift command from anywhere. Now let’s generate the Hbase thrift modeule from the Hbase.thrift config file.
thrift --gen py $HBASE_HOME/src/java/org/apache/hadoop/hbase/thrift/Hbase.thrift
This command will create gen-py folder in your thrift folder (/home/horcrux/Software/thrift-0.3.0).
5) Add HBase thrift python module to pythonpath
We need to add gen-py folder to python path. You can do so by multiple ways
a) You can add it directly at the top of your python file
import sys
sys.path.append('/home/horcrux/Software/thrift-0.3.0/gen-py')
or
b) If you are using an IDE like pydev, add it as a pythonpath source folder.
or
c) add it to pythonpath environemnt variable in your .bashrc.
export PYTHONPATH=$PYTHONPATH:/home/horcrux/Software/thrift-0.3.0/gen-py
6) Start HBase thrift server
You can simply start the thrift server by executing the following command:
$HBASE_HOME/bin/hbase thrift start
This will start HBase thrift server on port 9090 (default port).
7) Use the client!
Here is a sample code that will print all the table names on your HBase server:
from thrift.transport.TSocket import TSocket
from thrift.transport.TTransport import TBufferedTransport
from thrift.protocol import TBinaryProtocol
from hbase import Hbase

transport = TBufferedTransport(TSocket('localhost', 9090))
transport.open()
protocol = TBinaryProtocol.TBinaryProtocol(transport)
client = Hbase.Client(protocol)
print(client.getTableNames())

That’s it.

Fastest HBase Write using HBase Bulk Load

While you are trying to put Millions and even billions of key-values into HBase from your MR job, you can feel, even TableOutPutFormat is not that much efficient.

In such cases you can use HBase's Bulk load feature, which is tremendously faster than TableOutPutFormat.

The bulk load feature uses a MapReduce job to output table data in HBase's internal data format, and then directly loads the generated StoreFiles into a running cluster.
The process consists of 2 main steps.

• Preparing data via a MapReduce job

Data here refers to as the HBase data files(StoreFiles).
To achieve the same we need to change the OutPutFormat class of our MR job to HFileOutputFormat, which writes out data in HBase's internal storage format.

The following are the main changes that you have to make in your MR job,
.....
        mapRedJob.setMapOutputKeyClass(ImmutableBytesWritable.class);
        mapRedJob.setMapOutputValueClass(Put.class);

        mapRedJob.setInputFormatClass(TextInputFormat.class);
        mapRedJob.setOutputFormatClass(HFileOutputFormat.class);
.....
   //HBase configuration
   Configuration hConf = HBaseConfiguration.create(hadoopConf);
        hConf.set("hbase.zookeeper.quorum", zookeeper);
        hConf.set("hbase.zookeeper.property.clientPort", port);
        HTable hTable = new HTable(hConf, tableName);
        HFileOutputFormat.configureIncrementalLoad(mapRedJob, hTable);
.....

A test map method would look like the following,
.....
   public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
            Put row = new Put(Bytes.toBytes(value.toString()));
            row.add(Bytes.toBytes("CF"), Bytes.toBytes("C"), Bytes.toBytes(value.toString()));
            try {
                context.write(new ImmutableBytesWritable(Bytes.toBytes(value.toString())), row);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
.....

• Loading the Data into the HBase Table

Data can be loaded into the cluster using the command line tool 'completebulkload'.
The format is as follows,
$ hadoop jar hbase-VERSION.jar completebulkload [-c /path/to/hbase/config/hbase-site.xml] /user/myoutput mytable.
You can also load these files from your MR job programmatically by using the following code,

LoadIncrementalHFiles lihf = new LoadIncrementalHFiles(hConf);
         lihf.doBulkLoad(new Path(hfileOutPutPath), hTable);

Try it and feel the performance improvement.

SSRS Reporting for months

In SSRS,

For reporting ,formatting can b changed for a text box by right clicking a text box and going to format.Go to 3 ellipses near format code and choose currency to display $ before costs.

By default ,if you generate monthly reports, it will generate report in alphabetical order eg april,august,december like that.To order it in proper format,right click the text box
go to properties turn on the interactive sort option.
Now right click again and go to edit group,In sorting give the column datepart(mm,date)  and display would be datename(mm,date) .

Shared network drive and SSIS

When we copy a big file from a server to a remote location (NAS drive or something) through ssis,there comes a network connection unavailable error.
To fix this go to my computer and map a network drive .Add that remote drive.Now the job willl execute properly through SSIS.

Executing a package is easy but running a job from sql server gives error.To resolve that

Use the UNC path when specifying the destination-- the SQL Agent doesn't have a concept of "mapped" "drives". Also, SQL Agent typically runs as "Local Service" or "Local System" and, as such, doesn't have rights to remote shares on other computers. You have a couple of choices: Run SQL Agent as a role account in the domain. Grant that account permission to write to the directory / share where you'd like the backups stored. Run SQL Agent as "Network Service". It will authenticate to the sharing server with the domain computer account of the machine the service is running on. Grant that account permission to write to the directory / share where you'd like the backup stored. If you don't have a domain, create an account with the same username and password on both the machine hosting SQL Agent and the machine hosting the backup files. Change SQL Agent to run as this "role" account, and grant that account permission to write to the directory / share where you'd like the backup stored. (The "poor man's domain"...)      or else create a batch file pkgexec.bat having "C:\Program Files (x86)\Microsoft SQL Server\100\DTS\Binn\dtexec.exe" /file c:\SceneVisitReports.dtsx  and schedule in windows task manager c:\>pkgexec.bat

How to copy reports from sql server to a remote disk drive

For SSIS package to work with remote disk drive go to my computer and map a network drive to a local drive

Open a new notepad and write a command
copy c:\*.txt \\kp\nas\report and save it as "filename.bat"

In SSIS package
execute process task ,put this bat file in executables and run it.

The Excel Connection Manager is not supported in the 64-bit version of SSIS

The Excel Connection Manager is not supported in the 64-bit version of SSIS

When i am using SSIS Excel connection manager in my 64bit dev environment it is giving me the following error..
[Connection manager "Excel Connection Manager"] Error: SSIS Error Code DTS_E_OLEDB_EXCEL_NOT_SUPPORTED: The Excel Connection Manager is not supported in the 64-bit version of SSIS, as no OLE DB provider is available.
To fix this:
Go to Project –> Project Properties –> then Set Run64BitRuntime = False.
thats it…

Increasing java heap space

While running jasper reports i got error of insufficient java heap space.
Here is the solution for same.

In linux systems,set this

export JVM_ARGS="-Xms1024m -Xmx1024m"

In windows

Running Java applications in computers takes some memory during the process which is known as Java memory (Java heap). Frequently, it is necessary to increase that heap to prevent throttling the performance of the application.

1. Go to Control Panel. Click on "Start" button. Then click on "Control Panel."
   
   
2. 2
   

   Find Programs
   Select Programs. In the left side of Control Panel click on "Programs." Pleas click on the "Programs" written with green color, not the "Uninstall a program," which is in blue color.
3. 3
   

   Find Java
   Go to Java settings. In the next dialog click on "Java," usually at the bottom of the other Programs; "Java Control Panel" dialog pop-up opens.
4. 4
   

   Find Java Tab
   Select "Java" tab. Inside the Java tab, click on "View" button. It opens the "Java Runtime Environment Settings"
5. 5
   

   View Java Runtime Environment
   Change amount of heap. In the "Runtime Parameters" column change the value, or if it is blank decide for the new value, of the Java memory.
6. 6
   

   Modify Runtime Parameter
   Modify the parameter. To modify the parameter, double click in the "Runtime Parameters" column and,
   • type -Xmx512m that assigns 512MB memory for the Java.
   • type -Xmx1024m that assigns 1GB memory for the Java.
   • type -Xmx2048m that assigns 2GB memory for the Java.
   • type -Xmx3072m that assigns 3GB memory for the Java, and so on.
   • Please note, it begins with a minus sign and ends to an m.
   • Also note, there is no blank space between characters.
7. 7
   Close the dialogue box. Click on "OK" button on the "Java Runtime Environment Settings" to close it.
8. 8
   Close Java dialogue box. "Apply" button in the "Java Control Panel" has been enabled now. You should click on "Apply" to finalise the new Java memory. Then click on the "OK" button.
9. 9
   Close the Windows 7 Control Panel.