|
|
|
Sunday, 16 September 2012
hbase site
Sample core-site.xml
Sample Mapred-site.xml
they be compressed? Should be one of NONE, RECORD or BLOCK.
Cloudera's Distribution for Hadoop switches this default to BLOCK
for better performance.
SocketFactory settings interfering with the socket factory associated
with the actual daemons.
to be available for jobs in this queue.
account in scheduling decisions.
allocated to a user at any given time, if there is competition for them.
This user limit can vary between a minimum and maximum value. The former
depends on the number of users who have submitted jobs, and the latter is
set to this property value. For example, suppose the value of this
property is 25. If two users have submitted jobs to a queue, no single
user can use more than 50% of the queue resources. If a third user submits
a job, no single user can use more than 33% of the queue resources. With 4
or more users, no user can use more than 25% of the queue's resources. A
value of 100 implies no user limits are imposed.
of the job queue.
account in scheduling decisions by default in a job queue.
for the job queue at any given point of time by default.
of the job queue.
the job queues for jobs to initialize.
Initialization poller to initialize jobs in a set of queue.
If number mentioned in property is equal to number of job queues
then a single thread would initialize jobs in a queue. If lesser
then a thread would get a set of queues assigned. If the number
is greater then number of threads would be equal to number of
job queues.
to be available for jobs in this queue.
account in scheduling decisions.
allocated to a user at any given time, if there is competition for them.
This user limit can vary between a minimum and maximum value. The former
depends on the number of users who have submitted jobs, and the latter is
set to this property value. For example, suppose the value of this
property is 25. If two users have submitted jobs to a queue, no single
user can use more than 50% of the queue resources. If a third user submits
a job, no single user can use more than 33% of the queue resources. With 4
or more users, no user can use more than 25% of the queue's resources. A
value of 100 implies no user limits are imposed.
of the job queue.
to be available for jobs in this queue.
account in scheduling decisions.
allocated to a user at any given time, if there is competition for them.
This user limit can vary between a minimum and maximum value. The former
depends on the number of users who have submitted jobs, and the latter is
set to this property value. For example, suppose the value of this
property is 25. If two users have submitted jobs to a queue, no single
user can use more than 50% of the queue resources. If a third user submits
a job, no single user can use more than 33% of the queue resources. With 4
or more users, no user can use more than 25% of the queue's resources. A
value of 100 implies no user limits are imposed.
of the job queue.
Sample hdfs-site.xml
|
Decommissioning nodes in a hadoop cluster
How to decommission nodes/blacklist nodes
HDFS
Put following config in conf/hdfs-site.xml:Use following command to ask HDFS to re-read host exclude file and decommission nodes accordingly.dfs.hosts.exclude /full/path/of/host/exclude/file
./bin/hadoop dfsadmin -refreshNodes
MapReduce
Put following config in conf/mapred-site.xmlUse following command to ask Hadoop MapReduce to reconfigure nodes.mapred.hosts.exclude /full/path/of/host/exclude/file
./bin/hadoop mradmin -refreshNodes
Whitelist/Recommission
Also you can "whitelist" nodes. In other words, you can specify which nodes are allowed to connect to namenode/jobtracker.HDFS
Put following config in conf/hdfs-site.xml:Use following command to ask Hadoop to refresh node status to based on configuration.dfs.hosts /full/path/to/whitelisted/node/file
./bin/hadoop dfsadmin -refreshNodes
MapReduce
Put following config in conf/mapred-site.xmlUse following command to ask Hadoop MapReduce to reconfigure nodes.mapred.hosts >/full/path/to/whitelisted/node/file
./bin/hadoop mradmin -refreshNodes
Support of -mradmin was added in 0.21.0. See JIRA issue https://issues.apache.org/jira/browse/HADOOP-5643 for details.
Subscribe to:
Posts (Atom)