Jan
23
2008
You have a file with the following format:
type-1 2008/01/12 20:04:40 xxx yyy
type-2 2008/01/11 10:05:20 aaa bbb
type-2 2008/01/01 11:15:10 ccc ddd
You want to count for each type, how many rows have time stamp > 2008/01/10
BEGIN{
ta1=0;
ta2=0;
}
{
split($2, a, " ");
split(a[1], ymd, "/");
split(a[2], hms, ":");
t0 = mktime("2008 01 10 00 00 00");
t = mktime(sprintf("%s %s %s %s %s %s", ymd[1], ymd[2], ymd[3], hms[1], hms[2], hms[3]));
if( t > 0) {
if($1 == "type1") {
if(t > t0) {ta1 = ta1 +1 }
}
if($1 == "type2") {
if(t > t0) {ta2 = ta2 +1 }
}
}
END {
print "type1", ta1, "type2", ta1;
}
Jan
03
2008
The Following code will get sum of column 1 :-)
cat filename | awk -F "\\t"
'BEGIN { t1=0; t2=0 } { if($1<10) t2+=$1; t1+=$1 } END { print t1, t2, t2/t1 }'
Jan
03
2008
To use so you need LWP::Protocol::http::SocksChain.
There is a number of such protocol packages, but usually, installing any one of them requires a number of other prerequisites:
Here is a process I have tested, if you do have super user privilege, you can ignore where "/path/to/install" is mentioned
0. $ sudo -s
0.1 setenv PERL5LIB /path/to/install/lib/site_perl
1. INSTALL OPENSSL
1.1 http://www.openssl.org/source/,
download http://www.openssl.org/source/openssl-0.9.8g.tar.gz
1.2 Unzip openssl-0.9.8g.tar.gz
1.3 Read INSTALL
1.4 $ ./config
1.5 $ make
1.6 $ make test
1.6 $ make install
2. INSTALL Net::SSLeay
2.1 Download it from http://search.cpan.org/~flora/Net-SSLeay-1.32/,
download http://search.cpan.org/CPAN/authors/id/F/FL/FLORA/Net-SSLeay-1.32.tar.gz
2.2 Unzip Net-SSLeay-1.32.tar.gz
2.3 $ perl Makefile.PL PREFIX=/path/to/install
2.4 $ make install # the Makefile requires -lz, on certain system, you can remove it
2.5 $ cd examples
2.6 $ get_page.pl www.cryptsoft.com 443 /
3. INSTALL IO::Socket::SSL
3.1 Download it from http://search.cpan.org/dist/IO-Socket-SSL/,
download http://search.cpan.org/CPAN/authors/id/S/SU/SULLR/IO-Socket-SSL-1.12.tar.gz
3.2 Unzip IO-Socket-SSL-1.12.tar.gz
3.3 $ perl Makefile.PL PREFIX=/path/to/install
3.4 $ make
3.5 $ make test
3.6 $ make install
4. INSTALL Net::SC
4.1 Download it from http://search.cpan.org/~gosha/Net-SC-1.20/,
download http://search.cpan.org/CPAN/authors/id/G/GO/GOSHA/Net-SC-1.20.tar.gz
4.2 Unzip Net-SC-1.20.tar.gz
4.3 $ perl Makefile.PL PREFIX=/path/to/install
4.4 $ make
4.5 $ make test
4.6 $ make install
5. INSTALL
5.1 Download it from http://search.cpan.org/~gosha/LWP-Protocol-http-SocksChain-1.4/,
download http://search.cpan.org/CPAN/authors/id/G/GO/GOSHA/LWP-Protocol-http-SocksChain-1.4.tar.gz
5.2 Unzip LWP-Protocol-http-SocksChain-1.4.tar.gz
5.3 $ perl Makefile.PL PREFIX=/path/to/install
5.4 $ make
5.5 $ make test
5.6 $ make install
Now you can use LWPGet
#!/usr/bin/perl -w
require 5.8.0;
use strict;
use lib "/path/to/install/lib/site_perl/5.8.7";
use LWP::Simple; use LWP::UserAgent;
use LWP::Protocol::http::SocksChain;
LWP::Protocol::implementor( http => 'LWP::Protocol::http::SocksChain' );
@LWP::Protocol::http::SocksChain::EXTRA_SOCK_OPTS = (
Chain_Len => 1,
Debug => 0,
Random_Chain => 1,
Chain_File_Data => [
'ip_of_socks_proxy:port:::5',
],
Auto_Save => 0,
Restore_Type => 0 );
my $ua = new LWP::UserAgent $ua->agent('Mozilla/4.0 (compatible; MSIE 6.0; Windws NT 5.1)');
my $Url = $ARGV[0];
my $response = $ua->get($Url);
if($response->is_success) {
my $page = $response->content;
print $page, "\\n";
} else {
print STDERR "Fail to get $Url\\n";
}
Sep
28
2007
Step 1: Download the package
Step 2: Install the Binary
shell> /usr/sbin/groupadd mysql
shell> /usr/sbin/useradd -g mysql mysql
shell> cd /usr/local
shell> gunzip < /PATH/TO/MYSQL-VERSION-OS.tar.gz | tar xvf -
shell> ln -s FULL-PATH-TO-MYSQL-VERSION-OS mysql
shell> cd mysql
shell> scripts/mysql_install_db --user=mysql
shell> chown -R root .
shell> chown -R mysql data
shell> chgrp -R mysql .
shell> bin/mysqld_safe --user=mysql & # start mysql daemon
Step 3: Post-installation
shell> su mysql
shell> mysql/scripts/mysql_install_db
shell> mysql -u root -p
mysql> use mysql
mysql> GRANT ALL PRIVILEGES ON *.* TO 'user'@'localhost' IDENTIFIED BY 'password' WITH GRANT OPTION;
mysql> set PASSWORD FOR 'user'@'localhost' = PASSWORD('password');
mysql> FLUSH PRIVILEGES;
mysql> select * from mysql.user;
mysql> create database db_name;
mysql> SHOW DATABASES;
mysql> use db_name;
mysql> SHOWTABLES;
Sep
25
2007
The screen command lets you start new terminals, detach them and bring them back later. So you do not need to worry about how to bring a process to background any more. Here, I will give a very brief introduction on it, which is good enough for one to see the beauty of it. If you want to learn more, do a search "screen man" on any search engine.
To start screen: enter screen -RD at Unix prompt.
To exit screen: you need to close all the terminals attach to it. Hit ctrl-d or enter exit for each terminal, screen will exit when the last terminal exits.
To detach screen (sent it to background): simply close the terminals window if you are using any ssh/telnet clients, or to do it explicitly by hitting ctrl-a then d.
To re-attach screen and bring you previous Unix session to foreground: enter screen -RD at Unix prompt. If you only have one screen running on the host, it will be brought to foreground right away. Otherwise, see below.
You can start multiple screens on one host, detach them to send them to background. When you enter screen -RD next time, screen will list all the detached ones, and you can choose to re-attach any one of them by screen -RD screen_name
Sep
22
2007
| To find size under a dir |
|
du -sk * |
| To find # of files in a dir |
|
find . -type f | wc |
|
find . | cut -f 2 -d | uniq -c | sort n |
|
find . | cut -f 2-3 -d | uniq -c | sort n |
| To find and do sth to the results |
|
find dir_name -name filename* | xargs cat | sort | uniq > filename_new |
| To find all symbolic links |
|
ls -lR | grep '\->' |
| o use tar, gzip, unzip, bzip2 |
|
tar cvf - . | gzip > filename.tar.gz |
|
gzip -dc filename.tar.gz | tar xvf - |
|
bzip2 -d filename.bz2 |
| To sort file |
|
sort +0 -nr input_file > output_file -- sort with first column(+0), using number order(-n), reversed order(-r) |
| To remove duplicated lines |
|
uniq input_file > output_file |
| To get first/last k lines of a file |
|
head -n 10 input_file > output_file |
|
tail -n 10 input_file > output_file |
| To get certain lines in a file |
|
sed -n -e 20,30p filename -- only print lines 20~30 |
| To replace all blanks after 1,2,3 with just 1,2,3 |
|
cat filename | sed -e 's/\([1-3]\)\([ ]*\)/\1/' |
| awk |
|
Line level handling |
|
|
awk ' {split ($0, a, "\t"); printf("%s\t%s\n", a[2], a[3])} ' input_file > output_file |
|
|
awk '{if ( ($2=="xxxxxxx") && ($1>150)) print $0}' input_file > output_file -- given condition |
|
To right justify 1st column |
|
|
cat filename | awk '{printf "%10d %s\n",$1,$2}' |
|
To sum numbers in a column |
|
|
cat filename | awk 'BEGIN { tot=0 } { tot+=$1 } END { print tot }' |
|
Generate batch processing scripts |
|
|
Move ls -1 * | awk '{ print "mv " $1 " " $1}' | sed -e 's/EXT_A\(.*\)EXT_A/EXT_A\1EXT_B/' |
|
|
Copy ls -1 * | awk '{ printf "scp host:/path/%s %s\n", $1, " ."}' |
|
To compare two sorted files |
|
|
diff old_file new_file | grep ">" | awk '{print $2}' > diff.added |
|
|
diff old_file new_file | grep "<" | awk '{print $2}' > diff.deleted |
| vi |
|
To change to UPPER and lower cases |
|
|
:%s/.*/\U&/ |
|
|
:$s/.*/\L&/ |
|
To delete ^M |
|
|
:1,$ s/Ctl-VCtl-M//g |
|
To view HEX value and recover |
|
|
:%!xxd |
|
|
:%!xxd -r |