Monday, November 15, 2004
Remove Hidden Folders Recursively in Windows
C:\temp>for /f "tokens=* delims=" %x in ('dir /s /b /a:d *cache') do rd /s /q "%x"
Alternatively we can create .bat batch file so we can just click it to run the script, but the token needs to be changed from %x to %%x:
for /f "tokens=* delims=" %%x in ('dir /s /b /a:d *cache') do rd /s /q "%%x"
Friday, November 12, 2004
C# vs Java
1. C# uses namespace to organize the class by "using" keyword. Not like Java's package, C#'s namespace is purely logical without any implication of file/folder relation.
2. No "throws" in C# since C# doesn't have checked exceptions. C# only handles exception when it's actually thrown (not may be thrown). While in Java you must try-catch a potential exception or throws it explicitly, otherwise your code won't compile.
3. C# has "internal" and "protected internal" access modifiers. You can define a class with more granular scale.
3. No primitive type in C#. All C# types are rooted from System.Object. Following code is fine in C# but not in Java:
ArrayList list = new ArrayList
list.Add(123); // Java compile error
int number = (int)list[0]; // Java compile error
Type type = number.GetType(); // Java compile error
5. C# property is more intuitive than Java's accessor methods.
6. C# has "struct" while Java doesn't. .NET will put structs in stack instead of heap.
7. C# has "virtual" functions and you have to use "override" modifier explicitly to override a method in subclasses. A "new" function in C# creates a new method with same signature in subclasses, which is only visible in that specific class. Java doesn't have "virtual", "override" and "new" function modifiers. Function is picked based on class type checking.
8. C# doesn't have "final" constants but it has "const" and "readonly" for constant declaration. The difference between "const" and "readonly":
const
- Can't be static.
- Value evaluated at compile time.
- Initialized at declaration only.
- Can be static or instance.
- Value evaluated at run time.
- Declaration or initialized in the constructor.
9. C# has delegate - a type-safe method pointer. This delegate is very useful from my opinion. You can do interesting things similar to C/C++ pointers. Java doesn't have equivalent function.
10. Passing by values is default in C#. In addition C# has option to pass by references using keyword of "ref" or "out" (you have to initialize the "out" parameters). You can not pass by references in Java (for objects, Java creates a copy of parameter pointing to the same object in the heap, which leads to confusion in some cases).
Looks like C# is more powerful than Java at a glance, or just some syntax sugar you would argue?
Friday, October 08, 2004
HPCBench Now Open Source
Overview
Hpcbench is a Linux-based network benchmark evaluating the high performance networks such as Gigabit Ethernet, Myrinet and QsNet. Hpcbench measures the network latency and achievable throughput between two ends. Hpcbench is able to log the kernel information for each test, which includes the CPU and memory usage, interrupts, swapping, paging, context switches, network cards' statistics, etc.
Hpcbench consists of three independent packages that test UDP, TCP and MPI communications respectively. A kernel resources tracing tool "sysmon" is also included, whose output is similar to that of vmstat, but has more information of network statistics.
Programming language: C, MPI.
Recommended OS and compiler: Linux kernel 2.4 and gcc.
UDP communication:
- Microsecond resolution
- Roundtrip time test (UDP ping)
- Throughput test
- Unidirectional and Bidirectional test
- UDP traffic generator (can run in single mode)
- Fixed size and exponential test
- Log throughputs and process resource usage of each test
- Log system resources information of client and server (Linux only)
- Create plot configuration file for gnuplot
- Configurable message size
- Other tunable parameters:
- Port number
- Client and server's UDP socket buffer size
- Message size
- Packet (datagram) size
- Data size of each read/write
- QoS (TOS) type (Pre-defined six levels)
- Test time
- Test repetition
- Maximum throughput restriction (Unidirectional and UDP traffic generator)
TCP communication:
- Microsecond resolution
- Roundtrip Time test (TCP ping)
- Throughput test
- Unidirectional and Bidirectional test
- Blocking and non-blocking test
- Fixed size and exponential test
- Linux sendfile() test
- Log throughputs and process resource usage of each test
- Log system resources information of client and server (Linux only)
- Create plot configuration file for gnuplot
- Configurable message size
- Other tunable parameters:
- Port number
- Client and server's TCP socket buffer (window) size
- Message size
- Data size of each read/write
- Iteration of read/write
- MTU (MSS) setting
- TCP socket's TCP_NODELAY option setting
- TCP socket's TCP_CORK option setting
- QoS (TOS) type (Pre-defined six levels)
- Test time
- Test repetition
MPI communication:
- Microsecond resolution
- Roundtrip Time test (MPI ping)
- Throughput test
- Unidirectional and Bidirectional test
- Blocking and non-blocking test
- Fixed size and exponential test
- Log throughputs and process resource usage of each test
- Log system resources information of two processes (nodes) (Linux only)
- Create plot configuration file for gnuplot
- Tunable parameters:
- Message size
- Test time
- Test repetition
Monday, August 02, 2004
Linux Tunneling
Linux Tunneling
If pop3 server is sshd enabled:
% ssh -f -N -L 1234:localhost:110 user@POP3_serverAccess web server behind firewall if the web server is accessible through ssh:
% telnet localhost 1234
% ssh -L 8080:localhost:80 129.100.19.33When there is a ssh gateway such as gate.csd.uwo.ca:
% ssh -f -N -L 5678:129.100.19.34:80 huang@gate.csd.uwo.caLinux Reverse ssh tunnel
% telnet localhost 5678 // Or open we bsite by: http://localhost:5678
When only 129:100.19.34 can ssh to cat.sharcnet.ca:
% ssh -R 5678:localhost:22 cat.sharcnet.caCat ssh to 129.100.19.34 : % ssh -p 5678
script:
#!/bin/shMake 129.100.19.33 web service accessible via 129.100.19.34:
# $COMMAND is to create the reverse ssh tunnel
COMMAND='ssh -N -R 5678:localhost:22 cat.sharcnet.ca'
# Is the tunnel up?
CHECK_TUNNEL=`ps -ef args | grep "$COMMAND" | grep -v grep`
# If the tunnel is not up, create the tunnel
if [ -z "$CHECK_TUNNEL" ] ; then
$COMMAND
fi
(in 129.100.19.34) % ssh -fNR 8080:129.100.19.33:80 cat.sharcnet.caRemote X tips: (if ssh -X not work)
(in cat.sharcnet.ca) http://localhost:8080/a.html
1. Disable firewall: # /etc/init.d/iptables stop
2. Run: % ssh -R 6001:localhost:6000 huang@labmachine
3: After logon : % export DISPLAY = ":1"
Windows tunneling tips:
Access windows RDP behind firewall (such as gate.csd.uwo.ca) through Secure SSH client:
Connection: gate.csd.uwo.ca (Port 22, check "Request tunnels only")
Tunneling: Add RDP, Type->TCP, Listen Port->3456, Destination Host->129.100.19.34, Destination Port->3389
After connecting to gate.csd.uwo.ca, open Remote Desktop Client with host: localhost:3456.
Saturday, July 10, 2004
Gmail is Awesome
Hotmail, Yahoo Mail, along with many other free email providers, offer only free 20-50M of mailbox. I know many corporate email systems also have a limit size of 100M for employees. Google sets the bar to 1G and it's free. It definitely is a big hit.
That makes sense. Google has expertise in building huge distributing system with commodity Linux boxes, and they could provide the same capacity with much lower price.
The only thing is the maximum size of attachment is 10M in gmail. How great it's if 20M or even 50M could be supported? I may consider using gmail as my personal online storage not just an email account.
Monday, June 21, 2004
Performance Analysis of High Performance Computing Networks
(Oct. 2004 updated). The full thesis can be downloaded here.
Following is the table of context of my work:
Network Performance Measurement and Analysis in High Performance Computing Environments
Chapter 1 Introduction.
Chapter 2 Background. 3
2.1 HPC history and Its Convergence to Cluster Computing.
2.2 HPC networking.
2.2.1 High Performance Network Technologies.
2.2.2 Networking of HPC Clusters.
2.3 Message Passing Interface (MPI
2.3.1 MPI Introduction.
2.3.2 MPICH.
2.4 Job Management System.
2.4.1 Goals of JMS.
2.4.2 LSF (Load Share Facility).
2.5 File Systems in HPC Clusters
2.5.1 Storage Networking.
2.5.2 Cluster File Systems.
2.5.3 Network Storage in SHARCNET.
2.6 Test-bed specifications
Chapter 3 Implementation of Hpcbench.
3.1 A Survey of Network Measurement Tools.
3.2 Metrics.
3.3 Communication Model.
3.4 Timers and Timing.
3.5 Iteration Estimation and Communication Synchronization.
3.6 System Resource Tracing.
3.7 UDP Communication Measurement Considerations.
3.8 An overivew of Hpcbench.
Chapter 4 Investigation of Gigabit Ethernet in HPC Clusters.
4.1 A Closer Look at Gigabit Ethernet
4.1.1 Protocol Properties.
4.1.2 Interrupts Coalescence and Jumbo Frame Size.
4.1.3 Data Buffers and Zero-Copy Technique.
4.2 Network Performance Analysis of Gigabit Ethernet
4.2.1 Examining Network Protocols Communication Internal
4.2.1.1 Alpha SMP Architecture.
4.2.1.2 Intel Xeon SMP Architecture.
4.2.2 Network Performance vs. Computer Performance.
4.2.3 Blocking and Non-blocking Communication.
4.2.4 Local Communication.
4.2.5 Network Protocols Latency.
4.2.6 TCP/IP Communication Throughput
4.3 A Comparison with Myrinet and Quadrics Interconnects
Chapter 5 Conclusions and Future Work.
5.1 Summary and Conclusions
5.2 Future Work.
Reference
94. 106
Tuesday, April 06, 2004
Sysmon - A Linux Tool to Monitor System Resources
So what's Sysmon anyway? In short, Sysmon is a lightweight Linux-based system resource tracing tool, as vmstat does, but has more information. The output includes the CPU/memory usage, swapping/paging, interrupts and each network card's statistics, which includes interrupts to kernel, packets and bytes that received and sent in a specified interval. Sysmon can run as a daemon and log the system information for a long period. Help menu:
Usage: sysmon [-hwWbk] [-i interface-name] [-r repeat] [-t test-interval] [-T test-time]
[-h] Printout help messages.
[-w] Write all results to a file. Disable by default.
[-W] Write statistics of each network device to separate files. Disable by default.
[-b] Background (daemon) mode. Only valid when write option is defined.
[-k] Kill the sysmon background process (daemon). Disable by default.
[-i interface-name] Define the network device name (e.g. eth0). Monitor all if no interface defined.
[-r repeat] Repetition of monitoring. 10 times by default.
[-t test-interval] The interval (sample time) between each tracing in seconds. 2 seconds by default.
[-T test-time] The duration of system monitoring in minutes. Valid only write option defined.
[-o output] Specify the output (log) filename. Implies the write option.
Note: Default logfile has format of host-start-time.log if write option (-w) is defined.
Use "ifconfig" to check the network devices in your computer.
Possible names: eth0, wlan0, elan0, etc (defined by NETNAME in util.h). loop is for loopback address.
Example 1 (Monitor all network devices, very long format): % sysmon
Example 2 (Only monitor eth0): % sysmon -r 10 -t 2 -i eth0
Example 3 (Log every 10 minutes for one week, run as daemon): % sysmon -bw -i eth0 -t 600 -T 10080
Output format:
Network information: [interrupts] [recv-packets] [recv-bytes] [sent-packets] [sent-bytes]
Tuesday, March 16, 2004
MPICH Cluster Setup
This is a test to setup a cluster with two nodes using my home machines.
/etc/hosts (Redhat 9.0 with kernel 2.4.20) files:
Dell Inspiron 8100 (master node): 192.182.1.2 node1
Dell Dimension L600 (secondary node): 192.182.1.3 node2
Download the MPICH 1.2.5.2 from http://www-unix.mcs.anl.gov/mpi/mpich/ , follow the instruction to install on both machines. MPICH uses rsh or ssh to communicate with each other. The default is rsh. If you like to use ssh (secure shell) instead, you should configure with following parameters in the MPICH install directory:
[root]# ./configure --with-device=ch_p4 --prefix=/usr/local/mpich --rsh=ssh
[root]# make
After installation, add the /mpich_install_dir/bin and /mpich_install_dir/util to your $PATH environment. To let the master node ( laptop node1) know the other secondary nodes, add all nodes in the file /mpich_install_dir/util/machines/machines.LINUX:
[huang]$ cat machines.LINUX
# Change this file to contain the machines that you want to use
# to run MPI jobs on. The format is one host name per line, with either
# hostname
# or
# hostname:n
# where n is the number of processors in an SMP. The hostname should
# be the same as the result from the command "hostname"
#localhost.localdomain
node1
node2 To enable rsh (remote shell), edit the /etc/xinetd.d/rsh, change the line of "disable = yes" to "disable = no". To be convenient, I also enable the rlogin service. After the modification, you have to restart the xinetd daemon:
[root]# /etc/rc.d/init.d/xinetd restart
To let the node1 (master node) be able to run the programs in node2 automatically without password prompt, add .rhosts file in user's home directory of node2:
[huang] $ cat ~/.rhosts
node1 huang
Also, the /etc/hosts.allow and /etc/hosts.deny files must be correctly configured to allow the rsh service. For simplicity reason, I add following line on the /etc/hosts.allow file to accept all services between two machines:
ALL: node1 node2 192.182.1.0/255.255.255.0, 127.0.0.1
To allow the super user root to use the rsh and rlogin services, add another line on file /etc/securetty:
rsh, rlogin, rexec, pts/0, pts/1
The authentication file /etc/pam.d/rsh should also be modified:
[root]# cat /etc/pam.d/rsh
#%PAM-1.0
# For root login to succeed here with pam_securetty, "rsh" must be
# listed in /etc/securetty.
auth sufficient /lib/security/pam_nologin.so
auth optional /lib/security/pam_securetty.so
auth sufficient /lib/security/pam_env.so
auth sufficient /lib/security/pam_rhosts_auth.so
account sufficient /lib/security/pam_stack.so service=system-auth
session sufficient /lib/security/pam_stack.so service=system-auth We could verify the rsh service in master node (node1):
[huang]$ rsh node2 "ps -ef"
Then the running processes in node2 will be showed on node1. Now it's ready to run parallel programs. There are some sample programs in /mpich_install_dir/examples/basic directory, enter the directory and compile the source code with command make (in both machines), e.g., cpi is MPI program to compute the PI value:
[huang]$ mpirun -np 2 cpi
Process 0 of 2 on node1
pi is approximately 3.1415926544231318, Error is 0.0000000008333387
wall clock time = 0.001943
Process 1 of 2 on node2 Make sure the executable files in each machine must be in the same directory structure. We could also specify a configure file instead of using the default machines.LINUX configuration:
[huang]$ cat my.conf
node1 0 /home/huang/cpi
node2 1 /huang/cpi
[huang]$ mpirun -p4pg my.conf cpi
Process 0 of 1 on node1
pi is approximately 3.1415926544231318, Error is 0.0000000008333387
wall clock time = 0.002097
Process 1 of 2 on node2
P4 procgroup file is my.conf.
[huang]$ Enjoy the power of parallel computing!
Thursday, March 11, 2004
Wireless Setup (WPC11V4) in RedHat Linux
Occasionally I worked in Middlesex college with my Dell Inspiron 8100 notebook, but I couldn't get the wireless connection there with my Linux bootup (Redhat 9.0 with kernel 2.4.20). It's Linksys Wireless-B network card (WPC11 V4). Notice the WPC11 V4 network card has different chipset (Realtek 8180) from previous models that are based on prism2. Getting some wireless cards to work in Linux is not a trivial thing, and I spent a couple of days fiddling around how it work. A good link about wireless LAN resources for Linux comes from Jean Tourrilhes' excellent wireless collection.
Go to Realtek's dowload website, search for 8180L driver. There is one for Linux kernel 2.4.20. Download and unzip it. Compile it as a root user, there will be a driver file named "rtl8180-24x.o" created when there were no errors. If there were something wrong, it might be caused by a messed up kernel, or source that doesn't match the kernel you are running. You have to download a new kernel (2.4.20-8) and compile the kernel. When the driver file is created, copy it to the system's module library:
[root]# cp rtl8180_24x.o /lib/modules/`uname -r`/kernel/drivers/net/wireless/
[root]# cardctl insert
[root]# depmod -ae
[root]# modprobe rtl81880_24x
[root]# cardctl ident
Socket 0:
no product info available
Socket 1:
product info: "Realtek", "Rtl8180"
manfid: 0x0000, 0x024c
function: 6(network)
[root]# Now the device is recognized. To boot it up in the wireless network, we have to do some configuration work. Here is my script to enable the wireless card:
[huang]$ cat /etc/init.d/wlanup
# Load wireless lan driver
#/sbin/insmod -f rtl8180_24x.o
/sbin/modprobe rtl8180_24x
# Work as AP mode & Assign SSID and operation channel.
# Channel 1, 2, 10 are working fine in UWO
/sbin/iwpriv wlan0 wlan_para ssid=uwo
/sbin/iwpriv wlan0 wlan_para ssid2scan=uwo
/sbin/iwpriv wlan0 wlan_para channel=2
# Configure WEP. UWO doesn't use it, a "blue socket" instead
/sbin/iwpriv wlan0 wlan_para encmode=off
/sbin/iwpriv wlan0 wlan_para wepmode=off
# Configure debugging message
/sbin/iwpriv wlan0 msglevel 0
# Enable wireless lan driver
/sbin/iwpriv wlan0 enable
sleep 2
# Get IP address from DHCP server
/sbin/dhclient -1 -q -lf /var/lib/dhcp/dhclient-wlan0.leases \
-pf /var/run/dhclient-wlan0.pid wlan0
echo "$(/sbin/ifconfig wlan0)"
[huang]$ Notice in UWO's campus wireless network, the WEP (Wired Equivalent Privacy) is disable, instead, the school's username and password are used for authentication. The SSID (Service Set Identifier) is lowercase "uwo". After run the wlanup script, a network IP address of the wireless card would show up if the connection is established. To shutdown the connection, another script is used:
[huang]$ cat /etc/init.d/wlandown
# Shut down wlan0 net interface
/sbin/ifconfig wlan0 down
# Disable wireless lan driver
/sbin/iwpriv wlan0 disable# Unload module
/sbin/rmmod rtl8180_24x
[huang]$ su -
[root]# cardctl eject
Finally, we add a file named "S99wireless" with one line of "/etc/init.d/wlanup" into "/etc/rc.d/rc3.d/" directory, then the script would run automatically during Linux's booting up.
Tuesday, March 02, 2004
Resolve Hotmail Login Problem
I had a trouble when logging in some websites such as hotmail with Mozilla (Firefox).
The hardware is Motorola SB5100 cable modem and Netgear RIP614 router (firmware 4.15RC4)The systems worked properly with Sympatico and Execulink ADSL high-speed connections. After switching to Rogers Cable high speed Internet, logging on to hotmail's web-mail failed and was pending on "transferring data from server".
Solution: reduce the MTU size for the connections.
Linux: [root] # /sbin/ifconfig eth0 mtu 1418
Windows XP: Modify the Windows' Registry by command "regedit": in the tree of "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces", find out the network card's entry, something like "{C42A4EA1-8C79-48F4-9309-99D7DE35D462}", add a DWORD value with name of "MTU", select "Decimal" base and input the value data of "1418"; save and quit.
A MTU size of 1418 bytes is small enough to solve the problem in most cases. Basically a MTU of 1448-1472 bytes should work for most network environment.