Category Archives: Unix & Shell script

Shell script in Unix or Cygwin

Standford Startup Engineering online course – week1, 2 summary & experience

Standford Starup Engineering

Strategy

A long waiting is over. I am more determined to complete this course with following reasons : Potential, Interesting, and Collaborative Learning.  As course title suggested, at the end of course, I need to come up with a HTML 5 enabled mobile application that can bring a potential crowd sourcing.

Technologies in the course are relevant to my current work. Even though applications that I am dealing with are not public facing, certain technologies are relevant and very interesting : AWS, Cloud, Heroku, Node.js, Unix CLI, Git, GitHub, emacs, Bootstrap, Backbone.js, Coffeescript, REST, JSON, NoSQL, BitCoin, Social and payment integration. So many topics that I wanted to master/learn, but hard to get to because of time constraint and lack of systematical approach. For this reason, I thought about collaborative learning. Learn and discuss in a group and maximizing a benefit of peer pressure to keep everyone on the topics and achieve the learning objectives in the end. I have motivated and talented 8 co-workers, who are taking this course together, so it will be perfect time for me to experiment collaborative learning.

Week 1, 2 Experience

Each week’s objectives are clearly defined and lecture slides are very well prepared. Of course, the professor is an entrepreneur himself and very technical. First two weeks topics are sticking to AWS, Cloud, Heroku, Node.js, Git, and Unix commands. Without having in-depth knowledge about Unix command, I don’t have a problem to complete the homeworks. I think having more than 115,000 students from all over the world, professors are having a great challenge. Probably, it can be a learning experience for them by dealing with so many students (big data) and analyze the data in useful ways.

Summary – my notes

FileSystem Basics

  • pwd #Print working directory
  • touch myfile # Create a blank file called myfile
  • ls myfile
  • ls -alrth myfiel # list metadata on myfile
    • -a show you all files, even hidden
    • -l show you hugh amounts of information
    • -r reverse the order of how the files are displayed
    • -t show you the files in the modification time
    • -h ?
  • alias ll=’ls -alrth’ # set up an alias to save typing
  • echo “line1” >> myfile # append via ‘>>’ to a file

There are two ways of see the current value of PATH :

  • env | grep “^PATH”
  • echo $PATH

Read one or more files and print them to standard output

  • cat [options] [files]

Retrieve from remote location

  • wget [url]

Change permission of a file. Total number can be presented as a parameter to allow certain user group with privilege.

  • chmod
    • 400 read by owner
    • 040 read by group
    • 004 read by anybody (other)
    • 200 write by owner
    • 020 write by group
    • 002 write by anybody
    • 100 execute by owner
    • 010 execute by group
    • 001 execute by anybody

Connect to remote machines

  • ssh -i [private key]

Create a single or multiple directories. With -p, create parents if doesn’t exit.

  • mkdir -p [directories]

Remove all

  • rm -rf :
  • rmdir [directory]

Create alias – create a file and add below information.

  • Host awshost1
  • HostName ec2-34-228-105-7.us-west-2.compute.amazonaws.com
  • User ubuntu
  • IdentityFile “~/.ssh/xxkey.pem”

Copy hello.txt from local to remote home

  • scp hello.txt awshost1:~/

Copy ~/foo.txt from remote to local

  • scp awshost1:~/foo.txt .

sudo : Perform following command with super-user (root) capabilities.

sudo apt-get update : download the package lists from the repositories and updates them to get info on the newest versions of packages and dependencies.

Linux has two major ways to install software : Binary packages, Source.

  • Install a Binary package :
  • sudo apt-get install

Binary Package Disadvantages

  • Not able to customize binaries
  • Only old versions are available in binary format

sudo apt-get upgrade : fetch new version of packages existing on the machine if APT knows about these new versions by way of apt-get update.

sudo apt-get dist-upgrade: same as apt-get upgrade, plus it will handle dependencies.

sudo add-apt-repository ppa:chris-lea/node.js : a command which allows us to add a new repository link to the apt sources (/etc/apt/sources.list). Handy to add PPA archives. ppa:<repository-name>

*PPA (Personal Package Archive) : a special software repository for uploading source packages to be built and published as an A|PT repository by Launchpad or similar application – exclusive to Ubuntu.
*APT (Advanced Packaging Tool) : a free user interface that works with core libraries to handle the installation and removal of software on the GNU/Linux distribution. APT simplifies the software on Unix-like system by automating retrieval, configuration and installation of software packages.
* https://help.ubuntu.com/community/Repositories/CommandLine

#!/usr/bin/env node : Writting an executable scirpt with Node beginss with shebang : #!, followed by the path to a Node executable.

Advertisements

Cygpath utility issue

So, I got the Cygwin started without any issue. However, when I execute the command to compile java code, it complains NoClass Def Found exception. Cygwin can’t find ant.jar in the classpath. I looked at the classpath, it was defintely wrong. Previous “.” was translated into “c;” and all other directories are prefixed with Cygwin home directory.

I found that there are some issues with new version of Cygwin utility command “cygpath”. I used –path –windows to convert the CLASSPATH to Cygwin format, but it didn’t work. Only solution that I found was to use two steps approach. First, convert CLASSPATH to Unix path, then convert Unix path to Windows path.

Environment.function.sh

function Classpath.build()
{
local aPath=$1
local tempPath=””

osType=$(uname -s)
#
# Now make sure it is converted to work with Cygwin
#
case ${osType} in
CYGWIN* )
# This is required since Java is running in windows and not CYGWIN
tempPath=”$(cygpath –path –unix ${aPath})”
pathValue=”;$(cygpath –path –mixed ${tempPath})”
;;
* )
pathValue=”:${aPath}”
;;
esac

echo “${pathValue}”
}

This works beautifully and now all the jar files are loaded into the CLASSPATH.

* in front of path and unix are double dash

Cygwin upgrade

We have an old version of Cygwin 1.2.x. I downloaded newer version of Cygwin and installed. Package manager was little bit tricky. By default, it didn’t download all plugins. Especially, at first, it was even missing “mintty.exe”, which allows Cygwin to open a terminal window. Also, after executed “mkpasswd -l > /etc/passwd” command, it didn’t generate the my account folder under Cygwin home. Eventually, I re-install older version of Cygwin and everything worked fine under the old version of Cygwin. After I went through old version uninstallation and re-installation of new version by manually checking “mintty, it started to show the terminal console.

There was another issue after that. New Cygwin complained about ‘\r’ character from the existing Shell Scripts. I looked at Cygwin help page and found out that it was due to DOS vs. Unix file type difference. All existing Shell scripts have ‘\r’ character at the end of line.

tr -d ‘\r’ < existing_script_file > new_script_file

Another issue was that these scripts are scattered in different file locations. They are more than 50, but less than 100. I don’t want to navigate to each folder and execute the script. I had to write a script to loop through files and remove ‘\r’. I haven’t officially studied Shell script previously, so it was extra challenge for me. Eventually, I got it working with below script.

#!/bin/bash
# Shell to read source file and remove ‘\r’ character from sh script
# First thing that you want to do is convert .bash_profile by executing below command manually in Cygwin
#
# tr -d ‘\r’ < .bash_profile > .bash_profile_unix
# mv .bash_profile_unix .bash_profile
#
# Place Convert.win7.sh to C:\Deployed folder and C:\DevApps\cygwin\home\$User folder
# Make sure all the files are writable.

file_dir=JADE-Tools-JEE5/DEV/Database

cd $file_dir
str=”_unix”
for fn in *.sh
do
echo $fn
tr -d ‘\r’ < $fn > $fn””$str
mv $fn””$str $fn

Cygwin 1.7.5

If I save above script as Convert.win7.sh, then I can execute the script as below.

$ bash Convert.win7.sh