Using bash scripts to become a more efficient web developer
Here are a few scripts, that I find really useful for speeding up my web development time.
I’ve been building up this list as I needed to use them – so they maybe a little raw.
For example often clients send me images with filenames that don’t match my naming standard, so running the appropriate script really helps keep me focussed on the job in hand and not waste too much time reformatting filenames etc.
Finally if you have any useful little bash scripts why don’t you add them to the comments below?
Bash script to append a .txt extension to a filename
Iterate over the current directory, get all files with .log and append .txt to the end of the entire filename:
for i in *.log*; do mv "$i" "$i.txt"; done
Script to make filenames lowercase
Converts all the file names in a directory and converts them to lowercase.
echo version:
for i in *.log*; do echo mv \"$i\" \"`echo $i| tr [A-Z] [a-z]`\"; done
real version:
for i in *.txt; do mv "$i" "`echo $i| tr [A-Z] [a-z]`"; done
Quick note on how to use grep to search for a string in file
-o print Only the matched parts of a matching line, with each part on a separate line. -H print the filename for each match. -n prefix each line of output with the 1 based line number. -R, -r recursively read all files under subdirectories.
grep -oHnr "some pattern" *.txt
{txt,log} this globs the file extensions together. 2>/dev/null this passes all errors to a blackhole, so that they won’t be displayed.
grep 123741 ./adapter_logs/*.{txt,log} 2>/dev/null
Find all occurrences of 123741 in all .txt and .log files in all subdirectories.
for i in 'find . -type d'; do grep 123741 $i/*.{txt,log}; done 2>/dev/null
List all files that match this regular expression in the current direcory
All files with the number 5 in their name.
ls | grep -e .*5.*
Bash script to check whether a tag exists in a subversion repository
svn ls http://www.mysvnserver.co.uk/myproject/tags | grep mytag-0.7.0.1/
The [ ] is known as a Test, by default it tests integers Also to end an if statement you use fi
if [ 1 = 1 ] ; then echo YES; else echo NO; fi
if [ "`svn ls http://mysvnserver/mysvnrepository/myproject/tags | grep R-9.20.400.0/`" = "R-9.20.400.0/" ]; then echo YES; else echo NO; fi
Note need the / in the evaluation part of “it-0.9.0.7/”, don’t really need it in grep it-0.9.0.7/
$ if [ "`svn ls http://www.mysvnserver.co.uk/myproject/tags | grep it-0.7.0.1/`" = "it-0.7.0.1/" ]; then echo YES; else echo NO; fi
Quick note on how to increment a variable, in a bash script
j=0; j=$(($j+1))
incrementing in a loop
j=0;for i in *.txt; do echo "some_file_"$j".txt"; j=$(($j+1)); done
Using a bash for loop to list all text files and append whoopee
for i in *.txt; do echo $i"whoopee"; done
Change all filenames to be a specific incremented name, starting from file 16
j=16;for i in *.jpg; do mv "$i" "gallery_"$j".jpg"; j=$(($j+1)); done;ls
Script to loop through all .txt filenames and make them lower case
for i in *.txt; do mv "$i" "`echo $i| tr [A-Z] [a-z]`"; done
Get all JPG files and create the appropriate HTML list tags for them and add them to a file
Create an HTML version:
for i in *.jpg; do echo "\t<li>\r\t\t<img src='images/$i' alt='' />\r\t</li>"; done > list_items.html
Create a HAML version:
for i in *.jpg; do echo "\t%li\r\t\t%img{ :src => 'images/$i', :alt => '' }\r"; done > imgs.haml
Batch change a misspelled filename of certain files, using string replacement
for i in aples*.jpg; do mv $i ${i/aples/apples} ; done
Script to batch create files based on filenames in a file
Create a file called something like “my_files.txt” with the following content:
archive_error_112480_0040.txt archive_error_114390_0043.txt etc…
Commands:
for i in `sed -n -e 'p' my_files.txt`; do mkdir a_test/$i; done
or
while read x; do touch $x; done < my_files.txt
or
cat my_files.txt | xargs touch
(xargs passes each line as an argument into EACH touch)
Using bash Expansions, and allow escapes
-e is an undocumented flag that allows escapes Expansion ${command/parameter/substitution} everything else needs to be escaped.
echo -e ${PATH//\:/\\n}
Executing a .bash_script after making changes
source .bash_profile
or
. .bash_profile
NB: you can ‘include’ other bash .sh scripts in a .bash_profile file by using ’source’ or ‘.’
Quick tip on using curl for downloading files off the web
Use -O to name the remote file, curl will save it locally but remove the rest of the path.
curl -O http://rubyforge-files.ruby-forum.com/rubygems/rubygems-0.9.0.tgz
Bash command for testing the existence of files and directories
If a directory does NOT exist create it
[ -d "/var/cache/git" ] || mkdir /var/cache/git
If a file exists echo a message:
[ -f "./new_file.txt" ] && echo "its there"
Bash command to remove spaces from filenames
for i in *.html; do mv "$i" "`echo $i| tr -d ' '`"; done
replace spaces for underscores
for i in *.html; do mv "$i" "`echo $i| tr ' ' '_'`"; done
Vim regex to replace spaces in images file names of ah HTML page
:%s:\(images\/\)\(\w\+\)\s\(\w\+\.jpg\):\1\2_\3:g
Copy text to the clipboard
echo 'hello world'|pbcopy
cmd + v
Conclusion
Some of these are quite basic but useful none-the-less, please add your own to the comments below. Many thanks, and good luck.
Updated…
Script function for .bash_profile to cd to the last opened finder location
This is a script written by a colleague. James Power. Add the following function to your .bash_profile in Apple OSX. Now open a directory in Finder, then open terminal and type cdf, your terminal will now change directory to the same one as the last Finder opened.
# function to change directory to the one set in the last opened finder.
cdf () {
currFolderPath=$( /usr/bin/osascript <<" EOT"
tell application "Finder"
try
set currFolder to (folder of the front window as alias)
on error
set currFolder to (path to desktop folder as alias)
end try
POSIX path of currFolder
end tell
EOT
)
echo "cd to \"$currFolderPath\""
cd "$currFolderPath"
}
Also for a much more comprehensive list of commands check out: http://www.commandlinefu.com/commands/tagged/34/bash
Ruby Rush command-line tool
If you use Ruby maybe check out Rush, it allows you do use Ruby in a bash command-line like way.
emson… » 18 Useful bash scripts for web developers…
linux shell commands for web developers…
Excellent list, instantly going into my delicious account!
Here’s a few I use quite often:
Convert DOS EOL to UNIX style EOL (SVN pukes on unix machines when you try to svn add a file with DOS EOL’s); note that it runs on EVERY file! So, be aware that you will either need to add in functionality to detect if it is hitting a binary file, or, just run it on a directory in question with text/source files only.
find . * -type f -exec dos2unix {} \;This one I’ve used to remove .svn files from the root of a directory recursively. At first I tried to use find . -name .svn -delete but it wasn’t running properly and I didn’t want to spend the time figuring it out:
find . -name .svn -exec rm -rf {} \;The following command can also be used to clean up any directories/files that have not been added/modified to the repo (can also replace rm -rf with any other command):
svn status | grep ? | awk '{print $2}' | xargs rm -rfEasier way to increment:
$ x=1; let x++
Excellent thanks. You all might want to checkout the comments on HackerNews: http://news.ycombinator.com/item?id=643113
As there are one or two useful commands there to.
Oh, grep, seems you are using Markdown, not Textile. Please delete my above comment.
A note on
For easy patterns like this you don’t even need
grep,lsis sufficient:Haha, no, that’s not entirely it, too.
What is it you are using for comment formatting?
[...] ben put an intriguing blog post on emson… » 18 Useful bash scripts for web developersHere’s a quick excerptI’ve been building up this list as I needed to use them – so they maybe a little raw. For example often clients send me images with filenames that don’t match my naming standard, so running the appropriate script really helps keep me … [...]
One thing that occurs to me is that a lot of your examples could be simplified by the “rename” command. Availability varies – it’s in stock debian and I believe CentOS includes it as prename. It’s a rename command that takes a perl-compatible regex to rename. Briefly on why it’s worthwhile – you can use -n to dry run your action (frequently worthwhile) and I usually find it easier to dash off…
That said -
append a .txt extension to files with
rename 's/$/.txt/' *Or convert filenames to lower case with:
rename 'y/A-Z/a-z/' *You could do the same thing to txt files only by changing * to *txt
Or remove underscores
rename 's/_/\ /' *Fixing a typo in a filename is left as an exercise to the reader
In general I try to embrace the unix ethos and learn individual tools that do one thing and learn how to compose them. I’m working on tr, sed, awk and friends. For the specific case of munging filenames, however, regexps are the tool I want and rename allows me to apply a regexp to a set of my files with minimum fuss. Very worth knowing…
Ollie: even easier way to increment:
((x++)) # echo $x shows “1″
You can also shorten this:
mv file file.txt
to this:
mv file{,.txt}
PS: feel free to
a) provide some sort of link next to the comment box that details what formatting options are available; and
b) reformat my comment appropriately
cheers,
Some of the for loops could be replaced with xargs.
ie.
( find . -iname \.txt -o -iname \.log -print0 | xargs -0 grep 123741 ) 2>/dev/null
You may also use find … -exec cmd … instead of -print0:
http://codesnippets.joyent.com/posts/show/2042
[...] emson… » 18 Useful bash scripts for web developers (tags: web development reference tools linux tips webdev cheatsheet bash shell cli unix scripts sysadmin commandline) [...]
[...] emson… » 18 Useful bash scripts for web developers — 13:39 via [...]
[...] : Delicious-IT | Date : Jun 05 2009 | Views : 1 views | Total Word : 9 | Print this Page! | Permalink! [...]
For what it’s worth, -e is not an undocumented option of echo.
man echo|grep '\-e' -e enable interpretation of backslash escapes If -e is in effect, the following sequences are recognized:It’s also unnecessary to use -e for shell expansions.
You do need it in the example above, but that’s unrelated to the expansion replacement. You need it there because you are trying to replace the character with a newline, and the
Very useful indeed, one thing that always baffles me is the lack of some kind of cohesive package of common scripts based on what you want to do with your linux box, I always thought something like that would make a good package.
pgl: Good call. But, for what it’s worth, let works on sh but double parenthesis syntax is bash-specific.
[...] emson… » 18 Useful bash scripts for web developers awesome collection of scripts for Web developers, mac and Linux users (tags: bash shell linux scripts development reference unix commandline tools web sysadmin tips cheatsheet) [...]
[...] emson… » 18 Useful bash scripts for web developers (tags: bash shell scripts development reference) [...]
[...] emson… » 18 Useful bash scripts for web developers [...]
[...] via emson… » 18 Useful bash scripts for web developers. [...]
Nice collection!
Here’s yet another method to increment:
for i in {1..10}; do echo $i done
Cheers
Patrick
Thanks for the useful info. It’s so interesting
[...] Some very useful bash scripts for web developers [...]
I have written previously about: “” Why not using JavaScript as a scripting language, but outside browsers !? http://www.jroller.com/dmdevito/entry/why_not_using_javascript_as “”
Do you know that Netscape Enterprise Server proposed “Server-Side JavaScript (SSJS) — an extended version of JavaScript that enables back-end access to databases, file systems, and servers” ?
So, here, I continue my idea: why not proposing a way to write Unix scripts using JavaScript ? The underlined idea here is to develop a common JS library to provide, for example, file system access, a similar way than Netscape Enterprise Server years ago.
Then, web developers, a more important community than Unix script developers, would be able to develop their own Unix scripts using their favorite language.
Why not ?
[...] Bash Manual Useful bash one liner scripts [...]
[...] Des scripts bashs utiles aux webdeveloppeurs MailPress, un plugin de gestion de newsletter pour WordPress Expose, un plugin jQuery pour mettre des blocs en valeur… ca flashe ! WordPress Multi Language, un plugin multilingue pour WordPress Charles, un proxy de débuggage HTTP WordPress Plugin Template, une base de départ pour coder un plugin WordPress [...]
[...] 18 Useful bash scripts for web developers — 11:03am via [...]
Here’s one for recursively touching files starting from the current directory -
find . -type f -exec touch ‘{}’ \;
also in bash, you can increment a var with: x=0 $((x++))
Hello. Thank you for this great info! Keep up the good job!
thank you! I really liked this post!
[...] emson… » 18 Useful bash scripts for web developers blog.emson.co.uk/2009/06/18-useful-bash-scripts-for-web-developers – view page – cached Here are a few scripts, that I find really useful for speeding up my web development time. — From the page [...]
Thanks for the tips… lots seems very useful..some of them to me too..!! Thanks again..
Thx! Very useful!
Thanks for adding your colleague’s “Script function for .bash_profile to cd to the last opened finder location” to the end of this article. It works great, except that I had to first remove an extraneous “EOT” after the “end tell”.
[...] http://blog.emson.co.uk/2009/06/18-useful-bash-scripts-for-web-developers/ [...]
18 Useful bash scripts for web developers …
Visit Emson.co.uk for having useful tips on bash scripts for web developers. …
+1 for Metapundit .. Use rename instead of all those ocward for.. in … mv
This fact related to this topic seems to be hot! Hence people do not should compose the thesis writing or just thesis proposal by their own efforts, they will use your help.
Here are some points on the semantics
.5.gets expanded inline, -unless- it fails to matchWhen using test(1) without then, as a course of proper style, I prefer to use test rather than [. This leads to better lines IMO:
which is simply the same semantic as:
which interestingly, could be written:
[ is not known as a test, [ is test(1). less(1) is more(1), and so on.
if, then, else, and fi are bash(1) special, and in most sh(1) implementations, actually sh(1) special too.
Instead of always writing echo 'yes' and so on, you could make an alias or a command, called 'h' (for 'human') or some such) which takes an exit status, and outputs a human oriented value, e.g.
test -e $DIR; h # => 'YES'
"(xargs passes each line as an argument into EACH touch)"
on the contrary, xargs by default appends all line separated arguments to a single call to it's given command line.
The xargs from a text file example could simply be re-written:
touch
cat my_file_list.txtPeople often start using xargs where they don't need to...
replacing spaces in file names:
for i in *\ *; do mv $i ${i// /_}; done
As far as echo's -e argument being undocumented goes, well, that's not true, if you read the manpage for echo(1) on most systems, it'll say:
"Some shells may provide a builtin echo command which is similar or identical to this util-ity."
And find(1) has an -exec argument.
[...] emson… » 18 Useful bash scripts for web developers – Here are a few scripts, that I find really useful for speeding up my web development time. [...]
[...] : Advanced Bash-Scripting Guide LearningBash-Scripting: Advanced Bash-Scripting Guide Learningemson… 18 Useful bash scripts for web developersUsing bash scripts to become a more efficient web developer … Finally if you have any useful [...]
This line still tries to create the directory despite its existance (giving an error saying that the directory exists)
[ -d "/var/cache/git" ] || mkdir /var/cache/git
[ -d "/var/cache/git" ] || mkdir /var/cache/git
I recommend using “mkdir -p” instead – in your example, if /var or /var/cache were missing it would create these parts of the path too, otherwise your command might fail…