Running linux command periodically

Sometimes we need to monitor something in our servers like memory usage or hard drive space during a deployment process that is failing or a program that it´s filling all our memory and this implies to run a command in our consoles more than once to see the changes along the time.

A very useful command in this situations is watch. This command allows us to execute periodically a given command. For example, imagine that we want to visualize our memory. Without the watch command, we should execute the free command a lot of times one after the other. But with the watch command we can do this:

watch -n 2 --differences free
  • -n: Interval of seconds.
  • –differences: Highlight the difference between updates.

Executing this line the free command will be executed each two seconds and, in addition, the differences between the updates will be highlighted.

Unfortunately, for OS X users, there are bad news, this command is not available in the Terminal. But, we can solve it with a simple shell script.

#!/bin/bash
# usage: watch.sh <your_command> <sleep_duration>

while :; 
  do 
  clear; 
  echo "$(date)"
  $1; 
  sleep $2; 
done

It´s not perfect because you cannot execute command with parameters but, at least, you can have something similar to execute for example the memory_pressure command. Because, another bad news, the free command is not available in OS X systems.

See you.

Advertisements
Running linux command periodically

Find big files

A very common problem, especially in development environments where a lot of people and even different teams share the same servers, is running out of space in them. In general, due to a lack of maintenence tasks for these servers. Old logs never deleted, temporal files that have been there forever, this kind of things. All this things makes sometimes impossible to deploy our applications because we are running out of space. In these cases, we need to find what is exactly consuming this space.

To do that, we have in *nix systems some useful console commands or combination of commands.

The first one, it is useful to now the status of the space in our servers.

df -h

The result should be something similar to this. Probably with more lines because I have only pasted a couple of them.

Filesystem      Size   Used  Avail Capacity  iused    ifree %iused  Mounted on
/dev/disk0s2   111Gi   51Gi   60Gi    47% 13428490 15666812   46%   /
/dev/disk1s2   149Gi   63Gi   86Gi    43% 16540268 22448478   42%   /Volumes/HD

The first combination of command that we are going to see is useful to find the biggest X files or directories in a specific location and its subfolders. In this case, I am going to find the biggest ten files or directories in the location /Volumes/HD:

du -a /Volumes/HD | sort -n -r | head -n 10

The result should be something like:

87657984	/Volumes/HD/folder001
44418920	/Volumes/HD/folder002
29209808	/Volumes/HD/folder003
13658736	/Volumes/HD/folder004
10394096	/Volumes/HD/folder005
10394096	/Volumes/HD/folder001/file001
10361792	/Volumes/HD/folder002/file002
10360368	/Volumes/HD/folder001/folder003/file003
10100704	/Volumes/HD/folder001/folder003/folder004/file004
10100704	/Volumes/HD/file005

Another very useful command that we already know from previous post is find. It works slightly different, this command can help us to find files bigger than a given size. For example, let’s try to find files with more than two gigabytes in our system

find /Volumes/HD -size +2G -exec ls -sd {} +

The result should be something like:

10394096	/Volumes/HD/folder001/file001
10361792	/Volumes/HD/folder002/file002
10360368	/Volumes/HD/folder001/folder003/file003
10100704	/Volumes/HD/folder001/folder003/folder004/file004
10100704	/Volumes/HD/file005
10100217	/Volumes/HD/file006

With these two combinations of commands probably your life will be easier next time you run out of space in your development server.

In any case, a good advice should be take care of your development servers, do not leave them to arrive to this situation because this kind of things always happens in the worst moments.

See you.

Find big files

Search text inside files

Sometimes when we work in *nix systems we need to find some text inside files, to perform this task quickly and easily we have some command line tools that can help us.

One way to do it, it is using the tool find.

find . -type f | xargs grep "searchTerm"

Where we have:

  • find: Command line tool in *nix systems that search for files in a directory hierarchy.
  • path: In our case “.”. Folder where the search is going to start.
  • -type: Type of file that are going to be taken in consideration during the search.
  • xargs: Command line tool in *nix systems that build and execute command lines from standard input.
  • grep: Command line tool in *nix systems that print lines matching a pattern.
  • searchTerm: Word that we are looking for.

When we execute this command, a list of results with file names and the content of the lines where our searchTerm appears will be showed.

Another option is to use the command grep.

grep -ir "searchTerm" .

Where we have:

  • grep: Command line tool in *nix systems that print lines matching a pattern.
  • -i: Case insensitive.
  • -r: Recursive.
  • -l: Print only file names (Not in the above command but, sometimes, interesting option).
  • searchTerm: Word that we are looking for.

When we execute this command, a list of results with file names and the content of the lines where our searchTerm appears will be showed, except if we have used the option -l, in this case, only file names will be showed.

See you.

Search text inside files

Changing file character sets

One of the big problems I usually find when I export or import information into/from databases in text file is the character set used in the result file created by the operation. To solve this, we have a in our Unix or Unix based system a great command line tool, iconv.

If you do not know the use of this tool, it is very simple and easy to use. And, trust on me, It can save your live.

The available and useful options are:

  • f: the encoding of the input.
  • t: the encoding of the output.

The use will be:

iconv -f UTF-8 -t ISO-8859-15 in.txt > out.txt

A tool as simple as useful.

See you.

Changing file character sets

Testing REST with cURL

Nowadays is very common for developers, especially for back-end developers, develop some REST services to be consumed by the front-end developers. At the end of the development cycle usually you have (or you should have) some beautiful test suits that allow you to test your services and check if everything is working well. But, what happen during the development? The developer usually needs to test the services punctually, needs to test the connectivity or needs to perform a few test to check the REST API that he is investigating or trying to consume. There are some different solution to do this, create a little application to do it, use one of the plugins for the different browsers to use REST services, or use cURL. In this post, I am going to explain how to use cURL to perform these little checks in a quick way.

If you do not know cURL, cURL is a command line tool and library for transferring data with URL syntax, supporting a lot of different formats and options. Usually, it is installed by default in Unix and Unix based systems. In windows systems you need to install it or if you are using Cygwin you probably have it installed.

The first step before to start to use cURL should be check if it is installed in our system and take a look to the manual or help information that the tool provide us.

Checking if cURL is install in our system:

curl

This command should returns us something like:

curl: try 'curl --help' or 'curl --manual' for more information

Now, we can read the information about the tool and how to use it, but this information is huge and for this reason I am going to explain below the options that we are going to need. To read the information about the tool we have some different options:

man curl
curl --help
curl --manual

After read all this documentation the main options that we are going to need to test a basic REST services implementing CRUD operations are:

  • i – shows response headers.
  • H – passes request headers to the resource.
  • X – passes the HTTP method name. Default option is GET.
  • d – passes in parameters enclosed in quotes; multiple parameters are separated by ‘&’.

Then, it is time to start with the example.

In this case, we have available some REST services implementing the CRUD operations for Books. The available verbs are:

  • GET: Recover all the items or a particular item.
  • POST: To create a new book.
  • PUT: To update the book information.
  • DELETE: To delete a concrete book.

The available URLs are:

  • /books (GET, POST)
  • /books/{id} (GET, PUT, DELETE)

The object book will be a very simple object with three properties:

  • id: It will be auto generated by the back-end.
  • author: It should be provided in creation time.
  • title: It should be provided in creation time.

Having this clear in our minds we can start.

Recover all the books (/books, GET) It is empty right now

curl -i -H "Accept: application/json" http://localhost:4567/books
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Content-Length: 0
Server: Jetty(9.0.2.v20130417)

Create a new book (/books, POST)

curl -i -H "Accept: application/json" -X POST -d "author=john&title=unknown" http://localhost:4567/books
HTTP/1.1 201 Created
Content-Type: text/html; charset=UTF-8
Content-Length: 10
Server: Jetty(9.0.2.v20130417)

1354591741

Recover a concrete book (/books/{id}, GET)

curl -i -H "Accept: application/json" http://localhost:4567/books/1354591741
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Content-Length: 28
Server: Jetty(9.0.2.v20130417)

Title: unknown, Author: john

Update a book (/books, PUT)

curl -i -H "Accept: application/json " -X PUT -d "title=Sunshine" http://localhost:4567/books/1354591741
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Content-Length: 33
Server: Jetty(9.0.2.v20130417)

Book with id '1354591741' updated

Recover updated book (/books/{id}, GET)

curl -i -H "Accept: application/json" http://localhost:4567/books/1354591741
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Content-Length: 29
Server: Jetty(9.0.2.v20130417)

Title: Sunshine, Author: john

Delete a book (/books/{id}, DELETE)

curl -i -H "Accept: application/json" -X DELETE http://localhost:4567/books/1354591741
HTTP/1.1 200 OK
Content-Type: text/html; charset=UTF-8
Content-Length: 33
Server: Jetty(9.0.2.v20130417)

Book with id '1354591741' deleted

Recover deleted book (/books/{id}, GET)

curl -i -H "Accept: application/json " http://localhost:4567/books/1354591741
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 14
Server: Jetty(9.0.2.v20130417)

Book not found

With all the previous instructions we can test and check REST services implementing CRUD operations. It is a very easy and quick way, and if you use to use the console or terminal you will find it easier than use other tools.

Obviously, there are more ways to use it, for example, if your REST services expect a file, but the purpose of this post was only do a little introduction to the tool and to have a place to check the syntax quickly.

See you.

Testing REST with cURL