Categories
Career

The power to find what you need

As part of what I do, I often end up working in the command line in Linux.  As anyone that works in IT can tell you log files are very important.  Something that they might not mention is, they also take up a lot of space depending on the level of logging being done.  Recently I had need to free up some space for log file on the server, I didn’t want to get rid of any files, just compress the older files. Chances are I wouldn’t need them, but if I did, they would still be there.  To accomplish this, I decided to use find, a Linus command line tool that, you guessed it, finds “stuff”.  Find is useful in that the command line arguments you pass it can quickly sort through and return a very unique set of results.  Find also automatically recurses downward through the sub directories, so you get a comprehensive list, you can change that through a command line option as well.

So, the first thing I wanted to see is all files (I’m not interested in directories) in the directory designated for logging that were older than 10 days (a completely random selection of number of days, a little more than a week, less than two weeks).  That’s easy enough to do, I just enter

find -type f -mtime +10

and instantly got a whole slew of files returned.  So, let’s break down what I did, “-type f” tells it I’m looking only for file, no directories and the “-mtime +10” tell the program to only give me file that have a “last modified” date OLDER (+) that 10 days.  I notice a few files that have a .gz at the end which means they’ve already been compressed, so I don’t need them in the list.  Doing a bit of searching, I find that by adding “-not” in front of any option it return the opposite (props to http://www.askdavetaylor.com/how_do_i_list_files_that_dont_match_a_pattern/).  Knowing that the “-name ” option will return files based on wild card matching I add that to the mix and end up with:

find -not -name "*.gz" -type f -mtime +10

That looks better, giving me a shorter list, and NONE of the files listed end in .gz. Now the next step is to do something with those files, and this is where the, IMHO, the true power of find comes into play.  You can pass a command line argument to find that tells it to “do something” with the files it finds.  Just to make sure I’m not making any crazy mistakes, the first thing I try is something simple and non harming, like listing the full information for the file:

find . -not -name "*.gz" -type f -mtime +10 -exec ls -alh {} \;

The -exec command line argument is great in that it will “execute” everything after “-exec” up to the “\;”.  The “{}” tells it to replace the result This means you can string along several commands, although at my level, I usually just want to do one thing at a time.  I like seeing each step and, by doing one step at a time, chances are I’ll find mistakes before it’s too late.

So, this is great, I have a listing of files, but what I really want to do is compress those files.  Now that I have a means of listing the files, and the results are what I’m expecting, I can replace the “command” with what I really want to do.  Final results are:

find . -not -name "*.gz" -type f -mtime +10 -exec gzip {} \;

The sweet part of this, I can run this command on a daily, weekly or monthly basis and it won’t attempt to recompress files that have already been compressed.  It’s really a minor thing, but why try to process files that don’t need it.

I hope you’ve found this helpful, and can build off of what I’ve shown.  The man page, if you’re so inclined to read it, for find can be read here http://linux.die.net/man/1/find.

 

 

Categories
Career Interests

Cleaning up

For whatever reason, there comes a time in life that a person needs to tidy up.  In this case, it happens to be a linux based web server that has miscellaneous files strewn throughout.  Some from editing with emacs/vim/vi which tend to leave backup files (those annoying files with a ~ after them or other marker depending on the editor).  So, here’s what I did to do some cleanup.  Your mileage may vary.

First, determine the kind of files you want to remove and how you want to remove them.  Me, I’m going to use find and I have files ending with ~ and beginning with a ._ that are left over from editing.  So, to make sure I don’t delete anything I don’t want to, I’ll do a check first.  I do this by running “find -name ‘._*’ -print“.  This will find all files beginning with ._ and print them to the screen.  I can then do a check to make sure everything is what I expect it to be.  Once I am sure I want to delete the files listed, I run “find -name ‘._*’ -delete“.  I continue this process for the other files I need to eliminate.

This same process can be ran to match file date, file size and many other attributes. To see the full options under find check out the man page at http://linux.die.net/man/1/find or just google for it.  Find also allows a person to run an individual script on each file found which could be really useful for processing text files.