1
$\begingroup$

I have a directory of FASTA files, some of which are empty (placeholders that were output from a pipeline) and I would like to delete the empty files using a for loop on the command line.

From what I have seen so far I can use the wc -l [file] approach to see "if there are lines present, delete the file" but I want to make sure the syntax is right before deleting important files. I was thinking it might be easier for the command to look for the " > " symbol found at the header in FASTA files and if it's not present delete the file.

Here is what I have so far:

for file in *.fasta; do "if wc -l < 1 rm file"; done

Or

for file in *.fasta; do "grep > | rm file"; done
$\endgroup$
3
  • 1
    $\begingroup$ You'll want to use the find command, as the answer from @Ram RS shows. But on a separate note, neither of the above commands are syntactically correct. For a safe way to use these commands & test them, substitute ls for rm, to simply list files rather than delete them. $\endgroup$
    – Scot
    Jan 20 at 21:24
  • $\begingroup$ If you try find with -links and let me know $\endgroup$
    – M__
    Jan 21 at 13:16
  • $\begingroup$ What operating system are you using? Not all find implementations have the -empty or -delete flags. $\endgroup$
    – terdon
    Jan 23 at 14:49

4 Answers 4

2
$\begingroup$

This isn't the most efficient approach, using find will be easier and faster as long as your find supports the -empty option, but since this is what you asked for, here is how to do it based on the number of lines:

for file in *.fasta; do 
 [[ $(wc -l < "$file") -eq 0 ]] && echo rm -- "$file"
done

If you want to recursively descend into subdirectories, use:

shopt -s globstar
for file in  *.fasta */**.fasta; do.
  [[ $(wc -l < "$file") -eq 0 ]] && echo rm -- "$file"
done

If that prints the right files, remove the echo to actually delete them.

Note that this will also delete files that are not empty but don't have a newline character. These are not valid "text files", but strictly speaking they aren't empty, they just have no lines as lines are defined by the \n character. For example, this file would be deleted:

printf 'aaa' > file
$\endgroup$
4
  • $\begingroup$ The potential problem with this approach in Nextflow is there are "extensive" (or more than I think reasonable) numbers of subdirectories in the output and the names get complicated, which precisely is why I avoid it (I'd rather construct my own Dockerfiles). find will run straight through the directory structure and the depth can be set. Thats why in my opinion it's the favoured solution here. $\endgroup$
    – M__
    Jan 23 at 15:45
  • $\begingroup$ @M__ yes, absolutely. If the OP's find supports the -empty and -delete, it is the best tool for the job. You could easily make this recursive with shopt -s globstar; for file in *.fasta */**.fasta; do.... $\endgroup$
    – terdon
    Jan 23 at 15:51
  • $\begingroup$ thank you so much for this line of code!! and the comments really help too. I am running this with a mac OS which is why I think the find function doesn't work for me. It definitely seems like the easier option so I might try and find a workaround with an aws instance in the future but for now this works like a charm. Thank you! $\endgroup$
    – rimo
    Jan 24 at 15:55
  • 1
    $\begingroup$ @rimo yes, exactly, macOS ships with a different find implementation that might not support all of the flags used like -maxdepth. -empty and -delete. $\endgroup$
    – terdon
    Jan 24 at 16:08
3
$\begingroup$

Always Google before asking on forums. I googled "delete empty files linux" and this was the second result: https://www.baeldung.com/linux/delete-empty-files-dirs

# Print and delete empty files just in the current directory
$ find . -maxdepth 1 -type f -empty -print -delete
# Print and delete empty files in the current directory and all sub-directories
$ find . -type f -empty -print -delete
$\endgroup$
7
  • $\begingroup$ I saw that link too and tried it but it didn't work for some reason $\endgroup$
    – rimo
    Jan 20 at 20:48
  • $\begingroup$ What happened when you tried it? $\endgroup$
    – Scot
    Jan 20 at 21:58
  • $\begingroup$ it doesn't do anything...nothing output in the STDOUT and nothing changed in the directory $\endgroup$
    – rimo
    Jan 20 at 22:33
  • $\begingroup$ Are you running the command from the right directory? If it does not delete anything, but you're sure you have "empty" files, check if the files have blank spaces or some such invisible characters. cat -A them. $\endgroup$
    – Ram RS
    Jan 20 at 23:24
  • $\begingroup$ @rimo Ram's solution only works on Linux systems, what are you using? $\endgroup$
    – terdon
    Jan 23 at 14:49
2
$\begingroup$

As already stated, one way is to use the GNU find (the default on Linux) command. This will find and print all empty files in the current directory:

find . -maxdepth 1 -type f -empty -print

Append the -delete action to the command above when you're ready to remove them. If you need to find (and then delete) empty files recursively, simply remove the -maxdepth 1 option.



In a comment, you mention that these are actually "alias" files which are the results of a Nextflow pipeline. Unless the default publishDir mode has been changed, Nextflow will publish output files to the publishDir using symbolic links to the files in the working directory (i.e. ./work). I suspect what you are trying to do is to find and remove the links to these empty files, but not the files themselves. In this case, I think what you want is:

find . -type l -exec test ! -s '{}' \; -print

Again, append the -delete action to the command above to actually remove the symlinks. If you instead need to delete files from Nextflow's working directory, use nextflow clean in conjunction with nextflow log to safely clean up the failed runs.

$\endgroup$
1
$\begingroup$

@Ram RS has identified the solution here via find, if that does not work then the following will work,

find . -maxdepth 1 -type f -empty | xargs rm 

Edit 2

The solution is obvious ...

find . -type l -delete

The code under Edit is is better if the OP knows in advance precisely how many links the symbolic link file has, basically its always one. However, we don't know, thus -type l will delete every symbolic link.


To address @Steve's point of inadvertent deletion in case of a null return ... The code was checked prior release and this didn't occur (and also now rechecked).

$\endgroup$
8
  • $\begingroup$ I think the issue I'm facing is that these files are "Alias" files which are the results from a Nextflow script I wrote. Therefore, the file has size (~180 bytes) but is ultimately empty inside. I think the solution I am looking for is more specific than using the "find" command because it's not working for me. I would like to try and use a solution that actively looks in the files and sees whether there is text there (i.e. wc -l < 1) and if not, delete the file. $\endgroup$
    – rimo
    Jan 20 at 22:33
  • 1
    $\begingroup$ Or change the scope of what you are doing to only make softlinks to files that are greater than 0 bytes in size, rather than deleting the ones you already have made. $\endgroup$
    – Scot
    Jan 21 at 3:44
  • $\begingroup$ If find doesn't return any results, a pipe to xargs rm -rf could remove everything in your current directory. $\endgroup$
    – Steve
    Jan 21 at 11:36
  • $\begingroup$ @Steve the point was addressed above $\endgroup$
    – M__
    Jan 21 at 12:40
  • 1
    $\begingroup$ I don't understand how an Alias file (or even a symlink for that matter) can be considered empty. Is the alias/link empty when the file pointed to is empty? Or is it empty when the file pointed to does not exist? I'm unable to understand OP's situation. $\endgroup$
    – Ram RS
    Jan 24 at 15:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.