0
$\begingroup$

I have used the following script for calculating the median for one file (gene). However, I have 20,000 files for which I want to run this R script and get 200 separate output files

 #!/usr/bin/R
 setwd("/location folder")

 d <- read.table("tgf.txt", header=FALSE)

 x <- aggregate(d[, 2], list(d$V3), median)

 write.table(x, "tgf.output.txt", quote=FALSE, row.names=TRUE)

 q()
$\endgroup$
1
$\begingroup$

Another option is to use list.files and grep to select only the .txt in your directory. A for loop can read-aggregate and save the files one at the time.

all.files <- list.files("/location folder")
txt.files <- grep(".txt",all.files,value=T)

for(i in txt.files){
  d <- read.table(paste("/location folder",i,sep=""),header=F)
  x <- aggregate(d[, 2], list(d$V3), median)
  write.table(x, paste0(i,".output.txt"), quote=FALSE, row.names=TRUE)
}
$\endgroup$
6
  • 1
    $\begingroup$ Why not just list.files(pattern = "*.txt")? The grep is not required. $\endgroup$
    – Ram RS
    Jul 2 '19 at 15:07
  • $\begingroup$ And looping when vectorized operations are available is de-optimization. $\endgroup$
    – Ram RS
    Jul 2 '19 at 15:07
  • $\begingroup$ And there's some inconsistent usage: paste(..., sep="") on line 5 and paste0() on line 7. $\endgroup$
    – Ram RS
    Jul 2 '19 at 15:10
  • 1
    $\begingroup$ considering the nature of the question I thought that it was more important to show possibilities rather than one single point of view. paste and paste0 achieve the same but in different ways (e.g. sep=""). I tend to split commands involving list.files and grep as it makes easier to read and to reuse. The content of all.files is hard to miss. Finally the usage of the applyfamily is not easy to grasp for new user, while loops are much intuitive. Furthermore, apply is not always better than a nice for, especially in an example like this. $\endgroup$
    – fra
    Jul 2 '19 at 15:21
  • 1
    $\begingroup$ Showing possibilities is great! However, I don't think a loop would perform as well as or better than vectorized operations. While a loop is easier for people that know basic programming, the apply family solves a lot of problems, especially when there is a learning opportunity such as this one, where the task is well defined and how apply works can be easily understood. $\endgroup$
    – Ram RS
    Jul 2 '19 at 15:32
3
$\begingroup$

Please do not use setwd in a script, it changes working folder without explicit specifications from the user, so the results could be unpredictable for people that are not familiar with the content of the script.

Create a tab-separated file with input filename and output filename, or at least a file with all input files, unless you can pick them using a list.files(pattern = "<pattern_here>"). Read this file as a data.frame and use it with an apply, so a function processes the df row-by-row. Let's say the file (named metadata.txt) is:

in_file out_file
tgf.txt tgf.output.txt
abc.txt abc.filtered.txt
xyz.txt xyz.randomname.txt

Define a function called process_row() in the Rscript:

process_row <- function(r) {
    d <- read.table(r[[1]], header = FALSE);
    x <- aggregate(d[, 2], list(d$V3), median);
    write.table(x, r[[2]], quote=FALSE, row.names=TRUE);
}

And do the actual work:

meta_data <- read.table('./metadata.txt', sep = ' ', header = TRUE, stringsAsFactors = FALSE);
apply(meta_data, 1, process_row(r))
# You don't need a q() in an Rscript because it exits after the last line of code anyway.
$\endgroup$
4
  • $\begingroup$ I tried the above script but it is giving me following error Error in read.table(r[[1]], header = FALSE) : 'file' must be a character string or connection Called from: read.table(r[[1]], header = FALSE) $\endgroup$
    – Megha
    Jul 2 '19 at 9:57
  • $\begingroup$ What is the content of ./metadata.txt, which is in the working directory where you're running R? $\endgroup$
    – Ram RS
    Jul 2 '19 at 13:40
  • $\begingroup$ It worked. Thank you $\endgroup$
    – Megha
    Jul 3 '19 at 14:45
  • $\begingroup$ Please accept the answer if it worked. Thank you $\endgroup$
    – Ram RS
    Jul 3 '19 at 17:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.