1
$\begingroup$

Now jupyter installed on the server and I am using below code to plot the rarefaction plot but I am still getting some error. could you please suggest how I can get rid of it?

    #Run it in jupyter lab or notebook


import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import os

#%matplotlib inline 

qza = 'alpha-rarefaction-20000.qzv' # path to rarefaction file from qiime2
os.system("unzip qza")

digest = a[1].split('/')[0].replace('  inflating: ','')
inf = digest + '/data/observed_otus.csv'
otus = pd.read_csv(inf,index_col=0, sep=',')
os.system('rm -r $digest')

cols = [col for col in otus.columns if 'iter' not in col]
mean,data = otus[cols],pd.DataFrame(columns=cols)
depths = [col.split('_')[0] for col in otus.columns if 'depth' in col]
otus = otus.drop(cols,axis=1)
otus.columns = depths
for depth in depths:
    mean['ASV'] = otus[depth].mean(axis=1)
    mean['depth']= depth.split('-')[-1]
    data = pd.concat([data,mean])
# here provide colors for each item that will be plotted
pal={'Irrigated':'brown','Rainfed':'red'} 
fig,ax = plt.subplots(figsize=(2.5,2),dpi=600,tight_layout=True)
sns.set(style='ticks',rc={"lines.linewidth":0.7,"axes.linewidth":0.5})
# use your column name to plot here
sns.lineplot(x='depth',y='ASV',data=data,palette=pal,hue='Season',sort=False,err_style='bars',\
             dashes=True,style='Season',ci=67)
ax.set_xlabel('Sequencing depth',fontsize=8)
ax.set_ylabel('Observed ASVs',fontsize=8)
ax.tick_params(axis='x', labelrotation=90)
ax.tick_params(axis='both',which='major',length=2,pad=0.5,labelsize=6)
handles, labels = ax.get_legend_handles_labels()
ax.legend(handles=handles[1:],labels=labels[1:],fontsize=5,frameon=False,numpoints=4,borderaxespad=0,handletextpad=0.2,loc=2,)
plt.savefig('Infected_observed_ASVs.png', bbox_inches='tight')

Error;

 $ python3 myscript.rarefaction.py
Traceback (most recent call last):
  File "myscript.rarefaction.py", line 4, in <module>
    import seaborn as sns
  File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/__init__.py", line 2, in <module>
    from .rcmod import *  # noqa: F401,F403
  File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/rcmod.py", line 7, in <module>
    from . import palettes
  File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/palettes.py", line 9, in <module>
    from .utils import desaturate, get_color_cycle
  File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/seaborn/utils.py", line 11, in <module>
    import pandas as pd
  File "/opt/apps/seaborn/0.11.0/python3-3.7.4/lib/python3.7/site-packages/pandas/__init__.py", line 11, in <module>
    __import__(dependency)
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 1101, in <module>
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/lazy.py", line 101, in _lazy
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 1098, in <genexpr>
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 114, in resource_exists
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/libs/matplotlib_python34/1.5.1/gcc-4.8.5+python3-3.4.3+numpy_python34-1.11.3/python/lib/python3.4/site-packages/pytz-2019.1-py3.4.egg/pytz/__init__.py", line 102, in open_resource
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 668, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 638, in _load_backward_compatible
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2927, in <module>
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2913, in _call_aside
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2952, in _initialize_master_working_set
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 956, in subscribe
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2952, in <lambda>
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2515, in activate
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2097, in declare_namespace
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2047, in _handle_ns
  File "/opt/gridware/depots/54e7fb3c/el7/pkg/apps/setuptools/24.0.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-24.0.1-py2.7.egg/pkg_resources/__init__.py", line 2066, in _rebuild_mod_path
AttributeError: '_NamespacePath' object has no attribute 'sort'

         
     
    
    metadata file ;
    #SampleID       Villages        Region  Treatment       Batch   Season
    PN0086D.1.S1    Kholabari       Mymensingh      T1      batch1  Irrigated
    PN0086D.2.S2    Bogajan Mymensingh      T1      batch1  Irrigated
    PN0086D.3.S3    Bonkua  Mymensingh      T1      batch1  Irrigated
$\endgroup$
  • $\begingroup$ Just ask your cluster admin / manager if it's possible to run notebooks via the queueing system / workload manager, if it is not, you won't get it running anyhow. $\endgroup$ – Pallie Sep 28 at 14:36
2
$\begingroup$

This is a bash script (#!/bin/bash) and you try to run python in it. You should save your python script (e.g. in "myscript.py") and call it within your bash script. Maybe first try to see if you are able to run it in your terminal and then run it on your server with SLURM.

EDIT1: Also I don't think you need 3 nodes to run such script and this will probably not be even possible here. I think you need some specific modules to parallelize a python script with MPI (see https://stackoverflow.com/questions/25792357/how-to-distribute-a-python-script-across-multiple-nodes), but again I think this would be an overkill. Try first with #SBATCH -c 1 instead.

Edit2: Finally it might be possible to run jupyter-notebook on a node allocated by SLRUM instead of putting this into a python script by using jupyter-notebook --no-browser --port=${port} --ip=${node} command in your bash slurm script. Maybe you'll find this doc interesting: https://docs.ycrc.yale.edu/clusters-at-yale/guides/jupyter/ .

| improve this answer | |
$\endgroup$
  • $\begingroup$ thanks I am running it as python script after loading jupyter and python module but still shows an error? I am going to update the script and the error. $\endgroup$ – bioinfonext Sep 24 at 13:01
  • $\begingroup$ %matplotlib inline is a jupyter notebook command. Try to comment this line. $\endgroup$ – jeremy Sep 24 at 13:08
  • $\begingroup$ Again error. it is updated now. $\endgroup$ – bioinfonext Sep 24 at 13:10
  • $\begingroup$ Those are again some jupyter notebook writings to use bash commands within a notebook. You should try with subprocess module instead or build a bash script that does those steps and then call your python script. I think you should think by yourself now and googling until you are really stuck, this is a good training to write real python script ;) $\endgroup$ – jeremy Sep 24 at 13:19
  • $\begingroup$ Hi @bioinfonext please don't forget to upvote this post and if you feel it answered your question to mark as 'resolved' if resolved. $\endgroup$ – Michael Sep 24 at 13:24
0
$\begingroup$

I'll make a contribution and keep editing it rather than use 'comments', the error is about

import numpy as np

At the moment this isn't about Jupyter, its about getting Python working on your server. Is that a good assessment? With Jupyter notebook you write a little bit of code, check it, write a bit more. That's how it is designed to operate. The other way in is to use Python in your shebang statement otherwise it will default to sh or bash.

The code you supplied doesn't explicitly use numpy. Pandas does use numpy and np is the standard shorthand, Multiarray is probably something pandas would use. Did you get this code working on your local machine?

I am wondering whether the numpy module on your server is up to date? Did you creat a specific Python environment, use pip or conda? Checking the Python version on your server and whether numpy needs updating is perhaps where I might start.

It is not pretty getting a HPC to run with your stuff and can take real work.


I read your new post, I would simply reinstall Python from Anaconda3 and run the script for now from shell pointing the shebang into Anaconda3 (I can't remember if you need to do this, I can check if needed). This a more professional way of using a HPC than using the resident Python. In good old days of Perl, when everything was nice and simple, perlbrew was essentially the Perl equivalent and no-one would use the system Perl.

The reason why I'm saying this is I get this feeling it will be an endless cycle of import errors. Anaconda3 will give you control of your Python system without admin privelidge.

Just to explain if you are not calling np explicitly (and looking at your code I don't think you are) then explicitly stating import numpy isn't needed. The fact you wrote it and it then jumped onto a different error is a bit worrying. I would just use your own version of Python. In Jupyter I would guess, but don't know they use their own Python ... but from a data science point of view Jupyter is the front end to AWS and GCS because those guys want some control of what you are doing, not running riot. In DNN jupyter is big. If you want an absolute diagnosis then Stackoverflow might help, but I think you'll just another bug and another etc... One thing is that pandas changes alot and the version you are using might be ahead of the curve, thats my guess

Final point is if you are just plotting the ggplot2 is really good, but that is R. seaborne is really good, matplotlib but hmmmm, but nothing is as clear or explicit as ggplot2. Python is good at lots of things, pandas is fabulous, but plotting isn't on par with ggplot2


What is 100% clear is that you are loading the HPC version of Python and this is out of date (3.4) should be 3.7 (or even 3.8?). The thing I don't understand is why is it loads Python 2 modules?? This is a complete mystery to me, I thought Python 2 and Python 3 were incompatible, this is suggesting backwards compatibility or else the reason it is falling over. I have no idea, but I wouldn't use the resident Python.

| improve this answer | |
$\endgroup$
  • $\begingroup$ script and error updated after importing numpy. $\endgroup$ – bioinfonext Sep 25 at 15:14
  • $\begingroup$ @bioinfonext commented below $\endgroup$ – Michael Sep 25 at 15:42
  • $\begingroup$ Yes, thanks, this is all very confusing, R is much easier to adopt for potting.But I could not able to find any good R code for rarefaction plot that can plot it based on metadata column. $\endgroup$ – bioinfonext Sep 26 at 6:10
  • $\begingroup$ @bioinfonext answered above $\endgroup$ – Michael Sep 26 at 12:09
  • $\begingroup$ @bioinfonext answered above $\endgroup$ – Michael Sep 26 at 12:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.