HomeНаука и техникаRelated VideosMore From: APMonitor.com

Import Data and Analyze with Python

816 ratings | 213422 views
Python programming language allows sophisticated data analysis and visualization. This tutorial is a basic step-by-step introduction on how to import a text file (CSV), perform simple data analysis, export the results as a text file, and generate a trend. See https://youtu.be/pQv6zMlYJ0A for updated video for Python 3.
Html code for embedding videos on your blog
Text Comments (100)
Salah Oum (3 months ago)
can i ask you ?
APMonitor.com (3 months ago)
Sure, it is no problem to ask a question.
SAGAR SINGH RAJPOOT (3 months ago)
Amazing video
Matthew Schouw (4 months ago)
Great video
Andrej158158 (5 months ago)
finally useful tutorial
Ashish Kalyan (9 months ago)
i have data in ascii format and when i doing this same process i interface with following Traceback (most recent call last): File "C:\Users\Ashish\Desktop\assp.py", line 4, in <module> data = np.genfromtext ('ch00_B0833-45_20150612_191438_010_1', delimiter=',') AttributeError: 'module' object has no attribute 'genfromtext' >>>
APMonitor.com (9 months ago)
Leave out the extra 'e' in the function call: https://docs.scipy.org/doc/numpy-1.13.0/reference/generated/numpy.genfromtxt.html
Kundan (11 months ago)
how can we plot derivative plot of discrete data points....please help
APMonitor.com (11 months ago)
I'd recommend changing the style to circles 'o' or points '.' - here are tutorials on plotting: http://apmonitor.com/che263/index.php/Main/PythonPlots
Sudo Linux (1 year ago)
So is this data science? This looks very interesting.
Sudo Linux (1 year ago)
APMonitor.com thanks I’ll check it out!
APMonitor.com (1 year ago)
Data science is a lot more than just this video but it is a good start. Check out http://apmonitor.com/che263
Asif Nizamani (1 year ago)
sir how you learn python or matlab programing language i was learned to try to learning programing last 1.5 years 1st 2 month i learn matlab saw various lectures, then move to the python their liabraries numpy, pandas , matplotlib i almost forgot most of the things i saw various lectures videos on you tube but now i just limited on labview please tell me advice for me and all the world students how to learn any programing language please sir share also your experience and any tips and free pdf or notebook.
APMonitor.com (1 year ago)
If you need help with starting a programming language such as python, there are many wonderful tutorials online such as code.org. labview is a nice environment for data acquisition but I don't recommend it as a programming language. There are fewer courses that treat numerical python. If you are interested in learning more about python I recommend that you go through the homework on my course. http://apmonitor.com/che263/index.php/Main/CourseHomework
Rim Oo (1 year ago)
i need a video like this for mac please asap
APMonitor.com (1 year ago)
Python should work the same on all platforms including Windows, MacOS, or Linux. I recommend that you try it and mention anything that is different in the comments.
Bose s (1 year ago)
Awesome work...nice and clean example. Please go ahead with some complex analysis example. BTW learnt a lot from this one .
APMonitor.com (1 year ago)
More content is available on the free course website at http://apmonitor.com/che263
Abhishek Mehta (1 year ago)
Informative in simple words and step by step... very crisp and concise
Muggins (1 year ago)
Do you by any chance have a video on importing data sets with missing entries? I get the following error: ValueError: Some errors were detected ! Line #41 (got 5 columns instead of 12) I'm trying to use the 1872 Hidalgo stamp data, which is in a txt file with 12 columns but the last row has only 5 entries.
APMonitor.com (1 year ago)
+Muggins, you may want to open the file, read one line at a time, count the number of delimiters (comma or tab are common), and then do a try...except on converting the elements to numbers or leaving as strings. There are lots of examples on stack exchange.
Masroor Ahmad (1 year ago)
how to run custom datasets in python?
APMonitor.com (1 year ago)
There are a few examples here: http://apmonitor.com/che263/index.php/Main/PythonDataAnalysis
Piyush Bhedurkar (1 year ago)
how do we give extension name .py........i tried but not working
APMonitor.com (1 year ago)
See this video for changing windows file extensions: https://youtu.be/YlN9ax7yC0A You may have a file name such as myScript.py.txt and you just need to remove the .txt after you show the file extensions.
Polo C (1 year ago)
i'd have used pandas module .
APMonitor.com (1 year ago)
+Paul Cai, oops, Here is the correct link: http://apmonitor.com/che263/index.php/Main/PythonDataAnalysis
APMonitor.com (1 year ago)
+Paul Cai, NumPy and Pandas are both great. Here is a page that shows how to use both: http://apmonitor.com/wiki/index.php/Main/ApplicationWebinars
Mingguang Tuo (1 year ago)
For the txt saving, how do you add something like \n so that you start the new time in a different line? Thanks.
Mingguang Tuo (1 year ago)
Thanks for your kind reply. For your case here, you are using variable time_col which actually represents a column. I am still confused a little bit how you can process in separate rows. Thanks again.
APMonitor.com (1 year ago)
+Mingguang Tuo, you can store the data in separate rows. The savetxt function will automatically put it on the next line.
Pradumn Kumar (1 year ago)
in the fourth line of your code i.e. time=data[:,][:,0] what is the role of [:,]
Hossam Anany (1 year ago)
Maybe it is, but he is doing a video for you to learn so I guess you should appreciate whatever he does
foo bar 167 (1 year ago)
APMonitor.com -- I've checked it and string "[:,]" is useless for Python 2.7.13 too. Just for your knowledge.
APMonitor.com (1 year ago)
Great comment. This tutorial was for an older version of Python and Numpy. Here is a new video that I created without the [:,] and several other new features for Python 3: https://youtu.be/pQv6zMlYJ0A
foo bar 167 (1 year ago)
APMonitor.com -- Thank you for the tutorial! There is no need to type: time=data[:,][:,0] It is better to type: time=data[:,0] This is shorter and faster as I know. String "[:,]" is useless, it does nothing, but consumes computer resources. Also instead of sensors[:,][:,1] it is better to type sensors[:,1]
Pradumn Kumar (1 year ago)
Shawn Driscoll (1 year ago)
Very cool. Don't know what it all means, but very cool doing plots with Python.
Nice video! Can you explain why the following can happen: time = time - time[0] i mean time is defined. Do you redefine it by the time module minus the first point and now that is the new definition of time?
Thanks alot great video
APMonitor.com (1 year ago)
Simon Chemnitz-Thomsen, yes - it is just for convenience. t[0] is very large so I just time shift everything to start at t[0]=0.
MrPariah88 (1 year ago)
Super helpful, thank you very much!
very useful and helpful for me. Where i can download sample data according to the video ? Nice to hear your feedback :) Thanks,
APMonitor.com (1 year ago)
+Andreas Victor Simanungkalit, the source code and data file is available from http://apmonitor.com/che263/index.php/Main/PythonDataAnalysis
preeningPARK (1 year ago)
Hello, first of all i wanna say thanks for the video, i have two questions:What version of python are you using, Where does the txt file have to be saved to be called by the command? Thanks!
APMonitor.com (1 year ago)
I used Python 2.7 for this example although it can also be run with Python 3+ if you just add parenthesis around the print statement at 3:28. The txt file needs to be in your working directory (same as myScript.py) for this example. It is possible to navigate to different directories but this example requires the data (txt) file in the same directory. There is another example using Pandas along with source code posted here: http://apmonitor.com/che263/index.php/Main/PythonDataAnalysis
pooja mahindrakar (2 years ago)
Hello sir, can you please guide on how to import image data and analyze with python in jupyter notebook?
APMonitor.com (2 years ago)
+pooja mahindrakar, here are some tips to get started: http://stackoverflow.com/questions/3735553/how-do-i-read-an-image-file-using-python
G Dunken (2 years ago)
Well worth 12 mins. Thanks.
Population Channel (2 years ago)
import numpy as np : ImportError; no module named numpy? ;////
APMonitor.com (2 years ago)
Here are a couple videos on how to install modules such as NumPy. I recommend using: python pip install numpy from the command line. You can also do: import pip pip.main(['install','numpy']) in a Python script.
MImran Khan (2 years ago)
would you like to give me this notepad file
APMonitor.com (2 years ago)
+MImran Khan, notepad++ is freely available from https://notepad-plus-plus.org/ You can download this particular data set from http://apmonitor.com/che263/index.php/Main/PythonDataAnalysis
Ramsey Selim (2 years ago)
Very useful and helpful video. Thanks for taking the time to prepare this material.
Crazed Thinker (2 years ago)
My "Text Document" doesn't have the "txt" extension on the end and doesn't change when I add the "py" when I rename it. It just stays as a normal "Text Document". Any other way I can change it to "py"?
Crazed Thinker (2 years ago)
APMonitor.com Thanks it worked and thanks for the great video.
APMonitor.com (2 years ago)
+New Rider NT, you need to show Windows file extensions. Navigate to "Folder Options"..."View" and unselect the option to hide extensions for known file types. This will allow you to view and change file extensions.
x_s (2 years ago)
Ravi Shankar Dadhich (2 years ago)
Thanks for this kind help :) :)
price279 (2 years ago)
Thanx a lot! This video saved me.
tracy grygiel (2 years ago)
Awesome video -- this really helped me a lot!
Julkey - (2 years ago)
Hi there, I am trying to figure this one out. I've managed to import the data etc, but since I dont have any headers for the values, how will the program now what each one is (time or sensors, for example)? These are my headlines for each column that is not included in the txt file: Subject ID, Rating, Reaction time, Pupill. How can I tell the computer that each one of these represents each column?
APMonitor.com (2 years ago)
You'll need to assign each column to a separate variable name such as shown at 2:37 in the video when I take a slice of the data for time and the individual sensors. Alternatively, if your data file has headers, you can use Pandas to import the data and reference by column name. See https://youtu.be/FXhED53VZ50 for information on how to use the Pandas package.
R M (2 years ago)
Thanks that was a great tut...i found that watching the earlier videos from the series made what you did in this one much clearer( just a recommendation for those wanting to really understand what going on in this one). Thanks a lot for sharing!
APMonitor.com (2 years ago)
+Rich Mash thanks for the feedback!
Sana Khan (2 years ago)
Good one
GAURAV SANKAR (2 years ago)
I downloaded numpy... where do I place the files?? my code says "No module named numpy"
APMonitor.com (2 years ago)
Thanks for the command line example. When Python is called with -m, the given module (pip) is located on the Python module path and executed as a script.
GPC™ (2 years ago)
WINDOWS: python -m pip install module_name (I don't even know what -m means...)
GAURAV SANKAR (2 years ago)
+APMonitor.com awesome, nailed it! thank you Sir! :)
APMonitor.com (2 years ago)
Here are instructions on installing numpy: https://youtu.be/FKwicZF7xNE?t=10m16s if you are on Windows. If you are on Linux or MacOS, try "pip install numpy" on a command line.
Aneesh Lale (2 years ago)
Thanks you so much for this video! Made life a lot easier.
Kisore (3 years ago)
thx for the quick reply and can u show me some examples to split the letters and numbers in a text file ?
APMonitor.com (3 years ago)
+APMonitor.com You may also want to switch to pandas if you have mixed data such as floats and strings. See https://youtu.be/FXhED53VZ50
APMonitor.com (3 years ago)
+kisore hari Yes, just designate a delimiter character like I did at 1:38. Most text files use a space, tab, or comma.
Kisore (3 years ago)
can u plz help me with this error? Traceback (most recent call last): File "<pyshell#0>", line 1, in <module> data[0:4] NameError: name 'data' is not defined and can u show me the video for separating numbers and texts in a one file to the another file plz. My mail id is [email protected]
APMonitor.com (3 years ago)
+kisore hari It appears that the file import wasn't successful or that the array "data" wasn't created. There is another tutorial and some example files here: http://apmonitor.com/che263/index.php/Main/PythonDataAnalysis Please download the example files and let me know if you still have further questions.
Vishal Vashistha (3 years ago)
I Have three data length(200 to 500 nm) , width (300 to 900 nm) and Transmission value form 0 to 1. How to visualize this data in colormap plot. I have data in .csv form. Can you please mail me Python script. My mail id [email protected]
APMonitor.com (3 years ago)
+Vishal Vashistha Okay, a contour plot may be a better option. There are a couple examples here: http://apmonitor.com/me575/index.php/Main/InteriorPointMethod in Python
Vishal Vashistha (3 years ago)
+APMonitor.com , Sorry dear but that is not useful for my case. I want to study the parametric variation effect (height and width) on transmission. I want 3 D plot on 2 D scale. There is no option to upload it, otherwise I can send you screen shot of that graph.
APMonitor.com (3 years ago)
+Vishal Vashistha Check out this material on plotting functions in Python: http://apmonitor.com/che263/index.php/Main/PythonPlots There is information on colormaps in Python here: http://matplotlib.org/examples/color/colormaps_reference.html
Navtej Singh Saini (3 years ago)
what is the screen recording software you are using?
Navtej Singh Saini (3 years ago)
thanks :)
APMonitor.com (3 years ago)
Camtasia Relay on a Surface Pro Tablet.
Yulia Kim (3 years ago)
Great video!
APMonitor.com (3 years ago)
+Yulia Kim Thanks for your feedback. There are other Python videos on the Engineering Programming Course at http://apmonitor.com/che263
Abdullah ALSHEHRI (3 years ago)
Well, it's awesome video. Thanks for that.
APMonitor.com (3 years ago)
+Abdullah ALSHEHRI You're welcome - I'm glad that you enjoyed it. Check out other videos and course material at http://apmonitor.com/che263
Abdel Latif (3 years ago)
Hi, thanks for this tutorial, can you please post the data text file ? Thanks
APMonitor.com (3 years ago)
+Abdel Latif Abdel, I don't think I have the original data file. However, the following should work if you copy it into a text file such as data_file.txt: 1.000E+01,4.053E+02,3.926E+02,3.930E+02,4.026E+02,nan,nan,nan,nan 1.050E+01,3.904E+02,3.997E+02,4.084E+02,4.100E+02,nan,nan,nan,nan 1.100E+01,4.091E+02,3.934E+02,3.923E+02,4.035E+02,nan,nan,nan,nan 1.150E+01,4.015E+02,4.011E+02,4.044E+02,4.059E+02,nan,nan,nan,nan 1.200E+01,3.904E+02,3.925E+02,4.025E+02,3.948E+02,nan,nan,nan,nan 1.250E+01,4.095E+02,4.040E+02,3.960E+02,4.051E+02,nan,nan,nan,nan 1.300E+01,4.018E+02,3.916E+02,3.939E+02,3.979E+02,nan,nan,nan,nan 1.350E+01,3.931E+02,3.984E+02,3.950E+02,3.939E+02,nan,nan,nan,nan 1.400E+01,4.066E+02,3.927E+02,4.000E+02,3.992E+02,nan,nan,nan,nan 1.450E+01,3.953E+02,4.076E+02,4.098E+02,3.966E+02,nan,nan,nan,nan 1.500E+01,4.095E+02,3.976E+02,3.901E+02,4.018E+02,nan,nan,nan,nan 1.550E+01,3.948E+02,4.036E+02,3.997E+02,4.015E+02,nan,nan,nan,nan 1.600E+01,4.018E+02,3.983E+02,4.056E+02,4.047E+02,nan,nan,nan,nan 1.650E+01,4.000E+02,3.923E+02,3.952E+02,4.055E+02,nan,nan,nan,nan 1.700E+01,3.928E+02,3.907E+02,3.960E+02,3.909E+02,nan,nan,nan,nan 1.750E+01,3.921E+02,3.978E+02,4.060E+02,3.996E+02,nan,nan,nan,nan 1.800E+01,4.070E+02,4.056E+02,3.955E+02,4.068E+02,nan,nan,nan,nan 1.850E+01,4.010E+02,3.913E+02,4.019E+02,3.944E+02,nan,nan,nan,nan 1.900E+01,4.085E+02,3.949E+02,4.031E+02,4.090E+02,nan,nan,nan,nan 1.950E+01,3.968E+02,4.018E+02,3.975E+02,3.963E+02,nan,nan,nan,nan 2.000E+01,4.037E+02,4.037E+02,4.036E+02,3.907E+02,nan,nan,nan,nan
TURKI F (3 years ago)
Amazing . thanks. that what you call it tutorial . not just export and import also was plot, math, and re-shape data .....what genuine ,,, very nice
TURKI F (3 years ago)
+APMonitor.com thank you my friend
APMonitor.com (3 years ago)
+Turki Aljrees Turki, thanks for the positive feedback. More videos on Python are posted to the online course http://apmonitor.com/che263 - I hope you enjoy those as well.
vaibhav bhagat (3 years ago)
numpy is not working for 64 bit windows :( please help
APMonitor.com (3 years ago)
+vaibhav bhagat Numpy is only 32-bit and requires that you install the 32-bit version of Python. Even though you are running a 64-bit processor, you can still run the 32-bit applications. I recommend uninstalling Python and starting with the 32-bit versions. Also, you'll need Python 2.7. I don't think Numpy works with Python 3.4 yet. For Windows, this is a collection of files that you can download that will work with this example: http://apmonitor.com/che263/index.php/Main/PythonIntroduction
APMonitor.com (4 years ago)
George, you'll also need to install numpy with Python. Here are all of the installation files for Windows: http://apmonitor.com/che263/index.php/Main/PythonIntroduction
George Willcox (4 years ago)
When I run this it says, no module name numpy
APMonitor.com (1 year ago)
Thanks, Javed! Here is some additional help on installing Numpy for George: https://youtu.be/-llHYUMH9Dg
Javed Ayoob (1 year ago)
George Willcox use this on command prompt pip install numpy

Would you like to comment?

Join YouTube for a free account, or sign in if you are already a member.