How can we turn Emacs as full featured Python IDE ? How to use Emacs for data science ? Many packages are explained below. You don't have to install all of them one by one, see the all-in-one solutions, like Elpy, starter kits and the Spacemacs layer.
An important library you'll want to get familiar with is Jedi, an autocompletion and analysis library for Python. It helps at autocompletion, finding definitions, showing documentation, etc. Elpy makes good use of it.
We also have Rope that allows cross-project refactoring and smaller ones (change the method signature,…). Again Elpy bases on it and other packages like emacs-traad make it easy to install and use.
See also Django support.
There are a number of python modes for Emacs. fgallina's python.el is the python.el included in Emacs 24.2 and onwards.
emacs-for-python is a bundle of the above modes (and more), and it's an easy way to turn Emacs into a Python IDE.
Elpy is a collection of elisp packages for Python too.
It permits, among others, to:
- refactor code (using rope),
- send a region or a buffer to a python interpreter,
- run unit tests, run the test the cursor is in, with a test runner of choice (django, pytest, …),
and of course
- on-the-fly checks with flymake,
- code completion with company-mode and either rope or jedi,
- on the fly code helper with eldoc (and rope or jedi),
- code navigation,
- change of virtual environment inside Emacs (pyvenv), etc
EMP: EMacs Python
The goal of emp is to take all the best development components available for Python, and combine it conveniently under Emacs. It builds on top of elpy, and provides even more bindings.
Emacs-bootstrap is a website where you can select a few programming languages, some options, and have it generate a full .emacs.d folder for you.
rope & ropemacs
rope is a library for refactoring and manipulating Python code, pymacs is an interface between emacs lisp and python, and ropemacs is an Emacs interface to the rope library which uses rope and pymacs.
If you do some search and replace of code objects in your code and find it sometimes tricky, and/or you need to do it in more than one file, then you should consider a good tool of refactoring.
But Rope can do more than this simple example, it can:
- Rename anything
- Extract method/local variable
- Change method signature
- Perform cross-project refactorings
- Support Mercurial, GIT, Darcs and SVN in refactorings
Rope can also help IDE's with:
- Finding definition location
- Getting pydoc
- Finding occurrences
- Organizing imports (removing unused and duplicate imports and sorting them)
- Generating python elements
Note: see also emacs-traad below that has an easy installation process.
So an easy way is to use an el-get recipe: M-x el-get-install ropemacs. It installs and build Pymacs, rope and ropemacs. The difference from the solution using pip is that it setups the Emacs side of Pymacs correctly.
You can follow the instructions as given here ,if you don't want to use marmalade you can use the following instructions .
sudo pip install rope ropemacs
As of February 2014 you can't install Pymacs via pip, therefore:
git clone http://github.com/pinard/pymacs cd pymacs make check sudo make install
make check ensures the prerequisites. sudo make install puts Pymacs module in Python local modules path, usually /usr/local/lib/python2.7/dist-packages/. People generally frown upon make install, because if you lose the Makefile, it can become hard to cleanly uninstall. But you shouldn't fear it, as pip uses the same directory, so pip uninstall pymacs will work.
After you do the make install of pymacs do not delete the folder.Copy the file pymacs.el to ~/.emacs.d/site-lisp/
Now add path of pyamcs.el to your ~/.emacs file,using this code:
(setq emacs-config-path "~/.emacs.d/") (setq base-lisp-path "~/.emacs.d/site-lisp/") (setq site-lisp-path (concat emacs-config-path "/site-lisp")) (defun add-path (p) (add-to-list 'load-path (concat base-lisp-path p))) (add-path "") (add-to-list 'load-path "~/.emacs.d")
You can now lazily load ropemacs or load it at runtime.For lazy evaluation add this to your .emacs file:
(defun load-ropemacs () "Load pymacs and ropemacs" (interactive) (require 'pymacs) (pymacs-load "ropemacs" "rope-") ;; Automatically save project python buffers before refactorings (setq ropemacs-confirm-saving 'nil) ) (global-set-key "\C-xpl" 'load-ropemacs)
use c-xpl to load ropemacs.Read more about it here
Rope & traad
Traad is a client-server approach to using the rope Python refactory library. It involves two basic components:
- A HTTP server exposing the rope API via JSON, and
- Client libraries for talking to the server
Since rope is written in Python, any tool that wants to use rope needs to either embed Python, or it needs to find a way to communicate with a Python process running rope. The embedding approach is difficult if the target environment is not written in Python, and it also faces challenges when dealing with more than one Python version.
So traad aims to simplify communication with rope running in an independent process. HTTP communication and JSON data handling is well supported in many, many languages, so any environment that wants to use rope should be able to easily communicate with traad.
emacs-traad, easy installation and use
emacs-traad (simply traad in melpa) makes it very easy to install traad, and thus rope. Install traad from melpa and then M-x traad-install-server. Then start a connection with traad-open and enjoy. Furthermore it installs traad in its own virtualenv, so it won't clutter your current virtualenv, and you don't have to re-install it in every project.
It is possible to remove a method's argument, normalize arguments, see the occurrences of a method, organize imports, some more python refactoring, and also see the history of traad changes.
Other Python "refactoring" tools
Projectile - replace in project
This is not specific to Python but is very helpful anyway. Projectile has a function projectile-replace which interactively offers to replace all occurrences of a term in all the files of the project. So it's handy to rename a method.
Iedit - replace occurences of a symbol
Iedit isn't specific to Python and isn't a "real" "intelligent" refactoring, but it helps editing similar portions of text in the current buffer, with highlighting and simultaneous editing, like multicursors. Go on a variable you want to rename and call iedit (bound to C-;). You can restrict to the current method with a 0 prefix argument: C-u 0 M-x iedit. You can use C-\' (control quote) to only show the lines that will be edited.
Red4e - change the method signature, add decorators
red4e (not a package) is a little utility, considered a proof of concept, that helps at modifying the current method signature:
- add a decorator, remove one, automatically adding or removing "self" for a staticmethod,
- add an argument (and rename it inside the method body), remove an argument, add the couple "*args, **kwargs",
- rename/copy/comment/kill the current method (and rename it in the project too),
- quickly go to another method (with imenu),
- toggle arguments on one or multiple lines,
- and more, to be continued.
It uses under the hood (mostly, not everywhere) the redbaron python library that allows to manipulate an FST (Full Syntax Tree).
Pygen - code generation
See pygen. Unmaintained.
Refactor with Elpy
See Elpy's documentation: http://elpy.readthedocs.io/en/latest/ide.html?highlight=django#refactoring
One can refactor code (with rope), format code, edit all the occurences of the symbol at point simultaneously, and fix imports.
Jedi.el is a Python auto-completion package for Emacs. It aims at helping your Python coding in a non-destructive way. It also helps you to find information about Python objects, such as docstring, function arguments and code location.
Jedi is simple to install and it works out of the box.
See screenshots and get the full documentation : http://tkf.github.io/emacs-jedi/released/
Jedi's official page: http://jedi.jedidjah.ch/en/latest/
Install Jedi.el via el-get, Marmalade or MELPA (see install for more info) and add this to your Emacs configuration:
(add-hook 'python-mode-hook 'jedi:setup) (setq jedi:setup-keys t) ; optional (setq jedi:complete-on-dot t) ; optional
or call M-x jedi:setup
Note: it's nice to use it in a python interpreter inside emacs :)
Anaconda-mode is a mode for code navigation, documentation lookup and completion for Python.
It runs on emacs 24.3 with python >= 2.6.
- context-sensitive code completion for Python
- jump to definition
- find references
- view documentation
- virtualenv management
Flymake and Flycheck
To run pep8, pylint, pyflakes and unit-tests (with nose), you can be interested in using flymake-python.
Don't forget about pylint too.
Indexing sources allows you to do neat things, like going to the definition of a function or finding which functions are calling another one.
Etags (Exuberant Ctags) generates an index (or tag) file of language objects found in source files that allows these items to be quickly and easily located by a text editor or other utility. A tag signifies a language object for which an index entry is available (or, alternatively, the index entry created for that object). Etags is a multilingual implementation of ctags.
The primary use for the tags files is looking up class/method/function/constant/etc declaration/definitions. Cscope is more powerful (see below).
Generate the tags with this command at the root of your project:
find . -name "*.py" -print | etags -
it creates the file TAGS.
cscope is a much more powerful beast. While it operates on more or less the same principle (generating a file of useful metadata) it allows you do some fancier things like find all references to a symbol, see where a function is being invoked, etc.+ (you can find definitions as well).
It was originally designed for C and C++, but thanks to version 0.3 of pycscope, pythonistas can make use of it.
The following commands should get you running:
apt-get install cscope pip install pycscope # in project base dir: find . -name '*.py' > cscope.files cscope -R
now install the xcscope emacs package with ELPA and require it:
When done, you can either enable the mode with cscope-minor-mode (which will add a menu you are free to explore) or call some functions like M-x cscope-find-global-definition, M-x cscope-find-functions-calling-this-function, etc.
Interactivity with helm-cscope
You can do all that interactively with helm-cscope (in melpa): when you are on symbol, just call M-x helm-cscope-find-global-definition (for example) and enjoy the nice interactive interface.
Elpy has commands to run either all tests of the current buffer or only the test we are on. It supports launchers for Django (thus using "manage.py test"), pytest and nose. See http://elpy.readthedocs.io/en/latest/ide.html?highlight=django#testing
If you wish to modify the way it runs test, set the following variable:
(setq elpy-test-<django, nose, pytest>-runner-command '("./manage.py" "test"))
See this snippet: https://github.com/abo-abo/hydra/wiki/Elpy
Running Make, with completion of commands
It can be handy to call a Makefile target from wherever we are in a project, and even more handy with fuzzy completion of the targets. This is easily done with helm-make. It is also included in Django mode, which does the same for management commands.
Well, let's start with the debuggers documentation:
Ipdb, ipython debugger
If you call ipdb, the execution will stop and give a nice ipython-looking prompt. Just add the line
import ipdb; ipdb.set_trace()
Now you can use the usual commands: c to continue the execution, n to execute the next one, s to step into the next function call, l to list the source code, ! to execute a statement in the current context, etc.
Note: if you like IPython/Jupyter, look at the Emacs IPython Notebook.
Note: You can use yasnippet to quickly enter this sentence, or just a simple function:
(defun python-add-breakpoint () "Add a break point" (interactive) (newline-and-indent) (insert "import ipdb; ipdb.set_trace()") (highlight-lines-matching-regexp "^[ ]*import ipdb; ipdb.set_trace()"))
You can use the following to highlight this line, and not forget it :
; Highlight the call to ipdb ; src http://pedrokroger.com/2010/07/configuring-emacs-as-a-python-ide-2/ (defun annotate-pdb () (interactive) (highlight-lines-matching-regexp "import ipdb") (highlight-lines-matching-regexp "ipdb.set_trace()")) (add-hook 'python-mode-hook 'annotate-pdb)
And just delete every breakpoints of the buffer:
(defun ipdb-cleanup () (interactive) (save-excursion (replace-regexp ".*ipdb.set_trace().*\n" "" nil (point-min) (point-max)) ;; (save-buffer) ))
If you use emacs-for-python given above, you can track your source code while debugging with pdbtrack.
A tool to use in a non-emacs terminal would be pdbpp
Realgud, a front end to many debuggers to stay in the source window
Realgud is a graphical front-end to many python debuggers. The goal is to call the debugger from within Emacs and manipulate it (continue, next commands, setting breakpoints, etc) within the source window, with single key presses. We can also inspect variables with mouse clicks, set and clear breakpoints (from the source, with mouse clicks), evaluate expressions, etc.
Redefine the code being run in the debugger session
The tip in this article makes use of the pdb feature to execute code in the context of the current stack frame with the "!" command (which can actually be omitted. See the pdb doc) and it uses an emacs package to make it much easier to write code blocks for the (i)pdb prompt (à la org-mode code blocks).
M-x compile and breakpoints
You may run your script from a real terminal, from emacs with M-x shell or with M-x compile. If you like the latter you'll notice that it hangs if you set breakpoints inside your code. The function below checks if your code has breakpoints and if so, it gives you the hand for the debugger prompt:
; thanks to https://masteringemacs.org/article/compiling-running-scripts-emacs (slightly modified) (defadvice compile (before ad-compile-smart activate) "Advises `compile' so it sets the argument COMINT to t if breakpoints are present in `python-mode' files" (when (derived-mode-p major-mode 'python-mode) (save-excursion (save-match-data (goto-char (point-min)) (if (search-forward "ipdb.set_trace()" (point-max) t) (progn (ad-set-arg 1 t) (switch-to-buffer-other-window "*compilation*")))))))
Now just run M-x compile as usual.
Insert Sphinx docstrings
The package sphinx-doc (on melpa) offers, with one shortuct, to insert or update Sphinx-like docstrings for the current method's arguments.
Eldoc: argument list in the echo area
Eldoc is a MinorMode which shows you, in the echo area, the argument list of the function call you are currently writing. It works out of the box for elisp editing and for modes that implement its support, such as python-mode. You just have to call eldoc-mode.
To always enable it:
(add-hook 'python-mode-hook 'eldoc-mode)
Jedi: show the method signature, show documentation
This pops up the function signature when you're between parentheses for a function call (not the default): M-x jedi:get-in-function-call.
There is also M-x jedi:show-doc to show the docstring for the symbol at point in another window.
See more in the Jedi documentation.
helm-pydoc is a little utility that lets you interactively select a module you want read the doc and displays it in another buffer.
If you've ever used the emacs lisp documentation system, you may find that the pydoc is poorer in functionnalities:
- there is no direct link to code source
- there is no "back" button
the Scimax starter kit
See Scimax, a starter kit for scientists and engineers, for people interested in reproducible research and publishing. Scimax is just Emacs that has been configured extensively to make it act like we need it to for research documentation and publication. It provides a lot of customization of org-mode, improvements to bibliography, tools to send emails, manage a contacts database, evaluating code asynchronously, and more.
Ipython / Jupyter notebooks
Notebooks allow you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more. This is an example of result. You can try it online.
lpy - eval a region, step through expressions
lpy is a little tool that offers to:
- press e in the right position (before an expression) to eval it,
- press j to go to the next expression
and more ! It's a new tool in development (by a great developer, rest assured the doc will follow ! :p)
You'll have to read its documented source. Lpy is actually is an attempt to implement a variant of lispy-mode for Python. Unfortunately, Python isn't nearly as well-structured as LISP. But Python is ubiquitous, and the less powerful `lpy-mode' is better than nothing at all. The basic idea of `lpy-mode' is to increase the editing efficiency by binding useful navigation, refactoring and evaluation commands to unprefixed keys, e.g. "j" or "e". But only in certain point positions, so that you are still able to use uprefixed keys to insert themselves. Example, here "|" represents the point position:
print |("2+2=%d" % (2 + 2))
Here, if you press the key "e", the whole line will be evaluated and "2+2=4" will be printed in the Echo Area.
Send the current paragraph to the REPL, go to the next paragraph
(defun my-python-para-send-and-step () "Sends the current paragraph to the python REPL and goes to the next one" (interactive) (mark-paragraph) (python-shell-send-region) (forward-paragraph))
Send code to the REPL (isend-mode, eval-in-repl)
- sk's utilities for datascience: several helper functions, similar to
ess-describe-object-at-point, to get the shape, type and sum of the symbol at point. It works very well for my needs and doesn't need any extra package - just the built-in python-mode. There are also keybindings setup to send various chunks of code.
Some tools allow to apply the changes of virtual environment activation inside emacs. They make M-x compile aware of the environment, etc.
- virtualenvwrapper emulates much of the functionnality of virtualenvwrapper, integrates well with M-x shell or eshell, is aware of hooks, has an automatic activation by project and integrates in the mode line.
- pyvenv sets the python path, uses virtualwrapper's hooks, ...
Python versions (pyenv)
Once it's installed, we just have to switch with M-x pyenv-mode-set.
Pymacs is an Emacs extension that enables the use of Python alongside with Emacs Lisp.
include import statements or remove unused
Ropemacs (see above) is a plugin for performing python refactorings in emacs. It uses rope library and pymacs. It has rope-auto-import, so if you write
and then execute M-x rope-auto-import,
from shutil import rmtree
is inserted at the top of the file.
Unfortunately Ropemacs can not do non-relative imports, it can only create imports of the "from X import Y" variety (if you type shutil.rmtree it doesn't write import shutil).
pyimport can add import statements or remove the unused ones (with pyflakes).
Elpy too can import missing ones.
See also a snippet to add many imports at once.
Sort import statements
See the py-isort Melpa package to automatically sort import statements.
pip-requirements (in MELPA) is a major mode for editing pip requirements files, with the following features:
- syntax highlighting
- Auto completion of package names from PyPI
- togglable comments
pippel (in melpa) provides a similar package menu as "package-list-packages", to list, install, upgrade, remove etc pip packages.
pip-utils (not in melpa) provides some functions with completion of packages and of the requirements files of the project:
- choose one requirements file of the project if many and install its packages,
- chooses the right virtual env,
- install a new package and add it into a requirements file (with interactive choice, needed when there is a "dev-requirements" for instance),
- check the current package version (with pip freeze),
- open the package page on pypi.
Code coverage with Python 3
Let's put them on Python libraries. Hints to write Emacs packages for Python projects.
You'll be certainly interested in the following packages (that you will find on ELPA or MELPA):
Magit, a git interface
Emacs + git is magic : see magit
Then for cross-project code navigation see xcscope that answers questions like "show all methods that call this one".
See indent-tools (in MELPA) that allows to move and fire actions by levels of indentation, for code source like Python or Yaml. For example: go to the next line that has the same indentation level, comment/kill/fold/… the current indentation level, the current paragraph, etc.
Yasnippet, a template system
helm-pydoc to browse the documentation of installed packages and import one
pungi to integrate jedi, virtualenv and buildout
How to do code folding
other ELPA packages
If you're running Emacs with package support (usually Emacs 24 and later), check out what is available in ELPA:
- M-x list-packages
- and see flymake-pyhon-pyflakes, flymake-shell, abl-mode (a python TDD minor-mode), nose (easy python test-running in emacs), visual-regexp / visual-regexp-steroids, python-magic and more.