Python

How can we turn Emacs as full featured Python IDE? How to use Emacs for data science? Many packages are explained below. You don't have to install all of them one by one, see the all-in-one solutions, like Elpy, starter kits and the Spacemacs layer.

An important library you'll want to get familiar with is Jedi, an autocompletion and analysis library for Python. It helps at autocompletion, finding definitions, showing documentation, etc. Elpy makes good use of it.

We also have Rope that allows cross-project refactoring and smaller ones (change the method signature,…). Again Elpy bases on it and other packages like emacs-traad make it easy to install and use.

Default modes

There are a number of python modes for Emacs. fgallina's python.el is the python.el included in Emacs 24.2 and onwards.

All-in-one solutions

emacs-for-python

emacs-for-python is a bundle of the above modes (and more), and it's an easy way to turn Emacs into a Python IDE.

emacs-for-python at github

Elpy

Elpy is a collection of elisp packages for Python too.

https://github.com/jorgenschaefer/elpy/wiki/Features

It permits, among others, to:

refactor code (using rope),
send a region or a buffer to a python interpreter,
run unit tests, run the test the cursor is in, with a test runner of choice (django, pytest, …),
jump to a symbol definition (elpy-goto-definition),…

and of course

on-the-fly checks with flymake,
code completion with company-mode and either rope or jedi,
on the fly code helper with eldoc (and rope or jedi),
code navigation,
change of virtual environment inside Emacs (pyvenv), etc

EMP: EMacs Python

The goal of emp is to take all the best development components available for Python, and combine it conveniently under Emacs. It builds on top of elpy, and provides even more bindings.

https://kootenpv.github.io/2016-01-01-emacs-python

Emacs-bootstrap

Emacs-bootstrap is a website where you can select a few programming languages, some options, and have it generate a full .emacs.d folder for you.

Language Server Protocol support

See eglot, a polyglot LSP mode for Emacs (by the author of Yasnippet,…),

and pyls specifically for Python.

Refactoring

rope & ropemacs

Presentation

rope is a library for refactoring and manipulating Python code, pymacs is an interface between emacs lisp and python, and ropemacs is an Emacs interface to the rope library which uses rope and pymacs.

If you do some search and replace of code objects in your code and find it sometimes tricky, and/or you need to do it in more than one file, then you should consider a good tool of refactoring.

But Rope can do more than this simple example, it can:

Rename anything
Extract method/local variable
Change method signature
Perform cross-project refactorings
Support Mercurial, GIT, Darcs and SVN in refactorings

Rope can also help IDE's with:

Auto-completion
Finding definition location
Getting pydoc
Finding occurrences
Organizing imports (removing unused and duplicate imports and sorting them)
Generating python elements

Installation

Note: see also emacs-traad below that has an easy installation process.

With el-get

So an easy way is to use an el-get recipe: M-x el-get-install ropemacs. It installs and build Pymacs, rope and ropemacs. The difference from the solution using pip is that it setups the Emacs side of Pymacs correctly.

Manually

You can follow the instructions as given here ,if you don't want to use marmalade you can use the following instructions .

   sudo pip install rope ropemacs

As of February 2014 you can't install Pymacs via pip, therefore:

   git clone http://github.com/pinard/pymacs
   cd pymacs
   make check
   sudo make install

make check ensures the prerequisites. sudo make install puts Pymacs module in Python local modules path, usually /usr/local/lib/python2.7/dist-packages/. People generally frown upon make install, because if you lose the Makefile, it can become hard to cleanly uninstall. But you shouldn't fear it, as pip uses the same directory, so pip uninstall pymacs will work.

After you do the make install of pymacs do not delete the folder.Copy the file pymacs.el to ~/.emacs.d/site-lisp/

Now add path of pyamcs.el to your ~/.emacs file,using this code:

   (setq emacs-config-path "~/.emacs.d/") 
   (setq base-lisp-path "~/.emacs.d/site-lisp/")
   (setq site-lisp-path (concat emacs-config-path "/site-lisp"))
   (defun add-path (p)
     (add-to-list 'load-path (concat base-lisp-path p)))
   (add-path "")
   (add-to-list 'load-path "~/.emacs.d")

You can now lazily load ropemacs or load it at runtime.For lazy evaluation add this to your .emacs file:

   (defun load-ropemacs ()
     "Load pymacs and ropemacs"
     (interactive)
     (require 'pymacs)
     (pymacs-load "ropemacs" "rope-")
     ;; Automatically save project python buffers before refactorings
     (setq ropemacs-confirm-saving 'nil)
   )
   (global-set-key "\C-xpl" 'load-ropemacs)

use c-xpl to load ropemacs.Read more about it here

Rope & traad

Traad is a client-server approach to using the rope Python refactory library. It involves two basic components:

A HTTP server exposing the rope API via JSON, and
Client libraries for talking to the server

Since rope is written in Python, any tool that wants to use rope needs to either embed Python, or it needs to find a way to communicate with a Python process running rope. The embedding approach is difficult if the target environment is not written in Python, and it also faces challenges when dealing with more than one Python version.

So traad aims to simplify communication with rope running in an independent process. HTTP communication and JSON data handling is well supported in many, many languages, so any environment that wants to use rope should be able to easily communicate with traad.

emacs-traad, easy installation and use

emacs-traad (simply traad in melpa) makes it very easy to install traad, and thus rope. Install traad from melpa and then M-x traad-install-server. Then start a connection with traad-open and enjoy. Furthermore it installs traad in its own virtualenv, so it won't clutter your current virtualenv, and you don't have to re-install it in every project.

usage

It is possible to remove a method's argument, normalize arguments, see the occurrences of a method, organize imports, some more python refactoring, and also see the history of traad changes.

Other Python "refactoring" tools

Projectile - replace in project

This is not specific to Python but is very helpful anyway. Projectile has a function projectile-replace which interactively offers to replace all occurrences of a term in all the files of the project. So it's handy to rename a method.

Iedit - replace occurences of a symbol

Iedit isn't specific to Python and isn't a "real" "intelligent" refactoring, but it helps editing similar portions of text in the current buffer, with highlighting and simultaneous editing, like multicursors. Go on a variable you want to rename and call iedit (bound to C-;). You can restrict to the current method with a 0 prefix argument: C-u 0 M-x iedit. You can use C-\' (control quote) to only show the lines that will be edited.

Red4e - change the method signature, add decorators

red4e (not a package) is a little utility, considered a proof of concept, that helps at modifying the current method signature:

add a decorator, remove one, automatically adding or removing "self" for a staticmethod,
add an argument (and rename it inside the method body), remove an argument, add the couple "*args, **kwargs",
rename/copy/comment/kill the current method (and rename it in the project too),
quickly go to another method (with imenu),
toggle arguments on one or multiple lines,
and more, to be continued.

It uses under the hood (mostly, not everywhere) the redbaron python library that allows to manipulate an FST (Full Syntax Tree).

Pygen - code generation

See pygen. Unmaintained.

Refactor with Elpy

See Elpy's documentation: http://elpy.readthedocs.io/en/latest/ide.html?highlight=django#refactoring

One can refactor code (with rope), format code, edit all the occurences of the symbol at point simultaneously, and fix imports.

Auto-Completion

Jedi

Jedi.el is a Python auto-completion package for Emacs. It aims at helping your Python coding in a non-destructive way. It also helps you to find information about Python objects, such as docstring, function arguments and code location.

Jedi is simple to install and it works out of the box.

See screenshots and get the full documentation : http://tkf.github.io/emacs-jedi/released/

Jedi's official page: http://jedi.jedidjah.ch/en/latest/

Installation

Install Jedi.el via el-get, Marmalade or MELPA (see install for more info) and add this to your Emacs configuration:

   (add-hook 'python-mode-hook 'jedi:setup)
   (setq jedi:setup-keys t)                      ; optional
   (setq jedi:complete-on-dot t)                 ; optional

or call M-x jedi:setup

Note: it's nice to use it in a python interpreter inside emacs :)

Anaconda

Anaconda-mode is a mode for code navigation, documentation lookup and completion for Python.

It runs on emacs 24.3 with python >= 2.6.

It provides:

context-sensitive code completion for Python
jump to definition
find references
view documentation
virtualenv management

The package is available in MELPA. For more information, read its documentation.

Code Checker

Flymake and Flycheck

Flymake (in Emacs) is a on-the-fly syntax checker for Emacs. We can use it with flycheck (in melpa), that aims at being a replacement.

To run pep8, pylint, pyflakes and unit-tests (with nose), you can be interested in using flymake-python.

Don't forget about pylint too.

MyPy checks

With mypy-mode we get a compilation-mode based support for python 3.6+ type hints checks.

As a side note, we can also get flycheck to work, at the same time as pylint, with:

(flycheck-define-checker
    python-mypy ""
    :command ("mypy"
              "--ignore-missing-imports" "--fast-parser"
              "--python-version" "3.6"
              source-original)
    :error-patterns
    ((error line-start (file-name) ":" line ": error:" (message) line-end))
    :modes python-mode)

(add-to-list 'flycheck-checkers 'python-mypy t)
(flycheck-add-next-checker 'python-pylint 'python-mypy t)

Indexing sources: ctags, cscope, pycscope

Indexing sources allows you to do neat things, like going to the definition of a function or finding which functions are calling another one.

Note that Elpy does the "go to definition" feature (elpy-goto-definition), without the need of indexing sources.

etags, ctags

Etags (Exuberant Ctags) generates an index (or tag) file of language objects found in source files that allows these items to be quickly and easily located by a text editor or other utility. A tag signifies a language object for which an index entry is available (or, alternatively, the index entry created for that object). Etags is a multilingual implementation of ctags.

The primary use for the tags files is looking up class/method/function/constant/etc declaration/definitions. Cscope is more powerful (see below).

usage

Generate the tags with this command at the root of your project:

    find . -name "*.py" -print | etags -

it creates the file TAGS.

Note that projects like Projectile or Helm provide an integrated use of etags (finding one, re-generating the index, etc).

cscope

cscope is a much more powerful beast. While it operates on more or less the same principle (generating a file of useful metadata) it allows you do some fancier things like find all references to a symbol, see where a function is being invoked, etc.+ (you can find definitions as well).

It was originally designed for C and C++, but thanks to version 0.3 of pycscope, pythonistas can make use of it.

usage

The following commands should get you running:

   apt-get install cscope
   pip install pycscope
   # in project base dir:
   find . -name '*.py' > cscope.files
   cscope -R

now install the xcscope emacs package with ELPA and require it:

   (require 'xcscope)

When done, you can either enable the mode with cscope-minor-mode (which will add a menu you are free to explore) or call some functions like M-x cscope-find-global-definition, M-x cscope-find-functions-calling-this-function, etc.

Interactivity with helm-cscope

You can do all that interactively with helm-cscope (in melpa): when you are on symbol, just call M-x helm-cscope-find-global-definition (for example) and enjoy the nice interactive interface.

Running tests

Elpy has commands to run either all tests of the current buffer or only the test we are on. It supports launchers for Django (thus using "manage.py test"), pytest and nose. See http://elpy.readthedocs.io/en/latest/ide.html?highlight=django#testing

If you wish to modify the way it runs test, set the following variable:

(setq elpy-test-<django, nose, pytest>-runner-command '("./manage.py" "test"))

A Hydra to choose what tests to run, navigate errors and switch to the console

See this snippet: https://github.com/abo-abo/hydra/wiki/Elpy

Running Make, with completion of commands

It can be handy to call a Makefile target from wherever we are in a project, and even more handy with fuzzy completion of the targets. This is easily done with helm-make. It is also included in Django mode, which does the same for management commands.

Debugging

Well, let's start with the debuggers documentation:

pdb doc
pdb++ has a few more commands
ipdb to make use of ipython (aka jupyter)

Ipdb, ipython debugger

If you call ipdb, the execution will stop and give a nice ipython-looking prompt. Just add the line

   import ipdb; ipdb.set_trace()

Now you can use the usual commands: c to continue the execution, n to execute the next one, s to step into the next function call, l to list the source code, ! to execute a statement in the current context, etc.

Note: if you like IPython/Jupyter, look at the Emacs IPython Notebook.

Note: You can use yasnippet to quickly enter this sentence, or just a simple function:

(defun python-add-breakpoint ()
  "Add a break point"
  (interactive)
  (newline-and-indent)
  (insert "import ipdb; ipdb.set_trace()")
  (highlight-lines-matching-regexp "^[ ]*import ipdb; ipdb.set_trace()"))

You can use the following to highlight this line, and not forget it :

; Highlight the call to ipdb
; src http://pedrokroger.com/2010/07/configuring-emacs-as-a-python-ide-2/
(defun annotate-pdb ()
  (interactive)
  (highlight-lines-matching-regexp "import ipdb")
  (highlight-lines-matching-regexp "ipdb.set_trace()"))
(add-hook 'python-mode-hook 'annotate-pdb)

And just delete every breakpoints of the buffer:

(defun ipdb-cleanup ()
    (interactive)
    (save-excursion
      (replace-regexp ".*ipdb.set_trace().*\n" "" nil (point-min) (point-max))
      ;; (save-buffer)
      ))

pdb track

If you use emacs-for-python given above, you can track your source code while debugging with pdbtrack.

A tool to use in a non-emacs terminal would be pdbpp

Realgud, a front end to many debuggers to stay in the source window

Realgud is a graphical front-end to many python debuggers. The goal is to call the debugger from within Emacs and manipulate it (continue, next commands, setting breakpoints, etc) within the source window, with single key presses. We can also inspect variables with mouse clicks, set and clear breakpoints (from the source, with mouse clicks), evaluate expressions, etc.

Redefine the code being run in the debugger session

The tip in this article makes use of the pdb feature to execute code in the context of the current stack frame with the "!" command (which can actually be omitted. See the pdb doc) and it uses an emacs package to make it much easier to write code blocks for the (i)pdb prompt (à la org-mode code blocks).

M-x compile and breakpoints

You may run your script from a real terminal, from emacs with M-x shell or with M-x compile. If you like the latter you'll notice that it hangs if you set breakpoints inside your code. The function below checks if your code has breakpoints and if so, it gives you the hand for the debugger prompt:

; thanks to https://masteringemacs.org/article/compiling-running-scripts-emacs (slightly modified)
(defadvice compile
    (before ad-compile-smart activate)
  "Advises `compile' so it sets the argument COMINT to t if breakpoints are present in `python-mode' files"
  (when (derived-mode-p major-mode 'python-mode)
    (save-excursion
      (save-match-data (goto-char (point-min))
                       (if (search-forward "ipdb.set_trace()"
                                           (point-max)
                                           t)
                           (progn (ad-set-arg 1 t)
                                  (switch-to-buffer-other-window "*compilation*")))))))

Now just run M-x compile as usual.

Documentation

Insert Sphinx docstrings

The package sphinx-doc (on melpa) offers, with one shortuct, to insert or update Sphinx-like docstrings for the current method's arguments.

Eldoc: argument list in the echo area

Eldoc is a MinorMode which shows you, in the echo area, the argument list of the function call you are currently writing. It works out of the box for elisp editing and for modes that implement its support, such as python-mode. You just have to call eldoc-mode.

To always enable it:

   (add-hook 'python-mode-hook 'eldoc-mode)

Jedi: show the method signature, show documentation

This pops up the function signature when you're between parentheses for a function call (not the default): M-x jedi:get-in-function-call.

There is also M-x jedi:show-doc to show the docstring for the symbol at point in another window.

See more in the Jedi documentation.

helm-pydoc

helm-pydoc is a little utility that lets you interactively select a module you want read the doc and displays it in another buffer.

pydoc linkification

If you've ever used the emacs lisp documentation system, you may find that the pydoc is poorer in functionnalities:

there is no direct link to code source
there is no "back" button

John Kitchin's pydoc aims at fixing that with more enhancements, like a bit of source fontification. See more at his presentation and on github.

info format

Another way to read the python documentation is in the texinfo format. You'll need to install the package python-info on MELPA and go through the installation process.

Datascience

the Scimax starter kit

See Scimax, a starter kit for scientists and engineers, for people interested in reproducible research and publishing. Scimax is just Emacs that has been configured extensively to make it act like we need it to for research documentation and publication. It provides a lot of customization of org-mode, improvements to bibliography, tools to send emails, manage a contacts database, evaluating code asynchronously, and more.

Ipython / Jupyter notebooks

You have online Jupyter notebooks, also a Qt console and an emacs ipython notebook !

Notebooks allow you to create and share documents that contain live code, equations, visualizations and explanatory text. Uses include: data cleaning and transformation, numerical simulation, statistical modeling, machine learning and much more. This is an example of result. You can try it online.

lpy - eval a region, step through expressions

lpy is a little tool that offers to:

press e in the right position (before an expression) to eval it,
press j to go to the next expression

and more ! It's a new tool in development (by a great developer, rest assured the doc will follow ! :p)

You'll have to read its documented source. Lpy is actually is an attempt to implement a variant of lispy-mode for Python. Unfortunately, Python isn't nearly as well-structured as LISP. But Python is ubiquitous, and the less powerful `lpy-mode' is better than nothing at all. The basic idea of `lpy-mode' is to increase the editing efficiency by binding useful navigation, refactoring and evaluation commands to unprefixed keys, e.g. "j" or "e". But only in certain point positions, so that you are still able to use uprefixed keys to insert themselves. Example, here "|" represents the point position:

     print |("2+2=%d" % (2 + 2))

Here, if you press the key "e", the whole line will be evaluated and "2+2=4" will be printed in the Echo Area.

Send the current paragraph to the REPL, go to the next paragraph

(defun my-python-para-send-and-step ()
  "Sends the current paragraph to the python REPL and goes to the next one"
  (interactive)
  (mark-paragraph)
  (python-shell-send-region)
  (forward-paragraph))

Send code to the REPL (isend-mode, eval-in-repl)

This is actually a default feature of python.el. See its menu to send a line or a region to the repl.

Besides, isend-mode (in MELPA) works with many languages and REPLs, and so does eval-in-repl (in MELPA). With the latter, just use C-RET.

Other tools

Virtual environments

Some tools allow to apply the changes of virtual environment activation inside emacs. They make M-x compile aware of the environment, etc.

virtualenvwrapper emulates much of the functionnality of virtualenvwrapper, integrates well with M-x shell or eshell, is aware of hooks, has an automatic activation by project and integrates in the mode line.

pyvenv sets the python path, uses virtualwrapper's hooks, ...

Python versions (pyenv)

We can change python versions with pyenv-mode (in melpa) and automatically when there is a .python-version file with pyenv-mode-auto.

Once it's installed, we just have to switch with M-x pyenv-mode-set.

Pymacs

Pymacs is an Emacs extension that enables the use of Python alongside with Emacs Lisp.

Pymacs homepage

include import statements or remove unused

Ropemacs (see above) is a plugin for performing python refactorings in emacs. It uses rope library and pymacs. It has rope-auto-import, so if you write

   rmtree

and then execute M-x rope-auto-import,

   from shutil import rmtree

is inserted at the top of the file.

Unfortunately Ropemacs can not do non-relative imports, it can only create imports of the "from X import Y" variety (if you type shutil.rmtree it doesn't write import shutil).

pyimport can add import statements or remove the unused ones (with pyflakes).

Elpy too can import missing ones.

See also a snippet to add many imports at once.

Sort import statements

See the py-isort Melpa package to automatically sort import statements.

pip-requirements mode

pip-requirements (in MELPA) is a major mode for editing pip requirements files, with the following features:

syntax highlighting
Auto completion of package names from PyPI
togglable comments

pip utilities

pippel (in melpa) provides a similar package menu as "package-list-packages", to list, install, upgrade, remove etc pip packages.

pip-utils (not in melpa) provides some functions with completion of packages and of the requirements files of the project:

choose one requirements file of the project if many and install its packages,
chooses the right virtual env,
install a new package and add it into a requirements file (with interactive choice, needed when there is a "dev-requirements" for instance),
check the current package version (with pip freeze),
open the package page on pypi.

Code coverage with Python 3

There's a tool to highlight in real time the lines that are (not) covered by your unit tests. It's called pycoverage.el and it is in melpa.

Libraries

Let's put them on Python libraries. Hints to write Emacs packages for Python projects.

Python

Default modes

All-in-one solutions

emacs-for-python

Elpy

EMP: EMacs Python

Emacs-bootstrap

Language Server Protocol support

Refactoring

rope & ropemacs

Presentation

Installation

With el-get

Manually

Rope & traad

emacs-traad, easy installation and use

usage

Other Python "refactoring" tools

Projectile - replace in project

Iedit - replace occurences of a symbol

Red4e - change the method signature, add decorators

Pygen - code generation

Refactor with Elpy

Auto-Completion

Jedi

Installation

Anaconda

Code Checker

Flymake and Flycheck

MyPy checks

Indexing sources: ctags, cscope, pycscope

etags, ctags

usage

cscope

usage

Interactivity with helm-cscope

Running tests

A Hydra to choose what tests to run, navigate errors and switch to the console

Running Make, with completion of commands

Debugging

Ipdb, ipython debugger

pdb track

Realgud, a front end to many debuggers to stay in the source window

Redefine the code being run in the debugger session

M-x compile and breakpoints

Documentation

Insert Sphinx docstrings

Eldoc: argument list in the echo area

Jedi: show the method signature, show documentation

helm-pydoc

pydoc linkification

info format

Datascience

the Scimax starter kit

Ipython / Jupyter notebooks

lpy - eval a region, step through expressions

Send the current paragraph to the REPL, go to the next paragraph

Send code to the REPL (isend-mode, eval-in-repl)

See also

Other tools

Virtual environments

Python versions (pyenv)

Pymacs

include import statements or remove unused

Sort import statements

pip-requirements mode

pip utilities

Code coverage with Python 3

Libraries

See also

Magit, a git interface

Code navigation: jump to function definitions

Code navigation: move by indentation levels

Yasnippet, a template system

helm-pydoc to browse the documentation of installed packages and import one

pungi to integrate jedi, virtualenv and buildout

How to do code folding

other ELPA packages

Navigation menu

Search