Category: Emacs

My Blog Workflow

This blog, which I started in 2013, is hosted on wordpress.com. WordPress sites are both websites and blogs, and in late 2020 I translated my website over to this same domain name (nhigham.com).

org2blog_logo-color-multi.png

The whole site is maintained as Emacs Org mode files that I export to WordPress using the Emacs org2blog package. I do not use the WordPress editor, except occasionally to copy the html for a figure back into an Org file for tweaking and re-exporting as html (usually in order to have text next to an image, as here).

WordPress supports \LaTeX math mode, which must be typed within single dollar signs with the word “latex” right after the first dollar sign. Emacs Org mode supports \LaTeX equations and Org2blog does an excellent job of exporting to WordPress with the necessary conversions. For example, to produce the displayed equation

\notag X = \begin{bmatrix} a_{11} & a_{12} \\ a_{12} & a_{22} \end{bmatrix}^{-1}

I type

\begin{equation}\notag
 X =  \begin{bmatrix}
       a_{11} & a_{12} \\ 
       a_{12} & a_{22} 
      \end{bmatrix}^{-1}
\end{equation}

in my source and this is converted into

wordpress_eqn_example.jpg

in WordPress. (This fragment is included as an image because if I include the text directly I cannot stop WordPress interpreting it as \LaTeX!)

The beauty of this workflow is that I can export this same source to html, \LaTeX (then PDF), or even a Word file using Org mode’s export dispatcher. This is how I produce the PDF versions of the posts in the What Is series.

Advantages of this workflow are

  • I can work almost entirely in Emacs and avoid using the WordPress editor.
  • Org mode files are very readable and easy to edit.
  • Org2blog automatically uploads images into the WordPress media library and links to them.
  • Raw html can be included.

Drawbacks of the workflow are

  • \LaTeX macros cannot be defined, so \LaTeX commands must always be typed in full.
  • Displayed equations must be typed in an equation environment for reliable results.
  • All math is formatted in text style, so \displaystyle must be put in front of every \frac, \sum, etc. to obtain display style (so that formulas in fractions are not set in a smaller font, for example).
  • Inline math has awkward vertical spacing.
  • In the current version of org2blog, equation* (for an unnumbered equation) is not supported.

The latter three disadvantages would be avoided if the \LaTeX was interpreted by MathJax, but this requires a MathJax plugin, and the Business Plan is needed to be able to install plugins (I have the Premium plan).

Here is what the Emacs Org mode source code looks like for the post What Is the Sylvester Equation? After the properties drawer, which contains information about the post on WordPress, the text looks like standard \LaTeX, with the exception that a comment line begins with the # symbol rather than %.

what-is-kron_src.jpg

Org Mode Syntax Cheat Sheet

org_mode_logo.jpg

I’m a keen user of Emacs and Org mode for a variety of tasks, including

  • note taking,
  • generating documents for exporting to LaTeX, Word, or html.
  • creating blog posts (notably for this blog, using Org2blog).

Although Org mode is usually associated with Emacs, it is a markup language in its own right, and one that is far more powerful and more standardized than the Markdown language.

I recently came across the excellent blog post Org-Mode Is One of the Most Reasonable Markup Language to Use for Text by Org enthusiast Karl Voit. In the post he includes a simple example displaying some of the most important aspects of Org syntax. I was struck by how much information can be conveyed in a short piece of Org code. I have adapted Karl’s example into this longer version:

#+TITLE: Org Mode Syntax Cheat Sheet
#+OPTIONS: toc:nil 
# Adapted from http://karl-voit.at/2017/09/23/orgmode-as-markup-only/

* Top Level Heading
** Second Level Heading
*** Third Level Heading

# A comment line.  This line will not be exported.

Paragraphs are separated by at least one empty line.

*bold* /italic/ _underlined_ +strikethrough+ =monospaced=

[[https://nickhigham.wordpress.com/][Link description]]

https://nickhigham.wordpress.com/ A link without a description.

A DOI (digital object identifier) link: 
[[doi:10.1093/comnet/cnv016][Matching Exponential-Based and Resolvent-Based Centrality Measures]]

A horizontal line, fill-width across the page:
-----

- First item in a list.
- Second item.
  - Sub-item
    1. Numbered item.
    2. Another item.
- [ ] Item yet to be done.
- [X] Item that has been done.  

LaTeX macros can be included: $x_2 = \alpha + \beta^2 - \gamma$.

**** TODO A todo item.
**** DONE A todo item that has been done.

#+BEGIN_QUOTE
This text will be indented on both the left margin and the right margin.
#+END_QUOTE

: Text to be displayed verbatim (as-is), without markup 
: (*bold* does not change font), e.g., for source code. 
: Line breaks are respected. 

Some MATLAB source code:
#+BEGIN_SRC matlab
>> rand(1,3)
ans =
   5.5856e-01   7.5663e-01   9.9548e-01
#+END_SRC

Some arbitrary text to be typeset verbatim in monospace font:
#+BEGIN_SRC text
Apples, oranges,
cucumbers, tomatoes
#+END_SRC

# Table and spreadsheet.  The column headed "Ratio" is automatically
# calculated by hitting C-c C-c in Emacs on the #+TBLFM line.

|----------------+-----------+-----------+-------|
| Country        | Abstracts | Downloads | Ratio |
|----------------+-----------+-----------+-------|
| United States  |         7 |       497 |  71.0 |
| Unknown        |         4 |        83 |  20.8 |
| United Kingdom |         3 |        41 |  13.7 |
| Germany        |         3 |        29 |   9.7 |
| Netherlands    |         2 |        21 |  10.5 |
| Japan          |         1 |        18 |  18.0 |
|----------------+-----------+-----------+-------|
#+TBLFM: $4=$3/$2;%.1f

Include an image:
file:nickhighamwordpress.jpg

I have put the source on GitHub along with the results of exporting the file to txt, LaTeX, PDF (direct link), and html. I include conversions done two ways:

  • With Emacs: the recommended way.
  • With Pandoc. This is useful if you do not use Emacs or want an easy way to automate the conversions. However, Pandoc does not support all Org syntax and has different defaults, so the conversions are not identical.

For more about Org see my previous writings and videos such as Using Emacs 2 – org and Getting Started With Org Mode.

Distraction-Free Editing with Emacs

In recent years there has been a trend towards distraction-free writing, in which text is typed in a plain, full-screen window without distractions such as a web browser, email client, and notifications. The idea is that the writer can focus on the writing without being interrupted or distracted. New editors with these attributes have been produced for both desktop and mobile devices. The review mentioned in the Tweet to the right looked at nine such editors, ranging in cost from free to $44.99. Usually overlooked in such reviews is the possibility of configuring an existing editor to be distraction free. In this regard there is no better option than Emacs, which is free and trumps all the specialized editors for features.

Here is how to make Emacs distraction free. First, turn off unnecessary screen furniture by putting these lines in your .emacs configuration file:

(scroll-bar-mode 0)    ; Turn off scrollbars.
(tool-bar-mode 0)      ; Turn off toolbars.
(fringe-mode 0)        ; Turn off left and right fringe cols.
(menu-bar-mode 0)      ; Turn off menus.

Then, if you haven’t already done so, select a suitable color scheme by installing an appropriate Emacs theme; see the Emacs Themes site for lots of examples. Finally, type

M-x toggle-frame-full-screen

to make the Emacs window occupy the whole screen. Type it again to revert to the original window size.

Here is how the top left-hand corner of my screen looks as I write this post after carrying out the above steps (everything to the right of what is shown is black). 160110-emacs.jpg

It’s possible to go even further in simplifying the Emacs screen: see the excellent article Emacs, naked by Bastien Guerry.

One of the features that reviews of distraction-free editors look for is word counts. The Emacs command

M-x count-words

shows the number of characters, words, and lines, and there is an Emacs mode wc-mode that gives a constantly updated display of these statistics in the mode line.

Another relevant feature is support for Markdown. Emacs has a Markdown mode, as well as a much more powerful markup language called Org mode (used to write the posts on this blog).

“Distraction free” can also mean focusing on the part of the document you are editing, and this brings me to a final feature that is rarely discussed: narrowing. Narrowing means seeing only the logical unit that you are editing, which could be a paragraph, subsection, section, or a logical unit in a markup language. Emacs can intelligently narrow down to the appropriate unit. I recommend Artur Malabarba’s narrow-or-widen-dwim command. The easiest way to narrow to a paragraph is to mark it as a region with M-x mark-paragraph then type M-x call narrow-or-widen-dwim. Of course, all the Emacs commands beginning M-x above can be assigned to a key and invoked with a single keypress.

One possible objection to Emacs is that it does not run on mobile devices. In fact, there are projects to port Emacs or implement Emacs-like editors, as Googling will reveal. And of course one can always SSH into a remote machine running Emacs.

Managing BibTeX Files with Emacs

Here are a couple of tips on exploiting Emacs when working with BibTeX databases. I do not use a BibTeX database manager (though I have tried Jabref). I prefer to edit my bib files entirely within Emacs, and have had no trouble doing so even with my largest database of around 6800 entries.

1. Cleaning Downloaded Bib Entries

Back in the 1980s and 1990s BibTeX users had little choice but to create bib entries from scratch. Nowadays most journal websites have an “export citation” option that allows you to download a bib entry for a paper you are looking at. Unfortunately, these bib entries are of widely variable quality and are almost never entirely correct. What’s more, they will not match your own house style as regards ordering of fields and use of abbreviations. A great help in reformatting bib entries is the bibtex-clean-entry command in the BibTeX-mode of Emacs. I use it to

  • remove unwanted blank lines and spaces,
  • put the fields in the order I prefer,
  • change field delimiters from curly brackets to quotes,
  • delete delimiters around numerical fields,
  • change double dashes to single dashes in page ranges (BibTeX will convert single dashes to double as necessary),
  • change the entry type and field names to lower case,
  • set the indentation of field to two spaces.

These and other aspects of the formatting are customizable. Here is an example of a bib entry downloaded from JSTOR.

@article{10.2307/2323313,
 ISSN = {00029890, 19300972},
 URL = {http://www.jstor.org/stable/2323313},
 author = {Dan Scott, Donald R. Peeples},
 journal = {The American Mathematical Monthly},
 number = {7},
 pages = {651-653},
 publisher = {Mathematical Association of America},
 title = {A Constructive Proof of the Partial Fraction Decomposition},
 volume = {95},
 year = {1988}
}

After I hit C-c C-c in Emacs the entry is transformed to

@article{10.2307/2323313,
  author = "Dan Scott, Donald R. Peeples",
  title = "A Constructive Proof of the Partial Fraction Decomposition",
  journal = "The American Mathematical Monthly",
  volume = 95,
  number = 7,
  pages = "651-653",
  year = 1988,
  issn = "00029890, 19300972",
  url = "http://www.jstor.org/stable/2323313",
  publisher = "Mathematical Association of America"
        }

Further work is needed (such as replacing the comma by “and” in the author field), but the entry is now much closer to the format I require.

BibTeX-mode is not perfect. While bibtex-clean-entry can even generate a new key for the entry, I have not been able to get it to match my key-generating algorithm. And I have not found a way to make bibtex-clean-entry delete fields (such as issn, or publisher for the article type) that I do not use.

If you would like to see how I customized BibTeX-mode, take a look at the .emacs and .emacs-custom_windows files in my dot-emacs setup.

2. Bib Search and PDF Display with Helm-BibTeX

The Emacs RefTeX package provides some useful features for handling citations within a \LaTeX document, providing an intelligent link between the \LaTeX document and the relevant BibTeX databases. One thing it does not do is enable easy access to PDF files corresponding to bib entries. The Helm-BibTeX package can do this and much more.

To explain how I use Helm-BibTeX, suppose I have the following lines in my dot emacs.

(setq helm-bibtex-bibliography '("~/texmf/bibtex/bib/la.bib"
                                 "~/texmf/bibtex/bib/njhigham.bib"
                                 ))

(setq helm-bibtex-library-path '("~/pdf_papers" "~/pdf_books"))

These tell Helm-BibTeX to look in the first path for bib files and in the second path for PDF files whose names match a specified bib key.

If I type M-x helm-bibtex and type dongarra in the minibuffer then all bib entries in la.bib and njhigham.bib containing dongarra in the author or title fields are displayed, one per line. I can narrow the list down by typing a space and then another search term. Hitting tab displays several options, which include options to insert a \cite command for the item under the cursor, to insert a formatted reference for the item under the cursor, and to go to that entry in the bib file. The fourth column shows a looped square symbol (⌘) that indicates the presence of a matching PDF file. Hitting enter (or f1, or Ctrl-j) opens the PDF file in the default PDF application.

Here is Helm-BibTeX in action. helm-bibtex.jpg

Helm-BibTeX relies on the Helm package, which provides a powerful framework for incremental completion and selection narrowing.

My Dot Emacs

I have been using Emacs on Unix systems since the 1980s. The vast majority of my work is done on PCs, though, and from about 1988 until 2011 I used an editor for the PC called Qedit, later renamed TSE Pro, from The SemWare Corporation. TSE Pro is small, fast, and highly customizable via its own macro language. I customized it to my preferences over many years, in particular for use with \LaTeX, and wrote four books and many papers with it. When I acquired a Mac laptop in 2009 I realized that I needed to switch to an editor that runs on both Windows and Mac, so after trying Vim and Emacs (and temporarily using TextMate on the Mac) I switched to Emacs.

My past use of Emacs had mainly been to send emails or edit web pages, often over a terminal connection, and I had not realized how fast Emacs was—nor that one could issue commands by typing Alt- instead of the two consecutive keys Esc .

dot-emacs-screen.jpg
Emacs interrupted by a train, courtesy of https://github.com/mtoyoda/sl

Since 2011 I have used Emacs exclusively and have built up my dot emacs to its current size of nearly 3000 lines. For non-Emacs users, a dot emacs (.emacs) file is an Emacs Lisp file that is read and executed when Emacs loads, and it is where one sets customizations and loads packages.

I have now made my dot emacs file available on Github.

A few notes:

  • Some of the code uses conditional tests to see whether Emacs is running on a Windows machine or a Mac machine and take different action in each case.
  • The code is lightly documented with links to sources for the things I am using.
  • I have recently started using John Wiegley’s use-package for loading packages. Not everything is converted and I may convert other package to load this way. Use-package has several advantages:
    • It makes the dot emacs more readable.
    • It makes it easy to disable a package, by adding a line “:disabled t”.
    • It speeds up the loading of Emacs by delaying loading of packages until they are first used. When I first started using use-package my Emacs load time dropped from 8 seconds to 5 seconds.
  • I do not use the Emacs package manager, but rather download packages manually and place them in ~/dropbox/elisp. The one exception is the Helm package, which I was unable to install manually.
  • I spend most of my time in Emacs in org-mode, latex-mode, or bibtex-mode, so quite a lot of my dot emacs concerns those modes.
  • I recently started using Oleh Krehel’s hydra package, which was released at the start of the year. It provides a nice way to set up key bindings based on a prefix key and provides a reminder at the bottom of the screen about possible subsequent keypresses.

You should not expect to be able to take my dot emacs and use it as is, because many of the packages that are loaded will not be present on your system in the expected location. However, the main benefit of sharing Emacs configurations is that one can copy and adapt pieces of code.

My dot emacs is long, and I have trouble remembering all the features I have set up, even though I have an even longer emacs.org file that contains notes and a summary of my customized keypresses.

I still use TSE Pro: to edit my dot emacs file when an error introduced therein causes Emacs to fail to start properly! (Of course, I could also use emacs -q, which inhibits loading of the .emacs, but I like to keep my hand in with TSE Pro).

tse-pro-screen2.jpg

Emacs and Org Mode: What People are Saying

For a couple of years I’ve been collecting tweets about Emacs and Org mode. With the Twitter app’s new ability to provide code to embed tweets I decided to create a post listing the collection. If you are not an existing user of Emacs or Org mode these tweets should give you a feel for whether you might want to explore further. If you are already a convert then many of the sentiments expressed here will be familiar. Note that the links and hashtags below are clickable. (Like all of this Blog, this post was written in Org mode.)

https://twitter.com/K1Hedayati/status/504539678310424576

https://twitter.com/NeilDunbar66/status/364698349166800896

https://twitter.com/rodet/status/326667805497901059

Solutions to Some Emacs Problems

I’ve been using Emacs exclusively for a couple of years, on both Windows and Mac machines. Prior to that I had used it intermittently, mainly on Unix machines, for over 20 years.

In setting up and customizing Emacs I’ve hit a few Emacs problems whose solutions were not easily found on the web. I’m listing my solutions here in the hope that others will find them useful. The issues covered here are very specific. I’ll be writing about aspects of Emacs of more general interest in the future.

Copy and Paste Problems on Mac

I encountered an intermittent problem on the Mac, originally when exporting from Org mode, giving the strange error 

"quit: pasteboard doesn't contain valid data"

This error was then produced every time I did a kill (copy) or yank (paste). Pasting anything from another app (say Firefox) seemed to cure the problem, at least for a while. I eventually tracked the problem down to this configuration line that I’d put in my .emacs: 

(setq save-interprogram-paste-before-kill nil)

Making that code execute only when Emacs is running under Windows cured the problem.

Executables

On Windows I had trouble getting Emacs to find the git executable, despite the relevant directory being on the exec-path. In the end I had to set, in my .emacs, 

(custom-set-variables
...
 '(magit-git-executable "c:\\program files (x86)\\git\\bin\\git.exe")
 '(vc-git-program "c:\\program files (x86)\\git\\bin\\git.exe")
...)

I run git from magit; the second of those lines is needed for vc-annotate.

System Name

In my .emacs I set the Emacs window size based on the system-name variable, as I want a different window size for a laptop versus a large desktop screen. When travelling I’ve found that the window on my MacBook Pro sometimes opens too large and looks very strange, with a missing top line of the window. It seems that being on different wireless networks can change the value of system-name. I have not found a solution, other than to resize the window with the mouse when Emacs starts.

Take Command Shell

On Windows, I use the excellent Take Command shell instead of the CMD shell that comes with Windows. In order to get LaTeX to work as I wanted I needed to tell Emacs to use this shell: 

(setq TeX-shell "d:/tcmd/tcc.exe")

Emacs Org Mode Version 8: Upgrading and Some Tips

As I mentioned in my post Emacs: The Ultimate Editor?, one of the things I love about Emacs is Org mode, which provides excellent facilities for working with plain text and exporting it to a variety of other formats. Recently I’ve used Org mode to prepare a number of tables within documents that I then export to \LaTeX and compile to PDF. Key here is Org’s ability to easily add or remove rows and columns, sort rows, and even transpose a table (see below). This blog is written in Org mode and exported to WordPress using org2blog.

A couple of months ago, version 8 of Org was released. It has many improvements over earlier versions but also some changes in syntax. In particular, the export engine has been rewritten. These changes are quite likely to break older Org files. Indeed the release notes say Org 8.0 is the most disruptive major version of Org.

Here is a list of problems I’ve experienced and the fixes. I’m currently using Org 8.0.3.

  • Export to Beamer didn’t work until I added 
    (require 'ox-beamer)

    to my .emacs.

  • org2blog was broken in Org 8. A new branch for Org 8 was released at https://github.com/ptrv/org2blog/tree/org-8-support. In my tests org2blog/wp-post-subtree did not work properly: the title was being copied as a section heading. This was quickly fixed by author Peter Vasil earlier this week and org2blog is now working fine for me with Org 8.
  • The syntax for \LaTeX table alignments has changed. In Org <8: 
    #+ATTR_LaTeX: align = |l|...

    In Org 8: 

    #+ATTR_LaTeX: :align |l|...

Finally, here are a couple of useful, but easy to miss, features of Org.

Table Transpose

A new command org-table-transpose-table-at-point in Org 8 provides the array transpose function. With the cursor in the table

a11 a12 a13 a14
a21 a22 a23 a24
a31 a32 a33 a34

M-x org-table-transpose-table-at-point produces

a11 a21 a31
a12 a22 a32
a13 a23 a33
a14 a24 a34

This could be particularly useful in a \LaTeX file, provided orgtbl-mode is being used, as there is no easy way to transpose a \LaTeX table.

Shortcuts

I’m not sure if this is new to ORG 8, but in any case it’s new to me. Type <s followed by tab and an empty source block magically appears: 

#+BEGIN_SRC 
#+END_SRC

Very useful! The following table shows all the available expansions: 

|----------+------------------|
| Sequence | Expands to       |
|----------+------------------|
| <s       | #+BEGIN_SRC      |
| <e       | #+BEGIN_EXAMPLE  |
| <q       | #+BEGIN_QUOTE    |
| <v       | #+BEGIN_VERSE    |
| <V       | #+BEGIN_VERBATIM |
| <c       | #+BEGIN_CENTER   |
| <l       | #+BEGIN_LaTeX    |
| <L       | #+LaTeX          |
| <h       | #+BEGIN_HTML     |
| <H       | #+HTML           |
| <a       | #+BEGIN_ASCII    |
| <A       | #+ASCII:         |
| <i       | #+INDEX:         |
| <I       | #+INCLUDE:       |
|----------+------------------|

Emacs: The Ultimate Editor?

I started using Emacs about 1990 but have been using it exclusively for just two years. Prior to that my main editor was TSE Pro – a fast, customizable Windows-only editor that evolved from a 1980s DOS editor called Qedit. The motivation for switching to Emacs was that I wanted to be able to work in the same way on both Windows and Mac machines. After looking at the possibilities I settled on Emacs as the ideal solution.

Although Emacs dates from the 1980s, it seems to have enjoyed renewed popularity in the last few years, with regular new releases (currently version 24.3), many new packages appearing, several excellent Emacs blogs posting regularly, and even an Emacs conference held in London in March 2013.

For me the main advantages of Emacs are

  • Excellent LaTeX integration through AucTeX and RefTeX.
  • ORG mode, an incredibly powerful mode for working with plain text. Among its uses are
    • making notes and outlines,
    • TODO lists,
    • writing documents in a simple markup language that can be exported to LaTeX, html, and various other formats,
    • writing and managing WordPress blogs, via org2blog (this blog is produced entirely from within Emacs, apart from some tweaking in WordPress).

    In all these cases, the ability to narrow the view to certain parts of the buffer, and to reorder logical units via simple keypresses, provide tremendous usability.

  • Complete built-in documentation, with the ability to see the Emacs Lisp source code for all functions except the small number of low-level functions written in compiled C.
  • The ability to customize every aspect of Emacs, and in particular to reassign almost any keypress.
  • The use of Emacs is essentially system-independent; in particular Emacs has its own file management functions, which bypass the Windows, Mac or Linux file open/save dialog boxes.

I’ll write about some of these aspects in future posts.

For now, here are some videos that provide more information:

And for an excellent perspective on the eternal “Emacs/Vi versus the latest hot editor” debate I recommend the post Good tools by James Bennett, which appeared just as I was about to publish this post.