Building personal search infrastructure for your knowledge and code [see within blog graph]

For the most part, I keep things in Org mode, and I use Emacs to work with it. Apart from regular means of plaintext search (I'll write about it later), for me it's important to search over tags:

• org-tags-view is available by default and an easy way to run simple tag searches

• org-ql

  It's a relatively new package, with new query syntax as the main feature, which is much easier to use and remember than builtin Org query syntax: comparison.

  I mainly use these commands:

  • helm-org-ql for incremental search in the current buffer
  • org-ql-search interactively prompts you for the search target, sort and grouping

Another notable mention is org-rifle, which is an entry based search, presenting headings along with the matched content in Helm buffer. However as the author mentioned it might be obsoleted by org-ql soon.

Here are some typical workflows with my org-mode:

• tags for friends

  I see an interesting article or think of something which would be good to share with a friend, but at the moment it's not quite a good time to send it. I can just capture it and attach a tag (e.g. ann or jeremy). That way next time we chat I can just look up things under their tag and send them.

  It works the other way around as well: imagine they sent me a link or asked me to do something, but I can't do it immediately. I have a special script that converts chat messages into todo items and automatically attaches the corresponding tag. I write more about it here.

• assembling blog posts

  Unfortunately, I can't just sit and write comprehensible texts without preparation. Typically I have thoughts on the topic now and then, which I just note down and mark with the tag. When I feel it's time to prepare the post, I can just search by the tag (e.g. tags:search for this post), and refile the items into the file with the post draft.

As I mentioned, I find having to switch to the browser, wait till the website loads and cope with crappy search implementations very distracting and frustrating.

What is more, often you don't even remember whether exactly you were discussing something: on Telegram or Facebook or Reddit? So having a single point of entry to your information and unified search over all of your stuff is extremely helpful.

For instant messaging, I'm using plaintext mirrors, so chat history is always available in plaintext on my computers:

• Telegram messages Didn't bother with org-mode because files would be too huge and there isn't much structure anyway.
• Vkontakte messages Sadly export tool stopped working because of API restrictions, but I'm not using VK much anymore either. At least I got historic messages.

Most services where I can comment, write or leave annotation, I'm mirroring as org-mode. I write about it in detail here: part I, part II.

That gives me source data for a search engine over anything I've ever:

• tweeted
• bookmarked on Pinboard
• highlighted in Instapaper or Kobo
• saved or upvoted on Reddit
• etc., etc.

All these files are either non-Org or somewhat heavy for structured Org-mode search. In addition, I have many old files from my pre-orgmode era when I was using Gitit or Zim.

To search over them, I'm using Emacs and Ripgrep (you can read why later):

my/search runs ripgrep against my/search-targets variable contains paths to notes, chat logs, Orger outputs etc.

The interesting bit about my/search is --my/one-off-helm-follow-mode call. It's a somewhat horrible hack that automatically enables helm-follow mode so you don't have to press C-c C-f every time you invoke helm.

Finally, to make sure I can invoke search in an instance, I'm using a global keybinding.

Here's a demo gif (5Mb) of using this to search 'greg egan' in my knowledge repository. You can see that as a result, I'm getting my Kobo highlights (kobo.org), my reading list (read.org) and even some video (youtube.org)!

I can't recommend it enough, it's got many things done right, very fast and the code is extremely readable and well tested so it's easy to contribute.

It has its own small query language (at the time org-ql didn't exist).

You can save search queries, which ends up being pretty similar to custom Org-mode agendas. Searches can be displayed as persistent widgets, e.g. I find convenient to have a phone screen dedicated to 'Buy' search (t.buy query) or 'Do at work' search (t.@work query).

As I described above, I keep few saved search queries for some friends so I can recall what I wanted to discuss with them.

Docsearch is a not very well-known tool (e.g. zero search results on Reddit or Twitter), but I don't know any alternatives for it.

It's a fulltext indexing and search app for plaintext files, but apparently it even supports EPUBs and PDFs. Here's how matches list looks. Screenshots on Google Play give a pretty good idea what the app does.

I find it convenient for quick search over things that are not imported in Orgzly, e.g. .txt chat logs (Telegram, VK) and huge org-mode files I described above.

It's a bit backwards in terms of UI (even though I like that it's compact and functional), but main downside is it's not opensource. I'd be extremely happy to replace this with some open source application, so please let me know if you know one!

On the rare occasions when I need to search in pdfs or books (which I don't sync on my phone) , I just use Recoll Web UI that I'm selfhosting.

I find it convenient to enable 'highlight all' for search within a page.

When I used Chrome, one thing that annoyed me was that it populates search engines automatically, and there is not way to disable it.

There is a nice open source extension that prevents Chrome from doing it.

I've got lots of personal projects, experiments, data processing and backup scripts on my computer. I also tend to create a git repository at a slightest opportunity primarily as a means of code backup/rollback and progress tracking, but often it results in actual projects, so I would need a repository anyway. Naturally, these repositories end up scattered across the whole filesystem, making it tricky to remember where I've put the code or that it even existed in the first place.

It's very convenient to have some sort of code search engine if you're in a similar situation to mine for multiple reasons:

• Doing potentially breaking code changes

  For instance, I want to remove some unused function or refactor something in my package, which is a Python library to access my personal data. It's used in lots of scripts or dashboards that run in Cron every day.

  I could just go for it, remove the function and hope nothing fails, but if it does then I'd have to deal with fixing it again. It's frustrating and I'd rather search for function usages in all of my code and make sure it's actually safe to remove.

• Reusing code snippets and tricks

  When you're getting familiar to some new library or framework, you often end up googling how to solve problems twice. Sometimes you remember solving the problem you've already had, but don't quite recall where.

  For instance for me, such library is Sqlalchemy. It's very convenient for handling databases, but I only need it infrequently, so can never remember how to work with it. Reading documentation all over again is not very helpful because I've got very few usecases and queries that are specific to my purposes.

  If I can search for sqlalchemy in my code, it shows every repository where I used it so I can quickly copy bit of code I'm interested at.

• Forgotten code

  It happens that I remember writing code for some purpose, but don't quite recall where I put it. Even if you keep all your repos in the same location, you might forget how you named it.

  Full text search, however, allows to find it if you remember some comments or class/function names.

• Help and documentation

  However good is library's documentation, sometimes it just isn't covering your typical needs. If you're a power user, docs are almost never enough and you end up reading the code to bend the library into doing what you want.

  For me such libraries are Org mode or Hakyll, so I often had to search in their code on Github. Searching on Github however is quite awkward. It's slow, it's not incremental and lacks navigation.

  If I have a local clone of the repository on my disk, I can search over it in an instant (without having it opened in the first place) and use familiar tools (e.g. IDE) for navigation.

At the time, I was just using recursive grep and then opening some of results in vim to refine. That's a pretty pathetic workflow.

My ideal code search tool would:

• run against code on my filesystem

  Just any source files, so it wouldn't have to fetch repositories from Github and keep them somewhere separately.

• realtime indexing

  Ideally, inotify-based, but any means of refreshing search index without having to commit first would be nice.

• semantic search in definitions/variables etc with fallback to simple search if the language isn't supported

So, I wanted some code search and indexing tool that could watch over all the source file on my filesystem and let me search through them.

It sounds as a fairly straightforward wish, but to my surprise, none of existing projects I found and tried do the job:

• Sourcegraph

  Lets you index Github/Bitbucket/Gitlab repos etc, but the process for adding local repositories is extremely tedious. Also apparently, it clones repositories first so it's not exactly realtime indexing.

  Overall, I feel that it only makes sense for companies that use few monorepos.

• OpenGrok: setup looks extremely heavy, doesn't support realtime search in arbitrary paths
• Hound: doesn't support recursive repository discovery.
• zoekt: manual is pretty confusing and also looks tailored for huge standalone repos
• Livegrep: tailored to huge monorepos (see HN discussion)

As you can see, none of these are convenient for searching in personal code.

Disappointed, I figured that least I could do is at somehow improve my workflow with grep.

So, what are the problems with using grep?

• running it against all of code results in false positives. node_modules, minified javascript, etc., you name it
  • you probably want to at least ignore anything that's ignored by .gitignore
• getting bunch of output lines in terminal is not interactive
  • you have to repeat the command to refine the results
  • you can't quickly navigate to the result, check it and go back
• running it recursively against your filesystem root is ridiculously slow, even if you use an SSD
  • you probably want to restrict your search to directories that look like a project (e.g. repositories), and again, exclude files ignored by version control

As it turns out, ripgrep is the tool!

• respects .gitignore files, so by maintaining .gitignore properly (e.g. adding node_modules/venv etc) you can make sure you only get meaningful matches when searching for code.
• respects .ignore files. Sometimes code has to be under version control, but you don't want it to show up in search (e.g. could happen if you have vendorized code or minified javascript or static html files). In that case you can use .ignore files with the same syntax to exclude certain patterns from ripgrep's reach without messing with .gitignore.
• it's very fast, both by benchmarks and subjective experiments. You can read more comprehensive benchmarks here.

If you just use ripgrep instead of grep, code search becomes magnitude more pleasant, but it's still not interactive. Long story short, we can use helm in Emacs to achieve interactivity and incremental search.

The only thing that's left is restricting the search to git repositories only. Ripgrep relies on regexes, so we can't do something like Xpath queries and tell it to only search in directories, that contain .git directory. I ended up using a two step approach:

• first, my/code-targets returns all git repositories it can reach from my/git-repos-search-root.

  I'm using fd to go through the disk and collect all candidate git repositories.

  Even though fd is already ridiculously fast, this step still takes some time, so I'm caching the repositories. Cache is refreshed in the background every five minutes so we don't have to crawl the filesystem every time. That saves me few seconds on every search.

• then, my/search-code keybindings invokes ripgrep against all my directories with code, defined in my/code-targets function.

So, literally running grep against my code turned out to be a pretty good solution. I've got about 350 repositories and it works in a blink. Note, however, that I'm using SSD.

Ripgrep searches in real files on my disk, so any changes are reflected immediately, which removes the need for indexing (apart from performance concerns if you've got too many files to search in). It would still be nice to avoid unnecessary disk operations, and of course, semantic search would be great, and that is definitely going to require some sort of indexer.

I've got a global keybinding to invoke Emacs with a prompt to search in code, so I can do in in a blink.

Here's a gif (3.5 Mb) showing it in action: say, I am working on testing a browser extension, and need to interact with in via hotkeys. I remember using pyautogui for automating Grasp tests, but I forgot which function I actually need to use. Searching for 'pyautogui' brings me all the repositories where I'm using it and lets me quickly find out the command I need without having to read the documentation all over again.

It might be convenient to always have the daemon running, for that I'm using a systemd unit in ~/.config/systemd/user/emacs-daemon.service.

[Unit]
Description=Emacs daemon

[Service]
Type=forking
# running via bash -l makes it pick up .profile, which sets up PATH etc
ExecStart=/bin/bash -l -c '/usr/bin/emacs --daemon'
ExecStop=/usr/bin/emacsclient --eval "(kill-emacs)"
Environment=SSH_AUTH_SOCK=%t/keyring/ssh
Restart=always

[Install]
WantedBy=default.target

It's nothing unusual perhaps apart from using bash -l so Emacs picks up your .profile file. Without it, you almost certainly will encounter issues with missing binaries because they would not be in your PATH.