Sublime Blog http://www.sublimetext.com/blog Sublime Text News Tue, 2 May 2019 01:00:00 +0000 en hourly 1 Use mmap With Care https://www.sublimetext.com/blog/articles/use-mmap-with-care Tue, 2 May 2019 01:00:00 +0000 Benjamin Schaaf When we implemented the git portion of Sublime Merge, we chose to use mmap for reading git object files. This turned out to be considerably more difficult than we had first thought. Using mmap in desktop applications has some serious caveats, and here's why:

You can follow along with the example project here.

Say you were reading some binary format, data you need throughout some application. You take the easiest approach and use a simple read to get the full contents of the file. You release your software and someone comes along and says: "I need to parse this 3GB file, but I only have 2GB of memory. Could you make it so that you're not using so much memory?". Of course you want to help them, so you do some searching and come to the conclusion that memory mapping is the perfect solution to this problem.

Memory mapped files work by mapping the full file into a virtual address space and then using page faults to determine which chunks to load into physical memory. In essence it allows you to access the file as if you had read the whole thing into memory, without actually doing so. Crucially it requires only a small change to the codebase:

And with that, you've fixed the memory usage issue. You've fixed the bug, people are happy and all is well until you get another support ticket. Your program is crashing with a SIGBUS.

Caveat 1: SIGBUS

SIGBUS (bus error) is a signal that happens when you try to access memory that has not been physically mapped. This is different to a SIGSEGV (segmentation fault) in that a segfault happens when an address is invalid, while a bus error means the address is valid but we failed to read/write.

As it turns out, the ticket comes from someone using a networked drive. Their network happened to disconnect while your memory mapped file was open, and since the file no longer existed the OS couldn't load it into ram for you and gave you a SIGBUS instead.

Because the OS is loading the file on demand, you now have this wonderful issue where any arbitrary read from an address into the memory mapped file can and will fail at some point.

Luckily on POSIX systems we have signal handlers, and SIGBUS is a signal we can handle. All you need to do is register a signal handler for SIGBUS when the program starts and jump back to our code to handle failures there.

Sadly our code actually has some edge cases we should consider:

Signal handlers are global, but signals themselves are per-thread. So you need to make sure you're not messing with any other threads by making all our data thread local. Let's also add some robustness by making sure we've called setjmp before longjmp.

Using setjmp and longjmping from a signal handler is actually unsafe. It seems to cause undefined behaviour, especially on MacOS. Instead we must use sigsetjmp and siglongjmp. Since we're jumping out of a signal handler, we need that signal handler to not block any future signals, so we must also pass SA_NODEFER to sigaction.

This is starting to get quite complicated, especially if you were to have multiple places where a SIGBUS could happen. Let's factor things out into functions to make the logic a little cleaner.

There, now you just need to remember to always call install_signal_handlers for every application, and wrap all file accesses with safe_mmap_try. Annoying, but manageable. So now you've covered POSIX systems, but what about Windows?

Caveat 2: Windows

Windows doesn't have mmap, but it does have MapViewOfFile. Both of these implement memory mapped files, but there's one important difference: Windows keeps a lock on the file, not allowing it to be deleted. Even with the Windows flag FILE_SHARE_DELETE deletion does not work. This is an issue when we expect another application to delete files from under us, such as git garbage collection.

One way around this with the windows API is to essentially disable the system file cache entirely, which just makes everything absurdly slow. The way Sublime Merge handles this is by releasing the memory mapped file on idle. Its not a pretty solution, but it works.

Windows also does not have a SIGBUS signal, but you can trivially use structured exception handling in safe_mmap_try instead:

Now all is well, your application functions on Windows. But then you decide that you would like some crash reporting, to make it easier to identify issues in the future. So you add Google Breakpad, but unbeknownst to you you've just broken Linux and MacOS again…

Caveat 3: 3rd Parties

The problem with using signal handlers is that they're global, across threads and libraries. If you have or have added a library like Breakpad that uses signals internally you're going to break your previously safe memory mapping.

Breakpad registers signal handlers at initialization time on Linux, including one for SIGBUS. These signal handlers override each other, so installation order is important. There is not a nice solution to these types of situations: You can't simply set and reset the signal handler in safe_mmap_try as that would break multithreaded applications. At Sublime HQ our solution was to turn an unhandled SIGBUS in our signal handler into a SIGSEGV. Not particularly elegant but it's a reasonable compromise.

On MacOS things get a little more complicated. XNU, the MacOS kernel, is based on Mach, one of the earliest microkernels. Instead of signals, Mach has an asynchronous, message based exception handling mechanism. For compatibility reasons signals are also supported, with Mach exceptions taking priority. If a library such as Breakpad registers for Mach exception messages, and handles those, it will prevent signals from being fired. This is of course at odds with our signal handling. The only workaround we've found so far involves patching Breakpad to not handle SIGBUS.

3rd party libraries are a problem because signals are global state accessible from everywhere. The only available solutions to this are unsatisfying workarounds.

Caveat 4: 32bit Support

Memory mapping may not use physical memory, but it does require virtual address space. On 32bit platforms your address space is ~4GB. So while your application may not use 4GB of memory, it will run out of address space if you try to memory map a too large file. This has the same result as being out of memory.

Sadly this doesn't have a workaround like the other issues, it is a hard limitation of how memory mapping works. You can now either rewrite the codebase to not memory map the whole file, live with crashes on 32bit systems or not support 32bit.

With Sublime Merge and Sublime Text 3.2 we took the "no 32bit support" route. Sublime Merge does not have a 32bit build available and Sublime Text disables git integration on 32bit versions.

An Alternative

I mentioned before that you can rewrite your code to not use memory mapping. Instead of passing around a long lived pointer into a memory mapped file all around the codebase, you can use functions such as pread to copy only the portions of the file that you require into memory. This is less elegant initially than using mmap, but it avoids all the problems you're otherwise going to have.

Through some quick benchmarks for the way Sublime Merge reads git object files, pread was around ⅔ as fast as mmap on linux. In hindsight it's difficult to justify using mmap over pread, but now the beast has been tamed and there's little reason to change any more.

If you think we've missed something or made a mistake, please leave us a message on the forums. We hope this helps some of you in your future endeavours. If you haven't already, check out our git client Sublime Merge.

]]>
Sublime Text 3.2 https://www.sublimetext.com/blog/articles/sublime-text-3-point-2 Wed, 13 Mar 2019 14:00:00 +1000 Will Bond img.screenshot-32 { width: 600px; margin: 1em 0 0; display: block; filter: drop-shadow(3px 3px 5px #9998); } p.caption-32 { margin-top: 0.5em; color: #999; font-style: italic; }

The past year has been busy here at Sublime HQ. We've grown our engineering team, shipped Sublime Text 3.1 and released our second product, Sublime Merge. If you use Git and like the ethos of Sublime Text, we think you'll love it.

Today we're pushing out Sublime Text 3.2, which builds off of a bunch of work we've done in Sublime Merge. This includes: first-class Git integration, incremental diffing, new theme functionality and block caret support. Beyond that is a slew of other enhancements, stability improvements and performance gains. See the full changelog below for details.

Sublime Text 3.2 is available from the Download page.

Git status badges in the side bar and incremental diff markers in the gutter

Changelog

Git Integration New!

  • Files and folders in the sidebar will now display badges to indicate Git status
  • Ignored files and folders are visually de-emphasized
  • The current Git branch and number of modifications is displayed in the status bar
  • Commands have been added to open a repository, see file or folder history, or blame a file in Sublime Merge
  • Themes may customize the display of sidebar badges and status bar information
  • The setting show_git_status allows disabling Git integration
  • All file reads are done through a custom, high-performance Git library written for Sublime Merge
  • Read the documentation

Incremental Diff New!

  • All changes to a document are now represented by dedicated markers in the gutter
  • Diff markers show added, modified and deleted lines
  • The setting mini_diff controls incremental diff behavior
  • In coordination with the new Git functionality, diffs can be calculated against HEAD or the index
  • The git_diff_target setting controls base document source
  • API methods View.set_reference_document() and View.reset_reference_document() allow controlling the diff
  • The following diff-related commands were added:
    • Next Modification
    • Previous Modification
    • Revert Modification
  • Full inline diffs of each change can be displayed via the right-click context menu, or keyboard shortcuts
  • Inline diff presentation can be changed by customizing a color scheme
  • Read the documentation

Editor Control

  • Added block_caret setting
  • Improve positioning and sizing of gutter icons in some situations
  • Fixed draw_minimap_border setting not working
  • Linux: Improved input method (IM) support - fcitx, ibus, etc
  • Linux: Fixed a crash when using GTK_IM_MODULE=xim
  • Linux: Tweaked behavior of up/down when on the first and last lines of a file to better match platform conventions
  • Windows: Improved IME support

Themes/UI

  • Enhanced the .sublime-theme format:
    • Added variables support and associated revised JSON format with variables key
    • Added extends keyword to have one theme derive from another
    • Colors may be specified via CSS syntax
  • Improved performance with large numbers of rules in a .sublime-theme
  • Linux: Moved to GTK3
  • Linux: Various high DPI fixes
  • Mac: Added Mojave support
  • Mac: Add full support for macOS native tabs
  • Mac: Ensure context menus are shown without scrolling
  • Mac: Error message dialogs can now be closed with the escape key
  • Mac: Improved window placement
  • Mac: Improved resize performance
  • Windows: Fixed minimized and maximized state not restoring
  • Windows: Fixed a bug where auto complete entries would contain an ellipsis when not required

Text Rendering

  • Support for Unicode 11.0
  • Improved rendering of combining characters
  • Fixed a caret positioning bug when non-trivial graphemes are present
  • Fixed some cases of incorrect glyph positions on Windows and Mac
  • Linux: Color glyphs are now drawn properly on light backgrounds
  • Windows: Fixed a rendering issue with certain combining characters
  • Windows: Fixed some fonts having an incorrect baseline

Color Schemes

  • Added block_caret key to use in conjunction with block carets
  • caret values now respect alpha as expected, rather than pre-blending against the background color
  • Added the foreground_adjust property to rules with a background. Accepts CSS color mod adjusters to manipulate the saturation, lightness or opacity of the foreground color.

Syntax Highlighting

  • Many syntax highlighting improvements, including significant improvements to:
  • Fixed a crash that could occur when nesting embed patterns in .sublime-syntax files
  • Syntax Tests: Allow syntax test files to have a UTF-8 BOM

Files and Folders

  • Improve performance of file watching for ignored paths on Windows and Mac
  • Windows: Fixed Open File treating paths as case-sensitive
  • Windows: Properly unlock directories after contained files are closed

API

  • Added View.set_reference_document() and View.reset_reference_document() to control diff generation
  • Phantoms are now drawn correctly in conjunction with draw_centered
  • Various minor improvements related to plugin module loading and unloading
  • Added support for hwb() colors to minihtml
  • Added a custom min-contrast() adjuster for the CSS color mod function in minihtml
  • Mac: Fixed a plugin_host crash when running a process that itself crashes

Miscellaneous

  • Fixed a Goto Symbol in Project performance regression
  • F21..F24 keys can now be bound
  • Assorted minor fixes and stability improvements
  • Linux: Improved behavior of --wait command line argument when Sublime Text isn't currently running
]]>
Sublime Merge Build 1107 https://www.sublimetext.com/blog/articles/sublime-merge-build-1107 Tue, 12 Mar 2019 20:00:00 +1000 Jon Skinner Sublime Merge, our Git Client and Merge Tool, is out now with improvements to the Merge Tool, Tree viewing, and Clone dialog. See the full post for more details.

Sublime Text 3.2 will be coming very soon now, featuring Git integration and more.

]]>
Sublime Merge Build 1092 https://www.sublimetext.com/blog/articles/sublime-merge-build-1092 Wed, 29 Nov 2018 20:00:00 +1000 Jon Skinner Sublime HQ's Git Client, Sublime Merge, is out now with Interactive Rebase, a new Contents View, and Word Wrap - see the full post for more details.

]]>
Sublime Merge Documentation https://www.sublimetext.com/blog/articles/sublime-merge-documentation Mon, 05 Nov 2018 02:00:00 -0500 Will Bond While we continue to add more features and polish to the default Sublime Merge experience, we know that developers love the ability to tweak their tools. Sublime Merge is built on the same foundation as Sublime Text, so you can tweak key bindings, menus, command palette entries, and even the look and feel of the UI.

To assist, we’ve just rolled out a number of pages of documentation for users who wish to customize Sublime Merge to look and function a little differently. In addition, we’ve added a new documentation section for the dev builds of Sublime Text. It covers all of the features we added during the development of Sublime Merge.

]]>
Sublime Merge Build 1070 https://www.sublimetext.com/blog/articles/sublime-merge-build-1070 Wed, 10 Oct 2018 19:00:00 +1000 Jon Skinner Sublime Merge Build 1070 is out now with Commit Folding - see the full post for more details.

]]>
Sublime Merge Roadmap https://www.sublimetext.com/blog/articles/sublime-merge-roadmap Tue, 2 Oct 2018 10:00:00 +1000 Jon Skinner Just a quick post about what we're going to be adding to Sublime Merge. This isn't a comprehensive list, but it should give an indication of what we're focusing on right now.

  • Branch Visualization. Sublime Merge itself handles large repositories well, but there's more the UI could do to help users with a lot of branches on the go. We've got some nice things cooking for this that will be coming in the near future. Stay tuned!
  • Submodule Management. Sublime Merge understands submodules, but doesn't provide any UI for managing them. We'll be changing that soon.
  • Tabs. Tabs have been frequently requested by users working on several repositories at once. The Switch Repository functionality (under the File menu) helps with this, but tabs have their own benefits, so they're on their way.
  • Git Flow Integration. Git Flow is a branching strategy with an accompanying set of command line tools. We've had a lot of requests for UI support here, so it'll be making it in too.
  • Interactive Rebase. Sublime Merge currently supports editing commit messages and squashing commits (available from the commit context menu). We'll be adding extra commit wrangling functionality, including reordering commits, moving commits between branches, and amending arbitrary commits.
  • Plugins. Sublime Text style plugins are on our radar. This isn't at the top of the list, but they're coming.

We'll also be getting a new Sublime Text Dev Build out this week, rolling in all the updates from Sublime Merge, including the updated theming system and Mojave support.

]]>
Sublime Merge - Git, Done Sublime https://www.sublimetext.com/blog/articles/sublime-merge Mon, 20 Sep 2018 10:00:00 +0000 Jon Skinner img.screenshot-sm { width: 600px; margin: 1em 0 0; display: block; filter: drop-shadow(3px 3px 5px #9998); } p.caption-sm { margin-top: 0.5em; color: #999; font-style: italic; }

(You can download Sublime Merge from https://www.sublimemerge.com/download)

There's a company that makes photography accessories, called Really Right Stuff. They make lovely equipment, but what I really like is the name. It embodies the idea of building something that goes beyond the minimum: making it as good as it can be, paying attention to the details, and getting it really right.

When it comes to software, getting it really right goes beyond functionality. The feel, aesthetics, and performance all have to be there.

There's a real pleasure using software that gets it really right, as a lot of the time, it doesn't. We're all too familiar with clunky layouts, unresponsive buttons, choppy scrolling, tedious splash screens, and flickering on every interaction.

After typing git add -p in the terminal one too many times, I thought to myself: we've got some pretty great tech in Sublime Text. What if we used it to build a Git client? Could we make it fast? Could we make it buttery smooth, without flickering or blocking? Could we make something that's really, really right?

Today, I'd like to introduce Sublime Merge. It combines the UI engine of Sublime Text, with a from-scratch implementation of Git*. The result is, to us at least, something pretty special.

Sublime Merge - Commit Dialog

You can download Sublime Merge, and try it for yourself - there's no time limit, no accounts, no metrics, and no tracking. The evaluation version is fully functional, but is restricted to the light theme only. Individual purchases are buy once, use forever, with 3 years of updates included in the purchase. Business licenses are available on a subscription basis. Sublime Merge runs on Windows, Mac and Linux.

It's still early days for Sublime Merge - it has only been used by us and our small team of beta testers so far. We'd love to hear what you think. We'll be on the Forums listening to any feedback - let us know how you get on with it!


* We have a custom implementation of Git for reading repositories, which drives a lot of our high performance functionality. However we defer to Git itself for operations that mutate the repository (Staging, Committing, Checking out branches, etc).

]]>
Sublime Text 3.1 Released https://www.sublimetext.com/blog/articles/sublime-text-3-point-1 Mon, 7 May 2018 10:00:00 +0000 Will Bond and Jon Skinner Since version 3.0 we've been shipping frequent dev builds, refining Sublime Text for a more polished experience all-around. Those dev builds are now rolling out to everyone as Sublime Text 3.1. The full changelog for 3.1 includes every detail, but the following are some we'd like to highlight:

UI

  • Windows 10/8.1 per-monitor DPI settings for handling mixed monitor configurations and on-the-fly DPI switching
  • Lots of improvements for high DPI support on all platforms, including 8k monitor and @3x texture support

Text Rendering

  • Ligature support, with controls for customizing OpenType features and stylistic sets

    An example of ligatures with Fira Code

  • Improved text rendering on Windows, now respecting ClearType tuning
  • Better handling of emojis in the editor control  🎉

Color Schemes

  • A new color scheme format based on JSON, with easier customization and new features like hashed syntax highlighting

    Hashed syntax highlighting with the Celeste color scheme

  • Improved selection rendering with expanded customization via color schemes

Syntax Definitions

  • Syntax definitions for Git file formats to allow Sublime Text to be a better core.editor
  • Highlighting of fenced code blocks in Markdown

    Python highlighted in a fenced code block

  • Lots of TC39-track features have been added to the JavaScript syntax

User Interaction

  • Goto References to augment Goto Definition

    Goto References shown in the symbol popup

  • An improved Command Palette that can accept arbitrary user input and complex interactions

File System

  • Expanded options for filtering files and folders from the sidebar
  • Improved handling of filesystem notifications and symlinks

Performance

  • Significantly improved memory usage - up to 30% in some cases
  • Lots of little performance improvements and bug fixes

More!

Many of these changes were developed in response to the community's feedback, and in some cases with your help! As may be evident, this dev cycle spent a good amount of time polishing some foundational elements of the editor and making it more robust for use on different configurations and for different purposes.

We've got some exciting things planned for the future, which will further build upon the foundation we've laid with 3.0 and 3.1! If you want to preview the latest changes and help provide feedback during the next dev cycle, install the latest dev build and you'll get notifications as we release new updates. ]]> The Adaptive Theme https://www.sublimetext.com/blog/articles/the-adaptive-theme Fri, 06 Oct 2017 06:00:00 +0000 Will Bond With the release of Sublime Text 3.0, we refreshed the visual design for the application, icon, and website. The new icon ties into the colors and shapes of the old, but is more abstract and fits well with other modern applications. The updated Default theme is still distinctly Sublime Text, but has full high DPI support, and works well with both dark and light color schemes. The three new color schemes take advantage of the work that has gone into modernizing and enhancing the syntax definitions included with Sublime Text.

Sublime Text 3 Build 3126
(Old) Default theme, IDLE color scheme
Sublime Text 3.0
(New) Default theme, Sixteen color scheme

As well as an overhaul of the default theme, we've also include a new theme, Adaptive, which uses the colors of your color scheme and applies variations of them to the side bar and various panels. The new color scheme setting accent is used to highlight selected options in the find panel, and to highlight modified tabs. Furthermore, on recent versions of macOS the title bar is also styled by the Adaptive theme to follow your selected color scheme.

The result is a theme that acts as if it was custom designed for your color scheme. Check out this sample of Adaptive in action:

To try Adaptive out for yourself, open the Command Palette and type Select Theme. Once you've picked Adaptive, use the Select Color Scheme command to preview the available color schemes.

Probably the best part is that all of the power of the Adaptive theme is available for any theme to use. Additionally, we wrote up full documentation about all of the theme engine features and syntax. Having comprehensive documentation will hopefully make theming accessible to a much wider audience. If you are looking for help in customizing a theme, or creating your own, drop by the forum or ask some questions on the Discord server.

]]>