Making Elixir Recompile When External Files Change

In Elixir, we can execute code at both compile time and runtime. This separation allows us to offload large static computations to compile time, trading a slightly slower compile for the ability to just return values and not run calculations while in runtime. This approach works well when dealing with a static set of data that can be used to define module attributes or functions.

A good example of this is handling files or having a list of values we want to add module attributes for or build our functions off of. Plug.Conn.Status employs this technique to define its functions, simplifying maintenance and reducing repetitive boilerplate compared to manually defining each status function.

Let’s consider a practical example: building a library for icons. We might want to load all icon files from a folder and return them, a naive approach would be to do this during runtime like so:

defmodule MyApp.MyIcons do
  base_path = :my_app
    |> :code.priv_dir()
    |> Path.join("icons")

  @spec fetch(String.t()) :: {:ok, String.t()} | :error
  def fetch(icon_name) do
    base_path
      |> Path.join(icon_name)
      |> File.read
  end
end

while this works, it means we have to read the file in every time the function is called, causing IO to be used for each function call. Instead we would normally want to offload this to compile time so that the IO is only done then, and not while calling the function. To do this we could we could parse the folder and store the results in a module attribute, and simply fetch the items out of the module attribute at runtime like so:

defmodule MyApp.MyIcons do
  base_path = :my_app |> :code.priv_dir() |> Path.join("icons")

  @icon_map base_path |> File.ls!() |> Enum.reduce(%{}, fn icon_file, acc ->
    # Extract icon name: "star.svg" => :star
    icon_name = icon_file |> Path.basename(".svg") |> String.to_atom()

    # Read the SVG file content into a string
    svg = base_path |> Path.join(icon_file) |> File.read!()

    Map.put(acc, icon_name, svg)
  end)

  @spec fetch(String.t()) :: {:ok, String.t()} | :error
  def fetch(icon_name) do
    Map.fetch(@icon_map, icon_name)
  end
end

Another alternative approach which doesn’t even require a simple Map.fetch, would be to dynamically define functions for our icons, like so:

defmodule MyApp.MyIcons do
  # Set the path to the "./priv/icons" directory
  base_path = :my_app |> :code.priv_dir() |> Path.join("icons")

  for icon_file <- File.ls!(base_path) do
    # Extract icon name: "pizza.svg" => :pizza
    icon_name = icon_file |> Path.basename(".svg") |> String.to_atom()

    # Read the SVG file content into a string
    svg = base_path |> Path.join(icon_file) |> File.read!()

    # Use `unquote` to define functions named after the icons
    def unquote(icon_name)() do
      unquote(svg)
    end
  end
end

These methods are powerful because we can define functions directly from icon files or create a lookup map of file contents by file names, allowing us to offload the work of reading files in our compilation as opposed to every function call. Doing this however, introduces a dependency on external files, which can lead to specific issues around when the file changes.

Handling External File Changes

Once we have our functions setup to run off our files, we encounter the first issue: changes to the content these existing file (e.g., modifying an icon) won’t trigger a recompilation of the module. Elixir won’t automatically detect these changes. While running mix compile --force resolves this, it recompiles the entire project, which can be slow. In umbrella apps, using mix deps.compile my_icons_app --force narrows the scope, but it’s still inconvenient and slow for larger applications.

To address this, we can leverage the @external_resource attribute which is a special module attribute Elixir will recognize:

defmodule MyApp.MyIcons do
  app_priv_dir = code.priv_dir(:my_app)
  base_path = Path.join(app_priv_dir, "icons")

  for icon_file <- File.ls!(base_path) do
    @external_resource Path.join(base_path, icon_file)

    icon_name = icon_file |> Path.basename(".svg") |> String.to_atom()
    svg = base_path |> Path.join(icon_file) |> File.read!()

    def unquote(icon_name)() do
      unquote(svg)
    end
  end
end

Adding @external_resource icon_file flags the file so that if any flagged file changes, the module will recompile, reflecting the updated content. One important note here is to remember that paths are relative to the applications mix.exs, meaning our paths would look like ./priv/icons/#{icon_file} if we wanted them to be relative, though in our case we’re using absolute paths which allows us to ignore this all-together. While this is great for watching our changing files, adding or removing files in the directory still won’t trigger a recompilation. When a new file like video.svg is added, calling its corresponding function will result in an error:

** (UndefinedFunctionError) function MyApp.MyIcons.video/1 is undefined or private

This happens because the module has already been compiled and isn’t aware of the new directory structure. Similarly, deleting files won’t remove their functions.

Solving All Issues with Custom Recompilation

Starting in Elixir 1.12, we can use the __mix_recompile__?/0 function to trigger recompilation based on custom criteria. Here’s how we can handle changes in directory structure:

defmodule MyApp.MyIcons do
  base_path = :my_app |> :code.priv_dir() |> Path.join("icons")

  @icon_files_hash base_path |> File.ls!() |> :erlang.md5()

  for icon_file <- File.ls!(base_path) do
    # Known files are watched for changes
    @external_resource icon_file

    icon_name = icon_file |> Path.basename(".svg") |> String.to_atom()
    svg = base_path |> Path.join(icon_file) |> File.read!()

    def unquote(icon_name)() do
      unquote(svg)
    end
  end

  def __mix_recompile__? do
    # On recompile we check a hash of the file directory to be
    # the same as our current compile
    base_path |> File.ls!() |> :erlang.md5() !== @icon_files_hash
  end
end

Here, @icon_files_hash stores the MD5 hash of the file list at compile time. The __mix_recompile__?/0 function recalculates the hash when a recompile is done and compares it to the stored value. If the hashes differ, Elixir recompiles the module.

By using both @external_resource and __mix_recompile__?, we can have Elixir monitor both the file contents and the overall directory structure. Changes to files, as well as file additions or deletions will trigger recompilation which ensures the module reflects the current state of the folder without relying on mix compile --force.

Elixir’s ability to fine-tune recompilation allows us to build efficient and flexible functionality easily without requiring a ton of maintenance. It’s small features like this that make Elixir my favorite language to work with!!