Incredible read: https://eieio.games/essays/the-secret-in-one-million-checkboxes/
I failed many attempts at getting Sonnet to write code to display the folder structure of the output of a tree -F command using shortcodes.
After a lot of prompting, I wrote a mini-design doc on how the feature needed to be implemented and used it as context for Sonnet.
I tried several variants of instructions in the design including trying to improve it with the model itself for clarity.
I validated that the model could translate from the tree -F to the html markup directly.
It could.
That in fact is the example html target document in my design doc. Here is that doc:
File Tree Shortcode
This document outlines the requirements for a Hugo shortcode that generates an HTML representation of a file tree structure.
Input
The shortcode takes a multi-line string input representing the output of a tree -F command. For example:
./
├── Makefile
├── __pycache__/
│ ├── clean.cpython-312.pyc
│ ├── ex.cpython-312.pyc
│ ├── ex1.cpython-312.pyc
│ ├── ex2.cpython-312.pyc
│ ├── extract.cpython-312.pyc
│ └── main.cpython-312.pyc
├── ex.py
└── src/
├── __pycache__/
│ ├── clean.cpython-312.pyc
│ └── extract.cpython-312.pyc
├── clean.py
└── extract.py
Output
The shortcode should generate nested HTML lists representing the file tree structure:
- Folders are represented by
<li>elements with a<span class="caret">child containing the folder name. - Files are represented by simple
<li>elements. - The content of each folder is contained in a
<ul class="nested">element. - The root folder (’./’ in the example) should be treated as the topmost level.
Requirements
- Support arbitrary depth of nested folders.
- Use tree command characters (├, │, └, ─ and spaces) from the input to determine the level at which the file or folder should be nested.
- Distinguish between folders (ending with ‘/’) and files.
Assume the input will be well formed.
Usage
[shortcode begin]
./
├── Makefile
├── __pycache__/
│ ├── clean.cpython-312.pyc
│ ├── ex.cpython-312.pyc
│ ├── ex1.cpython-312.pyc
│ ├── ex2.cpython-312.pyc
│ ├── extract.cpython-312.pyc
│ └── main.cpython-312.pyc
├── ex.py
└── src/
├── __pycache__/
│ ├── clean.cpython-312.pyc
│ └── extract.cpython-312.pyc
├── clean.py
└── extract.py
[shortcode end]
Here is example html markup that shows how the above example of the shortcode should render.
<ul>
<li><span class="caret">./</span>
<ul class="nested">
<li>Makefile</li>
<li><span class="caret">__pycache__</span>
<ul class="nested">
<li>clean.cpython-312.pyc</li>
<li>ex.cpython-312.pyc</li>
<li>ex1.cpython-312.pyc</li>
<li>ex2.cpython-312.pyc</li>
<li>extract.cpython-312.pyc</li>
<li>main.cpython-312.pyc</li>
</ul>
</li>
<li>ex.py</li>
<li><span class="caret">src</span>
<ul class="nested">
<li><span class="caret">__pycache__</span>
<ul class="nested">
<li>clean.cpython-312.pyc</li>
<li>extract.cpython-312.pyc</li>
</ul>
</li>
<li>clean.py</li>
<li>extract.py</li>
</ul>
</li>
</ul>
</li>
</ul>
The above tree is depth three since src contains a folder called __pycache__ which contains files.
The shortcode should support file trees of arbitrary depth.
The output should also trim away the markup outputted by the tree command and just show the file names.
If it’s helpful, you can assume a folder will also end with a / in its name.
Here is the output of one such attempt which took several follow to even get it to render. It doesn’t work. I tried for a while and couldn’t get Sonnet to complete this task as described.
{{ $tree := .Inner | replaceRE "^\\./" "" | split "\n" }}
<ul>
{{ range $index, $line := $tree }}
{{ $indent := (len (replaceRE "[^│]" "" $line)) }}
{{ $isDir := (strings.HasSuffix $line "/") }}
{{ $name := (replaceRE "^[│ ]*├── |^[│ ]*└── |^[│ ]*│ |^[│ ]* " "" $line) }}
{{ if eq $indent 0 }}
{{ if $isDir }}
<li><span class="caret">{{ $name }}</span>
<ul class="nested">
{{ else }}
<li>{{ $name }}</li>
{{ end }}
{{ else if $isDir }}
{{ range $i, $v := seq 1 $indent }}
</ul>
</li>
{{ end }}
<li><span class="caret">{{ $name }}</span>
<ul class="nested">
{{ else }}
{{ range $i, $v := seq 1 (sub $indent 1) }}
</ul>
</li>
{{ end }}
<li>{{ $name }}</li>
{{ end }}
{{ end }}
{{ range $i, $v := seq 1 (len $tree) }}
</ul>
</li>
{{ end }}
</ul>
I also prompted Sonnet on whether it could identify any issues with the described approach, like if I somehow am suggesting it does something impossible (I’m not seeing it). It said
I believe the approach is generally sound and should work well.
Finally, I tried adding some of the Hugo docs to Cursor but this didn’t seem to be improving things much. Side note: I need to spend some time experimenting with Zed.
My takeaway is Sonnet wasn’t trained on enough Hugo templates to be good at it.
All the prompt engineering I could throw at it in a few hours didn’t seem to be enough to overcome these challenges.
Neither was trying gpt-4, llama-70b or one of the Gemini models (I forgot which I tried).
The fact that models could not solve this task violated my intuition for the problem. I was able to prompt the model to write Python code to output the desired html and it did so without issue.
def generate_file_tree_html(tree_output):
lines = tree_output.strip().split("\n")
html_output = []
stack = []
for line in lines:
indent_level = len(line) - len(line.lstrip(" │"))
is_dir = line.strip().endswith("/")
name = line.strip().split(" ", 1)[-1].strip("│ ├── └── ")
while stack and stack[-1][0] >= indent_level:
html_output.append("</ul></li>")
stack.pop()
if is_dir:
html_output.append(
f'<li><span class="caret">{name}</span><ul class="nested">'
)
stack.append((indent_level, name))
else:
html_output.append(f"<li>{name}</li>")
while stack:
html_output.append("</ul></li>")
stack.pop()
return "<ul>" + "".join(html_output) + "</ul>"
# Example usage
tree_output = """
./
├── Makefile
├── __pycache__/
│ ├── clean.cpython-312.pyc
│ ├── ex.cpython-312.pyc
│ ├── ex1.cpython-312.pyc
│ ├── ex2.cpython-312.pyc
│ ├── extract.cpython-312.pyc
│ └── main.cpython-312.pyc
├── ex.py
└── src/
├── __pycache__/
│ ├── clean.cpython-312.pyc
│ └── extract.cpython-312.pyc
├── clean.py
└── extract.py
"""
html_output = generate_file_tree_html(tree_output)
print(html_output)
This script outputs the expected html (without whitespace).
Lastly, I prompted Sonnet to translate this working Python code into a Hugo shortcode.
This approach also failed.
The model kept hallucinating functions like while that don’t exist in the Hugo templating language.
At this point, I didn’t have confidence in any of the models to write Hugo templates/shortcodes.
Before I called it, I figured if I were to try one more thing, I would pull together as many examples of Hugo template docs and usage and pass that as context to the model with the same instructions. I manually copied and pasted a ton of the examples from https://gohugo.io/functions then prompted the model similarly to before.
It output the following
{{- $tree := .Inner | strings.TrimRight "\n" | split "\n" -}}
{{- $root := index $tree 0 | strings.TrimRight "/" -}}
<ul>
<li><span class="caret">{{ $root }}</span>
<ul class="nested">
{{- $level := 0 -}}
{{- $stack := slice (dict "level" 0 "ul" "") -}}
{{- range after 1 $tree -}}
{{- $line := . -}}
{{- $name := $line | replaceRE "^[│├└]([─ ]+)" "" | strings.TrimRight "/" -}}
{{- $depth := len (findRE "[│├└]" $line) -}}
{{- $isDir := strings.HasSuffix . "/" -}}
{{- if gt $depth $level -}}
<ul class="nested">
{{- $stack = append $stack (dict "level" $depth "ul" "</ul>") -}}
{{- else if lt $depth $level -}}
{{- range seq (sub $level $depth) -}}
{{- $last := index $stack (sub (len $stack) 1) -}}
{{ $last.ul | safeHTML }}
{{- $stack = slice $stack 0 (sub (len $stack) 1) -}}
{{- end -}}
{{- end -}}
<li>
{{- if $isDir -}}
<span class="caret">{{ $name }}</span>
{{- else -}}
{{ $name }}
{{- end -}}
</li>
{{- $level = $depth -}}
{{- end -}}
{{- range seq (len $stack) -}}
{{- $last := index $stack (sub (len $stack) 1) -}}
{{ $last.ul | safeHTML }}
{{- end -}}
</ul>
</li>
</ul>
which also does not work. Maybe, I’ll get this working at some point.