Commit Graph

1206 Commits

Author SHA1 Message Date
Qi Xiao
b901b49f12 cmd/elvmdfmt: Split Markdown formatting into own command, supporting -w and -d.
Also change the behavior of the rendering command (elvmd) to still output HTML
when given -trace.
2022-11-04 16:37:47 +00:00
Qi Xiao
ef5acc9c4b pkg/md/fmt.go: Avoid trailing spaces in blank lines. 2022-11-04 14:23:38 +00:00
Qi Xiao
9c1c0c923e pkg/md: Fix more bugs found by fuzzing. 2022-11-04 12:51:54 +00:00
Qi Xiao
d30cf3e9ab pkg/md/fmt_test.go: Remove test for the idempotency of FmtCodec.
It is redundant for the existing check that the HTML output of formatted
markdown remains the same.
2022-11-03 23:53:26 +00:00
Qi Xiao
6fabd0f6a8 pkg/md: Fix numerous corner cases found by fuzzing. 2022-11-03 23:46:13 +00:00
Qi Xiao
f278185904 pkg/md/fmt.go: Fix how new stanzas are introduced. 2022-11-03 03:13:06 +00:00
Qi Xiao
4996be7221 pkg/md: Fix nil pointer access in processEmphasis. 2022-11-03 02:34:39 +00:00
Qi Xiao
57cde859b6 pkg/md: Don't support nested emphasis or strong emphasis in FmtCodec.
Correctly formatting nested emphasis or strong emphasis is just too hard.

Also don't support consecutive emphasis or strong emphasis.
2022-11-03 02:27:25 +00:00
Qi Xiao
6cb2a1f50d cmd/elvmd: Consolidate mdfmt and mdtrace into elvmd.
Also export pkg/md.HTMLCodec, and make it the default in elvmd.
2022-11-02 13:22:44 +00:00
Qi Xiao
1320f16b5f pkg/md: Fix more bugs discovered by fuzzing.
- When parsing starting markers, consider the rest of the line to be in a new
  paragraph after one marker is parsed.

- FmtCodec: Escape info string in fenced code blocks.

- FmtCodec: Use "_" as emphasis marker also after end of emphasis or strong emphasis.
2022-11-02 13:13:33 +00:00
Qi Xiao
d1f09b47de pkg/md: Fix and simplify insertion of hard line breaks. 2022-11-02 11:55:29 +00:00
Qi Xiao
96e4f75e43 Add a new "mdtrace" for debugging the md package with the trace codec. 2022-11-02 02:20:28 +00:00
Qi Xiao
b61dc859ad pkg/md: Add fuzz tests, and fix some bugs found by them.
- Avoid reformatting "- - *" as "- - -"; the former is a 3-level list with no
  content, the latter is a thematic break.

- Escape "~~~" at the start of line.

- When handling escapes inside ATX headings, use "i == 0" as a condition for
  start of line, since we are actually not at the literal start of line after
  writing the heading starter.

- Fix the definition of Unicode punctuations to be consistent with the spec.
2022-11-02 02:03:19 +00:00
Qi Xiao
dddc5d273e pkg/md: Handle code and HTML blocks with missing closers. 2022-11-02 01:05:56 +00:00
Qi Xiao
9b901b2508 pkg/md: Test that FmtCodec is idempotent. 2022-11-01 23:59:47 +00:00
Qi Xiao
eb4a84540a pkg/md/fmt.go: Improve escaping.
- Only escape "!" when followed by a link.

- Only escape "#" when it is either at the start of line in a paragraph and
  introduce an ATX heading, or is at the end of the line in an ATX heading and
  can be parsed as the optional ATX heading closer.

- Escape "-" and "+" if and only if they can introduce a bullet list.

- Escape "." and ")" if and only if they come after a number and can introduce
  an ordered list.

- Escape ">" when it can introduce blockquote.

- Escape spaces and tabs at start and end of line.
2022-11-01 23:46:47 +00:00
Qi Xiao
947bcc0eb6 pkg/md: Separate block and inline ops. 2022-11-01 20:43:51 +00:00
Qi Xiao
289f3dfb46 pkg/md/fmt_test.go: Enable most supplemental test cases. 2022-10-30 23:33:00 +00:00
Qi Xiao
e16ab05078 pkg/md/fmt.go: Format link destination and title. 2022-10-30 23:28:14 +00:00
Qi Xiao
db56c9b318 pkg/md: Alternate between * and _ for consecutive OpEmphasisStart's. 2022-10-30 18:41:09 +00:00
Qi Xiao
f3397a746f pkg/md: Alternate list marker's punctuation when it follows immediately after another. 2022-10-30 18:02:45 +00:00
Qi Xiao
806286cfa1 pkg/md: Comment and simplify the fields of FmtCodec. 2022-10-30 16:53:15 +00:00
Qi Xiao
614cdca1f9 Merge branch 'master' into builtin-elvdoc 2022-10-28 19:22:08 +01:00
Qi Xiao
a08ab691df pkg/md: Fix newlines around HTML blocks in FmtCodec.
Also:

- Show a trace of Op's when a test case fails.

- Don't loosify lists when comparing the HTML output of original and formatted.
2022-10-28 19:20:39 +01:00
Qi Xiao
89063ab095 Manually cherry-pick codespell changes from #1613. 2022-10-23 23:57:01 +01:00
Qi Xiao
a0be886c87 pkg/md: Clarify package godoc. 2022-10-23 23:34:44 +01:00
Qi Xiao
f584434735 Add md.FmtCodec and a mdfmt command.
A lot of tests are skipped for now.
2022-10-15 12:21:11 +01:00
Qi Xiao
ffc05aba8d pkg/md: Generate String method for OpType.
Also standardize the name of the file generated by stringer as "zstring.go" and
ignore all existing ones in .codecov.yml.
2022-10-15 11:39:27 +01:00
Qi Xiao
9d75552d0b pkg/md: Split test utilities from md_test.go into testutils_test.go. 2022-10-15 11:34:48 +01:00
Qi Xiao
b846cfa60b pkg/md/md_test.go: Add comment on the escape* functions. 2022-10-07 13:00:23 +01:00
Qi Xiao
871c0833b9 Add LICENSE for CommonMark spec.
Already clarify the wording of the license description in general.
2022-10-07 10:46:30 +01:00
Qi Xiao
4c27a9e551 Merge branch 'master' into builtin-elvdoc 2022-10-07 10:36:17 +01:00
Qi Xiao
dad7c6f035 README: Clarify license. 2022-10-05 01:29:52 +01:00
Qi Xiao
c928d16527 pkg/md: Add supplemental tests to cover all reachable code.
Also make the handling of raw HTML in image alt consistent with cmark.
2022-10-05 01:12:29 +01:00
Qi Xiao
6a7b189a51 pkg/md/md_test: Reorder supplemental tests to match section order in spec. 2022-10-05 00:23:18 +01:00
Qi Xiao
ab9054971d pkg/md: Only support a minimum set of HTML entities.
This avoids a dependency on the html package, which has a huge entity table.

The pkg/md package is not used by Elvish now, but its impact on the binary size
can be measured by exposing md.Render as a builtin function.

- With the dependency on the html package, Elvish's binary size would increase by
  206KiB.

- Without the dependency, the binary size would only increase by 125KiB.
2022-10-05 00:12:36 +01:00
Qi Xiao
a1a9193f93 pkg/md: Update package doc. 2022-10-04 23:28:08 +01:00
Qi Xiao
ca67bbd038 pkg/md: Remove source from HTML block condition 6.
This matches this post-0.30 change of the spec: 053924aa51
2022-10-04 23:21:30 +01:00
Qi Xiao
b7d72cc089 pkg/md: Rearrange code and add comments. 2022-10-04 23:20:54 +01:00
Qi Xiao
165f5ba261 pkg/md: Use a Python-like dedent function to make test cases more readable. 2022-10-04 21:55:52 +01:00
Qi Xiao
e545d88442 pkg/md: Simplify blank lines closing blockquotes. 2022-10-04 21:35:47 +01:00
Qi Xiao
30b2ec5540 pkg/md: Refactor and add comments.
- Use a Codec interface to specify output format.

  This gives more control over how output is built, and will make it possible to
  build a terminal renderer.

- Refactor block tree logic into its own type.

- Handle the opening and closing of list containers in one place.

- Use lookahead to implement the rule allowing only one blank line to start a
  list item.

- Remove handling for non-\n line endings.

  This is required by the spec, but there is no test for it. It was also not
  handled consistently before; if there is a need, it is easier to just
  normalize the line ending at the very beginning rather than handling it
  everywhere.
2022-10-04 21:09:22 +01:00
Qi Xiao
4e8ac3c12b pkg/md: Handle entities in links and autolinks correctly.
Entities in link destination and title need to be handled as part of the parsing
procedure, instead of after it. The spec doesn't have test cases for this, so
added new supplemental test cases.

Entity support in autolinks also don't have test cases in the spec, so add a
supplemental test case too.
2022-10-03 17:18:06 +01:00
Qi Xiao
3bef3fd55c pkg/md: Rearrange code and add some links to spec. 2022-10-03 06:17:19 +01:00
Qi Xiao
afaf977217 pkg/md: Split inline syntax into a separate file. 2022-10-03 05:58:32 +01:00
Qi Xiao
0b81091545 pkg/md: Support indented code blocks. 2022-10-03 05:52:41 +01:00
Qi Xiao
e091971598 pkg/md: Don't skip with regexp match.
Indented code blocks and link reference definitions are used a lot in other
sections, and they used to be skipped by matching the markdown with regexp
patterns. However, this has some false positives, so list the individual example
IDs instead.
2022-10-03 05:07:37 +01:00
Qi Xiao
3ff0e4bcd9 pkg/eval: Check whether variable name is empty when parsing lvalue.
This fixes #1560.
2022-10-03 04:34:05 +01:00
Kurtis Rader
e896ac3c0e
Improve some compilation errors (#1617)
Quote variable names appearing in some compilation errors to improve
the clarity of those errors.
2022-10-03 04:22:46 +01:00
Qi Xiao
ce87939ade pkg/md: Accept at most 3 leading spaces in thematic break and ATX heading. 2022-10-03 04:07:58 +01:00