Skip to content

Commit c10437e

Browse files
anatoliykmetyukanatoliykmetyuk
authored
Merge pull request #6786 from dotty-staging/dotty-knowledge
Dotty internals docs update, dotty knowledge collection system integration
2 parents 32012f7 + 2445bb2 commit c10437e

12 files changed

+481
-6
lines changed

‎README.md

+9
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ Dotty
22
=====
33
[![Build Status](http://dotty-ci.epfl.ch/api/badges/lampepfl/dotty/status.svg)](http://dotty-ci.epfl.ch/lampepfl/dotty)
44
[![Join the chat at https://gitter.im/lampepfl/dotty](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/lampepfl/dotty?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
5+
[![Log Knowledge](https://img.shields.io/badge/log-knowledge-blueviolet.svg)](https://github.com/lampepfl/dotty-knowledge/issues/new/choose)
56

67
* [Homepage](http://dotty.epfl.ch)
78
* [Documentation](http://dotty.epfl.ch/docs)
@@ -22,6 +23,14 @@ How to Contribute
2223
* [Awesome Error Messages](http://scala-lang.org/blog/2016/10/14/dotty-errors.html)
2324
* [Issues](https://github.com/lampepfl/dotty/issues?q=is%3Aissue+is%3Aopen+label%3A%22help+wanted%22)
2425

26+
## Contribute Internals-related Knowledge
27+
If you know anything useful at all about Dotty, feel free to log this knowledge:
28+
29+
- [📜Log the Knowledge](https://img.shields.io/badge/log-knowledge-blueviolet.svg)](https://github.com/lampepfl/dotty-knowledge/issues/new/choose)
30+
- [🎓More about Logging the Knowledge](https://github.com/lampepfl/dotty-knowledge/blob/master/README.md)
31+
32+
In short, no need to make it pretty, particularly human-readable or give it a particular structure. Just dump the knowledge you have and we'll take it from there.
33+
2534
License
2635
=======
2736
Dotty is licensed under the [3-Clause BSD License](https://github.com/lampepfl/dotty/blob/master/LICENSE.md)

‎docs/docs/contributing/contribute-knowledge.md

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# Contribute Internals-related Knowledge
2+
If you know anything useful at all about Dotty, feel free to log this knowledge:
3+
4+
- [📜Log the Knowledge](https://github.com/lampepfl/dotty-knowledge/issues/new)
5+
- [🎓More about Logging the Knowledge](https://github.com/lampepfl/dotty-knowledge/blob/master/README.md)
6+
7+
In short, no need to make it pretty, particularly human-readable or give it a particular structure. Just dump the knowledge you have and we'll take it from there.

‎docs/docs/contributing/debugging.md

+332
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,332 @@
1+
# Debugging Techniques
2+
- [Setting up the playground](#setting-up-the-playground)
3+
- [Show for human readable output](#show-for-human-readable-output)
4+
- [How to disable color](#how-to-disable-color)
5+
- [Reporting as a non-intrusive println](#reporting-as-a-non-intrusive-println)
6+
- [Printing out trees after phases](#printing-out-trees-after-phases)
7+
- [Configuring the printer output](#configuring-the-printer-output)
8+
- [Figuring out an object creation site](#figuring-out-an-object-creation-site)
9+
* [Via ID](#via-id)
10+
* [Via tracer](#via-tracer)
11+
- [Built-in Logging Architecture](#built-in-logging-architecture)
12+
* [Printers](#printers)
13+
* [Tracing](#tracing)
14+
* [Reporter](#reporter)
15+
16+
<small><i><a href='http://ecotrust-canada.github.io/markdown-toc/'>Table of contents generated with markdown-toc</a></i></small>
17+
18+
## Setting up the playground
19+
Consider the `../issues/Playground.scala` (relative to the Dotty directory) file is:
20+
21+
```scala
22+
object Playground {
23+
def main(args: Array[String]) = {
24+
println("Hello World")
25+
}
26+
}
27+
```
28+
29+
Then, you can debug Dotty by compiling this file via `dotc ../issues/Playground.scala` (from the SBT console) and collecting various debug output in process. This section documents techniques you can use to collect the debug info.
30+
31+
[This](https://github.com/lampepfl/dotty/blob/10526a7d0aa8910729b6036ee51942e05b71abf6/compiler/src/dotty/tools/dotc/typer/Typer.scala#L2231) is the entry point to the Typer. The job of the Typer is to take an untyped tree, compute its type and turn it into a typed tree by attaching the type information to that tree. We will use this entry point to practice debugging techniques. E.g.:
32+
33+
```scala
34+
def typed(tree: untpd.Tree, pt: Type, locked: TypeVars)(implicit ctx: Context): Tree =
35+
trace(i"typing $tree", typr, show = true) {
36+
println("Hello Debug!")
37+
/*...*/
38+
```
39+
40+
Then:
41+
42+
```
43+
dotc ../issues/Playground.scala
44+
```
45+
46+
The techniques discussed below can be tried out in place of `println("Hello Debug")` in that location. They are of course applicable thoughout the codebase.
47+
48+
## Show for human readable output
49+
Many objects in the compiler have a `show` method available on them via implicit rich wrapper:
50+
51+
```scala
52+
println(tree.show)
53+
```
54+
55+
This will output every single tree passing through the typer (or wherever else you inject it) in a human readable form. Try calling `show` on anything you want to be human-readable, and chances are it will be possible to do so.
56+
57+
## How to disable color
58+
Note that the `show` command above outputs the code in color. This is achieved by injecting special characters into the strings which terminals interpret as commands to change color of the output. This however may not be what you want, e.g. if you want to zero-in on a particular tree:
59+
60+
```scala
61+
if (tree.show == """println("Hello World")""")
62+
println(s"${tree.show}\n${pt.show}\n${tree.uniqueId}\n===\n")
63+
```
64+
65+
The intention above is to output an extended debug info on a tree that matches a particular human-readable representation. However, because of the color characters, the comparison will fail.
66+
67+
To disable color output from `show`, run `dotc` as follows:
68+
69+
`dotc -color:never ../issues/Playground.scala`
70+
71+
## Reporting as a non-intrusive println
72+
Consider you want to debug the `tree` that goes into `assertPositioned(tree)` in the `typed` method. You can do:
73+
74+
```scala
75+
println(tree.show)
76+
assertPositioned(tree)
77+
```
78+
79+
But you can also do:
80+
81+
```scala
82+
assertPositioned(tree.reporting(t => t.show))
83+
```
84+
85+
`(a: A).reporting(f: A => String): A` is defined on all types. Its job is to print a `String` produced by the argument function and return the value it is called on unchanged.
86+
87+
## Printing out trees after phases
88+
To print out the trees you are compiling after Frontend (scanner, parser, namer, typer) phase:
89+
90+
```
91+
dotc -Xprint:frontend ../issues/Playground.scala
92+
```
93+
94+
To print out the trees after Frontend and CollectSuperCalls phases:
95+
96+
```
97+
dotc -Xprint:frontend,collectSuperCalls ../issues/Playground.scal
98+
```
99+
100+
To print out the trees after all phases:
101+
102+
```
103+
dotc -Xprint:all ../issues/Playground.scala
104+
```
105+
106+
To find out the list of all the phases and their names, check out [this](https://github.com/lampepfl/dotty/blob/10526a7d0aa8910729b6036ee51942e05b71abf6/compiler/src/dotty/tools/dotc/Compiler.scala#L34) line in `Compiler.scala`. Each `Phase` object has `phaseName` defined on it, this is the phase name.
107+
108+
## Configuring the printer output
109+
Printing from the `show` and `-Xprint` is done from the Printers framework (discussed in more details below). The following settings influence the output of the printers:
110+
111+
```scala
112+
val printLines = BooleanSetting("-print-lines" , "Show source code line numbers.") withAbbreviation "--print-lines"
113+
val uniqid = BooleanSetting("-uniqid" , "Uniquely tag all identifiers in debugging output.") withAbbreviation "--unique-id"
114+
val XprintInline = BooleanSetting("-Xprint-inline" , "Show where inlined code comes from")
115+
val XprintTypes = BooleanSetting("-Xprint-types" , "Print tree types (debugging option).")
116+
val Ydebug = BooleanSetting("-Ydebug" , "Increase the quantity of debugging output.")
117+
val YdebugFlags = BooleanSetting("-Ydebug-flags" , "Print all flags of definitions")
118+
val YdebugMissingRefs = BooleanSetting("-Ydebug-missing-refs", "Print a stacktrace when a required symbol is missing")
119+
val YdebugNames = BooleanSetting("-Ydebug-names" , "Show internal representation of names")
120+
val YdebugPos = BooleanSetting("-Ydebug-pos" , "Show full source positions including spans")
121+
val YdebugTrace = BooleanSetting("-Ydebug-trace" , "Trace core operations")
122+
val YdebugTreeWithId = IntSetting ("-Ydebug-tree-with-id", "Print the stack trace when the tree with the given id is created", Int.MinValue)
123+
val YprintDebug = BooleanSetting("-Yprint-debug" , "when printing trees, print some extra information useful for debugging.")
124+
val YprintDebugOwners = BooleanSetting("-Yprint-debug-owners", "when printing trees, print owners of definitions.")
125+
val YprintPos = BooleanSetting("-Yprint-pos" , "show tree positions.")
126+
val YprintPosSyms = BooleanSetting("-Yprint-pos-syms" , "show symbol definitions positions.")
127+
val YprintSyms = BooleanSetting("-Yprint-syms" , "when printing trees print info in symbols instead of corresponding info in trees.")
128+
val YshowTreeIds = BooleanSetting("-Yshow-tree-ids" , "Uniquely tag all tree nodes in debugging output.")
129+
val YshowVarBounds = BooleanSetting("-Yshow-var-bounds" , "Print type variables with their bounds")
130+
val YtestPickler = BooleanSetting("-Ytest-pickler" , "self-test for pickling functionality; should be used with -Ystop-after:pickler")
131+
```
132+
133+
They are defined in [ScalaSettings.scala](https://github.com/lampepfl/dotty/blob/master/compiler/src/dotty/tools/dotc/config/ScalaSettings.scala). E.g. `YprintPos` is defined as:
134+
135+
```scala
136+
val YprintPos: Setting[Boolean] = BooleanSetting("-Yprint-pos", "show tree positions.")
137+
```
138+
139+
And is to be used as:
140+
141+
```scala
142+
dotc -Yprint-pos ../issues/Playground.scala
143+
```
144+
145+
If used, all the trees output with `show` or via `-Xprint:frontend` will also have positions attached to them, e.g.:
146+
147+
```scala
148+
package <empty>@<Playground.scala:1> {
149+
module object Playground {
150+
def main(
151+
args:
152+
Array@<Playground.scala:2>[String@<Playground.scala:2>]@<
153+
Playground.scala:2
154+
>
155+
@<Playground.scala:2>
156+
) =
157+
{
158+
println@<Playground.scala:3>("Hello World"@<Playground.scala:3>)@<
159+
Playground.scala:3
160+
>
161+
}@<Playground.scala:2>
162+
@<Playground.scala:2>
163+
}@<Playground.scala:1>
164+
}@<Playground.scala:1>
165+
<empty>@<Playground.scala:1>
166+
```
167+
168+
## Figuring out an object creation site
169+
### Via ID
170+
Every [Positioned](https://github.com/lampepfl/dotty/blob/10526a7d0aa8910729b6036ee51942e05b71abf6/compiler/src/dotty/tools/dotc/ast/Positioned.scala) (a parent class of `Tree`) object has a `uniqueId` field. It is an integer that is unique for that tree and doesn't change from compile run to compile run. You can output these IDs from any printer (such as the ones used by `.show` and `-Xprint`) via `-Yshow-tree-ids` flag, e.g.:
171+
172+
```
173+
dotc -Xprint:frontend -Yshow-tree-ids ../issues/Playground.scala
174+
```
175+
176+
Gives:
177+
178+
```scala
179+
package <empty>#1047 {
180+
final lazy module val Playground: Playground$#1049 =
181+
new Playground$#1049#1050#1051()#1052
182+
#1053
183+
final module class Playground$() extends Object#1090#1091#1092()#1093, _root_#
184+
1061
185+
.scala#1062.Serializable#1063 { this: Playground#1054.type#1055 =>
186+
def main(args: Array#1028[String#1033]#1034#1038): Unit#1039 =
187+
{
188+
println#1094("Hello World"#1041)#1095
189+
}#1096
190+
#1097
191+
}#1099
192+
}#1100
193+
```
194+
195+
You can then use these IDs to locate the creation site of a given tree using that ID via `-Ydebug-tree-with-id`, e.g.:
196+
197+
```
198+
dotc -Ydebug-tree-with-id 1049 ../issues/Playground.scala
199+
```
200+
201+
When the tree with the correspond id is allocated, the following prompt will appear:
202+
203+
```
204+
Debug tree (id=1049) creation
205+
Ident(Playground$)
206+
207+
208+
a)bort, s)tack, r)esume
209+
```
210+
211+
If you input `s`, you will get a stack trace like this:
212+
213+
```
214+
java.lang.Throwable
215+
at dotty.tools.dotc.reporting.Reporter$.loop$1(Reporter.scala:55)
216+
at dotty.tools.dotc.reporting.Reporter$.displayPrompt(Reporter.scala:63)
217+
at dotty.tools.dotc.ast.Positioned.printTrace$1(Positioned.scala:32)
218+
at dotty.tools.dotc.ast.Positioned.uniqueId_$eq(Positioned.scala:34)
219+
at dotty.tools.dotc.ast.Positioned.<init>(Positioned.scala:45)
220+
at dotty.tools.dotc.ast.Trees$Tree.<init>(Trees.scala:53)
221+
at dotty.tools.dotc.ast.Trees$DenotingTree.<init>(Trees.scala:266)
222+
at dotty.tools.dotc.ast.Trees$NameTree.<init>(Trees.scala:292)
223+
at dotty.tools.dotc.ast.Trees$RefTree.<init>(Trees.scala:298)
224+
at dotty.tools.dotc.ast.Trees$Ident.<init>(Trees.scala:375)
225+
at dotty.tools.dotc.ast.untpd$.Ident(untpd.scala:301)
226+
at dotty.tools.dotc.ast.desugar$.moduleDef(Desugar.scala:804)
227+
at dotty.tools.dotc.ast.desugar$.defTree(Desugar.scala:1038)
228+
at dotty.tools.dotc.typer.Namer.expand(Namer.scala:441)
229+
at dotty.tools.dotc.typer.Namer.index$$anonfun$1(Namer.scala:722)
230+
at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:15)
231+
at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:10)
232+
at scala.collection.immutable.List.foreach(List.scala:392)
233+
at dotty.tools.dotc.typer.Namer.index(Namer.scala:722)
234+
at dotty.tools.dotc.typer.Namer.recur$1(Namer.scala:484)
235+
at dotty.tools.dotc.typer.Namer.indexExpanded(Namer.scala:501)
236+
at dotty.tools.dotc.typer.Namer.index(Namer.scala:474)
237+
at dotty.tools.dotc.typer.FrontEnd.enterSyms$$anonfun$1(FrontEnd.scala:69)
238+
at dotty.runtime.function.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
239+
at dotty.tools.dotc.typer.FrontEnd.monitor(FrontEnd.scala:41)
240+
at dotty.tools.dotc.typer.FrontEnd.enterSyms(FrontEnd.scala:71)
241+
at dotty.tools.dotc.typer.FrontEnd.runOn(FrontEnd.scala:100)
242+
at dotty.tools.dotc.Run.runPhases$4$$anonfun$4(Run.scala:158)
243+
at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:15)
244+
at dotty.runtime.function.JProcedure1.apply(JProcedure1.java:10)
245+
at scala.collection.IndexedSeqOptimized.foreach(IndexedSeqOptimized.scala:36)
246+
at scala.collection.IndexedSeqOptimized.foreach$(IndexedSeqOptimized.scala:33)
247+
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:198)
248+
at dotty.tools.dotc.Run.runPhases$5(Run.scala:170)
249+
at dotty.tools.dotc.Run.compileUnits$$anonfun$1(Run.scala:178)
250+
at dotty.runtime.function.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
251+
at dotty.tools.dotc.util.Stats$.maybeMonitored(Stats.scala:102)
252+
at dotty.tools.dotc.Run.compileUnits(Run.scala:185)
253+
at dotty.tools.dotc.Run.compileSources(Run.scala:120)
254+
at dotty.tools.dotc.Run.compile(Run.scala:104)
255+
at dotty.tools.dotc.Driver.doCompile(Driver.scala:34)
256+
at dotty.tools.dotc.Driver.process(Driver.scala:172)
257+
at dotty.tools.dotc.Driver.process(Driver.scala:141)
258+
at dotty.tools.dotc.Driver.process(Driver.scala:153)
259+
at dotty.tools.dotc.Driver.main(Driver.scala:180)
260+
at dotty.tools.dotc.Main.main(Main.scala)
261+
```
262+
263+
So that tree was created at:
264+
265+
```
266+
at dotty.tools.dotc.ast.desugar$.moduleDef(Desugar.scala:804)
267+
```
268+
269+
Since all the stack frames above it are technical frames executing the tree creation command, and the frame in question is the location where the intent of the tree creation was expressed.
270+
271+
### Via tracer
272+
Some objects may not be `Positioned` and hence their creation site is not debuggable via the technique in the section above. Say you target a tree at `Typer`'s `typed` method as follows:
273+
274+
```scala
275+
if (tree.show == """println("Hello World")""") {
276+
val creationSite = "<creation site stack here>"
277+
println(creationSite)
278+
}
279+
```
280+
281+
In other words, you have a reference to the object and want to know were it was created. To do so, go to the class definition of that object. In our case, `tree` is a [`Tree`](https://github.com/lampepfl/dotty/blob/10526a7d0aa8910729b6036ee51942e05b71abf6/compiler/src/dotty/tools/dotc/ast/Trees.scala#L52). Now, create a new `val` member of that type:
282+
283+
```scala
284+
val tracer = Thread.currentThread.getStackTrace.mkString("\n")
285+
```
286+
287+
Then, from the `typed` method (or wherever you want to access the trace):
288+
289+
```scala
290+
if (tree.show == """println("Hello World")""") {
291+
val creationSite = tree.tracer
292+
println(creationSite)
293+
}
294+
```
295+
296+
## Built-in Logging Architecture
297+
Dotty has a lot of debug calls scattered throughout the code, most of which are disabled by default. At least three (possibly intertwined) architectures for logging are used for that:
298+
299+
- Printer
300+
- Tracing
301+
- Reporter
302+
303+
These do not follow any particular system and so probably it will be easier to go with `println` most of the times instead.
304+
305+
### Printers
306+
Defined in [Printers.scala](https://github.com/lampepfl/dotty/blob/10526a7d0aa8910729b6036ee51942e05b71abf6/compiler/src/dotty/tools/dotc/config/Printers.scala) as a set of variables, each responsible for its own domain. To enable them, replace `noPrinter` with `default`. [Example](https://github.com/lampepfl/dotty/blob/10526a7d0aa8910729b6036ee51942e05b71abf6/compiler/src/dotty/tools/dotc/typer/Typer.scala#L2226) from the code:
307+
308+
```scala
309+
typr.println(i"make contextual function $tree / $pt ---> $ifun")
310+
```
311+
312+
`typr` is a printer.
313+
314+
### Tracing
315+
Defined in [trace.scala](https://github.com/lampepfl/dotty/blob/10526a7d0aa8910729b6036ee51942e05b71abf6/compiler/src/dotty/tools/dotc/reporting/trace.scala). [Example](https://github.com/lampepfl/dotty/blob/10526a7d0aa8910729b6036ee51942e05b71abf6/compiler/src/dotty/tools/dotc/typer/Typer.scala#L2232) from the code:
316+
317+
```scala
318+
trace(i"typing $tree", typr, show = true) { // ...
319+
```
320+
321+
To enable globally, change [tracingEnabled](https://github.com/lampepfl/dotty/blob/10526a7d0aa8910729b6036ee51942e05b71abf6/compiler/src/dotty/tools/dotc/config/Config.scala#L164) to `true` (will recompile a lot of code).
322+
323+
You also need to set the printer referenced in the call (in the example, `typr`) to `default` as explained in the section on printers.
324+
325+
To enable for a single trace, do the following:
326+
327+
```scala
328+
trace.force(i"typing $tree", typr, show = true) { // ...
329+
```
330+
331+
### Reporter
332+
Defined in [Reporter.scala](https://github.com/lampepfl/dotty/blob/10526a7d0aa8910729b6036ee51942e05b71abf6/compiler/src/dotty/tools/dotc/reporting/Reporter.scala). Enables calls such as `ctx.log`, `ctx.error` etc. To enable, run dotc with `-Ylog:frontend` option.

‎docs/docs/contributing/release.md ‎docs/docs/contributing/procedures/release.md

File renamed without changes.

‎docs/docs/contributing/procedures/vulpix.md

+30
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Test Vulpix Framework
2+
If you are modifying the Vulpix framework and need a playground with dummy tests to try out your modifications, do the following.
3+
4+
Create the directory structure for the playground:
5+
6+
```bash
7+
mkdir -pv tests/playground/run tests/playground/neg
8+
echo "stuff" > tests/playground/neg/Sample.scala
9+
echo 'object Test { def main(args: Array[String]): Unit = {println("Hi")} }' > tests/playground/run/Sample.scala
10+
```
11+
12+
In `CompilationTests.scala`:
13+
14+
```scala
15+
@Test def exampleNeg: Unit = {
16+
implicit val testGroup: TestGroup = TestGroup("exampleNeg")
17+
compileFilesInDir("tests/playground/neg", defaultOptions).checkExpectedErrors()
18+
}
19+
20+
@Test def exampleRun: Unit = {
21+
implicit val testGroup: TestGroup = TestGroup("exampleRun")
22+
compileFilesInDir("tests/playground/run", defaultOptions).checkRuns()
23+
}
24+
```
25+
26+
SBT:
27+
28+
```scala
29+
testOnly dotty.tools.dotc.CompilationTests -- *example*
30+
```

0 commit comments

Comments
 (0)