Skip to content
This repository has been archived by the owner on Aug 24, 2019. It is now read-only.

Only a subset of syntax is highlighted #5

Open
zats opened this issue Jun 27, 2015 · 13 comments · May be fixed by #18
Open

Only a subset of syntax is highlighted #5

zats opened this issue Jun 27, 2015 · 13 comments · May be fixed by #18

Comments

@zats
Copy link

zats commented Jun 27, 2015

Hi, I was playing around with initial setup of the library.
It seems like the library recognises only a subset of the syntax (while the same file opened in TextMate is showing the full syntax highlight – not sure if it's relevant)

screen shot 2015-06-27 at 3 15 22 pm

This is the code I used (more or less copied from the readme, slightly modified for Swift 2):

let samplePath = bundle.URLForResource("Sample", withExtension: "swift")!
let sample = try! String(contentsOfURL: samplePath)

let languagePath = bundle.URLForResource("Swift", withExtension: "tmLanguage")!
let languageDictionary = NSDictionary(contentsOfURL: languagePath) as! [NSObject: AnyObject]
let language = Language(dictionary: languageDictionary)!

let themeURL = bundle.URLsForResourcesWithExtension("tmTheme", subdirectory: nil)!.first!
let themeDictioanry = NSDictionary(contentsOfURL: themeURL) as! [NSObject: AnyObject]
let theme = Theme(dictionary: themeDictioanry)!

let parse = AttributedParser(language: language, theme: theme)
let attributedString = parse.attributedStringForString(sample)

Following is the playground (please rename to .zip and extract).

Is anything I'm doing look obviously wrong?

And a follow up question: is there a smarter way to figure out the language used for a file than the file extension? Extensions might be used by several languages: .h file might be either Objective-C or plain C; while .m is used for both Objective-C and Matlab.

Thanks a lot!

@zats zats changed the title invalid results Invalid results Jun 27, 2015
@zats zats changed the title Invalid results Only a subset of syntax is highlighted Jun 27, 2015
@dehlen
Copy link

dehlen commented Jul 23, 2015

I got the exact same problem. Only parts are highlighted. Other are recognized but the attributes array is nil.
Could you please have a look into it. Provided Playground by zats replicates the problem although my problem ist that LaTeX comments and commands like \command{} do not get highlighted with some exceptions (\begin{document}, \usepackage{}, \newcommand{}) is working. Would it help you if i provided my current setup also ?

Or do you think that this is a problem concerning the tmLanguage, tmTheme files ? When I open my test file in Textmate a lot more is colored than in my own NSTextView.

@dehlen
Copy link

dehlen commented Jul 25, 2015

Just had a look at this this morning and for me it looks like the library is working just fine. It seems like the plist(tmLanguage) file is missing entries and therefore the code is only party highlighted.

@zats
Copy link
Author

zats commented Jul 25, 2015

I think I copied the bundle from the official repo. However I'll need to
check that
On Sat, Jul 25, 2015 at 11:30 AM David Ehlen notifications@github.com
wrote:

Just had a look at this this morning and for me it looks like the library
is working just fine. It seems like the plist(tmLanguage) file is missing
entries and therefore the code is only party highlighted.


Reply to this email directly or view it on GitHub
#5 (comment).

@dehlen
Copy link

dehlen commented Jul 25, 2015

Sure please report back. I only had a look at the Latex.tmLanguage file and it occured to me that there was no pattern for highlighting comments for example.

@zats
Copy link
Author

zats commented Jul 27, 2015

Yeah, as I suspected, I just copied the language file from the official
repo (I played around with swift definition as you can see from the
screenshots).
Link to the playground includes my setup (except for the SyntaxKit that has
to be part of the workspace for the playground to work)
Please let me know if I can provide any more information to help understand
the issue better.
Thanks!

On Sat, Jul 25, 2015 at 11:43 AM David Ehlen notifications@github.com
wrote:

Sure please report back. I only had a look at the Latex.tmLanguage file
and it occured to me that there was no pattern for highlighting comments
for example.


Reply to this email directly or view it on GitHub
#5 (comment).

@Orion98MC
Copy link

Hi,

Same for me.

I have downloaded Swift.tmLanguage from the tm gh repo as well as the twilight theme.
Only storage.type.swift and support.function.swift are tokenized

Tried a different tmLanguage, like markdown => No recognized pattern at all when parsing a md file

Let me know if I can help in any way.

Regards,
Thierry

@Orion98MC
Copy link

Adding to my comments:

The following Language initialization log is taken from Language() for Swift.tmLanguage.

Here is an excerpt of the plist parsing which results from the execution of the "repository" parsing block at Language.swift:34:

The log shows how the Swift.tmLanguage 'repository.optional-type' keyPath is parsed:

Pattern:
    name: Optional("meta.optional.swift")
    Match: Optional("\\b(Optional)(<)")
    Begin: nil
    End: Optional("(>)")
    BeginCaptures: CaptureCollection(captures: [2: SyntaxKit.Capture(name: "punctuation.definition.optional.begin.swift"), 1: SyntaxKit.Capture(name: "support.type.optional.swift")])
    EndCaptures: CaptureCollection(captures: [1: SyntaxKit.Capture(name: "punctuation.definition.optional.end.swift")])
    1 Pattern(s): [
           Pattern:
            name: nil
            Match: nil
            Begin: nil
            End: nil
       ]

On the other hand, the plist itself for the same keyPath is:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>beginCaptures</key>
    <dict>
        <key>1</key>
        <dict>
            <key>name</key>
            <string>support.type.optional.swift</string>
        </dict>
        <key>2</key>
        <dict>
            <key>name</key>
            <string>punctuation.definition.optional.begin.swift</string>
        </dict>
    </dict>
    <key>end</key>
    <string>(&gt;)</string>
    <key>endCaptures</key>
    <dict>
        <key>1</key>
        <dict>
            <key>name</key>
            <string>punctuation.definition.optional.end.swift</string>
        </dict>
    </dict>
    <key>match</key>
    <string>\b(Optional)(&lt;)</string>
    <key>name</key>
    <string>meta.optional.swift</string>
    <key>patterns</key>
    <array>
        <dict>
            <key>include</key>
            <string>$self</string>
        </dict>
    </array>
</dict>
</plist>

I have no knowledge of the tmLanguage format but I noticed that the patterns keyPath within the repository.optional-type keyPath is translated as an array of 1 pattern with all nil values which seems to be a false interpretation of the plist data.

Now, I'm pretty sure many other keyPaths are not parsed correctly. Which may have significant side effects on the recognition of patterns when the parser runs.

Also the Language.patterns (for both Swift.tmLanguage and markdown) does contain many all-nil-values Pattern-s which seems to confirm that the parsing of tmLanguage files is not optimal

Thanks and regards,
Thierry

@soffes
Copy link
Owner

soffes commented Sep 5, 2015

Ya there's definitely some outstanding bugs.

@dehlen
Copy link

dehlen commented May 30, 2016

Here is a definition of the textmate grammar: http://manual.macromates.com/en/language_grammars#language_grammars.

It looks like the include key is not parsed correctly in SyntaxKit. The different values a include key can hold are described in the link above. Is there any plan to fix this issue in the future ? It seems like this library was last updated quite a time ago.

Cheers,
David

@soffes
Copy link
Owner

soffes commented May 30, 2016

This is amazingly helpful @dehlen! I didn't know this existed. I'll be using this library again in a project I'm working on in about 2 months, so I'll get started with some some tests and full support when I start using it again.

Sorry I've been slow to work on this everyone. Watch the linked video above if you want to know why :)

@dehlen
Copy link

dehlen commented May 30, 2016

You don't have to be sorry. Open-Source is sharing and caring :) I am myself a bit stressed at the moment writing my thesis. However if I'll find time I'll might have a look at the outstanding bug.

Keep up the amazingly good work.

@soffes
Copy link
Owner

soffes commented May 30, 2016

Thanks <3

@aaronvegh
Copy link

The timing on this thread is really good. I came across this project last week and spent some time looking at the missing features in the Language.swift implementation. Updating it to support the new (?) syntax is going to be quite the challenge! I thought I'd take a crack at it, but it kicked my ass. So Sam, I will wish you all the best in getting it updated!

alehed referenced this issue in alehed/SyntaxKit Aug 7, 2016
Reuse the already matched results
@alehed alehed linked a pull request Aug 7, 2016 that will close this issue
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants