Add option to extract individual values raw #3

hoshsadiq · 2018-03-12T21:10:20Z

something like this:

<p><a href="/home">some url</a></p>

$ cascadia --in my.html --out --css 'p > a' --raw-text
some url
$ cascadia --in my.html --out --css 'p > a' --raw-attr 'href'
/home

suntong · 2018-03-12T21:13:48Z

I don't think CSS selection can select base on attributes though.

Do you want to give another example instead?

suntong · 2018-03-12T21:18:06Z

If you just want to raw CSS selection value,

The work around is to use one -p after -c:

  -p, --piece           sub CSS selectors within -css to split that block up into pieces
                        format: PieceName=[RAW:]selector_string

e.g.,

$ echo '<p><a href="/home">some url</a></p>' | cascadia -i -o -c 'p' -p 'ATag=a'
ATag
some url

$ echo '<p><a href="/home">some url</a></p>' | cascadia -i -o -c 'p' -p 'ATag=RAW:a'
ATag
<a href="/home">some url</a>

It's not perfect, but cascadia was not built to be perfect but for quick hacks.

hoshsadiq · 2018-03-12T21:22:42Z

The issue with using -p is that it creates columns (with headers). My proposal is getting attributes for an individual elements. in my case I'm downloading the latest version of a file automatically, there's no other way to get "latest" version. example:

<html>
<head></head>
<body>
...
<div>
<a href="/files/plugin-name/download?version=1.25">1.25</a>
</div>
...
</body>
</html>

I want to be able to retrieve only the value of href with nothing else, as well as innerText of that anchor.

hoshsadiq · 2018-03-12T21:23:22Z

Perhaps an alternative would be adding a --no-header option for --piece, that doesn't print out the headers?

suntong · 2018-03-12T21:23:50Z

Gotya. Lee me think it over...

hoshsadiq · 2018-03-12T21:24:34Z

Happy to raise a PR if needed

suntong · 2018-03-14T00:21:22Z

Uhm... I gave a careful thought about it, and was about to turn it down, because my believe in the "Unix philosophy" -- Write programs that

do one thing and do it well
to work together
to handle text streams

I.e., the attributes selection is impossible with CSS thus should be out of the scope of cascadia; and --no-header can be simply solved by sed 1d:

$ echo '<p><a href="/home">some url</a></p>' | cascadia -i -o -c 'p' -p 'ATag=a' | sed 1d
some url

I.e., it'd against my principle to complicate my code base for something so simple to solve. However, I do see a need in your request, and you offered a PR. So I'm OK with the PR, iff you are doing the correct way -- i.e., starting from cascadia.yaml, and use wireframe for the code gen.

If that doesn't deter you, then go ahead. :-)

Thx for your contribution.

suntong · 2020-04-19T20:52:28Z

closed for lack of activity, please reopen if there is more input...

mazznoer · 2021-09-19T09:38:56Z

@hoshsadiq

You can do it using pup.

pup -f file.html 'body div a attr{href}'

0xdevalias · 2023-06-14T08:35:24Z

Just stumbled onto this issue as I was attempting to extract all of the src attributes for the script tags from a page, and it sounds like that should be possible with --piece, yet it didn't work for me:

curl --silent https://example.com/somethingwithscripts | cascadia --in --out --css 'html > head > script' --piece url='attr[src]'

I also tried various variations of this none of which seemed to work:

curl --silent https://example.com/somethingwithscripts | cascadia --in --out --css 'html > head > script' --piece url='attr[src]:script'

curl --silent https://example.com/somethingwithscripts | cascadia --in --out --css 'html > head > script' --piece url='attr[src]:*'

curl --silent https://example.com/somethingwithscripts | cascadia --in --out --css 'html > head' --piece url='attr[src]:script'

Yet with pup, it was not only a far simpler syntax, but also just worked the first time I tried it:

⇒ curl --silent https://example.com/somethingwithscripts | pup 'html > head > script attr{src}'

/_next/static/chunks/polyfills-c67a75d1b6f99dc8.js
/_next/static/chunks/webpack-1eeae5c7aedde088.js
/_next/static/chunks/framework-e23f030857e925d4.js
/_next/static/chunks/main-35ce5aa6f4f7a906.js
/_next/static/chunks/pages/_app-0df67bf7d9e6e732.js
/_next/static/chunks/1f110208-cda4026aba1898fb.js
/_next/static/chunks/012ff928-bcfa62e3ac82441c.js
/_next/static/chunks/68a27ff6-a453fd719d5bf767.js
/_next/static/chunks/bd26816a-981e1ddc27b37cc6.js
/_next/static/chunks/692-a1e5a91f2cd1f1d0.js
/_next/static/chunks/434-6f11f27f549beeab.js
/_next/static/chunks/97-536ee884c863676e.js
/_next/static/chunks/734-30d5c00c7bdf11c1.js
/_next/static/chunks/pages/share/%5B%5B...shareParams%5D%5D-44619ef92ec8f3b5.js
/_next/static/a3Jc7aP-UMfeR9s4-iLvW/_buildManifest.js
/_next/static/a3Jc7aP-UMfeR9s4-iLvW/_ssgManifest.js

suntong · 2023-06-14T13:36:49Z

Hmm... please try giving a minimal reproducible example. Else I won't be able to guess what the problem is.

suntong · 2023-06-14T13:42:07Z

Also, you didn't say which version you're using. Are you using the v1.2.7?

Sorry, trying to rush out some reply before getting back to my burning issue at hand...

0xdevalias · 2023-06-16T01:26:49Z

Hmm... please try giving a minimal reproducible example. Else I won't be able to guess what the problem is.

<html>
<head>
  <script src="foo.js"></script>
  <script src="bar.js"></script>
  <script src="baz.js"></script>
</head>
</html>

⇒ pbpaste | cascadia --in --out --css 'html > head > script' --piece url='attr[src]'
url




⇒ pbpaste | cascadia --in --out --css 'html > head > script' --piece url='attr[src]:script'
url




⇒ pbpaste | cascadia --in --out --css 'html > head' --piece url='attr[src]:script'
url
foo.js

Expected outcome:

url
foo.js
bar.js
baz.js

Also, you didn't say which version you're using. Are you using the v1.2.7?

Version 1.2.7 built on 2023-01-08

suntong · 2023-06-16T02:56:46Z

Indeed, it looks like a bug. Will look into it when I have some time. Meanwhile,

CC: @himcc, I've replicated problem @0xdevalias reported, seems to be logic problem with attr selection. Do you have some time to look into it please?

Hope the final interface would be:
--css 'html > head > script' --piece url='attr[src]', which look more straightforward than the latter two...

0xdevalias · 2023-06-16T03:34:56Z

Hope the final interface would be:
--css 'html > head > script' --piece url='attr[src]', which look more straightforward than the latter two...

While it would sort of be a breaking change, in a sense it feels like the attr prefix is sort of redundant as well, particularly given in the implementation in cascadia it's getting used in a seperate context (--piece) rather than the main --css (whereas with pup it sort of needs to differentiate itself since it all appears in the one query)

The initial thing I would have expected/tried was just being able to use standard CSS attribute selector syntax within --piece, eg [src] (or perhaps also script[src] if you wanted to support the full syntax there as well):

https://developer.mozilla.org/en-US/docs/Web/CSS/Attribute_selectors

That way the --piece syntax would sort of end up being much closer to just standard CSS selector usage. And I could do something like:

⇒ pbpaste | cascadia --in --out --css 'html > head > script' --piece url='[src]'

⇒ pbpaste | cascadia --in --out --css 'html > head' --piece url='script[src]'

# etc

Skimming the codebase, the following areas look like they would be relevant/related to these changes:

https://github.com/suntong/cascadia/blob/master/cascadia_main.go
- OutputStyleATTR, AttrName
- With these 2 bits looking like the main implementations for it:
  - https://github.com/suntong/cascadia/blob/master/cascadia_main.go#L190-L192
  - https://github.com/suntong/cascadia/blob/master/cascadia_main.go#L233-L235
https://github.com/suntong/cascadia/blob/master/cascadia_test.go#L553-L559
- PieceAttrTest, PieceAttrTests, TestPieceAttr

0xdevalias · 2023-06-16T03:46:56Z

@suntong @himcc I haven't looked too deeply at the code/tested this assumption, but from a quick skim I noticed that what appears to be the section handling --piece hardcodes operating on cssa[0]:

cascadia/cascadia_main.go

Lines 165 to 200 in 4b56cde

    
           } else { 
        
           	// have sub CSS selectors within -css -- block selection mode 
        
           	// fmt.Printf("%v\n", piece) 
        
           	// https://godoc.org/github.com/PuerkitoBio/goquery 
        
           	// for debug 
        
           	//doc, err := goquery.NewDocumentFromReader(strings.NewReader(testhtml)) 
        
           	doc, err := goquery.NewDocumentFromReader(bi) 
        
           	abortOn("Input", err) 
        
           	// Print csv headers 
        
           	for _, key := range piece.Keys { 
        
           		fmt.Fprintf(bw, "%s%s", key, deli) 
        
           	} 
        
           	fmt.Fprintf(bw, "\n") 
        
           	// Process each item block 
        
           	doc.Find(cssa[0]).Each(func(index int, item *goquery.Selection) { 
        
           		//fmt.Printf("] #%d: %s\n", index, item.Text()) 
        
           		for _, key := range piece.Keys { 
        
           			//fmt.Printf("] %s: %s\n", key, piece.Values[key]) 
        
           			switch piece.OutputStyles[key] { 
        
           			case OutputStyleRAW: 
        
           				html.Render(bw, item.Find(piece.Values[key]).Get(0)) 
        
           				fmt.Fprintf(bw, deli) 
        
           			case OutputStyleATTR: 
        
           				fmt.Fprintf(bw, "%s%s", 
        
           					item.Find(piece.Values[key]).AttrOr(piece.AttrName[key], ""), deli) 
        
           			case OutputStyleTEXT: 
        
           				fmt.Fprintf(bw, "%s%s", 
        
           					item.Find(piece.Values[key]).Contents().Text(), deli) 
        
           			} 
        
           		} 
        
           		fmt.Fprintf(bw, "\n") 
        
           	}) 
        
           }

Whereas the seemingly non---piece code uses for _, css := range cssa

cascadia/cascadia_main.go

Lines 127 to 165 in 4b56cde

    
           if len(piece.Values) == 0 { 
        
           	// no sub CSS selectors -- none-block selection mode 
        
           	if textOut { 
        
           		doc, err := goquery.NewDocumentFromReader(bi) 
        
           		abortOn("Input", err) 
        
           		for _, css := range cssa { 
        
           			// Process each item block 
        
           			doc.Find(css).Each(func(index int, item *goquery.Selection) { 
        
           				//fmt.Printf("] #%d: %s\n", index, item.Text()) 
        
           				if textRaw { 
        
           					fmt.Fprintf(bw, "%s%s", 
        
           						item.Text(), deli) 
        
           				} else { 
        
           					fmt.Fprintf(bw, "%s%s", 
        
           						strings.TrimSpace(item.Text()), deli) 
        
           				} 
        
           				fmt.Fprintf(bw, "\n") 
        
           			}) 
        
           		} 
        
           	} else { 
        
           		doc, err := html.Parse(bi) 
        
           		abortOn("Input", err) 
        
           		for _, css := range cssa { 
        
           			c, err := cascadia.Compile(css) 
        
           			abortOn("CSS Selector string "+css, err) 
        
           			// https://godoc.org/github.com/andybalholm/cascadia 
        
           			ns := c.MatchAll(doc) 
        
           			if !beQuiet { 
        
           				fmt.Fprintf(os.Stderr, "%d elements for '%s':\n", len(ns), css) 
        
           			} 
        
           			for _, n := range ns { 
        
           				html.Render(bw, n) 
        
           				fmt.Fprintf(bw, "\n") 
        
           			} 
        
           		} 
        
           	} 
        
           } else {

Based on that, and the fact that in my above pbpaste | cascadia --in --out --css 'html > head' --piece url='attr[src]:script' example (Ref) it only output the first item; I suspect that this is where the current bug exists; and my assumption is that it would be fixed by also using code similar to for _, css := range cssa here too instead of cssa[0]

(Though personally, I still think it would be useful to simplify and improve the --piece syntax as well if we can)

suntong · 2023-06-16T13:45:15Z

it only output the first item; I suspect that this is where the current bug exists

Yep, I thought so too. Thanks for investating.

That way the --piece syntax would sort of end up being much closer to just standard CSS selector usage.

That's a brilliant idea. I've been searching for CSS Attribute selectors before many times, but all conclusings had been that it is not supported. Yeah, I fully agree with you that we should use the CSS Attribute selectors syntax instead.

suntong · 2023-06-16T20:22:14Z

Ops,

I've been searching for CSS Attribute selectors before many times, but all conclusings had been that it is not supported

I didn't look into the url closely, but having check it out again just now, I found that the a[title] in CSS Attribute selectors means <a> elements with a title attribute, IE, how to select the <a> elements, while here we need a syntax to return attributes, which is not supported by CSS selectors (still).

Will think it over...

0xdevalias · 2023-06-21T03:52:52Z

I didn't look into the url closely, but having check it out again just now, I found that the a[title] in CSS Attribute selectors means <a> elements with a title attribute, IE, how to select the <a> elements, while here we need a syntax to return attributes, which is not supported by CSS selectors (still).

@suntong Yup, that is how it works in normal CSS selector usage, and is how it would need to work (and just how it does work I believe) in the --query part of cascadia; but what I was proposing above is that we could leverage the same familiar semantics of that syntax, but when used within --piece, we could have it output the actual attribute being described.

So basically:

--query:
- [foo] would return all elements that have an attribute named foo
- a[foo] would return all elements that have an a attribute named foo
- etc
--piece
- [foo] would return the attribute named foo from all elements that have it
- a[foo] would return the attribute named foo from all a elements that have it
- etc

So then combined, you could use these like I described above:

That way the --piece syntax would sort of end up being much closer to just standard CSS selector usage. And I could do something like:
⇒ pbpaste | cascadia --in --out --css 'html > head > script' --piece url='[src]'

⇒ pbpaste | cascadia --in --out --css 'html > head' --piece url='script[src]'

# etc
Originally posted by @0xdevalias in #3 (comment)

0xdevalias · 2023-06-21T04:12:06Z

Another alternative would be to re-consider how --piece works in terms of the 'prior art' from pup, and how it has various 'Display Functions' (as they call them)

https://github.com/ericchiang/pup#display-functions
- Non-HTML selectors which effect the output type are implemented as functions which can be provided as a final argument.

They have:

text{}
attr{attrkey}
json{}

Personally I don't think json{} makes a lot of sense here (you could just run the HTML through an XML -> JSON tool).

I'm pretty sure the --text mode already covers what text{} would do.

So it basically seems to just leave the attr{attrkey} version.

Though personally I like the idea of keeping the CSS selector --query as 'pure selectors' (unlike pup also adding in the 'Display Functions') there.

suntong · 2023-06-22T02:30:47Z

Thanks for the great input. Make sense.
I might not be able to look into in 2~3 weeks, but I will...

0xdevalias · 2023-06-22T06:12:53Z

Thanks for the great input. Make sense.
I might not be able to look into in 2~3 weeks, but I will...

@suntong No worries, I appreciate it :)

0xdevalias · 2023-06-26T00:36:41Z

Another source of 'prior art', xq just implemented this recently; you can see their approach on this comment (and in the commits linked later in the timeline):

Allow CSS selectors to return the full html content, not just the text within it. sibprogrammer/xq#20 (comment)

suntong · 2023-06-29T16:15:26Z

pbpaste | cascadia --in --out --css 'html > head > script' --piece url='attr[src]'

See

cascadia/piece_test.go

Line 29 in 7a5fade

    
           "-i", "opt_piece_script.html", "-o", "-c", "html > head > script", "-p", "SourceJS=ATTR:src",

cascadia/test/script_src.ref

Lines 1 to 4 in 92fb160

    
           SourceJS	 
        
           foo.js	 
        
           bar.js	 
        
           baz.js

0xdevalias · 2023-06-30T08:31:07Z

Awesome! Will have to check it out once a new release is made! Thanks :)

suntong closed this as completed Apr 19, 2020

suntong reopened this Jun 16, 2023

0xdevalias mentioned this issue Jun 21, 2023

Allow CSS selectors to return the full html content, not just the text within it. sibprogrammer/xq#20

Closed

suntong closed this as completed in 7c00a1e Jun 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option to extract individual values raw #3

Add option to extract individual values raw #3

hoshsadiq commented Mar 12, 2018 •

edited

Loading

suntong commented Mar 12, 2018

suntong commented Mar 12, 2018 •

edited

Loading

hoshsadiq commented Mar 12, 2018 •

edited

Loading

hoshsadiq commented Mar 12, 2018

suntong commented Mar 12, 2018

hoshsadiq commented Mar 12, 2018

suntong commented Mar 14, 2018

suntong commented Apr 19, 2020

mazznoer commented Sep 19, 2021

0xdevalias commented Jun 14, 2023 •

edited

Loading

suntong commented Jun 14, 2023 •

edited

Loading

suntong commented Jun 14, 2023 •

edited

Loading

0xdevalias commented Jun 16, 2023 •

edited

Loading

suntong commented Jun 16, 2023

0xdevalias commented Jun 16, 2023 •

edited

Loading

0xdevalias commented Jun 16, 2023

suntong commented Jun 16, 2023 •

edited

Loading

suntong commented Jun 16, 2023 •

edited

Loading

0xdevalias commented Jun 21, 2023

0xdevalias commented Jun 21, 2023

suntong commented Jun 22, 2023

0xdevalias commented Jun 22, 2023 •

edited

Loading

0xdevalias commented Jun 26, 2023

suntong commented Jun 29, 2023 •

edited

Loading

0xdevalias commented Jun 30, 2023

Add option to extract individual values raw #3

Add option to extract individual values raw #3

Comments

hoshsadiq commented Mar 12, 2018 • edited Loading

suntong commented Mar 12, 2018

suntong commented Mar 12, 2018 • edited Loading

hoshsadiq commented Mar 12, 2018 • edited Loading

hoshsadiq commented Mar 12, 2018

suntong commented Mar 12, 2018

hoshsadiq commented Mar 12, 2018

suntong commented Mar 14, 2018

suntong commented Apr 19, 2020

mazznoer commented Sep 19, 2021

0xdevalias commented Jun 14, 2023 • edited Loading

suntong commented Jun 14, 2023 • edited Loading

suntong commented Jun 14, 2023 • edited Loading

0xdevalias commented Jun 16, 2023 • edited Loading

suntong commented Jun 16, 2023

0xdevalias commented Jun 16, 2023 • edited Loading

0xdevalias commented Jun 16, 2023

suntong commented Jun 16, 2023 • edited Loading

suntong commented Jun 16, 2023 • edited Loading

0xdevalias commented Jun 21, 2023

0xdevalias commented Jun 21, 2023

suntong commented Jun 22, 2023

0xdevalias commented Jun 22, 2023 • edited Loading

0xdevalias commented Jun 26, 2023

suntong commented Jun 29, 2023 • edited Loading

0xdevalias commented Jun 30, 2023

hoshsadiq commented Mar 12, 2018 •

edited

Loading

suntong commented Mar 12, 2018 •

edited

Loading

hoshsadiq commented Mar 12, 2018 •

edited

Loading

0xdevalias commented Jun 14, 2023 •

edited

Loading

suntong commented Jun 14, 2023 •

edited

Loading

suntong commented Jun 14, 2023 •

edited

Loading

0xdevalias commented Jun 16, 2023 •

edited

Loading

0xdevalias commented Jun 16, 2023 •

edited

Loading

suntong commented Jun 16, 2023 •

edited

Loading

suntong commented Jun 16, 2023 •

edited

Loading

0xdevalias commented Jun 22, 2023 •

edited

Loading

suntong commented Jun 29, 2023 •

edited

Loading