Implement a better method for generating svg images with TikZImage.pm.

[drgrice1]

The current implementation generates a pdf using pdflatex, and then converts the pdf to svg using imagemagick's convert utility.

This changes it so that it uses latex to generate a dvi with the pgfsysdriver redefined to pgfsys-dvisvgm.def, and then converts the dvi to svg using dvisvgm. This is the best way that I have found of creating svg images from tikz/tex.

This does require another external program (dvisvgm). That is added in a corresponding pull request to webwork2.

I have attached a couple of examples to compare the png image generation to the svg image generation.
examples.zip

To test this, make sure to set the dvisvgm external command in site.conf. See openwebwork/webwork2#1308.

[Alex-Jordan]

Just looking over the example .pg files for now. I guess you can put things like \( in the heredoc because WeBWorK expands that to \\( before the perl interpolation happens, is that right?

How do you put actual dollar, percent, and at signs in the tikz code? Like if they were part of the text labeling an axis or something? Is it possible without using ${DOLLAR} etc?

Can authors leave comments in the tikz code? Like % this is a comment?

Let's add the BEGIN_TIKZ substitution. If the early examples demonstrating this to new users avoid the heredoc syntax, it will be more comfortable.

[drgrice1]

Yeah, the \(...\) construct works. You can also do ~~$...~~$ and that works as well.

To add the dollar symbol you can do \node at (3,-3){\(~~\$50,000\)};. $DOLLAR doesn't work though.

Using % this is a comment works.

I am not sure that the BEGIN_TIKZ ... END_TIKZ idea will work. The TikZ code needs to be associated to a particular instance of a TikZImage object. It is not the same as with BEGIN_PGML ... END_PGML and such.

[drgrice1]

Yeah, so I looked into attempting to do the BEGIN_TEXT ... END_TEXT a bit. That certainly would not be an easy thing to accomplish. I don't think that is going to happen now.

[drgrice1]

Okay. So it wasn't that hard after all. I have to update the documentation, but I will commit it soon.

[drgrice1]

Note that using the heredoc still works (as in the attached examples in the initial comments on this pull request). You could also use a nowdoc if desired (assuming you don't need interpolation). The BEGIN_TIKZ/END_TIKZ construction is a heredoc though. Generally that is what you want so that you can interpolate variables for dynamic images.

[drgrice1]

Here is an example using the BEGIN_TIKZ/END_TIKZ construction. Note it is a little different than the other BEGIN/END constructions, as it needs an object.

BEGIN_TIKZ_eg.zip

[pstaabp]

Overall, this looks good (with pun intended since svg is cleaner).

I wonder if instead of calling:

$graph_image->ext('svg') if $displayMode ne 'TeX';

inside a problem, that we have setting for this instead. It seems like this should be a global or course-level setting rather than a problem setting. We could have a

$pg{specialPGEnvironmentVars}{tikz_image_output} = "svg";

inside defaults.config and override it in site.conf or course.conf. Would the other setting then be 'png'?

And then of course the conversion would need to handled somewhere instead of inside the problem itself.

[drgrice1]

This is definitely not a case where it makes sense to be a course level or system level setting. This is a problem authoring decision. The point is that an author must decide if the images created by their tikz code work well with SVG or not. I am fine with setting the default to SVG, and then having the author override that to PNG if their images don't work well. Generally SVG is the right way to go if that works.

I am not fond of having to set the override extension in the way that is done here, but there are limitations with the module that make this necessary.

[drgrice1]

Attached are updated examples for the latest changes.
examples.zip

[Alex-Jordan]

Here is a screenshot from a modified version of the testing problem BEGIN_TIKZ_eg.pg. The modification is that it displays the svg image, then changes to png and displays it again. Then back to svg and change the image dimension from 200x200 to 100x100. Is something is wrong with the scaling for svg? Or do I need to do something to the tikz code, knowing the pixel dimensions of the ultimate output image?

[drgrice1]

I am not entirely certain of what you are doing. I made a guess as follows, but it doesn't produce the output you show. Here is my example attempting to reproduce your image.

DOCUMENT();

loadMacros("PGstandard.pl", "PGML.pl", "PGtikz.pl");

TEXT(beginproblem());

$a = random(-10, 9);
$b = random(-10, 9);

$graph_image = createTikZImage();
$graph_image->tikzLibraries("arrows.meta");
$graph_image->tikzOptions("x=0.6cm,y=0.6cm");
$graph_image->BEGIN_TIKZ
\tikzset{>={Stealth[scale=1.8]}}
\filldraw[draw=blue,fill=white,rounded corners=14pt,thick,use as bounding box] (-11,-11) rectangle (11,11);
\begin{scope}
	\clip[rounded corners=14pt] (-11,-11) rectangle (11,11);
	\draw[line width=0.2pt,color=gray,step=1] (-11,-11) grid (11,11);
\end{scope}
\huge
\draw[<->,thick] (-11,0) -- (11,0) node[above left,outer sep=2pt]{\(x\)};
\draw[<->,thick] (0,-11) -- (0,11) node[below right,outer sep=2pt]{\(y\)};
\foreach \x in {-10,-8,...,-2,2,4,...,10} \draw[thin] (\x,5pt) -- (\x,-5pt) node[below]{\(\x\)};
\foreach \y in {-10,-8,...,-2,2,4,...,10} \draw[thin] (5pt,\y) -- (-5pt,\y) node[left]{\(\y\)};
\filldraw[red] ($a,$b) circle[radius=5pt];
% this is a comment
\node at (3,-3){\(~~\$50,000\)};
END_TIKZ

BEGIN_PGML
[@ image(insertGraph($graph_image), width => 200, height => 200) @]*

[@ $graph_image->ext('png'); image(insertGraph($graph_image), width => 200, height => 200) @]*

[@ $graph_image->ext('svg'); image(insertGraph($graph_image), width => 100, height => 100) @]*
END_PGML

ENDDOCUMENT();

[Alex-Jordan]

If I take BEGIN_TIKZ_eg.pg from your most recent examples.zip and just run it as is, I get:

That's using the 300 pixel width written into the problem, and apparently I am seeing some default height. The image of course is a zoomed in section of the intended graph.

If I change the BEGIN_PGML...ENDPGML part to the following, it is how I end up with the image I previously posted:

BEGIN_PGML

[@ image(insertGraph($graph_image), width => 200, height=>200, tex_size => 1000) @]*

END_PGML

$graph_image->ext('png');

BEGIN_PGML

[@ image(insertGraph($graph_image), width => 200, height=>200, tex_size => 1000) @]*

END_PGML

$graph_image->ext('svg');

BEGIN_PGML

[@ image(insertGraph($graph_image), width => 100, height=>100, tex_size => 1000) @]*

END_PGML

[Alex-Jordan]

If there is a discrepancy between what I get and what you get, it could be some issue about the version of latex or dvisvgm. This server has pdfTeX 3.14159265-2.6-1.40.18 (TeX Live 2017/Debian) and dvisvgm 2.1.3.

[drgrice1]

Argh! I tested this on my server which has Ubuntu 18.04 and the versions of texlive/dvisvgm installed that you have. I get the same crap. So apparently newer versions of TeX and dvisvgm (and probably ghostscript which dvisvgm uses) are needed.

Looking closely I see that the pgfsys-dvisvgm.def file in the older texlive distribution is quite different than the one in the newer tex distribution on Ubuntu 20.04. On Ubuntu 18.04 the dvisvgm binary is part of the texlive-binaries package, but on Ubuntu 20.04 it is in the separate dvisvgm package. So apparently things have changed quite a bit with dvisvgm.

[drgrice1]

I added a system configuration option as @pstaabp had suggested earlier to set the default image extension used by the PGtikz.pl macro. This is not ideal, but unfortunately needed as the versions of latex and dvisvgm on Ubuntu 18.04 are not good enough to generate 'svg' images. See openwebwork/webwork2#1320 for the settings.

[Alex-Jordan]

I never really tried making tikz images before this change. Would it make sense for the default to still be svg, but the configuration option is between the convert method or the dvisvgm method?

I'm not sure if the same or similar issues happen with the convert method. There must be something I do not understand about the svg production before this PR. With test files I try, even the images that are supposed to come out as svg still come out as png. Like with this test file from an old PR:

##DESCRIPTION
# TEST tikz from a pgml problem
##ENDDESCRIPTION
DOCUMENT();
loadMacros(
    "PGstandard.pl",
    "MathObjects.pl",
    "PGML.pl",
    "PGtikz.pl"
);
TEXT(beginproblem());
##############################################################
#  Setup
##############################################################
$drawing = createTikZImage();
$drawing->tikzOptions("main_node/.style={circle,fill=blue!20,draw,minimum size=1em,inner sep=3pt}");
$drawing->tex(<<END_TIKZ);
\node[main_node] (1) at (0,0) {1};
\node[main_node] (2) at (-1, -1.5)  {2};
\node[main_node] (3) at (1, -1.5) {3};
\draw (1) -- (2) -- (3) -- (1);
END_TIKZ
$path = insertGraph($drawing);
Context("Numeric");
##############################################################
#  Text
##############################################################

BEGIN_PGML
path = [$path]
path = [@ protect_underbar($path) @]
[@ $BR @]*
alias = [@ alias($path) @]*
alias = [@ protect_underbar(alias($path)) @]*

image = [@ image($path, width => 100, tex_size => 400) @]*

svg = [@ embedSVG($path) @]*
END_PGML
ENDDOCUMENT();

[pstaabp]

Something a bit crazy is going on. I'm looking at the png_svg_cmp.pg file that you have in the examples. It appears to default to PNG in that both of them are clearly png files.

If I add the line

$graph_svg->ext('svg');

then I do get it as a SVG file. I was looking if there is any config that I set, but it doesn't appear to. I'm guessing others are not seeing this.

[pstaabp]

I just looked at openwebwork/webwork2#1320 and I have not pulled that in, so that's not causing this.

[drgrice1]

@Alex-Jordan: The convert method always really generates png images, even for the svg extension. If you select the svg extension it just puts the png image into an svg container.

@pstaabp: Even without openwebwork/webwork2#1320 the default is now 'png'. I haven't updated the example file, so you will get 'png' from both. You need to add $image->ext('svg') to get 'svg' output.

[pstaabp]

@drgrice1 Thanks. I was walking through the code, trying to figure out what's going on. I got the impression that line 38 in TikZImage.pm set the default. Evidently not.

[Alex-Jordan]

OK, shooting through all of the options I know of, what about pdf2svg then? It's what PreTeXt uses for the PDF -> SVG conversion in a similar situation. Although there is movement afoot that may lead to using dvipsvgm.

Using pdf2svg (assuming it even works for this) could be a separate project.

[drgrice1]

@pstaabp: Line 38 of TikZImage.pm sets the underlying default of TikZImage package. However, that is overridden by line 99 of PGtikz.pl. That sets the extension to the tikzImageExtension if that is defined, and falls back to 'png' if not.

@Alex-Jordan: I would prefer to use dvisvgm as the current version does a better job in general. I guess I will switch this pull request to using pdf2svg for now though.

[Alex-Jordan]

@Alex-Jordan: I would prefer to use dvisvgm as the current version does a better job in general. I guess I will switch this pull request to using pdf2svg for now though.

I completely agree. I am wondering if the configuration option that you just introduced could control whether this process uses dvisvgm versus pdf2svg instead of controlling whether the default extension is svg or png.

[drgrice1]

I completely agree. I am wondering if the configuration option that you just introduced could control whether this process uses dvisvgm versus pdf2svg instead of controlling whether the default extension is svg or png.

I was thinking the same thing, and will try to implement that.

[drgrice1]

@pstaabp: Could you test with the latest commit, and see if that resolves the permissions issue that you are experiencing?

[pstaabp]

@drgrice1 This fixes that permissions problem now.

[drgrice1]

@pstaabp: So the images are properly generated on your local test server then?

[pstaabp]

@drgrice1 before this patch, it wouldn't generate the svg image using pdf2svg. Now it works fine.

[drgrice1]

Yeah, that is what I meant. I forgot that it was working with dvisvgm. Good deal.

[Alex-Jordan]

I decided to implement the second version of what I described in my last comment. That is now you can just do what you were trying to do and load the xcolor package with the cmyk option, and it will work.

Thanks. I got to thinking what other package option clashes might arise? Since the packages and their options are a hash reference, we cannot require the packages to load in the order that the problem author writes them. I thought to add an alternative where order would be enforced with an array reference. An extreme example:

$image->texPackages([
    {package1 => {option1a => value1a}},
    {
        package2 => {option2a => value2a}, 
        package3 => {option3a => value3a, option3b => value3b}
    }
])

but I think that is not easy to merge with the xcolor handling.

Probably it is incredibly rare that packages you might use with a tikz picture have the opportunity for a clash, so this observation is not all that important.

[drgrice1]

I got to thinking what other package option clashes might arise? Since the packages and their options are a hash reference, we cannot require the packages to load in the order that the problem author writes them. I thought to add an alternative where order would be enforced with an array reference. An extreme example:

Probably it is incredibly rare that packages you might use with a tikz picture have the opportunity for a clash, so this observation is not all that important.

You are right. This may not be that important, but could be an issue. How about just an array or arrays, where the inner arrays may have 1 or 2 elements. The first being the package name, and the second (if provided) the package options? So it would be used like

$image->texPackages([
    [package1, options1],
    [package2]
]])

Here options1 would be a string that is in the format that it would be passed to the package in latex already. So the latex that is output would be

\usepackage[options1]{package1}
\usepackage{package2}

Of course the xcolor package would be filtered out.

I don't see the need to separate the options into yet another embedded hash. Furthermore, latex package options do not always fit into the format that are assumed with that.

[drgrice1]

The last commit changes from hash to an array for the texPackages option of the TikZImage package.
An example of the new usage is

$image = createTikZImage();
$image->texPackages([
    "pgfplots",
    ["hf-tikz", "customcolors"],
    ["xcolor", "cmyk,table"]
]);

Note that the xcolor package breaks the order rule. It is always loaded, and is always loaded first. Passing it is only needed if you want to change the options for the package.

[drgrice1]

Here is the updated example that uses the pgfplots package. It doesn't load any other packages though, so it is not a good example of how to use the texPackages option.
tikz_tex_package_test.zip

[Alex-Jordan]

I looked at the most recent commit that moves to an array reference for packages, and it looks good. Still approve for merge.

[somiaj]

EDIT: Figured it out, my site.conf needed to be updated to include the binary for pdf2svg.

Now that this has been merged into PG-2.16, I have tested it out, using the .pg file in tizk_tex_package_test.zip. The .png file generates just fine but the .svg didn't with the default pdf2svg. Changing to dvisvgm worked. I also didn't notice any issues with dvisvgm, though my server is running Debian 10 with slightly older version at 2.6.1. This is what I have in my apache2 error logs when trying to create an svg with pdf2svg.

Use of uninitialized value in concatenation (.) or string at /home/webwork/pg/lib/TikZImage.pm line 176.
Convert operation failed. at /home/webwork/pg/lib/TikZImage.pm line 192.

[drgrice1]

@somiaj: Yeah, if you are testing things with the release candidate branch, make sure that your configuration files are all updated. Thanks for testing though. We need as many people testing things as we can get.

[somiaj]

@drgrice1 What is the preferred way to change convert options? I have been wanting to use Tikz to create some graphs, but the .png quality was always low. Adding -density 300 to the convert option in TikZImage.pm on line 181 has made the .png images that are being created better quality. Here I think my issue is my images are actually fairly small, so the defaults don't work (your example the defaults worked fine). If there isn't a method, maybe it could be useful to add an option to be able to pass to convert? (Though the svg images work just fine, so I can just use those).

I also assume that using $v_1$ for math cannot be supported (though \(v_1\) works, just means I have to update my graphs from my .tex files).

Anyways, thanks for the support, and all my tests appear to be working fine, these are just things I noticed.

[Alex-Jordan]

There are two open pull requests to handle convert options. One for pg, one for webwork2.

#566

openwebwork/webwork2#1334

[drgrice1]

Wait for #566. That adds the capability you are looking for. @Alex-Jordan just beat me to this post.

[drgrice1]

You can use $v_1$, just use ~~$v_1~~$. Yeah, it is a bit ugly, so \(v_1\) is nicer.

[somiaj]

@drgrice1 I was working with creating some images with this package and noticed the following behavior when working with .svg images (I am using pdf2svg for this).

• Write a problem to create an image and save it.
• View the problem in a homework set.
• Go edit the problem, and change the image.
• When I "View" the edited images everything works fine, but when I "Update" the problem it shows the old image in the assignment.

It appears that the old image is used if it is attached to an assignment (something is being cached?) thus making it so changes don't appear in the assignment. I tried deleting the image from htdocs/tmp, but that didn't change things, the old image was still generated. But when I unassigned the assignment and reassigned it, the new image was generated.

Edit: In testing this I found that it only seemed to happen with .svg files. I also decided to test pdf2svg vs dvisvgm, and after restarting the server I am no longer getting the behavior. So though I had this behavior happen on both my live server and my test server, once I restarted them I can no longer reproduce it. Leaving this here in case I run across this again.

[drgrice1]

@somiaj: Check things carefully to see that you are using the files you think you are using. That is, make sure you are using the actual PG file for the homework problem that you think you are. There was a bug in the PG problem editor that actually changed the file the instructor was using in homework sets when the instructor edited a PG file. That was fixed in a recent pull request, but since you are mentioning restarting the server it is possible your server was on an earlier git commit before it was last restarted.

[somiaj]

@drgrice1: That wasn't the issue, other changes in the file would be visible, it was only the image that would not change.

I tested this again this morning and got the same behavior again. I am able to see any changes to the problem except the image. But I think I was able to track down the issue, and it is due to my browser. Firefox was just caching the image, and since the random string that is generated is the same for the problem, the image name was the same and my browser was using the cached version. A hard reload got the correct image showing. Firefox doesn't seem to cache the image right away when I was actively testing it, but if I wait a bit and came back, it would use its cached version.

Thanks again for your support.