-
Notifications
You must be signed in to change notification settings - Fork 17
/
user_guide.shtml
836 lines (576 loc) · 40.8 KB
/
user_guide.shtml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
<html>
<head>
<meta http-equiv="Content-Language" content="en-us">
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
<title>Pythonizer user guide</title>
<style>
body {Background-color: #FFFFFF link: #0000CC vlink: #660066}
h1 { font-family: arial; text-align: Center; color: #CC0000 }
h2 { font-family: arial; text-align: Center; color: #0000CC }
h3 { font-family: arial; text-align: Center }
h4 { font-family: Arial; color: #006400}
h5 { font-family: Arial; color: #333300; font-style: italic}
h6 { font-family: Arial; font-size: 10pt; font-style: bold}
em { font-style: italic; font-weight: bold; color: #FF0000}
cite { font-style: italic; font-weight: bold; color: blue }
a {color: #0000FF;}
td {font-family:arial,helvetica,sans-serif; font-size:10pt;}
p.petit { font-family: arial; font-size: smaller }
table.swb {Background-color: cyan}
td.swb { border-right-style: solid; border-bottom-style: solid }
table.nb {Background-color: #ffffcc}
tr.nb { text-align: Center; font-weight: bold }
td.nb { border-style: solid }
kbd {font-family: Fixedsys; font-size: 12pt; color: #0000FF;}
tt {font-family: Fixedsys; font-size: 12pt; color: #0000FF;}
code {font-family: Fixedsys; font-size: 12pt; color: #0000FF;}
pre {font-family: Fixedsys; color: #0000FF; font-size: 12pt; margin: 1em 40px; }
pre.code {font-family: Fixedsys; color: #0000FF; margin: 1em 40px;}
blockquote {font-size: 10pt; font-family: arial; background: #F9F9F9}
</style>
</head>
<body>
<h1>Pythonizer user guide </h1>
<h3>Version 0.82 (Oct 06, 2020)</h3>
<table border="1" width="100%" bgcolor="#FFFFCC">
<tr>
<td width="14%" align="center" bordercolorlight="#000000" bordercolordark="#FFFFFF"><a href="#News">News</a></td>
<td width="14%" align="center" bordercolorlight="#000000" bordercolordark="#FFFFFF"> <a href="../index.shtml">Python for
Perl programmers</a></td>
<td width="14%" align="center" bordercolorlight="#000000" bordercolordark="#FFFFFF">
<p align="center"><b><a href="../../../../Bookshelf/Computers/python.shtml">Best Python books for system administrators</a></b></p>
</td>
<td width="14%" align="center" bordercolorlight="#000000" bordercolordark="#FFFFFF"><a href="#Recommended_Links">Recommended Links</a></td>
<td width="14%" align="center" bordercolorlight="#000000" bordercolordark="#FFFFFF"><b>
<a href="../Pl2py_reference/pl2py_functions_map.shtml">Perl
to Python functions map</a></b></td>
<td width="15%" align="center" bordercolorlight="#000000" bordercolordark="#FFFFFF"><b>
<a href="../Py_shell_commands_execution/index.shtml">Execution of commands and shell scripts using subprocess module</a></b></td>
<td width="15%" align="center" bordercolorlight="#000000" bordercolordark="#FFFFFF"><b>
<a href="protocol_of_translation_of_pre_pythonizer.shtml">Full protocol of translation of pre_pythonizer.pl by the current
version of Pythonizer</a></b></td>
</tr>
</table>
<p><em>NOTE: In version 0.8 you need to specify PERL5LIB variable pointing it to the directory with modules to run the
program. See installation section below</em></p>
<ul>
<li><b><a href="#Introduction">Introduction</a></b></li>
<li><b><a href="#Pre-pythonizer__implements_the_first_phase_of_translation">Pre-pythonizer implements the first phase of
translation</a></b></li>
<li><b><a href="#Pythonizer_implements_actual_transformation_of_Perl_into_Python">Pythonizer implements actual transformation of
Perl into Python</a></b><ul>
<li><b><a href="#Options">Options</a></b></li>
<li><b><a href="#Logs">Logs</a></b></li>
<li><b><a href="#Structure">Structure</a></b></li>
<li><b><a href="#Installation">Installation</a></b></li>
<li><b><a href="#Invocation">Invocation</a></b></li>
<li><b><a href="#History">History</a></b></li>
<li><b><a href="#Known_errors">Known errors</a></b></li>
<li><b><a href="#Submission_of_tickets">Submission of tickets</a></b></li>
</ul>
</li>
<li><b><a href="#Recommended_manual_transformations_of_Perl_code">Recommended manual transformations of Perl code</a></b><ul>
<li><a href="#Decision_to_remove_round_brackets">Decision to remove opening and closing round brackets
in Perl built-in function opened a can of worms</a></li>
<li><a href="#Other_recommended_transformation_of_Perl_code">Other recommended transformations of Perl code</a></li>
</ul>
</li>
<li><b><a href="#Debugging_generated_Python_code">Debugging generated Python code</a></b></li>
</ul>
<hr noshade color="#FF0000" width="33%" size="7">
<h3><a name="Introduction">Introduction</a></h3>
<p>Some organizations are now involved in converting their old Perl codebase into other scripting languages, such as Python. But a more common task is to maintain existing
Perl scripts, when the person who is assigned to this task known only Python. </p>
<p>University graduates now typically know Python but not Perl and that creates difficulties in the old codebase maintenance.
In this case, a program that "explains" Perl constructs in Python term would be extremely useful and, sometimes, a lifesaver. Of
course, Perl 5 is here to stay (please note what happened with people who were predicting the demise of Fortran ;-), and in most
cases, old scripts will stay too.</p>
<p>The other role is to give a quick start for system administrators who want to learn Python (for example, need to support
researchers who work with Python), but who currently knows only Perl -- many older school sysadmins dislike Python and for a reason
;-) </p>
<p>Yes another role is provide a proof that those two languages are mostly compatible and that program from one can be
translated into another with modest amount of effort. Although such translation is not necessary a best fit, in most cases it
is close enough and needs just minor manual editing. </p>
<p>Of course, complex constructs and idioms are often translated incorrectly. Several complex issues remains unresolved (implicit
conversion to strings in Perl is one such issues.) As experience with Google translation of natural languages attests there
are always around 10 to 20% of sentences (depending of the subject area of the text) that are translated incorrectly. And,
probably, 2-3% that have absurd or funny translation. </p>
<p>The idea here is that using "fuzzy translation" concept it is possible to create such a tool with relatively modest efforts. A tool, written with some knowledge of
compiler technologies, that falls into the category of "small language compliers" with the total effort around one man-year or less. Assuming ten lines per day of debugged code for the task of complexity comparable with the writing of compilers, the estimated
size should be around 3-5K lines of code (~1K line phase 1 and 2-3K line phase 2).</p>
<p>As of version 0.8 it looks like the initial idea was a sound one: within 5K LOC limit it is possible to create a
useful utility that transcribes Perl in Python with a decent quality. </p>
<p>As the currently code base exceeded 4K lines, it is close to the limit on which I can maintain this codebase as a hobby project, so
some enhancements need to be abandoned or moved to the pre-pythonizer phase. This first of all is true about insertion
of proper conversion of types. Right now it is left to the programmer to fix those issues. </p>
<p><em>NOTE: </em>Pythonizer now creates the list of variables that need to be declared global to preserve a part of the
global namespace that Perl subroutines access. This is a needed function as Python has different rules of variables visibility --
global variables are visible in all subroutines, but to be able to modify them you need to declare them global explicitly (and this
global variable should already exist and be initialized) </p>
<h3><a name="Installation">Installation</a></h3>
<p>For testing the main program and all modules should reside in a single directory from which you can run the program. </p>
<p>You need to download files or replicate the directory via git . Let's assume that you downloaded or replicated them into
<tt>/home/softpano/Pythonizer</tt><tt> </tt></p>
<p>After that you can run <tt>pythonizer</tt> from the install directory by specifying <tt>-I /home/softpano/Pythonizer</tt> option for Perl interpreter
to point to proper directory with modules
For example:</p>
<pre>perl -I /home/softpano/Pythonizer ~/Pythonizer/pythonizer script_to_be_translated.pl</pre>
<p>Alternatively you can specify PERL5LIB environment variable to point to this directory</p>
<font SIZE="3">
<pre>export PERL5LIB=<tt>/home/softpano/Pythonizer</tt></font></pre>
<p>If you already have this variable defined in your bashrc you can add this directory to the end after column, for example</p>
<pre>PERL5LIB=/home/softpano/perl5/lib/:<font SIZE="3">PERL5LIB=</font><tt>/home/softpano/Pythonizer</tt></pre>
<p>After that you do not need to use option -I of Perl interpreter and can invoke the program directly</p>
<pre>~/Pythonizer/pythonizer script_to_be_tranlated.pl</pre>
<p>In order to specify <tt>pythonizer</tt> without path you need to add the directory in which you installed it to the <tt>PATH </tt>
or copy the program to one of the directories already in <tt>PATH</tt>, for example<tt> /use/local/bin i</tt>f you are root<tt> or
~/bin </tt>if you are a regular user<tt>.</tt></p>
<p>Version 0.8 has a simpler installer using which you can quickly move the main script
and module to proper places in your directory tree. It is called <tt>installer</tt> and accept two arguments:</p>
<ul>
<li>
<p><b>path_to_pythonizer</b> -- directory in which you want to put <tt>pythonizer</tt> and <tt>
pre_pythonizer.pl</tt></li>
<li>
<p><b>path_to_modules</b> -- directory in which you want to put modules (should be defined iether in <tt>@INC</tt>
or in environment variable <tt>PERL5LIB</tt>) </li>
</ul>
<p>General format of invocation: </p>
<pre>installer path_to_pythonizer path_to_modules</pre>
<p>The directory with pythonizer and modules should be current. For example: </p>
<pre>cd ~/Pythonizer && installer /user/local/bin /user /usr/local/share/perl5</pre>
<p>By default if run as root it assumes <tt>/usr/local/bin</tt> as the target directory for pythonizer and<tt> /usr/local/share/perl5</tt>
for modules <p>If run as a regular user defaults are <tt>~/bin</tt> and <tt>~/Perl5/lib</tt>
<h3><a name="Invocation">Invocation</a></h3>
<p>You can run Pythonizer both in Cygwin and Linux. To "pythonize" the Perl script you need to set <font SIZE="3">
directory with modules in -I option of Perl interpreter, PERL5LIB, or to put modules in some directory already in @INC as
discussed in the <a href="#Installation">Installation</a> section.</font></p>
<p>The simplest invocation can look like: <font SIZE="3"> </font></p>
<pre><font SIZE="3">export PERL5LIB=</font>/opt/Pythonizer && /opt/Pythonizer/pythonizer /path/to/your/program/script_to_be_translated.pl</pre>
<p>If the program runs to the end you will get "pythonized" text in <tt>/path/to/your/program.py</tt></p>
<p>It also produces protocol of translation in <tt>/tmp/Pythonizer</tt> with size by side Perl and Python code, which allows you to
analyses the protocol detect places that need to be commented out translated
manually. </p>
<p>If <tt> __DATA__</tt> or <tt>__END__</tt> are used a separate file with the extension <b>.data </b> (<tt>/path/to/your/program.data </tt>
for the example above) will be created with the content on this section of Perl script.</p>
<h3><a name="Pre-pythonizer__implements_the_first_phase_of_translation">Pre-pythonizer implements the first phase of
translation</a></h3>
<p>Processing consists of two passes, which currently are not integrated in any away with <tt>pre_pythonizer</tt> mainly providing
refactoring of Perl code to create <tt>main</tt> procedure and move all subroutines up so that they are now defined before use. </p>
<p>Running this phase also reformat the Perl script in a way that slightly increases chances that the script will be translated with
fewer errors. Opening and closeting curvy bracket are put on single lines to ease the job of the lexical scanner in Pythonizer. </p>
<p>It can be used
iether as separate utility of in integrated way via <tt>-r</tt> (<b><i>refactor</i></b>) option in pythonizer. </p>
<pre>pre_pythonizer [options] <file> # currently performs is refactoring of Perl script,
# pushing subroutines to the top and creating <tt>main</tt> sub
# out of code not included into any subroutine.
pythonizer [options] <file></pre>
<p>The first pass is currently fully optional and need transformations of Perl code can be performed by other utilities.
It just slightly increase probability of more correct translation of the code. It reformat the code so that
curvy brackets were mostly on separate lines (this was essential only for pythonizer up to version 0.2; later versions do not depend
directly on this transformation.) </p>
<p>It can be used as a separate program, which transforms initial Perl script creating a backup with the extension <tt>.original.</tt>
The main useful function in the current version is refactoring of the Perl program by pushing subroutines up as in Python
subroutines needs to be declared before use. </p>
<p>It needs to run only once for each Perl script you want to translate to Python. Subsequence modifications of Perl script to make
it more "Python-compatible" can be performed on this text instead of the original script. </p>
<h3><a name="Pythonizer_implements_actual_transformation_of_Perl_into_Python">Pythonizer implements actual transformation of Perl
into Python</a></h3>
<p>Running <tt>pythonizer -h</tt> provides a list of options. </p>
<p>Here is example of the protocol
<a href="protocol_of_translation_of_pre_pythonizer020.shtml">Full protocol of translation of pre_pythonizer.pl by version 0.2 of
pythonizer</a></p>
<p>To increase chances of correct translation it is recommended to run the Perl script via
<span>
<a title="pre_pythonizer.pl" id="9413911e31717187d3a97255af3e0f95-f911160659ba8e3129cfb73f329fb2fd14950434" href="https://github.com/softpano/pythonizer/blob/master/pre_pythonizer.pl">
pre_pythonizer.pl</a></span></p>
<p><span>Parts that can't be translated during the first invocation can
be commented outs and iteratively you can modify simplify Perl code and reach the stage when the Perl script is translated more or
less OK. After that you can work in Python debugger and iron remaining inconsistencies and cases of incorrect translation of
statements. Sometimes algorithm of a part of the program need to be adapted to Python. </span></p>
<p><b><span>Some features:</span></b></p>
<ul>
<li>The Pythonizer is implemented in Perl 5.10+ and does not use any non standard Perl libraries. </li>
<li>Converts most of Perl 5 code typical for sysadmin scripts into Python. That includes loops, special variables, function
calls, lists, hashes, etc.</li>
<li>Recognizes some Perl idioms and special cases. Perform limited peephole optimization of Perl within individual statements in case this is possible (for example use
of replace function instead or regular expression in case both the source and target are plain strings. it also optimize some types of split function when the regex supplied to Perl actually is a string and does not contain any
metacharacters. </li>
<li>Whenever there is no equivalent Python function for a Perl function, the function is emulated. Currently this is done only in
cases when this does not require additional Python library.</li>
<li>Perl special variables are renamed to selected Python names and used consistently. But that does not mean that they are
correctly populated. </li>
<li>Limited translation of double quoted literals with scalars inside list elements and hash elements and here-documents is
supported. Most common special variable are also translated to their equivalents in double quoted operators. </li>
<li>Arguments to Perl subroutines are emulated via lists</li>
<li>In case of Perl control constructs that are missing in Python (for example <tt>(debug>0) && say $line</tt>; or <tt>open(SYSIN,'<',',$mytext)
|| die("Can't open the file $mytest\n");<span lang="ru">) </tt></span>they are translated into equivalent Python
constructs. There are some exceptions, for example Perl <tt>continue</tt> statement, but they are rarely used. </li>
<li>Tail data after <tt>__DATA__</tt> or <tt>__END__ </tt> are converted to a separate file with the same name as script and<tt>
.data</tt></li>
<li>List of statements for which pythonizer encountered difficulties and most probably did not translate correctly are produced
at the end. </li>
</ul>
<h4><a name="Limitations">Limitations</a></h4>
<ul>
<li>No OO constructs such as <tt>bless</tt> and <tt>package</tt> statement are translated. The <tt>use</tt> statement and import
and expert of variables from modules also need to be translated manually. </li>
<li>The content of BEGIN, END blocks is translated but blocks itself need to be emulated manually. Please note
that these blocks are executed at the time of execution of Perl interpreter, not at the time of execution of the script, so
opportunities to match Perl semantic here are limited. </li>
<li>References are not translated correctly. In many cases of usage of references in Perl this is very difficult to express
in Python </li>
</ul>
<h3><a name="Options">Options</a></h3>
<p><em>NOTE: </em>All options with small numeric values can be expressed by repeating the letter, so<tt> -p 2</tt> is equivalent to <tt>-pp.</tt>
<tt>-d 3 -ddd</tt></p>
<p>Currently only few user options are supported (<tt>pythonizer -h</tt> provides a list of options): </p>
<ul>
<li><tt>-p</tt> -- The version of Python used for generation. Default is 3.8 (can be set explicitly by specifying <tt>-p 3</tt>)
You can set 2.7 by using <tt>-p 2</tt></li>
<li><tt>-v</tt> -- verbosity <tt>-v 0</tt> -minimal verbosity <tt>-v 3</tt> -- max verbosity; can also be expresses as <tt>-v <tt>
</tt>-vv</tt> and <tt>-vvv </tt>-- forms to which Unix user got used in other utilities</li>
<li><tt>-t</tt> -- tab size for generated code; the default<tt> -t 4</tt> </li>
<li><tt>-r</tt> -- refactor Perl code. If specified, before processing the script by the pythonizer, it invokes the <tt>
pre_pythonizer</tt>
preliminary pass on the source code. Of course this can be done manually
with more control but this provides an integrated way to refactor the program. </li>
</ul>
<p><b>Options for developers</b></p>
<ul>
<li><tt>-d</tt> -- debug option <em>for developers</em>; default<tt> -d 0</tt> </li>
<li><tt>-b</tt> -- breakpoint option<em> for developers</em>; default <tt>-b 99999</tt></li>
</ul>
<p>The same options work for the <tt>pre_pythonizer</tt>, but usually defaults are OK. There is no options to control
refactoring of the script. </p>
<h3><a name="Logs">Logs</a></h3>
<p>Logs are written to <tt>/tmp/Pythonizer </tt>You can redirect this to any directory via symlink.<tt> </tt>
Currently there is no option to customize the location of the log. </p>
<h3><a name="Structure">Structure</a></h3>
<p>Pythonizer consists of the main program called, as you can guess, <tt>pythonizer</tt> and three modules (which currently need to
reside in the same directory as the main program. Main program currently used three modules, which should be located so that the
main program can load then from @INC array set of directories: </p>
<ul>
<li><tt>Perlscan.pm</tt></li>
<li><tt>Pythonizer.pm</tt></li>
<li><tt>Softpano.pm</tt></li>
</ul>
<p>The total size of the codebase in version 0.8 is around 4.3K lines: </p>
<pre>
# wc -l Perlscan.pm Pythonizer.pm pythonizer Softpano.pm pre_pythonizer.pl
1123 Perlscan.pm
474 Pythonizer.pm
1615 pythonizer
340 Softpano.pm
797 pre_pythonizer.pl
4349 total
</pre>
<p>The total size of the codebase in version 0.7 is slightly over 4K lines:
<pre>
wc -l Perlscan.pm Pythonizer.pm pythonizer Softpano.pm pre_pythonizer.pl
1107 Perlscan.pm
382 Pythonizer.pm
1515 pythonizer
339 Softpano.pm
825 pre_pythonizer.pl
4168 total</pre>
<p>The total size of the codebase in version 0.5 is slightly below 4K lines:
</p>
<pre>
wc -l Perlscan.pm Pythonizer.pm pythonizer Softpano.pm pre_pythonizer.pl
1051 Perlscan.pm
317 Pythonizer.pm
1442 pythonizer
336 Softpano.pm
825 pre_pythonizer.pl
3971 total</pre>
<p>The total size of the codebase in version 0.4 is around 3.5K lines: </p>
<pre>wc -l Perlscan.pm Pythonizer.pm pythonizer Softpano.pm pre_pythonizer.pl
866 Perlscan.pm
292 Pythonizer.pm
1268 pythonizer
312 Softpano.pm
833 pre_pythonizer.pl
3571 total</pre>
<h3><a name="Recommended_manual_transformations_of_Perl_code">Recommended manual transformations of Perl code</a></h3>
<p>Recently Perl adopted Python stance of minimizing the number of parenthesis in standard function</p>
<p>The first and foremost under-parenthesized Perl script (the scripts which uses built-in functions without parenthesis, like
recommended by "Modern Perl" are translated with more errors by the current version of Pythonizer then in cases then they are
parenthesized. This topic is discussed below in more derails and some examples provided. </p>
<p>Of course, cases are different and a single function as statement is pretty safe without parenthesis. But in complex expression
your mileage can vary. In cases where Pythonizer complains adding parenthesizes often allows the correct translation. </p>
<p>Also sometime complex expressions are parsed incorrectly by Pythonizer. In such case you can factored then out and simplify in
order to get correct translation. Or translate them manually as in most such cases the proper Python equivalent is different from
close to "one to one" translation that Pythonizer provides, anyways. </p>
<p>But this is to be expected due to limitation of
the size and the complexity of this project. </p>
<p>Generally, Perl provide too much syntax rope and some programmers manage to hang themselves with it: the syntax variety allowed is just an
overkill and stimulate adoption of "perverted" constructs which detract from, not add to the clarity of the code. </p>
<p>The most typical
example of this kind of "syntactic perversions" is the excessive use of postfix if constructs. I would understand using them in loop to specify exist condition like </p>
<blockquote>
<pre>last if (substr($line,0,3) eq 'EOF');</pre>
</blockquote>
<p>but not like </p>
<blockquote>
<pre>$x++ if $i<$limit;</pre>
</blockquote>
or
<pre>
print SQL "SELECT column 1 FROM $var" if (defined $var);
</pre>
<p>in the first case writing</p>
<blockquote>
<pre>if $i < $limit { $x++; }</pre>
</blockquote>
<p>is only two symbols longer but is much clearer and modifiable. Pythonizer recognizes single statements with postfix if or
other conditional modifiers, and instead of single statement a comma separate list can be provided making them, essentially, a
block: </p>
<pre>$i=5,$j=10 if( $next iteration);</pre>
<p>IMHO provides way too much syntax flexibility ;-).You need to transform your such statements to prefix if statements if you want automatic
translation. Actually the result looks more readable</p>
<pre>if( $next iteration ){
$i=5;$j=10
}</pre>
Similary Pythonizer will complain in case of cacading my declarations:
<pre>
my $i=my $j=1
</pre>
<p>But this probably makes some sence so it might be corrected in later versions.
<h3><a name="Decision_to_remove_round_brackets">Decision to remove opening and closing round brackets<br>
in Perl built-in function opened a can of worms</a></h3>
<p>Situation with under-parenthesized built-in function, that is mentions above, is more complex and here I think Perl implementers made a blunder opening a can of
worms. Many statements written in this fashion are ambiguous. </p>
<p>So we will have problem with the language itself. Which exists independently of the level of the upper level complexity allowed
in Pythonizer and its limitations. </p>
<p>For example:</p>
<blockquote>
It looks like with parenthesis elimination drive Perl developers into the situation similar to Chinese situation with the
elimination of sparrows.<p>Today I encountered the following ambiguous statement written in "de-parenthesized" "Modern Perl":</p>
<pre>get_config(split /$s/,$_);</pre>
<p><a href="https://perlmonks.org/?abspart=1;displaytype=displaycode;part=1;node_id=11122556">[download]</a></p>
<p>Where it is completely unclear if</p>
<ol>
<li>We have two arguments to <tt>get_config</tt> and one argument to <tt>split</tt> (regular expression <tt>/$s/<tt>,
and split is assumed to be operating on $_) </li>
<li>We have two arguments <tt>to split (/$s/ </tt>and<tt> $_ )</tt> and one argument (expression) to <tt>get_config</tt>.
</li>
</ol>
<p>It is accepted by interpreter as case (2) which
IMHO is correct, but still pretty arbitrary, especially if you write
<pre>get_config(split /$s/,$i)</pre>
<p>instead.</p>
<pre>
DB<100> sub get_config{ print scalar(@_), join(' ',@_),"\n"}
DB<101> $s=' ';
DB<102> $_=qq(abba baba);
DB<103> get_config(split /$s/,$_);
2abba baba</pre>
<p>Unless this problem is fixed, this is an argument for sticking to earlier versions of Perl 5 and avoiding "de-parenthesized"
"Modern Perl". Looks like "Parenthesis eliminators" essentially harmed the language, while trying to help. That's happens, but
this is a rather sad situation.</p>
<p>In any case, versions after 5.26 probably will reach commercial Linuxes in 5 to 10 years time-frame so there is still a time
to fix this.</p>
</blockquote>
<h3><a name="Other_recommended_transformation_of_Perl_code">Other recommended transformations of Perl code</a></h3>
<p>While parenthesizing built-in function in statements which caused problems for Pythonizer is number one recommendation, there are
several other. </p>
<p>In many case you can get more correct translation via Pythonizer by simplifying Perl code. Some transformation can be done via macro generator like
M4 some via Perl itself using regular expressions.</p>
<p>If pre-pythonizer transformation of Perl code produces errors you need to fix them before submitting
the code to pythonizer.</p>
<p>Pythonizer expects syntaxically correct Perl code. </p>
<p>If it hangs or goes into infinite loop, then you need to comments out the offending line(s) and try again. You can use option -d
3 to find the line if it is not clear from the listing. </p>
<p><em>NOTE:</em> To increase changes of success in translation of long scripts with translation of which you encounter problems. you can split your code into several chunks (for example main program and subs) and try to convert
them chunk by chunk. Then you can merge the code and add global declarations manually. </p>
<p>Also the old <tt>open </tt>statement where the <i><b>opening mode</b></i> is prefixed to the file is better converted to
the new mode when it is a separate argument as not all modes are correctly recognized in the old format. Typically there are very
few open statement in the program so that can easily be done. </p>
<p>Excessive syntax flexibility might helps to explains flimsy error reporting by Perl interpreter, when it accepts clearly
syntaxically incorrect programs as valid. This situation is worse in the debugger then in interpreter itself. </p>
<p>Version 5.26.3 also refuse to function properly in some cases, probably hitting some internal bugs or corruption on internal data
struictures, which demonstrated itself that a certain statement was executed incorrectly. But it recovers if you change
your Perl code a little bit. I experienced one such situation with <tt>pythonizer</tt>, when it was slightly less 900 lines. Luckily
as I added code the problem disappeared. But I seriously I thought about
switching to GOlang at this point. </p>
<h3><a name="Debugging_generated_Python_code">Debugging generated Python code</a></h3>
<p>Python debugger is still inferior to Perl debugger, so working with it is more difficult and time consuming. </p>
<p>The standard invocation is something like </p><font SIZE="3">
<pre>python3.8 -m pdb maintest.py</pre>
</font>
<p>You can run the script till the first error with the command 'c' and if you are lucky at this point you have some useful
information what is wrong from the diagnostic message produced by Python interpreter. If not, you need to work step by step
and determine the context of the error yourself </p>
<p>See <a href="../../Debugging/index.shtml">Python Debugging</a> for some additional information and tips. </p>
<p>Generally to find the fist error you can run the generated code with the command c and proceed from this point. </p>
<p>But the fact that code runs to this point does not mean that the code executed correctly: you need to verify that. </p>
<p>For some additional ideas see <a href="../translating_perl_to_python.shtml">Perl to Python translation</a></p>
<blockquote>
<h3><a name="Known_errors">Known errors</a></h3>
</blockquote>
<ol>
<li>Pythonizer does not like some statements and standard functions calls with omitted round brackets (especially in complex
expressions) and can mark then untranslatable, while translating them correctly if
brackets are added. </li>
<li>For long scripts to increase changes of success you can split your code into several chunks (for example main program and subs)
and try to convert them chunk by chunk. </li>
<li>In version 0.8 filehandles are not included in the list of global variables and as such are not propagated correctly of file
operations on a given file handle spread into several different subroutines. </li>
<li>State variable are now simply mapped to global namespace which can create conflicts and corrupt their value at run time. So
the usage of state variable currently needs to ne manually reviewed to detect such conflicts. They are rare but they happen. One possible solutions to rename them using sub name as the prefix. For example in sub
<tt>maxi</tt> the state variable <tt>limit</tt>
can be renamed into <tt>maxi_limit</tt></li>
</ol>
<h3><a name="Submission_of_tickets">Submission of tickets</a></h3>
<p>Tickets can be opened on GitHub. </p>
<p>The ticket should be reproducible on the most recent uploaded version; non reproducible tickets related to earlier version will be ignored.
Information proved should be enough to reproduce the error: </p>
<ul>
<li>If case of internal errors, for example rejection of a valid statement, incorrect translation, or unlimited recursion please provide the current line and the context (say
10 "before" and 10 "after" lines.) </li>
</ul>
<p>It is recommended that you run pythonizer with the debugging option <tt>-d 3</tt> to generate additional output relevant fragment
(only relevant fragment) can be attaches as a file to the ticket.</p>
<p>Samples the cause internal errors also need to be attached as files to save my time. </p>
<h3><a name="History">History</a></h3>
<p><b>Version 0.8 uploaded</b></p>
<blockquote>
<p><b>Changes since version 0.7 </b></p>
<p>More correct translation of array assignments. Some non-obvious bugs in translation were fixed. Now you need to specify
PERL5LIB variable pointing it to the directory with modules to run the program. Global variable now are initialized after main sub to undef value
to create a global namespace. Previously this was done incorrectly. Simple installer for Python programmers who do not
know much Perl added: the problem proved to be useful as a help for understanding Perl scripts by Python programmers. </p>
<p><b>Changes in pre_pythonizer.pl</b></p>
<p>Unlike previous versions, the current version by default <em>does not create</em> <tt>main</tt> <em>subroutine out of statement found on nesting level
zero</em>, as it introduces some errors. You need specify option <tt>-m</tt> to create it. </p>
<p><em>NOTE:</em> All Python statements on nesting level zero should starts from the beginning of the line which is ugly, but
you can enclose them in the dummy <tt>if</tt> statement </p>
<pre>if True: </pre>
<p>to create the artificial nesting level 1</p>
</blockquote>
<p><b>Version 0.7 uploaded</b></p>
<blockquote>
<p><b>Changes since version 0.6. </b></p>
<p>This version creates of the list of global variables for each subroutine to maintain the same visibility in Python as in
Perl and generates <tt>global</tt> statement with the list of such variables that is inserted in each Python subroutine
definition if pythonizer determined that this subroutine access global variables. The list might be excessive. </p>
</blockquote>
<p><b>Version 0.6 uploaded</b></p>
<blockquote>
<p><b>Changes since version 0.5. </b></p>
<p>Regular expressions now are translated more correctly. Short cut if like <tt>(debug>0) && say $line</tt> are translated
in more general way then before. This is the first version that translates the main test (<tt>pre_pythonizer.pl</tt>) without syntax
errors. Generated source starts executing in Python interpreter till the first error. List on internal
functions created. Translation of backquotes and open statement improved. </p>
</blockquote>
<p><b>Version 0.5 uploaded</b></p>
<blockquote>
<p><b>Changes since version 0.4</b></p>
<p>
<span>Regular
expression and tr function translation was improved. Many other changes and error corrections. -r (refactor) option implemented to
allow refactoring Perl source via pre-pythonlizer.pl in integrated fashion.</span></p>
</blockquote>
<p><b>Version 0.4 uploaded</b></p>
<blockquote>
<p><b>Changes since version 0.3</b></p>
</blockquote>
<ul>
<li>Scanner is improved</li>
<li>f-strings are now generated for double quoted literals</li>
<li>Many errors fixed. </li>
</ul>
<p><b>Version 0.3 uploaded</b></p>
<blockquote>
<p><b>Changes since version 0.2: </b></p>
</blockquote>
<ul>
<li>default version of Python used is now version 3.8; </li>
<li>option -p
allows to set version 2 id you still need generation for Python 2.7 (more constructs will be untranslatable). </li>
</ul>
</blockquote>
<hr>
<!--#include virtual="/adv_news_pane.htm" -->
<hr>
<h2><a name="News">Old News</a> ;-) </h2>
<h4>[Oct 05, 2020] <a href="https://github.com/softpano/pythonizer">Version 0.8 uploaded</a> </h4>
<blockquote>
<p><b>Changes since version 0.7 </b></p>
<p>More correct translation of array assignments. Some non-obvious bugs in translation were fixed. Now you need to specify
<tt>PERL5LIB</tt> variable pointing it to the directory with modules to run the program. Global variable now are initialized after main sub to undef value
to create a global namespace. Previously this was done incorrectly. Simple installer for Python programmers who do not
know much Perl added: the problem proved to be useful as a help for understanding Perl scripts by Python programmers. </p>
<p><b>Changes in pre_pythonizer.pl</b></p>
<p>Current version by default <em>does not create</em> <tt>main</tt> <em>subroutine out of statement found on nesting level
zero</em>, as it introduces some errors. You need specify option <tt>-m</tt> to create it. </p>
<p><em>NOTE:</em> All Python statements on nesting level zero should starts from the beginning of the line which is ugly, but
you can enclose them in the dummy <tt>if</tt> statement </p>
<pre>if True: </pre>
<p>to create artificial nesting level 1</p>
</blockquote>
<h4>[Sep 18, 2020] <a href="https://github.com/softpano/pythonizer">Version 0.7 uploaded</a> </h4>
<blockquote>
<p><b>Changes since version 0.6</b></p>
<p>This version creates of the list of global variables for each subroutine to maintain the same visibility in Python as in Perl
and generates <tt>global</tt> statement with the list of such variables that is inserted in each Python subroutine
definition if pythonizer determined that this subroutine access global variables. </p>
<p>So far the specifics of Perl <tt>state</tt> variable is ignored and they are assumed to be yet another type of global
variables (they generally do not belong to the global namespace as while they have lifetime similar to global variables their
namespace is local). </p>
</blockquote>
<h4>[Sep 08, 2020] <a href="https://github.com/softpano/pythonizer">Version 0.6 uploaded</a></h4>
<blockquote>
<p>Regular expressions now are translated more correctly. Short cut if like <tt>(debug>0) && say $line</tt> are translated
in more general way then before. This is the first version that translates the main test (pre_pythonizer.pl) without syntax
errors. Generated source starts executing in Python interpreter till the first error. List on internal functions created.
Translation of backquotes and open statement improved. </p>
</blockquote>
<h4>[Aug 31, 2020] <a href="https://github.com/softpano/pythonizer">Version 0.5 uploaded</a></h4>
<blockquote>
<p><b>Changes since version 0.4</b></p>
<ul>
<li>
<span>Regular
expression and tr function translation was improved. </span></li>
<li>
<span>
Many</span><span> other changes and error corrections.
</span></li>
<li>
<span>-r (refactor) option implemented to
allow refactoring Perl source via pre-pythonlizer.pl in integrated fashion.</span></li>
</ul>
</blockquote>
<h4><b>[Aug 22, 2020] <a href="https://github.com/softpano/pythonizer">Version 0.4 uploaded</a></b></h4>
<blockquote>
<p><b>Changes since version 0.3</b></p>
<ul>
<li>Lexical scanner improved</li>
<li>f-strings are now generated for double quoted literals</li>
<li>Many errors fixed. </li>
</ul>
</blockquote>
<h4>[Aug 17, 2020] <a href="https://github.com/softpano/pythonizer">Version 0.3 was uploaded</a> </h4>
<blockquote>
<p><b>Changes since version 0.2: </b> </p>
<ul>
<li>default version of Python used is now version 3.8; </li>
<li>option -p
allows to set version 2 id you still need generation for Python 2.7 (more constructs will be untranslatable). </li>
</ul>
</blockquote>
<h2><a name="Recommended_Links">Recommended Links</a></h2>
<h3><a name="Top_articles">Top articles</a></h3>
<h3><a name="Sites">Sites</a></h3>
<ul>
<li><a href="../../Debugging/index.shtml">Python Debugging</a></li>
<li><a href="http://www.softpanorama.org/Scripting/Pythonorama/Python_for_perl_programmers/index.shtml">Python for Perl programmers</a></li>
<li><a href="protocol_of_translation_of_pre_pythonizer020.shtml">Full protocol of translation of pre_pythonizer.pl </a></li>
<li><a href="http://pleac.sourceforge.net/">PLEAC - Programming Language Examples Alike Cookbook</a></li>
<li><a href="https://wiki.python.org/moin/PyPerlish">PyPerlish - Python Wiki</a> (it looks like the library itself is no longer
available, only docs survived</li>
</ul>
</body>
</html>