Support literals encoding conversions according to the literal type. #162

AhmedMaher309 · 2024-08-03T20:29:01Z

Added the support for encoding conversion for National literals to be in UTF-16 and for alphanumeric literals to be in ISO-8859-15 and for the UTF-8 literals to be in UTF-8

Changes in details:

Handling of single-byte and UTF-8 encoded files in terms of literals construction
Implementation of internal conversions between ISO-8859–15, UTF-8 and UTF-16 according to literals type (Alphanumeric, National, and UTF-8 literals)
Enhanced support for National literals stored as UTF-16, and the alphanumeric literals as ISO-8859-15
New function for building UTF-8 literals to be stored as UTF-8
Command-line argument for specifying source file encoding or using the locale if no encoded specified in the command line

GitMensch

while the overall approach is fine (misses documentation and tests for utf8, but both should come after the new command line argument) there are some things to adjust as detailed below; please try to get to those until Tuesday and push when finished

cobc/typeck.c

tests/testsuite.src/syn_definition.at

cobc/typeck.c

cobc/tree.h

cobc/scanner.l

cobc/tree.c

cobc/typeck.c

GitMensch

nice next commits; testcase for that feature and Changelog for the last commits seems too be missin

cobc/cobc.c

cobc/flag.def

GitMensch · 2024-08-10T14:37:18Z

In this case (after my push) opening the file will create an error if that was really the encoding, and iconv will return an error on first use if the given encoding does not match the input.

GitMensch · 2024-08-10T21:36:31Z

cobc/ChangeLog

+2024-07-06 Ahmed Maher <ahmed.maher.ahmed.309@gmail.com>
+
+	* tree.c: added encoding conversion support to cb_build_national_literal and
+	cb_build_alphanumeric_literal


indent by two spaces

GitMensch · 2024-08-12T19:21:14Z

@AhmedMaher309: I've fixed the merge conflict for the Changelog so the CI can run here; note that I've also applied the Changelog format to your entries - please have a look to get an idea how this (quite commonly, especially in GNU space) is done; you may also want to check the GNU coding guide on this for a better understanding

I'm looking forward for your documentation entry on the encoding topic, even when we can't consider it for the final evaluation (because of the deadline now being over) [note: I plan to do the evaluation part until Thursday - it's just some quite busy week so far)

GitMensch · 2024-08-12T19:23:40Z

For the failing CI build - you may want to use { } around that :-)
For the UTF8 test - let's inspect that later this week.

GitMensch · 2024-08-13T08:01:24Z

I know that's a very late thought (post GSOC project timeline)...

But we definitely should add the option to set cb_iconv.alphanumeric via command line - otherwise we force it to ISO-8859-15 where previously people did use CP850 on Windows to draw TUI symbols which won't work any more and return iconv errors as these are not part of iso-8859-1 at all.
... and of course this may also have an effect at runtime... so we should at least add this to the cob_module structure as char *alphanumeric_encoding and set this if it isn't ISO-8859-15. A later addition in libcob would then need to handle a very specific case: encodings that have digits at the non-native place - 0x30-x39 vs: 0xF0-xF9, for ease this likely should be a separate flag and the conversion of digits won't need iconv but a simple lookup table.

GitMensch · 2024-08-21T10:27:51Z

You haven't seem to pushed your last changes, do you?

…Windows and Macos Co-authored-by: Fabrice Le Fessant <fabrice.le_fessant@ocamlpro.com>

…bol-3.x` Workflow setup shall be retargetted to branch `gnucobol-3.x` upon merge

to prevent this to be caught late, as happened with OCamlPro#85

…s_literal in one function cb_build_literal_by_category

cobc: * cobc.c, codegen.c, tree.c: fixed C89 errors * typeck.c (trimmed_size, is_blank): minor refactorings

* cobc.c, codegen.c, tree.c: fixed C89 errors * typeck.c (trimmed_size, is_blank): minor refactorings * typeck.c: provide the original buffer if encoding cannot be applied run_functions.at: testsuite drafts (currently failing) for encoding related issues testsuite environment update tests: * atlocal.in (set_utf8_locale): new function for the testsuite enabling tests to either run with UTF8 locale or skip * atlocal.in: use configure-setup for the grep binary * atlocal_win: updated to current atlocal.in [to be used for encoding and possibly screenio tests]

GitMensch · 2025-01-19T17:41:37Z

cobc/cobc.h

@@ -214,6 +221,7 @@ enum cb_std_def {
 	CB_STD_BS2000,
 	CB_STD_ACU,
 	CB_STD_RM,
+	CB_STD_GCOS,


drop this line...

GitMensch · 2025-01-19T17:41:48Z

tests/testsuite.src/configuration.at

@@ -331,7 +331,7 @@ AT_CHECK([$COMPILE_ONLY \
 -fstandard-define=99 prog.cob], [1], [],
 [configuration error:
 -fstandard-define=99: invalid value '99' for configuration tag 'standard-define';
-	maximum value: 9
+	maximum value: 10


... and that line

GitMensch requested changes Aug 4, 2024

View reviewed changes

GitMensch reviewed Aug 5, 2024

View reviewed changes

cobc/typeck.c Outdated Show resolved Hide resolved

GitMensch requested changes Aug 9, 2024

View reviewed changes

cobc/cobc.c Outdated Show resolved Hide resolved

cobc/cobc.c Outdated Show resolved Hide resolved

cobc/flag.def Show resolved Hide resolved

GitMensch reviewed Aug 10, 2024

View reviewed changes

AhmedMaher309 marked this pull request as ready for review August 11, 2024 13:02

AhmedMaher309 force-pushed the gcos4gnucobol-3.x branch from 5a2a864 to 7efa05c Compare September 29, 2024 21:14

ddeclerck and others added 19 commits January 19, 2025 17:56

GIT-specific settings, with CI setup and github workflow for Ubuntu, …

ebc0e65

…Windows and Macos Co-authored-by: Fabrice Le Fessant <fabrice.le_fessant@ocamlpro.com>

[GCOS] Add GCOS configuration file

dfb449a

Redispatch ChangeLog entries

50aedb0

Thanks OCamlPro contributors

dd0e532

Improve setup for CI jobs, with temporary focus on branch `gcos4gnuco…

acc6b97

…bol-3.x` Workflow setup shall be retargetted to branch `gnucobol-3.x` upon merge

Use github actions to emit a distribution archive and test logs

05bdc7b

Use ubuntu CI to measure coverage

270a5c6

Adjust macos & windows CIs

b183d9f

Disable automated windows CI workflow

45befa5

Fix handling of quotes in testsuite artifact name

0e2b01c

Fix ubuntu CI

85251d9

Add working Github action files, except for Windows (OCamlPro#79)

ecbb7d7

CI: check for c89 declaration (OCamlPro#97)

54e2a7a

to prevent this to be caught late, as happened with OCamlPro#85

Add/Update Windows workflows

f36a5c2

Update MSVC & MSYS1 CI

9163932

Upgrade versions of github actions used in CI

6019858

Adjust MSVC workflow now that error popups are disabled by default

4bfd3d5

Further CI adjustments

64a6827

Enforce a 45 minutes timeout on Windows CI workflows

7688453

AhmedMaher309 and others added 26 commits January 19, 2025 18:34

refactor scanner.l

32b50aa

merged cb_build_alphanumeric_for_figurative_constant and cb_build_gco…

1d2b225

…s_literal in one function cb_build_literal_by_category

added the command line argument for the source encoding

cf63837

changed the default source encoding from UTF-8 to ISO-8859-15

a65c5b0

refactor the command line argument for source encoding

beed135

reverted the changes for validate_alphabet

4cb65f5

code refactoring

b29282e

adding the test for utf8

2006da4

replaced spaces indentation with tabs

c97f8d2

refactored command line option for source encoding

576a0a6

added a chapter for encoding handling

9f5df14

modified the chapter for encoding handling in doc/gnucobol.texi

4e7e9ee

updated doc/gnucobol.texi

047ac11

fixed doc/gnucobol.texi

c133620

update gnucobol.texi file

a1a8fc3

minor adjustments to character encoding code

c2b488d

cobc: * cobc.c, codegen.c, tree.c: fixed C89 errors * typeck.c (trimmed_size, is_blank): minor refactorings

added command line option for the alphanumeric literals encoding

9716d84

added command line option for the alphanumeric literals encoding

79ae168

fixing build and test issues

8126cb3

fixing typeck/validate_alphabet

11ed412

fixed build errors

6691015

removed yywrap function

78b4b30

fixed the config directory

d9e22c6

fixed merge conflicts

01aab23

fix merge errors

b95e404

AhmedMaher309 force-pushed the gcos4gnucobol-3.x branch from c291c69 to b95e404 Compare January 19, 2025 17:35

GitMensch reviewed Jan 19, 2025

View reviewed changes

AhmedMaher309 added 2 commits January 19, 2025 19:46

fixing merge conflicts

44091e6

fixing merge conflict error

a9b05d2

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support literals encoding conversions according to the literal type. #162

Support literals encoding conversions according to the literal type. #162

AhmedMaher309 commented Aug 3, 2024 •

edited

Loading

GitMensch left a comment

GitMensch left a comment

GitMensch commented Aug 10, 2024 via email

GitMensch Aug 10, 2024

GitMensch commented Aug 12, 2024

GitMensch commented Aug 12, 2024

GitMensch commented Aug 13, 2024

GitMensch commented Aug 21, 2024

GitMensch Jan 19, 2025

GitMensch Jan 19, 2025

Support literals encoding conversions according to the literal type. #162

Are you sure you want to change the base?

Support literals encoding conversions according to the literal type. #162

Conversation

AhmedMaher309 commented Aug 3, 2024 • edited Loading

GitMensch left a comment

Choose a reason for hiding this comment

GitMensch left a comment

Choose a reason for hiding this comment

GitMensch commented Aug 10, 2024 via email

GitMensch Aug 10, 2024

Choose a reason for hiding this comment

GitMensch commented Aug 12, 2024

GitMensch commented Aug 12, 2024

GitMensch commented Aug 13, 2024

GitMensch commented Aug 21, 2024

GitMensch Jan 19, 2025

Choose a reason for hiding this comment

GitMensch Jan 19, 2025

Choose a reason for hiding this comment

AhmedMaher309 commented Aug 3, 2024 •

edited

Loading