Merge branch 'main' into 17532-nullness-issue---downcasting-should-pr…

…eserve-nullness-information

INTERNAL.md

@@ -64,15 +64,48 @@ it's a good idea to check the previous link for any old or stalled insertions in
 Update the `insertTargetBranch` value at the bottom of `azure-pipelines.yml` in the appropriate release branch.  E.g., when VS 17.3 snapped and switched to ask mode, [this PR](https://github.com/dotnet/fsharp/pull/13456/files) correctly updates the insertion target so that future builds from that F# branch will get auto-inserted to VS.
 
 ### When VS `main` is open for insertions for preview releases of VS:
-
-1. Create a new `release/dev*` branch (e.g., `release/dev17.4`) and initially set its HEAD commit to that of the previous release (e.g., `release/dev17.3` in this case).
+0. Disable auto-merges from `main` to **current** release branch, please make a change for the following file and create a pull request:
+https://github.com/dotnet/roslyn-tools/blob/6d7c182c46f8319d7922561e2c1586c7aadce19e/src/GitHubCreateMergePRs/config.xml#L52-L74
+> You should comment out the `main -> release/devXX.X` flow until step #4 is completed (`<merge from="main" to="release/dev17.13" />`)
+2. Create a new `release/dev*` branch (e.g., `release/dev17.4`) and initially set its HEAD commit to that of the previous release (e.g., `release/dev17.3` in this case).
    ```console
    git checkout -b release/dev17.4
    git reset --hard upstream/release/dev17.3
    git push --set-upstream upstream release/dev17.4
    ```
-3. Set the new branch to receive auto-merges from `main`, and also set the old release branch to flow into the new one.  [This PR](https://github.com/dotnet/roslyn-tools/pull/1245/files) is a good example of what to do when a new `release/dev17.4` branch is created that should receive merges from both `main` and the previous release branch, `release/dev17.3`.
-4. Set the packages from the new branch to flow into the correct package feeds via the `darc` tool.  To do this:
+3. Update versions in both `main` and new release branch **they need to match, so release notes bot knows which changelog file to check**
+4. Update target insertion branches in the `azure-pipelines.yml`:    
+   1. F# release branch
+   ```yaml
+   # Release branch for F#
+   # Should be 'current' release branch name, i.e. 'release/dev17.10' in dotnet/fsharp/refs/heads/main, 'release/dev17.10' in dotnet/fsharp/refs/heads/release/dev17.10 and 'release/dev17.9' in dotnet/fsharp/refs/heads/release/dev17.9
+   # Should **never** be 'main' in dotnet/fsharp/refs/heads/main, since it will start inserting to VS twice.
+   - name: FSharpReleaseBranchName
+     value: release/dev17.13
+   ```
+   2. VS insertion branch
+   ```yaml
+   # VS Insertion branch name (NOT the same as F# branch)
+   # Should be previous release branch or 'main' in 'main' and 'main' in release branch
+   # (since for all *new* release branches we insert into VS main and for all *previous* releases we insert into corresponding VS release),
+   # i.e. 'rel/d17.9' *or* 'main' in dotnet/fsharp/refs/heads/main and 'main' in F# dotnet/fsharp/refs/heads/release/dev17.10 (latest release branch)
+   - name: VSInsertionTargetBranchName
+     value: main
+   ```
+   > [!NOTE] Note  
+   > For the **old** release branch `VSInsertionTargetBranchName` should point to corresponding VS release target branch (e.g. should be `rel/d17.13` in the F# repo branch `release/dev17.13`)
+   > For both `main` and **new** release branch `VSInsertionTargetBranchName` should be `main`.
+   >
+   > `FSharpReleaseBranchName` should **always** be the most recent in `main` and corresponding release branch name in release branches.
+
+   3. Oneloc branch:
+   ```yaml
+   # Localization: we only run it for specific release branches
+      - ${{ if eq(variables['Build.SourceBranch'], 'refs/heads/release/dev17.13') }}:
+   ```
+5. Set the new branch to receive auto-merges from `main`, and also set the old release branch to flow into the new one.  [This PR](https://github.com/dotnet/roslyn-tools/pull/1245/files) is a good example of what to do when a new `release/dev17.4` branch is created that should receive merges from both `main` and the previous release branch, `release/dev17.3`. Old release branch should stop receiving updates from the `main`.
+
+6. Set the packages from the new branch to flow into the correct package feeds via the `darc` tool.  To do this:
    1. Ensure the latest `darc` tool is installed by running `eng/common/darc-init.ps1`.
    2. (only needed once) Run the command `darc authenticate`.  A text file will be opened with instructions on how to populate access tokens.
    3. Check the current package/channel subscriptions by running `darc get-default-channels --source-repo fsharp`.  For this example, notice that the latest subscription shows the F# branch `release/dev17.3` is getting added to the `VS 17.3` channel.

docs/release-notes/.FSharp.Compiler.Service/9.0.200.md

@@ -28,5 +28,6 @@
 * Better ranges for `inherit` error reporting. ([PR #17879](https://github.com/dotnet/fsharp/pull/17879))
 * Better ranges for `inherit` `struct` error reporting. ([PR #17886](https://github.com/dotnet/fsharp/pull/17886))
 * Better ranges for `inherit` objects error reporting. ([PR #17893](https://github.com/dotnet/fsharp/pull/17893))
+* Better ranges for #nowarn error reporting; bring back #nowarn warnings for --langVersion:80; add warnings under feature flag ([PR #17871](https://github.com/dotnet/fsharp/pull/17871))
 
 ### Breaking Changes

src/Compiler/Driver/CompilerConfig.fs

@@ -22,6 +22,7 @@ open FSharp.Compiler.DiagnosticsLogger
 open FSharp.Compiler.Features
 open FSharp.Compiler.IO
 open FSharp.Compiler.CodeAnalysis
+open FSharp.Compiler.Syntax
 open FSharp.Compiler.Text
 open FSharp.Compiler.Text.Range
 open FSharp.Compiler.Xml
@@ -91,24 +92,61 @@ let ResolveFileUsingPaths (paths, m, fileName) =
         let searchMessage = String.concat "\n " paths
         raise (FileNameNotResolved(fileName, searchMessage, m))
 
-let GetWarningNumber (m, warningNumber: string, prefixSupported) =
-    try
-        let warningNumber =
-            if warningNumber.StartsWithOrdinal "FS" then
-                if prefixSupported then
-                    warningNumber.Substring 2
-                else
-                    raise (new ArgumentException())
-            else
-                warningNumber
+[<RequireQualifiedAccess>]
+type WarningNumberSource =
+    | CommandLineOption
+    | CompilerDirective
+
+[<RequireQualifiedAccess>]
+type WarningDescription =
+    | Int32 of int
+    | String of string
+    | Ident of Ident
+
+let GetWarningNumber (m, description: WarningDescription, langVersion: LanguageVersion, source: WarningNumberSource) =
+    let argFeature = LanguageFeature.ParsedHashDirectiveArgumentNonQuotes
+
+    let parse (numStr: string) =
+        let trimPrefix (s: string) =
+            if s.StartsWithOrdinal "FS" then s[2..] else s
+
+        let tryParseIntWithFailAction (s: string) (failAction: unit -> unit) =
+            match Int32.TryParse s with
+            | true, n -> Some n
+            | false, _ ->
+                failAction ()
+                None
+
+        let warnInvalid () =
+            warning (Error(FSComp.SR.buildInvalidWarningNumber numStr, m))
+
+        if source = WarningNumberSource.CommandLineOption then
+            tryParseIntWithFailAction (trimPrefix numStr) id
+        elif langVersion.SupportsFeature(argFeature) then
+            tryParseIntWithFailAction (trimPrefix numStr) warnInvalid
+        else
+            tryParseIntWithFailAction numStr id
 
-        if Char.IsDigit(warningNumber[0]) then
-            Some(int32 warningNumber)
+    match description with
+    | WarningDescription.Int32 n ->
+        if tryCheckLanguageFeatureAndRecover langVersion argFeature m then
+            Some n
+        else
+            None
+    | WarningDescription.String s ->
+        if
+            source = WarningNumberSource.CompilerDirective
+            && not (langVersion.SupportsFeature argFeature)
+            && s.StartsWithOrdinal "FS"
+        then
+            warning (Error(FSComp.SR.buildInvalidWarningNumber s, m))
+
+        parse s
+    | WarningDescription.Ident ident ->
+        if tryCheckLanguageFeatureAndRecover langVersion argFeature m then
+            parse ident.idText
         else
             None
-    with _ ->
-        warning (Error(FSComp.SR.buildInvalidWarningNumber warningNumber, m))
-        None
 
 let ComputeMakePathAbsolute implicitIncludeDir (path: string) =
     try
@@ -934,7 +972,7 @@ type TcConfigBuilder =
     member tcConfigB.TurnWarningOff(m, s: string) =
         use _ = UseBuildPhase BuildPhase.Parameter
 
-        match GetWarningNumber(m, s, tcConfigB.langVersion.SupportsFeature(LanguageFeature.ParsedHashDirectiveArgumentNonQuotes)) with
+        match GetWarningNumber(m, WarningDescription.String s, tcConfigB.langVersion, WarningNumberSource.CommandLineOption) with
         | None -> ()
         | Some n ->
             // nowarn:62 turns on mlCompatibility, e.g. shows ML compat items in intellisense menus
@@ -949,7 +987,7 @@ type TcConfigBuilder =
     member tcConfigB.TurnWarningOn(m, s: string) =
         use _ = UseBuildPhase BuildPhase.Parameter
 
-        match GetWarningNumber(m, s, tcConfigB.langVersion.SupportsFeature(LanguageFeature.ParsedHashDirectiveArgumentNonQuotes)) with
+        match GetWarningNumber(m, WarningDescription.String s, tcConfigB.langVersion, WarningNumberSource.CommandLineOption) with
         | None -> ()
         | Some n ->
             // warnon 62 turns on mlCompatibility, e.g. shows ML compat items in intellisense menus

src/Compiler/Driver/CompilerConfig.fsi

@@ -17,6 +17,7 @@ open FSharp.Compiler.Diagnostics
 open FSharp.Compiler.DiagnosticsLogger
 open FSharp.Compiler.Features
 open FSharp.Compiler.CodeAnalysis
+open FSharp.Compiler.Syntax
 open FSharp.Compiler.Text
 open FSharp.Compiler.BuildGraph
 
@@ -926,7 +927,20 @@ val TryResolveFileUsingPaths: paths: string seq * m: range * fileName: string ->
 
 val ResolveFileUsingPaths: paths: string seq * m: range * fileName: string -> string
 
-val GetWarningNumber: m: range * warningNumber: string * prefixSupported: bool -> int option
+[<RequireQualifiedAccess>]
+type WarningNumberSource =
+    | CommandLineOption
+    | CompilerDirective
+
+[<RequireQualifiedAccess>]
+type WarningDescription =
+    | Int32 of int
+    | String of string
+    | Ident of Ident
+
+val GetWarningNumber:
+    m: range * description: WarningDescription * langVersion: LanguageVersion * source: WarningNumberSource ->
+        int option
 
 /// Get the name used for FSharp.Core
 val GetFSharpCoreLibraryName: unit -> string

src/Compiler/Driver/CompilerOptions.fs

@@ -832,15 +832,15 @@ let errorsAndWarningsFlags (tcConfigB: TcConfigBuilder) =
         CompilerOption(
             "nowarn",
             tagWarnList,
-            OptionStringList(fun n -> tcConfigB.TurnWarningOff(rangeCmdArgs, trimFS n)),
+            OptionStringList(fun n -> tcConfigB.TurnWarningOff(rangeCmdArgs, n)),
             None,
             Some(FSComp.SR.optsNowarn ())
         )
 
         CompilerOption(
             "warnon",
             tagWarnList,
-            OptionStringList(fun n -> tcConfigB.TurnWarningOn(rangeCmdArgs, trimFS n)),
+            OptionStringList(fun n -> tcConfigB.TurnWarningOn(rangeCmdArgs, n)),
             None,
             Some(FSComp.SR.optsWarnOn ())
         )

src/Compiler/Driver/ParseAndCheckInputs.fs

@@ -217,25 +217,23 @@ let PostParseModuleSpec (_i, defaultNamespace, isLastCompiland, fileName, intf)
         SynModuleOrNamespaceSig(lid, isRecursive, kind, decls, xmlDoc, attributes, None, range, trivia)
 
 let GetScopedPragmasForHashDirective hd (langVersion: LanguageVersion) =
-    let supportsNonStringArguments =
-        langVersion.SupportsFeature(LanguageFeature.ParsedHashDirectiveArgumentNonQuotes)
-
     [
         match hd with
-        | ParsedHashDirective("nowarn", numbers, m) ->
-            for s in numbers do
-                let warningNumber =
-                    match supportsNonStringArguments, s with
-                    | _, ParsedHashDirectiveArgument.SourceIdentifier _ -> None
-                    | true, ParsedHashDirectiveArgument.LongIdent _ -> None
-                    | true, ParsedHashDirectiveArgument.Int32(n, _) -> GetWarningNumber(m, string n, true)
-                    | true, ParsedHashDirectiveArgument.Ident(s, _) -> GetWarningNumber(m, s.idText, true)
-                    | _, ParsedHashDirectiveArgument.String(s, _, _) -> GetWarningNumber(m, s, true)
+        | ParsedHashDirective("nowarn", args, _) ->
+            for arg in args do
+                let rangeAndDescription =
+                    match arg with
+                    | ParsedHashDirectiveArgument.Int32(n, m) -> Some(m, WarningDescription.Int32 n)
+                    | ParsedHashDirectiveArgument.Ident(ident, m) -> Some(m, WarningDescription.Ident ident)
+                    | ParsedHashDirectiveArgument.String(s, _, m) -> Some(m, WarningDescription.String s)
                     | _ -> None
 
-                match warningNumber with
+                match rangeAndDescription with
                 | None -> ()
-                | Some n -> ScopedPragma.WarningOff(m, n)
+                | Some(m, description) ->
+                    match GetWarningNumber(m, description, langVersion, WarningNumberSource.CompilerDirective) with
+                    | None -> ()
+                    | Some n -> ScopedPragma.WarningOff(m, n)
         | _ -> ()
     ]
 
@@ -912,7 +910,8 @@ let ProcessMetaCommandsFromInput
                 state
 
             | ParsedHashDirective("nowarn", hashArguments, m) ->
-                let arguments = parsedHashDirectiveArguments hashArguments tcConfig.langVersion
+                let arguments = parsedHashDirectiveArgumentsNoCheck hashArguments
+
                 List.fold (fun state d -> nowarnF state (m, d)) state arguments
 
             | ParsedHashDirective(("reference" | "r") as c, [], m) ->

src/Compiler/Facilities/prim-lexing.fs

@@ -238,13 +238,13 @@ type internal Position =
         Position(x.FileIndex, x.Line, x.OriginalLine, x.StartOfLineAbsoluteOffset, x.StartOfLineAbsoluteOffset - 1)
 
     member x.ApplyLineDirective(fileIdx, line) =
-        Position(fileIdx, line, x.OriginalLine, x.AbsoluteOffset, x.AbsoluteOffset)
+        Position(fileIdx, line, x.OriginalLine + 1, x.AbsoluteOffset, x.AbsoluteOffset)
 
     override p.ToString() = $"({p.Line},{p.Column})"
 
     static member Empty = Position()
 
-    static member FirstLine fileIdx = Position(fileIdx, 1, 0, 0, 0)
+    static member FirstLine fileIdx = Position(fileIdx, 1, 1, 0, 0)
 
 type internal LexBufferFiller<'Char> = LexBuffer<'Char> -> unit
 

src/Compiler/Interactive/fsi.fs

@@ -3839,7 +3839,8 @@ type FsiInteractionProcessor
             istate, Completed None
 
         | ParsedHashDirective("nowarn", nowarnArguments, m) ->
-            let numbers = (parsedHashDirectiveArguments nowarnArguments tcConfigB.langVersion)
+            let numbers = (parsedHashDirectiveArgumentsNoCheck nowarnArguments)
+
             List.iter (fun (d: string) -> tcConfigB.TurnWarningOff(m, d)) numbers
             istate, Completed None
 

src/Compiler/SyntaxTree/SyntaxTreeOps.fs

@@ -1005,6 +1005,16 @@ let parsedHashDirectiveArguments (input: ParsedHashDirectiveArgument list) (lang
             | false -> None)
         input
 
+let parsedHashDirectiveArgumentsNoCheck (input: ParsedHashDirectiveArgument list) =
+    List.map
+        (function
+        | ParsedHashDirectiveArgument.String(s, _, _) -> s
+        | ParsedHashDirectiveArgument.SourceIdentifier(_, v, _) -> v
+        | ParsedHashDirectiveArgument.Int32(n, m) -> string n
+        | ParsedHashDirectiveArgument.Ident(ident, m) -> ident.idText
+        | ParsedHashDirectiveArgument.LongIdent(ident, m) -> longIdentToString ident)
+        input
+
 let parsedHashDirectiveStringArguments (input: ParsedHashDirectiveArgument list) (_langVersion: LanguageVersion) =
     List.choose
         (function

src/Compiler/SyntaxTree/SyntaxTreeOps.fsi

@@ -322,6 +322,8 @@ val synExprContainsError: inpExpr: SynExpr -> bool
 
 val parsedHashDirectiveArguments: ParsedHashDirectiveArgument list -> LanguageVersion -> string list
 
+val parsedHashDirectiveArgumentsNoCheck: ParsedHashDirectiveArgument list -> string list
+
 val parsedHashDirectiveStringArguments: ParsedHashDirectiveArgument list -> LanguageVersion -> string list
 
 /// 'e1 && e2'

tests/FSharp.Compiler.ComponentTests/CompilerDirectives/Line.fs

@@ -2,16 +2,20 @@ namespace CompilerDirectives
 
 open Microsoft.FSharp.Control
 open Xunit
-open FSharp.Test.Compiler
+open Internal.Utilities
+open FSharp.Compiler
 open FSharp.Compiler.CodeAnalysis
-open FSharp.Compiler.Text
+open FSharp.Compiler.DiagnosticsLogger
+open FSharp.Compiler.Features
+open FSharp.Compiler.Lexhelp
 open FSharp.Compiler.Syntax
+open FSharp.Compiler.Text
+open FSharp.Compiler.UnicodeLexing
 
 module Line =
 
-    let checker = FSharpChecker.Create()
-
     let parse (source: string) =
+        let checker = FSharpChecker.Create()
         let langVersion = "preview"
         let sourceFileName = __SOURCE_FILE__
         let parsingOptions =
@@ -63,4 +67,51 @@ printfn ""
             | ParsedInput.SigFile _ -> failwith "unexpected: sig file"
         if exprRange <> expectedRange then
             failwith $"case{case}: expected: {expectedRange}, found {exprRange}"
-
+
+
+
+
+
+    let private getTokens sourceText =
+        let langVersion = LanguageVersion.Default
+        let lexargs =
+            mkLexargs (
+                [],
+                IndentationAwareSyntaxStatus(true, false),
+                LexResourceManager(),
+                [],
+                DiscardErrorsLogger,
+                PathMap.empty,
+                true
+            )
+        let lexbuf = StringAsLexbuf(true, langVersion, None, sourceText)
+        resetLexbufPos "test.fs" lexbuf
+        let tokenizer _ =
+            let t = Lexer.token lexargs true lexbuf
+            let p = lexbuf.StartPos
+            t, FileIndex.fileOfFileIndex p.FileIndex, p.OriginalLine, p.Line
+        let isNotEof(t,_,_,_) = match t with Parser.EOF _ -> false | _ -> true
+        Seq.initInfinite tokenizer |> Seq.takeWhile isNotEof |> Seq.toList
+
+    let private code = """
+1
+#line 5 "other.fs"
+2
+#line 10 "test.fs"
+3
+"""
+
+    let private expected = [
+        "test.fs", 2, 2
+        "other.fs", 4, 5
+        "test.fs", 6, 10
+    ]
+
+    [<Fact>]
+    let checkOriginalLineNumbers() =
+        let tokens = getTokens code
+        Assert.Equal(expected.Length, tokens.Length)
+        for ((e_idx, e_oLine, e_line), (_, idx, oLine, line)) in List.zip expected tokens do
+            Assert.Equal(e_idx, idx)
+            Assert.Equal(e_oLine, oLine)
+            Assert.Equal(e_line, line)